Shopify — An AI Agent That Verifies Its Own Work

Shopify's River is an AI agent that closes the gap between “a security patch was written” and “the vulnerability is gone in production.” It re-verifies fixes against the live repo, and security merges rose from about 10% to 80%.

Key takeaways:

• A patch isn't a fix. A PR that passed CI can go stale before merge. River distrusts tracker records and re-checks the live code before anything is marked resolved.
• It does the tedious work. River rebases, retriggers CI and repairs lock files. On one run, it opened four Slack threads covering 35 findings within a minute.
• It knows when to hand off. Judgment calls go to engineers with the evidence attached.
• Results. In the first 11 days of the dependency workflow, the open-issue backlog fell about 70%. Security merges through the freshness-gated queue went from about 10% to 80%.
• What's new. Not that an AI writes code, but the discipline of tying evidence to commits before closing anything.

Chapters:

00:00 When “done” isn't done

00:47 Shopify's River

01:32 The gap between a patch and a fix

03:48 Stale PRs and a growing backlog

05:19 River rechecks the live code

07:41 Rebasing, CI and lock files

09:59 Many fixes at once

10:46 Handing off to humans

12:17 Evidence before “resolved”

13:03 Results: backlog down 70%

15:21 Beyond security patching

16:06 What's actually new

18:23 Recap

Sources:

• How River takes security work from a fix to merge (Erin Son and Kaiyi Li, Shopify Engineering, Sep 2026)

Related episode: E2: Shopify — An AI Agent That Trains Itself

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.