Shopify — An AI Agent That Verifies Its Own Work
Episode notes
Shopify's River is an AI agent that closes the gap between “a security patch was written” and “the vulnerability is gone in production.” It re-verifies fixes against the live repo, and security merges rose from about 10% to 80%.
Key takeaways:
- A patch isn't a fix. A PR that passed CI can go stale before merge. River distrusts tracker records and re-checks the live code before anything is marked resolved.
- It does the tedious work. River rebases, retriggers CI and repairs lock files. On one run, it opened four Slack threads covering 35 findings within a minute.
- It knows when to hand off. Judgment calls go to engineers with the evidence attached.
- Results. In the first 11 days of the dependency workflow, the open-issue backlog fell about 70%. Security merges through the freshness-gated queue went from about 10% to 80%.
- What's new. Not that an AI writes code, but the discipline of tying evidence to commits before closing anything.
Chapters:
00:00 When “done” isn't done
00:47 Shopify's River
01:32 The gap between a patch and a fix
03:48 Stale PRs and a growing backlog
05:19 River rechecks the live code
07:41 Rebasing, CI and lock files
09:59 Many fixes at once
10:46 Handing off to humans
12:17 Evidence before “resolved”
13:03 Results: backlog down 70%
15:21 Beyond security patching
16:06 What's actually new
18:23 Recap
Sources:
- How River takes security work from a fix to merge (Erin Son and Kaiyi Li, Shopify Engineering, Sep 2026)
Related episode: E2: Shopify — An AI Agent That Trains Itself
Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.
Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.
Made with AI-generated voices; curated by Peter Liu from the sources above.