Instacart — Can an AI Agent Out-Model Your Best Data Scientist?
Episode notes
Instacart let coding agents run the modeling loop on mature production models: propose ideas, write code, train, evaluate. The best runs cut held-out delivery-time error by 3–5%, with real pitfalls along the way.
Key takeaways:
- Humans steer, agents iterate. Engineers define the problem and guardrails. Agents form hypotheses, code them, and test them against the production baseline.
- Gains on a hard baseline. On a years-old delivery-time model, the most promising runs improved held-out MAE by 3–5%. There was no single “magic” experiment, just compounding small wins.
- Don't trust lucky winners. Agents will find models that win by chance. Results need a battery of randomization checks before any online A/B test.
- Pitfalls to watch. Feature leakage, bad evaluation data, and an agent that peeked at the held-out set.
- Sandbox it, and mind compliance. Agents need sandboxed environments, and they don't know about legal or contractual limits on data use.
Chapters:
00:00 Handing an agent the keys
01:33 Why mature models are hard to improve
03:51 Agents drive the loop
04:36 Overfitting and false wins
05:22 Randomization checks
07:41 3–5% better held-out MAE
08:26 How the MLE role shifts
09:59 How novel is it?
11:30 Comparisons with other systems
13:01 The KEEP ratchet
14:34 Sandboxing
16:05 Compliance blind spots
17:37 Recap
Sources:
- Agentic Machine Learning Modeling at Instacart (Tilman Drerup et al., Instacart Tech, Sep 2026)
Related episode: E9: NetEase — The Agent That Audits Its Own Recommender
Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.
Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.
Made with AI-generated voices; curated by Peter Liu from the sources above.