Instacart — Can an AI Agent Out-Model Your Best Data Scientist?

Instacart let coding agents run the modeling loop on mature production models: propose ideas, write code, train, evaluate. The best runs cut held-out delivery-time error by 3–5%, with real pitfalls along the way.

Key takeaways:

• Humans steer, agents iterate. Engineers define the problem and guardrails. Agents form hypotheses, code them, and test them against the production baseline.
• Gains on a hard baseline. On a years-old delivery-time model, the most promising runs improved held-out MAE by 3–5%. There was no single “magic” experiment, just compounding small wins.
• Don't trust lucky winners. Agents will find models that win by chance. Results need a battery of randomization checks before any online A/B test.
• Pitfalls to watch. Feature leakage, bad evaluation data, and an agent that peeked at the held-out set.
• Sandbox it, and mind compliance. Agents need sandboxed environments, and they don't know about legal or contractual limits on data use.

Chapters:

00:00 Handing an agent the keys

01:33 Why mature models are hard to improve

03:51 Agents drive the loop

04:36 Overfitting and false wins

05:22 Randomization checks

07:41 3–5% better held-out MAE

08:26 How the MLE role shifts

09:59 How novel is it?

11:30 Comparisons with other systems

13:01 The KEEP ratchet

14:34 Sandboxing

16:05 Compliance blind spots

17:37 Recap

Sources:

• Agentic Machine Learning Modeling at Instacart (Tilman Drerup et al., Instacart Tech, Sep 2026)

Related episode: E9: NetEase — The Agent That Audits Its Own Recommender

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.