NetEase — The Agent That Audits Its Own Recommender

NetEase's AutoLR uses AI agents to take recommender improvements from idea to Launch Review, at RMB 3–4 in LLM API cost per iteration. It also caught its own failure mode: noisy wins that quietly reset the baseline.

Key takeaways:

• Agents propose, a harness decides. LLM agents generate and analyze ideas. A deterministic harness runs code, evaluations and the ledger, so every decision can be audited at Launch Review.
• Specialists, then a roundtable. Separate code-constraint and domain agents critique each proposal before a synthesis step merges overlapping ideas.
• Cheap enough to run routinely. Moving routine iterations from Claude Opus-class models to a DeepSeek-V4-Pro/Flash stack brought LLM API cost to RMB 3–4 per iteration, excluding training compute.
• Results, read carefully. The audit covers 1,586 completed evaluations. Nine Launch Reviews in NetEase's Dashen app showed positive lifts summing to +5.75% consumption penetration, +10.83% consumption time and +5.55% valid views. These are descriptive sums, not one causal effect.
• The KEEP ratchet. Noisy offline wins become the new baseline and fake steady progress. The fix is protocol-matched repeats.

Chapters:

00:00 The factory behind your feed

01:31 Why recommender upgrades are slow

03:49 Rule of separation

05:23 Specialist agents vs. context dilution

06:54 The roundtable synthesis

09:11 Choosing what to test

11:29 Cutting LLM costs

12:14 Results across nine launches

13:45 The KEEP ratchet

15:17 Fix: protocol-matched repeats

17:34 Compared with freer agent systems

19:06 Closing thoughts

Sources:

• AutoLR: Automating the Path from Research to Launch Review in Industrial Recommender Systems (Zhang et al., NetEase; arXiv, Sep 2026)

Related episode: E10: Tencent — Teaching a Recommender to Reason About Cold Starts

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.