Kuaishou E-commerce — An Agent That Learns From Its Own A/B Tests

Kuaishou E-commerce built A/B Agent, an LLM agent that proposes recommendation strategies, learns from live A/B results and gets better each round. In production it lifted GMV 4.829% without hurting guardrail metrics.

Key takeaways:

• Institutional memory. Knowledge from past experiments is usually scattered. A/B Agent organizes historical strategies into a hierarchical experience tree by business scenario, recommendation stage, objective and context.
• Tree-RAG. Multi-path retrieval walks the tree to find transferable evidence, avoiding the mismatches of flat retrieval.
• Generate, test, evolve. The agent writes executable strategies, analyzes online A/B feedback to tune them, and updates the tree.
• Results. Built on a benchmark of 310 historical strategies from three Kuaishou scenarios. Online it delivered a 4.829% GMV lift with positive guardrail metrics.
• Caveat. The lift is author-reported, and the best of many automated tests can overstate the true effect.

Chapters:

00:00 The restaurant analogy

01:32 When AI runs the experiments

02:18 Hundreds of A/B tests at once

03:50 Experts don't scale

04:35 Historical amnesia

05:22 The experience tree

07:42 Tree-RAG retrieval

08:27 Generating strategies

09:58 Self-evolution and guardrails

10:43 Results: GMV up 4.829%

12:17 How novel is it?

14:36 Caveats: lucky winners

16:07 Closing question

Sources:

• A/B Agent: A Self-Evolving Agent for Strategy Iteration in Industrial A/B Testing (Jiang et al., Kuaishou; arXiv, Aug 2026)

Related episode: E9: NetEase — The Agent That Audits Its Own Recommender

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.