Jeunen (arXiv) — Faster A/B Tests by Ignoring Where Policies Agree

When the treatment and control in an A/B test pick the same action, that traffic adds noise but no signal. Olivier Jeunen shows how to exploit this policy overlap for unbiased, lower-variance tests of recommenders and rankers.

Key takeaways:

• The hidden inefficiency. Two ranking policies often agree on most decisions. Outcomes where they agree inflate confidence intervals without saying anything about the treatment effect.
• Assignment as a meta-policy. Treat the randomized assignment itself as a policy, log the probability of each action under both policies, and estimate the effect with Δ-off-policy estimation.
• Provably no worse. In the general case the method reduces to standard A/B testing. Its variance scales with how much the policies diverge, not with raw outcome noise, so it beats difference-in-means whenever the policies overlap.
• Complements CUPED. CUPED uses historical data to cut noise. This uses the structure of the two policies, so the two can in principle be stacked.
• Caveat. You need action probabilities from both policies, which is harder for black-box generative systems.

Chapters:

00:00 A great algorithm, a stalled test

00:45 Jeunen's paper

01:30 The A/B testing bottleneck

02:16 When policies agree

04:34 Logging probabilities, not just actions

05:19 Measuring policy overlap

06:51 Off-policy estimation

08:23 Does it work in practice?

09:54 Comparison with CUPED

12:14 The LLM problem

13:46 Related work

17:34 Caveats

19:07 Recap

Sources:

• Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap (Olivier Jeunen; arXiv, Jul 2026)

Related episode: E31: Stanford & Marketplace Researchers — Rerandomizing Switchback Experiments

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.