Academic — Why Your A/B Test Winner Might Be a Fragile Bet

“p < 0.05, ship it” answers the wrong question, argue Max Farrell, Malika Korganbekova and Sanjog Misra. Their robust A/B decision rule is as easy to compute as a t-test and cut regret across 552 real ad experiments.

Key takeaways:

• The wrong question. A t-test asks whether two arms differ in your sample. A business needs to know which arm pays off where it will actually be deployed.
• Ambiguity aversion. Each arm is judged by its worst plausible value over outcome distributions close to the one observed in the test, so high-variance “wins” get penalized.
• A closed-form shortcut. Thanks to the Donsker-Varadhan representation, the rule needs only standard A/B outcome data and one trust parameter for how much you trust the experiment.
• Real-world test. On an archive of 552 advertising experiments from a US online platform, the rule substantially reduced regret compared with significance testing.
• Caveat. It improves the decision step; it doesn't fix a badly run experiment, and the trust parameter still takes judgment.

Chapters:

00:00 A significant winner, then a flop

00:45 The paper

02:18 The standard ship-it workflow

03:04 What a t-test was built for

04:37 Why winners don't transfer

06:09 A decision-theory view

06:55 The ambiguity-averse traveler

08:27 Making it computable

09:59 The closed-form shortcut

11:32 The trust parameter

13:04 552 real ad experiments

13:50 Regret and fragile winners

16:07 How novel is it?

19:10 Caveats

20:40 Recap

Sources:

• Robust A/B Decisions (Max H. Farrell, Malika Korganbekova and Sanjog Misra; arXiv, Sep 2026)

Related episode: E17: Spotify — Why Bayesian A/B Testing Doesn't Pay Off

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.