Spotify — Why Bayesian A/B Testing Doesn't Pay Off

Bayesian A/B testing is pitched as more intuitive, safe to peek at, and smarter about multiple metrics. Spotify ran the numbers and found that, for its experimentation program, it doesn't beat what it already uses.

Key takeaways:

• Often the same math. With a flat prior, Bayesian and frequentist procedures give numerically identical results, just with different wording.
• Peeking still bites. The posterior stays coherent whenever you stop, but false-positive rates and estimates still depend on the prior and the stopping rule.
• Priors are hard to get right. A good historical prior needs a large (sometimes 200+ experiments), representative archive. Winner-only or pooled archives made estimates worse than plain group sequential testing.
• Even a perfect prior didn't win. In simulations, an oracle prior gave no power advantage over group sequential testing.
• The real question. Not “Bayesian or frequentist?” but whether a specific setup improves on what you have. For Spotify today, it doesn't.

Chapters:

00:00 The shiny new engine

01:32 Why Spotify isn't using Bayesian A/B tests

03:49 What Spotify's program needs

06:53 Priors, explained

08:24 Same numbers, different words

10:40 Myth: safe to peek

12:10 Myth: shrinkage fixes everything

14:31 Myth: better with many metrics

15:16 The 200-experiment archive

16:47 Simulating the best case

19:05 Bad pooling

20:38 What it means for everyone else

22:54 Recap

Sources:

• Why Spotify Is Not Using Bayesian A/B Testing (Mattias Frånberg and Mårten Schultzberg, Spotify Engineering, Sep 2026)

Related episode: E26: Academic — Why Your A/B Test Winner Might Be a Fragile Bet

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.