Spotify — Why Bayesian A/B Testing Doesn't Pay Off
Note sull'episodio
Bayesian A/B testing is pitched as more intuitive, safe to peek at, and smarter about multiple metrics. Spotify ran the numbers and found that, for its experimentation program, it doesn't beat what it already uses.
Key takeaways:
- Often the same math. With a flat prior, Bayesian and frequentist procedures give numerically identical results, just with different wording.
- Peeking still bites. The posterior stays coherent whenever you stop, but false-positive rates and estimates still depend on the prior and the stopping rule.
- Priors are hard to get right. A good historical prior needs a large (sometimes 200+ experiments), representative archive. Winner-only or pooled archives made estimates worse than plain group sequential testing.
- Even a perfect prior didn't win. In simulations, an oracle prior gave no power advantage over group sequential testing.
- The real question. Not “Bayesian or frequentist?” but whether a specific setup improves on what you have. For Spotify today, it doesn't.
Chapters:
00:00 The shiny new engine
01:32 Why Spotify isn't using Bayesian A/B tests
03:49 What Spotify's program needs
06:53 Priors, explained
08:24 Same numbers, different words
10:40 Myth: safe to peek
12:10 Myth: shrinkage fixes everything
14:31 Myth: better with many metrics
15:16 The 200-experiment archive
16:47 Simulating the best case
19:05 Bad pooling
20:38 What it means for everyone else
22:54 Recap
Sources:
- Why Spotify Is Not Using Bayesian A/B Testing (Mattias Frånberg and Mårten Schultzberg, Spotify Engineering, Sep 2026)
Related episode: E26: Academic — Why Your A/B Test Winner Might Be a Fragile Bet
Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.
Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.
Made with AI-generated voices; curated by Peter Liu from the sources above.