Stanford & Marketplace Researchers — Rerandomizing Switchback Experiments
Note sull'episodio
When platforms can't randomize users, they switch whole regions between treatment and control over time. A new design from Zeng, Johari, Wager and colleagues re-randomizes every period to keep switchback groups balanced.
Key takeaways:
- Why switchbacks struggle. Few units, big differences between markets, conditions that change over time, and carryover from earlier periods all make standard A/B tests unreliable.
- Re-randomize every period. SRSB redraws assignments each period to balance prognostic variables built from past data, such as lagged outcomes and covariates.
- Valid inference. The authors provide finite-sample randomization inference under a sharp null, plus asymptotic inference as the number of periods grows.
- Handling carryover. Blocked SRSB re-randomizes within strata defined by the previous treatment, creating stable “stay” groups to compare.
- Evidence so far. Extensive simulations show gains over standard switchback designs. Real-world deployment will test whether it holds up under fast-changing conditions.
Chapters:
00:00 Not as clear as an X-ray
00:45 The paper
01:32 Why marketplaces use switchbacks
03:06 Few units, big differences
04:37 The carryover problem
05:24 SRSB: re-randomizing each period
07:43 Blocked SRSB for carryover
10:00 Tighter confidence intervals
11:30 Pairing with regression adjustment
13:51 Choosing the switch cadence
14:36 Production caveats
16:07 Recap
Sources:
- Sequentially-Rerandomized Switchback Experiments (Zeng, Adjaho, Bucarey, Qin, Zhang, Hoban, Johari and Wager; arXiv, Apr 2026)
Related episode: E30: Jeunen (arXiv) — Faster A/B Tests by Ignoring Where Policies Agree
Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.
Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.
Made with AI-generated voices; curated by Peter Liu from the sources above.