Etsy — Using Post-Treatment Data to Sharpen A/B Tests

Etsy and UC Berkeley stack 23 screened in-experiment covariates on top of CUPAC, cutting A/B-test variance a further ~2.5–14% across 29 experiments.

Key takeaways:

• Two stages. Stage 1 is Etsy's unchanged CUPAC (one LightGBM model, 117 pre-experiment covariates). Stage 2 is a linear adjustment on 23 in-experiment covariates, fit to CUPAC's residuals.
• Mean equivalence, not independence. A post-treatment covariate is usable if its mean matches across arms. That's testable; if it fails by Δ, the bias is −γᵀΔ.
• What Etsy ran. Drop too-sparse count variables, Mann–Whitney U per experiment, Fisher's method across experiments. No FWER correction, no TOST. The screen is a diagnostic, not proof, paired with domain knowledge.
• Results. On conversion rate, √R² rose 0.02 to over 0.14 vs. CUPAC; the extra variance cut was ~2.5–14% per experiment on top of CUPAC's ~10–25% (Fig. 2).
• Note. The paper is from CLeaR 2026 (PMLR 323), not ICLR. ByteDance's CUPED paper (Jun 2026) is parallel work; Etsy's predates it.

Chapters:

00:00 Why Etsy's traffic still isn't enough

02:40 CUPED, CUPAC and “forbidden” data

04:57 Two stages: CUPAC plus a linear add-on

07:15 Mean equivalence, a testable condition

08:06 What Etsy ran, and what it didn't

11:06 A diagnostic, not a proof; the −γᵀΔ bias

13:38 Results across 29 experiments

15:56 Caveat: covariates can drift

16:28 What's new, and related work

18:44 Recap and closing question

Sources:

• Variance reduction combining pre-experiment and in-experiment data (Lin & Crespo; CLeaR 2026, PMLR 323; arXiv 2410.09027)
• Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED (ByteDance, related, Jun 2026)
• CUPED on Steroids: Multivariate Covariate Adjustment for Switchback Experiments (related, Aug 2026)

Related episode: S2E1: Pinterest — A Cheap LLM Judge Plus Smarter Sampling Cuts Search-Experiment MDE

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.