Pinterest — A Cheap LLM Judge Plus Smarter Sampling Cuts Search-Experiment MDE

Pinterest's fine-tuned XLM-RoBERTa relevance judge made stratified sampling affordable, cutting search A/B-test MDE from 1.3–1.5% to ≤0.25%.

Key takeaways:

• A small, fast judge. An XLM-RoBERTa-large cross-encoder fine-tuned on ~2.6M human-labeled query–Pin pairs labels 150,000 rows in 30 minutes on one A10G GPU, using text only (incl. BLIP image captions). Llama-3-8B was slightly better but ~6x costlier.
• Sampling is the payoff. Bigger samples, strata by query interest × popularity, Neyman allocation and a stratified estimator took MDE from 1.3–1.5% to ≤0.25%. At 2,000 queries, stratification alone cut the standard deviation 52%.
• Close to humans. 73.7% exact label match, 91.7% within one point; US sDCG@25 correlations τ 0.652, ρ 0.817. Humans still grade relevance-model tests.
• Our take. Novelty ~3/5: a standard model, smart measurement engineering. Watch for a judge that favors treatments built like itself.
• Note. τ and ρ compare how LLM- and human-based scores rank queries, not the order of Pins on a page. The 67% cut is the full design at 5,000 queries vs. 2,000 random ones. France/Germany (τ ≈ 0.47) is lower but still “strong” per the paper.

Chapters:

00:00 When an A/B test says nothing changed

01:07 Pinterest's LLM relevance paper

02:30 MDE: the rusty bathroom scale

04:01 Choosing the judge: Llama-3-8B vs. XLM-RoBERTa

06:26 A text-only judge on a visual platform

08:36 sDCG@25 and the long-tail problem

10:33 Stratified sampling and Neyman allocation

13:19 Results: MDE ≤0.25%, and agreement with humans

15:52 Our take: novelty and the language gap

18:12 Judge bias and the human safety net

19:34 Related work, recap, closing question

Sources:

• LLM-based Relevance Assessment for Web-Scale Search Evaluation at Pinterest (Wang et al.; RecSys ’25 EARL workshop, Sep 2025)
• Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects (related, Oct 2026)
• The Quest to Understand Metric Movements (Pinterest Engineering, 2025)

Related episode: E15: Netflix — Grading the Graders: Can an AI Judge Your Synopsis?

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.