Netflix — Grading the Graders: Can an AI Judge Your Synopsis?
Episode notes
Netflix's catalog has too many synopses for humans to check line by line, so it built LLM judges for tone, clarity, precision and factuality. They agree with Netflix's creative writers 85%+ of the time, and the writers agree with each other about 80%.
Key takeaways:
- One judge per criterion. A single prompt that grades everything gets overloaded, so each quality dimension gets its own dedicated judge.
- Consensus scoring. The tone and clarity judges score each synopsis five times and combine the results, which smooths out random misfires.
- Agents check the facts. For factuality, agents verify claims against show data (plot, metadata, talent, awards), and the lowest score wins.
- Calibrate the humans first. Writers ran calibration rounds until they agreed about 80% of the time, then built a golden set of about 600 synopses to test the judges against.
- Scores predict behavior. A/B tests tied the scores to take fraction and abandonment, so the judges can monitor the whole global catalog.
Chapters:
00:00 Choosing a show at 9:30 p.m.
01:30 Why synopses are hard
03:51 The cost of a bad synopsis
05:23 Four quality dimensions
06:56 Consensus scoring
07:43 Agents-as-a-judge for facts
09:17 Tiered rationales
10:52 The 600-synopsis golden set
11:38 The 80% human baseline
13:09 Do scores change viewing?
14:40 What's genuinely new
16:56 Other Netflix work
17:43 Beyond Netflix
19:16 Closing question
Sources:
- Evaluating Netflix Show Synopses with LLM-as-a-Judge (Gabriela Alessio, Cameron Taylor and Cameron R. Wolfe, Netflix Tech Blog, Apr 2026)
Related episode: E8: Airbnb — The LLM Eval Loop That Went From Weeks to a Day
Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.
Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.
Made with AI-generated voices; curated by Peter Liu from the sources above.