Airbnb — Turning Scientific Method Into AI Infrastructure

Before launching an AI customer-service assistant, Airbnb spent months mapping the rare, risky situations it might face. Its Insight Miner harness builds that scientific method into infrastructure, cutting investigations from months to days.

Key takeaways:

• The bottleneck. Each investigation took months of hand-built work, and repeating it for new languages, countries and products at a near-weekly pace outgrew the process.
• Method as infrastructure. At its core, Insight Miner extracts, embeds and clusters conversations. Around that sit proven techniques: prompt tuning, hard-example mining, contrastive labeling, and moving from unsupervised exploration to classification.
• Explore before you classify. Letting clusters emerge first surfaces issues a predefined list of categories would miss.
• More human judgment, not less. The mechanical steps scale, so experts spend their time on the most ambiguous and strategic data.
• Wide adoption. A year in, dozens of teams run hundreds of types of investigations, most users are outside technical roles, and separate agents help maintain the harness.

Chapters:

00:00 Drowning in conversation data

01:33 Why Airbnb needed this

03:04 When analysis outgrew humans

05:25 Insight Miner, the agent harness

06:55 Extract, embed, cluster

07:41 Clustering before classification

09:59 Hard-example mining

10:45 Leniency bias and contrastive labeling

12:17 Reproducible investigations

13:49 Adoption across teams

14:35 Can it scale bad questions?

16:08 How novel is it?

17:43 A new category of infrastructure

19:16 Recap

Sources:

• Beyond the model: Engineering AI infra with scientific judgement (Wren Dougherty, Airbnb Engineering, Sep 2026)

Related episode: E8: Airbnb — The LLM Eval Loop That Went From Weeks to a Day

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.