Notas del episodio
One of the most underrated ML skills — forecasting discipline, dressed up as a COVID story. Jordan and Riley unpack why "retrain" secretly hides three different decisions: refit (same model, new data), respecify (change structure/features/priors), and hold (do nothing) — and how a standing retrain cadence always silently picks refit. They cover the three failure modes: chasing noise (an off-cycle refit on a one-off spike overweights the least-understood window), carrying ghosts (a stale COVID cancellation-timing assumption stays switched on behind clean-looking refits), and respec-as-panic (rebuilding under deadline for a passing FX shock). The decision triage: params drifted but process same -> refit; process changed in a way the model can't represent (tell: one-sided, directional misses) -> respecify; miss inside normal range -> hold. Plus the real case where fixed-hierarchy pooling broke on divergent geographic recovery, a refit went unstable (~3x error, wouldn't settle = structural signal), and respecifying to borrow along geographic adjacency cut error by more than half. Takeaway: refitting keeps a model current but not honest — a good team learns to forget shocks on purpose. Source: How we knew COVID was over (and what our models had to unlearn) — Airbnb Engineering, Aug 19 2026, by Harrison Katz — https://medium.com/airbnb-engineering/how-we-knew-covid-was-over-and-what-our-models-had-to-unlearn-c606b9bdb0ab This is commentary/summary in the hosts' own words, not a reproduction of the article.