AI
Netflix's Real-Time Distributed Graph, Part 1 — Ingesting and Processing Data Streams
AI
The Daily Diff by Premchand Chidipoti
Episode notes
The origin story of Netflix's Real-Time Distributed Graph: why microservices left them with data silos across ads, live events, and games, and how member actions become graph nodes and edges. Jordan and Riley cover the API Gateway to Kafka to Flink pipeline (~1M msgs/sec per topic, 5M+ records/sec published downstream), and the two scaling calls Netflix made — one Flink job per Kafka topic, and a separate topic per node/edge type — trading operational overhead for independent scaling.
Source: How and Why Netflix Built a Real-Time Distributed Graph, Part 1: Ingesting and Processing Data Streams at Internet Scale — Netflix Tech Blog, Oct 17 2025 — https:/ ...
Keywords
Tech blogEngineering blogSoftware designSoftware engineering