1. Netflix (Video Streaming)
Challenges
- Petabytes of video
- Global delivery (latency)
- Multiple qualities (adaptive)
- Recommendation personalized
Video Pipeline
Upload → Transcode Service → Multiple bitrates (480p, 720p, 1080p, 4K) Multiple codecs (H.264, H.265, AV1) Multiple containers (MP4, WebM) Chunked (2-10s segments) — HLS/DASH ↓ Store in S3 + CDN (Open Connect at ISPs) ↓ Client adaptive bitrate based on bandwidth
Open Connect
Netflix own CDN — servers at ISPs। Popular content pre-pushed।
Recommendation
Batch (daily): Spark → collaborative filtering model Real-time: user events → Kafka → feature store Serving: TensorFlow Serving low-latency inference A/B test: thousands of experiments simultaneously
2. YouTube
Similar to Netflix but UGC (user-generated)।
Additional Challenges
- Upload pipeline at scale
- Content moderation (ML + human)
- Comments, live streaming
- Copyright detection (Content ID)
3. Google Search
Components
Crawler (Googlebot) → Fetch pages → Parser → Extract links, text, metadata ↓ Indexer → Inverted index: "hello" → [doc1, doc5, doc99] "world" → [doc2, doc5, doc42] ↓ Ranker (PageRank + 200+ signals) → ML model ↓ Query server → parse query → lookup index → rank → personalize → return top 10
Inverted Index
Forward: doc1: hello world foo doc2: hello bar baz Inverted: hello → [doc1, doc2] world → [doc1] foo → [doc1] bar → [doc2]
Scale
- 100B+ pages indexed
- Distributed across thousands of machines
- Shard by doc_id or by term
- Multi-layer cache (query, doc, snippet)
4. Recommendation System ⭐
Approaches
- Collaborative Filtering — similar users, similar items
- Content-based — item attributes similar
- Hybrid — combine both
- Deep Learning — neural collaborative filter
Pipeline
User events → Kafka → Feature Store Batch: Spark jobs → train model nightly Real-time: Flink → update user features Serving: low-latency API (<50ms) Candidate generation → Ranking → Re-ranking (diversity) Feedback loop: impressions, clicks → retrain
5. Data Pipeline / Lambda Architecture ⭐
Source (logs, events)
↓
Ingest (Kafka, Kinesis)
↓
┌────────────┬────────────┐
│ Batch Layer │ Stream Layer │
│ (Spark) │ (Flink, Storm) │
│ Hourly/daily │ Real-time │
│ Accurate, slow │ Fast, approx │
└─────┬──────┴──────┬─────┘
↓ ↓
Serving Layer (Cassandra, Druid)
↓
Dashboard / API
Kappa Architecture (modern)
Stream-only — Kafka as replay log, reprocess when needed।
Data Lake vs Warehouse
| Data Lake | Warehouse |
|---|---|
| Raw, unstructured | Processed, schema |
| S3, HDFS | Redshift, BigQuery, Snowflake |
| Schema on read | Schema on write |
| ML training | BI reporting |
Lakehouse
Databricks Delta Lake, Iceberg — lake flexibility + warehouse ACID।
Ad Platform (Google Ads)
Flow
Page loads → Ad request (with user context, page context) ↓ <100ms SLA Ad Server → Candidate selection (targeting match) Auction (generalized 2nd price) Budget check Return winning ad ↓ Impression log (Kafka) → analytics, billing
Design Trade-offs in Practice
| Decision | Trade-off |
|---|---|
| SQL vs NoSQL | ACID vs scale |
| Sync vs Async | Simplicity vs throughput |
| Microservice vs Monolith | Flexibility vs complexity |
| Strong vs Eventual consistency | Correctness vs availability |
| Cache vs fresh | Speed vs accuracy |
| SQL JOIN vs denormalize | Space vs speed |
Resources
- "Designing Data-Intensive Applications" — Kleppmann ⭐
- "System Design Interview" — Alex Xu
- High Scalability blog
- Netflix / Uber / Meta engineering blogs