---
title: A Comparative Overview of Data Pipeline Architectures: Lambda, Kappa, and the Modern Data Stack
created: 2026-04-01
updated: 2026-04-01
type: summary
tags: ["architecture", "data", "notes", "pipelines", "telegram"]
external: http://go/ui/posts/2317/
---

Here are the most common data pipeline architectures used by companies like Airbnb, Uber, Netflix, LinkedIn, and others — I'll show them as three distinct patterns.The **Lambda architecture** was dominant from ~2013–2020. The core tradeoff: the batch layer is accurate but slow (hours), the speed layer is fast but approximate. The serving layer merges both. The big pain point is maintaining the same logic twice.**Kappa** simplifies Lambda by removing the batch layer — everything is a stream. Reprocessing is done by replaying from an earlier Kafka offset. Much simpler to operate, but requires your stream processor (Flink) to be expressive enough to handle complex historical aggregations.The **Modern Data Stack** is the dominant pattern for most companies today. It bets on cloud warehouses being cheap enough that you can store raw data first and transform later (ELT vs ETL). dbt became hugely popular here because transformations are just SQL, version controlled, with a lineage graph built in.

---

**Quick comparison**:

| Pattern | Latency | Complexity | Best for |
|---|---|---|---|
| Lambda | Seconds–hours | High (dual codepaths) | Large scale, pre-2020 |
| Kappa | Sub-second | Medium | Event-driven products |
| Modern Data Stack | Minutes–hours | Low | Analytics, SaaS companies |

A common real-world setup you'd see at a mid-to-large company combines all three: **Kafka + Flink** for operational event streams (Kappa-style), feeding into a **Snowflake/BigQuery** warehouse for analytics (MDS-style), with **dbt** handling the transformation layer and Airflow orchestrating the whole thing.
