🗺️ What is the Modern Data Stack?
The modern data stack is a layered architecture for moving, storing, transforming, and querying large volumes of data reliably. It replaces the old world of monolithic ETL scripts dumped into a single database with a composable set of purpose-built tools — each doing one job extremely well.
Every data engineering system, from a 10 GB nightly export to a 10 billion event Kafka stream, passes through the same five fundamental layers. The tools change at different scales, but the layers don't. Understanding those layers is the difference between an engineer who can pick the right tool and one who reaches for Spark every time they touch a CSV file.
This lesson is your map. Read it once now, then come back after each arc and you'll see exactly which piece of the puzzle you just built.
🏗️ The 5 Layers Every Data Engineer Touches
Think of the stack as a vertical pipeline. Raw events enter at the bottom (Layer 1) and polished insights exit at the top (Layer 5). Each layer has a clear contract: it accepts data in one shape and hands it to the next layer in a better shape.
🔧 This Track's Tools Mapped to the Stack
Nine tools, five layers. Some tools span multiple layers — DuckDB is both a compute engine and a serving layer query tool. Here is the full map before you write a single line of code.
| Tool | Layer | Arc | What it handles |
|---|---|---|---|
| Kafka 3.7 | Ingestion (L1) | Arc C | Event streaming, producer/consumer, delivery guarantees (at-most-once, at-least-once, exactly-once) |
| PySpark 3.5 | Compute (L3) | Arc B, D, G | Batch + streaming DataFrame transformations over data too large for a single machine |
| DuckDB 1.0 | Compute + Serving (L3/L5) | Arc B, F, G | Fast local SQL on Parquet files; dbt backend for transformations; ad-hoc queries at the serving layer |
| Cassandra 5 | Storage — write-heavy (L2) | Arc E | Wide-column NoSQL, partition keys, clustering keys, high-throughput writes — when Postgres can't keep up with insert rate |
| OpenSearch 2 | Storage + Serving (L2/L5) | Arc E | Full-text search, aggregations, log analytics — stores indexed documents and serves search queries and dashboards |
| Airflow 2.9 | Orchestration (L4) | Arc F | DAGs, task scheduling, backfills, dependency management — the glue that makes pipelines run on time |
| dbt | Transformation (L4) | Arc F | SQL-first data transformations, automated tests, column-level lineage — turns raw storage into analytics-ready models |
| MinIO | Storage (L2) | Arc G | Local S3-compatible object store for Parquet and Iceberg files — no AWS account needed during development |
| Parquet / Iceberg | Storage (L2) | Arc B, G | Columnar file format (Parquet) + open table format with ACID transactions and time-travel (Iceberg) |
➡️ End-to-End Data Flow: All 5 Layers
Every pipeline in this track follows the same physical journey. An event born at the source travels through ingestion, compute, storage, transformation, and ends as a queryable insight. This diagram shows that journey at a glance.
Notice that Storage (L2) sits between Compute (L3) and Transformation (L4). This is intentional: compute first reads raw ingested data, processes it, and writes results back to storage. Transformation then reads from that same storage to produce the clean, modelled views that the serving layer queries. Storage is the shared handshake point between every other layer.
🛒 NaijaCart: All 5 Layers in Production
Tunde is a data engineer at NaijaCart, based in Lagos. Every day, 8 million product views, 400K orders, and 2 million search events land in Kafka (Layer 1 — Ingestion). A PySpark Structured Streaming job reads that Kafka topic and aggregates product popularity scores in near-real time (Layer 3 — Compute). Results land as Parquet files on MinIO, NaijaCart's local S3-compatible object store (Layer 2 — Storage). An Airflow DAG triggers a dbt transformation job every hour that refreshes the "trending products" view by joining popularity scores with inventory data (Layer 4 — Transformation). The merchandising team then queries this view in OpenSearch Dashboards to decide which products to feature on the homepage (Layer 5 — Serve).
Total latency from customer click to dashboard update: under 90 minutes. NaijaCart's Postgres database could never handle 10 million events per day and still serve sub-second analytics queries — the modern data stack makes both possible simultaneously.
⚡ Two Ways to Build the Pipeline: Batch vs Streaming
The five layers remain constant whether you build a batch pipeline or a streaming pipeline. What changes is the tool choice and the latency at each layer. Understanding this trade-off before writing code saves weeks of rework.
| Layer | Tool | Frequency | Latency |
|---|---|---|---|
| L1 Ingestion | Postgres CDC / file export | Nightly | Hours |
| L3 Compute | PySpark (batch mode) | Nightly | Minutes |
| L2 Storage | Parquet / Iceberg | Nightly | Hours (written after batch) |
| L4 Transform | dbt + DuckDB | On-demand | Seconds |
| L5 Serve | DuckDB / BI tool | Ad-hoc | Seconds |
| Layer | Tool | Frequency | Latency |
|---|---|---|---|
| L1 Ingestion | Kafka | Continuous | Milliseconds |
| L3 Compute | Spark Structured Streaming | Continuous | Seconds |
| L2 Storage | Cassandra (writes) + Parquet (archive) | Continuous | Seconds |
| L4 Transform | Spark SQL windowed aggregations | Continuous | Seconds |
| L5 Serve | OpenSearch Dashboards | Near-real-time | < 1 minute |
🗺️ Arc-to-Layer Map: What You Learn, When
Seven arcs, 36 lessons. Each arc focuses on one or two layers of the stack. After finishing all seven arcs, you will have built every layer from scratch on your own laptop.
| Arc | Lessons | Layer Focus | Key Tool |
|---|---|---|---|
| A — Foundations | 00–02 | Setup + mental models | Docker, pip, stack overview |
| B — PySpark & Formats | 03–13 | Compute (batch) — L3 | PySpark, DuckDB, Parquet |
| C — Kafka | 14–20 | Ingestion — L1 | Kafka 3.7 (KRaft mode) |
| D — Structured Streaming | 21–24 | Compute (real-time) — L3 | Spark Structured Streaming |
| E — NoSQL at Scale | 25–28 | Storage (NoSQL) — L2 | Cassandra, OpenSearch |
| F — Orchestration | 29–33 | Transformation & Orchestration — L4 | Airflow 2.9, dbt |
| G — Capstones | 34–35 | All 5 layers combined | Full stack — batch + streaming |
🧠 Mini Challenge: Which Layers Do You Need?
Before reading further, try to answer these three architecture questions using only what you know so far. No code required — just layer reasoning.
NairaSwift settles 1 million bank transfers per day. They need a nightly reconciliation report that lists every transfer, its status, and the net position per bank. Which layers do they need? Which can they skip?
Show hint
L1 may be optional — if Postgres already holds all transfer records, a nightly export or CDC is sufficient and Kafka adds unnecessary complexity. They definitely need L3 (PySpark batch over 1M rows), L4 (dbt model to compute net positions), and L5 (a BI tool or DuckDB query for the report). L2 is needed for Parquet intermediate storage. Streaming is not justified — the report runs once a night.
JengaTel generates 40 million CDR (Call Detail Record) events per day. Fraud analysts need to query "calls over 90 minutes from new SIM cards activated in the last 7 days" within 5 minutes of the call ending. Which layers are required?
Show hint
All five layers are required. L1: Kafka ingesting CDR events at millisecond latency. L3: Spark Structured Streaming with a sliding or tumbling window to detect the pattern. L2: Cassandra for fast write + read on the flagged call records; Parquet for archival. L4: Spark SQL windowed aggregation (the transform is embedded in the streaming job here). L5: OpenSearch Dashboards for the fraud analyst UI, updated every 60 seconds.
VidyaPath has 3 years of learning event logs (200 GB total) stored in S3 as Parquet files. Data scientists want to run ad-hoc SQL like "average lesson completion rate per course, per week" on their laptops. There is no real-time requirement. Which single tool from this track is the right choice?
Show hint
DuckDB, running directly on the Parquet files in S3. At 200 GB, DuckDB's columnar execution engine can handle this on a laptop without needing a Spark cluster, Airflow, dbt, or Kafka. Spinning up Spark for a one-person ad-hoc analytics workload is a classic over-engineering mistake. DuckDB is the right tool: it runs locally, reads Parquet natively, and completes aggregation queries over 200 GB in seconds.
💼 Interview Q&A
These are the architecture questions that come up in data engineering interviews at product companies. Practice answering them in 60–90 seconds each.
Bottom to top: Ingestion (L1) — raw data arrives via Kafka, file exports, or CDC. Storage (L2) — data is persisted in Parquet, Iceberg, Cassandra, or object stores. Compute (L3) — Spark or DuckDB transforms and aggregates data. Transformation & Orchestration (L4) — dbt models clean and model data; Airflow schedules the pipeline. Serving (L5) — analysts query via dashboards, BI tools, or SQL. The mnemonic: I-S-C-T-S (Ingest, Store, Compute, Transform, Serve).
Kafka belongs to Layer 1 — Ingestion. Its job is to reliably receive events from producers (application servers, IoT devices, databases) and durably buffer them until consumers are ready to process them. Kafka does not transform data, it does not store final analytical results, and it does not serve dashboards. It is a distributed, fault-tolerant message queue — which is exactly what the ingestion layer needs. The confusion arises because Kafka stores data on disk, but that storage is temporary and exists only to buffer events between producers and consumers, not to hold analytics data long-term.
The batch pipeline needs all five layers. L1: Kafka ingests click events (or a nightly Postgres export if streaming isn't required). L2: Parquet files on MinIO hold intermediate results. L3: PySpark batch job aggregates product views per hour. L4: Airflow triggers every hour; dbt refreshes the trending products view. L5: OpenSearch Dashboards shows the latest results to the merchandising team. For the 90-minute SLA, a batch pipeline running hourly is sufficient — no Structured Streaming required. If the SLA were under 10 minutes, you'd replace the hourly PySpark batch with a Spark Structured Streaming job reading directly from Kafka, which would cut end-to-end latency to under 2 minutes at the cost of significantly higher operational complexity.
A data warehouse (Snowflake, BigQuery) is a managed, vendor-hosted system that combines L2 (Storage), L3 (Compute), and sometimes L4 (Transformation) into one SaaS product. You pay per query or per compute slot. The benefit is zero infrastructure — no clusters to manage. The downside is vendor lock-in and cost at very high data volumes. A lakehouse (Iceberg + Parquet on MinIO/S3 + Spark) separates storage (L2, Iceberg/Parquet files you own) from compute (L3, Spark you run yourself). You pay only for object storage and the compute you actually use. The benefit is no lock-in and lower cost at scale. The downside is operational burden — you manage Spark clusters, Iceberg metadata, and schema evolution yourself. In this track, we build a lakehouse architecture (no cloud account required) and survey warehouses in Arc F as an alternative for teams that prefer managed infrastructure.
This is the dual-sink pattern, also called the Lambda architecture at the storage layer. The Spark Structured Streaming job writes to two sinks in the same micro-batch: (1) Cassandra, using the Spark-Cassandra connector, for low-latency random reads — fraud analysts query individual records by partition key within milliseconds. (2) Parquet files on MinIO, for long-term archival and batch reprocessing — the same data is stored column-compressed for efficient full-scan queries later. The key implementation detail: Spark uses foreachBatch to write to both sinks within a single atomic micro-batch, then commits the Kafka offset only after both writes succeed. If either sink fails, the offset is not committed and the batch is retried. This gives you at-least-once delivery to both sinks. For exactly-once semantics in Cassandra, you use idempotent inserts (upserts) with a unique row key so retries don't create duplicates.
📝 Quiz: 5 Questions
The stack makes more sense once you build it. Arc B is next.
⚠️ Common Mistakes
✅ Key Takeaways
- The modern data stack has 5 layers: Ingestion → Storage → Compute → Transformation → Serving. Every pipeline — batch or streaming — passes through all five.
- This track covers all 5 layers: Kafka (L1), Parquet + Cassandra + MinIO (L2), PySpark + DuckDB (L3), dbt + Airflow (L4), OpenSearch + DuckDB (L5).
- Batch pipeline: nightly cadence with hours of latency — simpler to operate, cheaper compute, right for most analytical workloads.
- Streaming pipeline: Kafka + Spark Structured Streaming with seconds of latency — justified when freshness genuinely matters (fraud detection, live rankings), not for convenience.
- Both capstone projects (Lessons 34 and 35) build end-to-end across all 5 layers — you will have built a complete data platform on your laptop by the end of this track.
Now you know where everything fits — let's build it layer by layer.
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →