🗺️ What is the Modern Data Stack?

The modern data stack is a layered architecture for moving, storing, transforming, and querying large volumes of data reliably. It replaces the old world of monolithic ETL scripts dumped into a single database with a composable set of purpose-built tools — each doing one job extremely well.

Every data engineering system, from a 10 GB nightly export to a 10 billion event Kafka stream, passes through the same five fundamental layers. The tools change at different scales, but the layers don't. Understanding those layers is the difference between an engineer who can pick the right tool and one who reaches for Spark every time they touch a CSV file.

What you will build in this track:
By the end of Arc G (Lesson 35), you will have a working, end-to-end data platform running entirely on your laptop: Kafka ingesting events, PySpark processing them, Parquet and Cassandra storing them, dbt and Airflow transforming them, and OpenSearch Dashboards querying them. All five layers, nine tools, zero cloud accounts required.

This lesson is your map. Read it once now, then come back after each arc and you'll see exactly which piece of the puzzle you just built.

🏗️ The 5 Layers Every Data Engineer Touches

Think of the stack as a vertical pipeline. Raw events enter at the bottom (Layer 1) and polished insights exit at the top (Layer 5). Each layer has a clear contract: it accepts data in one shape and hands it to the next layer in a better shape.

L5
Serving — Dashboards, APIs & Query Tools
Tools in this track: DuckDB (ad-hoc SQL on Parquet), OpenSearch Dashboards (real-time visualisations), BI tools like Metabase or Tableau. This is where analysts and product managers spend their time. The data engineer's job is to make this layer fast and fresh.
L4
Transformation & Orchestration — dbt models, Airflow DAGs, Spark SQL
Tools in this track: dbt (SQL transformations, tests, lineage), Airflow 2.9 (scheduling, backfills, dependency graphs). Raw storage data is cleaned, joined, and modelled here into the views that the serving layer queries.
L3
Compute — Batch processing and Real-time streaming engines
Tools in this track: PySpark 3.5 (distributed batch), Spark Structured Streaming (real-time), DuckDB 1.0 (fast local SQL — sits at both L3 and L5). This is where DataFrame transformations and aggregations happen at scale.
L2
Storage — Files, Tables & NoSQL Stores
Tools in this track: Parquet files (columnar, compressed), Iceberg/Delta (open table formats with ACID), Cassandra 5 (write-heavy NoSQL), MinIO (local S3-compatible object store). How you store data determines every downstream performance characteristic.
L1
Ingestion — Getting data into the pipeline
Tools in this track: Kafka 3.7 (streaming events, producers/consumers, delivery guarantees), batch file drops, Postgres CDC (change data capture). This layer answers: "how does raw data arrive, at what rate, and how do we not lose it?"
The key insight: a bad decision at a lower layer cascades upward. Choosing raw CSV files over Parquet at L2 will slow down every Spark query at L3, every dbt model at L4, and every dashboard at L5. Get the foundations right first.

🔧 This Track's Tools Mapped to the Stack

Nine tools, five layers. Some tools span multiple layers — DuckDB is both a compute engine and a serving layer query tool. Here is the full map before you write a single line of code.

Tool Layer Arc What it handles
Kafka 3.7 Ingestion (L1) Arc C Event streaming, producer/consumer, delivery guarantees (at-most-once, at-least-once, exactly-once)
PySpark 3.5 Compute (L3) Arc B, D, G Batch + streaming DataFrame transformations over data too large for a single machine
DuckDB 1.0 Compute + Serving (L3/L5) Arc B, F, G Fast local SQL on Parquet files; dbt backend for transformations; ad-hoc queries at the serving layer
Cassandra 5 Storage — write-heavy (L2) Arc E Wide-column NoSQL, partition keys, clustering keys, high-throughput writes — when Postgres can't keep up with insert rate
OpenSearch 2 Storage + Serving (L2/L5) Arc E Full-text search, aggregations, log analytics — stores indexed documents and serves search queries and dashboards
Airflow 2.9 Orchestration (L4) Arc F DAGs, task scheduling, backfills, dependency management — the glue that makes pipelines run on time
dbt Transformation (L4) Arc F SQL-first data transformations, automated tests, column-level lineage — turns raw storage into analytics-ready models
MinIO Storage (L2) Arc G Local S3-compatible object store for Parquet and Iceberg files — no AWS account needed during development
Parquet / Iceberg Storage (L2) Arc B, G Columnar file format (Parquet) + open table format with ACID transactions and time-travel (Iceberg)

➡️ End-to-End Data Flow: All 5 Layers

Every pipeline in this track follows the same physical journey. An event born at the source travels through ingestion, compute, storage, transformation, and ends as a queryable insight. This diagram shows that journey at a glance.

L1 Ingestion Kafka event / DB export L3 Compute Spark / DuckDB L2 Storage Parquet / Cassandra L4 Transform dbt models / Airflow DAGs L5 Serve OpenSearch / DuckDB query INGEST COMPUTE STORE TRANSFORM SERVE

Notice that Storage (L2) sits between Compute (L3) and Transformation (L4). This is intentional: compute first reads raw ingested data, processes it, and writes results back to storage. Transformation then reads from that same storage to produce the clean, modelled views that the serving layer queries. Storage is the shared handshake point between every other layer.

🛒 NaijaCart: All 5 Layers in Production

🛒
Real-world scenario
NaijaCart — Nigeria's largest e-commerce platform uses all 5 layers every day

Tunde is a data engineer at NaijaCart, based in Lagos. Every day, 8 million product views, 400K orders, and 2 million search events land in Kafka (Layer 1 — Ingestion). A PySpark Structured Streaming job reads that Kafka topic and aggregates product popularity scores in near-real time (Layer 3 — Compute). Results land as Parquet files on MinIO, NaijaCart's local S3-compatible object store (Layer 2 — Storage). An Airflow DAG triggers a dbt transformation job every hour that refreshes the "trending products" view by joining popularity scores with inventory data (Layer 4 — Transformation). The merchandising team then queries this view in OpenSearch Dashboards to decide which products to feature on the homepage (Layer 5 — Serve).

Total latency from customer click to dashboard update: under 90 minutes. NaijaCart's Postgres database could never handle 10 million events per day and still serve sub-second analytics queries — the modern data stack makes both possible simultaneously.

⚡ Two Ways to Build the Pipeline: Batch vs Streaming

The five layers remain constant whether you build a batch pipeline or a streaming pipeline. What changes is the tool choice and the latency at each layer. Understanding this trade-off before writing code saves weeks of rework.

Batch Pipeline — Arc G Capstone, Lesson 34
Layer Tool Frequency Latency
L1 Ingestion Postgres CDC / file export Nightly Hours
L3 Compute PySpark (batch mode) Nightly Minutes
L2 Storage Parquet / Iceberg Nightly Hours (written after batch)
L4 Transform dbt + DuckDB On-demand Seconds
L5 Serve DuckDB / BI tool Ad-hoc Seconds
Streaming Pipeline — Arc G Capstone, Lesson 35
Layer Tool Frequency Latency
L1 Ingestion Kafka Continuous Milliseconds
L3 Compute Spark Structured Streaming Continuous Seconds
L2 Storage Cassandra (writes) + Parquet (archive) Continuous Seconds
L4 Transform Spark SQL windowed aggregations Continuous Seconds
L5 Serve OpenSearch Dashboards Near-real-time < 1 minute
When to choose which:
Batch is simpler to build, debug, and operate. Use it when data that is a few hours stale is acceptable — nightly reconciliation reports, daily recommendation refreshes, weekly analytics. Streaming adds Kafka, watermark management, state management, and consumer-group complexity. Use it only when the business genuinely needs data within minutes — fraud detection, live leaderboards, real-time inventory.

🗺️ Arc-to-Layer Map: What You Learn, When

Seven arcs, 36 lessons. Each arc focuses on one or two layers of the stack. After finishing all seven arcs, you will have built every layer from scratch on your own laptop.

Arc Lessons Layer Focus Key Tool
A — Foundations 00–02 Setup + mental models Docker, pip, stack overview
B — PySpark & Formats 03–13 Compute (batch) — L3 PySpark, DuckDB, Parquet
C — Kafka 14–20 Ingestion — L1 Kafka 3.7 (KRaft mode)
D — Structured Streaming 21–24 Compute (real-time) — L3 Spark Structured Streaming
E — NoSQL at Scale 25–28 Storage (NoSQL) — L2 Cassandra, OpenSearch
F — Orchestration 29–33 Transformation & Orchestration — L4 Airflow 2.9, dbt
G — Capstones 34–35 All 5 layers combined Full stack — batch + streaming

🧠 Mini Challenge: Which Layers Do You Need?

Before reading further, try to answer these three architecture questions using only what you know so far. No code required — just layer reasoning.

Three Architecture Scenarios
Scenario 1 — NairaSwift (Lagos fintech)

NairaSwift settles 1 million bank transfers per day. They need a nightly reconciliation report that lists every transfer, its status, and the net position per bank. Which layers do they need? Which can they skip?

Show hint

L1 may be optional — if Postgres already holds all transfer records, a nightly export or CDC is sufficient and Kafka adds unnecessary complexity. They definitely need L3 (PySpark batch over 1M rows), L4 (dbt model to compute net positions), and L5 (a BI tool or DuckDB query for the report). L2 is needed for Parquet intermediate storage. Streaming is not justified — the report runs once a night.

Scenario 2 — JengaTel (Nairobi telecom)

JengaTel generates 40 million CDR (Call Detail Record) events per day. Fraud analysts need to query "calls over 90 minutes from new SIM cards activated in the last 7 days" within 5 minutes of the call ending. Which layers are required?

Show hint

All five layers are required. L1: Kafka ingesting CDR events at millisecond latency. L3: Spark Structured Streaming with a sliding or tumbling window to detect the pattern. L2: Cassandra for fast write + read on the flagged call records; Parquet for archival. L4: Spark SQL windowed aggregation (the transform is embedded in the streaming job here). L5: OpenSearch Dashboards for the fraud analyst UI, updated every 60 seconds.

Scenario 3 — VidyaPath (Bangalore edtech)

VidyaPath has 3 years of learning event logs (200 GB total) stored in S3 as Parquet files. Data scientists want to run ad-hoc SQL like "average lesson completion rate per course, per week" on their laptops. There is no real-time requirement. Which single tool from this track is the right choice?

Show hint

DuckDB, running directly on the Parquet files in S3. At 200 GB, DuckDB's columnar execution engine can handle this on a laptop without needing a Spark cluster, Airflow, dbt, or Kafka. Spinning up Spark for a one-person ad-hoc analytics workload is a classic over-engineering mistake. DuckDB is the right tool: it runs locally, reads Parquet natively, and completes aggregation queries over 200 GB in seconds.

💼 Interview Q&A

These are the architecture questions that come up in data engineering interviews at product companies. Practice answering them in 60–90 seconds each.

Easy Name the 5 layers of the modern data stack in order from bottom to top.
+

Bottom to top: Ingestion (L1) — raw data arrives via Kafka, file exports, or CDC. Storage (L2) — data is persisted in Parquet, Iceberg, Cassandra, or object stores. Compute (L3) — Spark or DuckDB transforms and aggregates data. Transformation & Orchestration (L4) — dbt models clean and model data; Airflow schedules the pipeline. Serving (L5) — analysts query via dashboards, BI tools, or SQL. The mnemonic: I-S-C-T-S (Ingest, Store, Compute, Transform, Serve).

Easy Which layer does Kafka belong to, and why?
+

Kafka belongs to Layer 1 — Ingestion. Its job is to reliably receive events from producers (application servers, IoT devices, databases) and durably buffer them until consumers are ready to process them. Kafka does not transform data, it does not store final analytical results, and it does not serve dashboards. It is a distributed, fault-tolerant message queue — which is exactly what the ingestion layer needs. The confusion arises because Kafka stores data on disk, but that storage is temporary and exists only to buffer events between producers and consumers, not to hold analytics data long-term.

Medium Tunde at NaijaCart needs trending products updated within 90 minutes. Which layers does the batch pipeline need, and what would change in a streaming approach?
+

The batch pipeline needs all five layers. L1: Kafka ingests click events (or a nightly Postgres export if streaming isn't required). L2: Parquet files on MinIO hold intermediate results. L3: PySpark batch job aggregates product views per hour. L4: Airflow triggers every hour; dbt refreshes the trending products view. L5: OpenSearch Dashboards shows the latest results to the merchandising team. For the 90-minute SLA, a batch pipeline running hourly is sufficient — no Structured Streaming required. If the SLA were under 10 minutes, you'd replace the hourly PySpark batch with a Spark Structured Streaming job reading directly from Kafka, which would cut end-to-end latency to under 2 minutes at the cost of significantly higher operational complexity.

Medium What is the difference between a data warehouse (Snowflake/BigQuery) and a lakehouse (Iceberg + Spark)? Which layers do they replace?
+

A data warehouse (Snowflake, BigQuery) is a managed, vendor-hosted system that combines L2 (Storage), L3 (Compute), and sometimes L4 (Transformation) into one SaaS product. You pay per query or per compute slot. The benefit is zero infrastructure — no clusters to manage. The downside is vendor lock-in and cost at very high data volumes. A lakehouse (Iceberg + Parquet on MinIO/S3 + Spark) separates storage (L2, Iceberg/Parquet files you own) from compute (L3, Spark you run yourself). You pay only for object storage and the compute you actually use. The benefit is no lock-in and lower cost at scale. The downside is operational burden — you manage Spark clusters, Iceberg metadata, and schema evolution yourself. In this track, we build a lakehouse architecture (no cloud account required) and survey warehouses in Arc F as an alternative for teams that prefer managed infrastructure.

Hard In the streaming pipeline: Kafka is at L1, Spark is at L3. What happens at L2 (storage) when you need both low-latency reads (Cassandra) AND long-term storage (Parquet)? How does Spark write to both simultaneously?
+

This is the dual-sink pattern, also called the Lambda architecture at the storage layer. The Spark Structured Streaming job writes to two sinks in the same micro-batch: (1) Cassandra, using the Spark-Cassandra connector, for low-latency random reads — fraud analysts query individual records by partition key within milliseconds. (2) Parquet files on MinIO, for long-term archival and batch reprocessing — the same data is stored column-compressed for efficient full-scan queries later. The key implementation detail: Spark uses foreachBatch to write to both sinks within a single atomic micro-batch, then commits the Kafka offset only after both writes succeed. If either sink fails, the offset is not committed and the batch is retried. This gives you at-least-once delivery to both sinks. For exactly-once semantics in Cassandra, you use idempotent inserts (upserts) with a unique row key so retries don't create duplicates.

📝 Quiz: 5 Questions

Question 1 of 5
Kafka belongs to which layer of the modern data stack?
Question 2 of 5
dbt (data build tool) primarily sits at which layer?
Question 3 of 5
Which tool in this track covers both the Compute and Serving layers?
Question 4 of 5
NaijaCart's pipeline achieves what end-to-end latency from customer click to dashboard update?
Question 5 of 5
In Arc G (the capstone arc), you build:
0
/ 5
Layer reasoning complete.

The stack makes more sense once you build it. Arc B is next.

⚠️ Common Mistakes

"Our data is streaming, so we must use Kafka"
Kafka is powerful but adds real operational complexity — consumer groups, partition management, offset commits, rebalancing. If your consumers run hourly anyway, a Postgres CDC export or a simple file drop is a far simpler ingestion layer. Use Kafka when you genuinely need millisecond-latency ingestion and multiple independent consumers reading the same events.
"The modern data stack means cloud-only tools"
Spark, DuckDB, Kafka, Cassandra, OpenSearch, Airflow, dbt, and MinIO all run entirely on your laptop via Docker. This entire track requires no cloud account. "Modern" refers to the architectural pattern — composable, purpose-built layers — not to the deployment target.
"dbt and DuckDB do the same thing — serving layer"
dbt transforms data (Layer 4 — Transformation). It reads from your storage layer and writes clean, modelled tables back to storage. DuckDB serves data (Layer 5) — analysts run ad-hoc SQL queries against it. They have different responsibilities. dbt is a build tool; DuckDB is a query engine. In this track, dbt uses DuckDB as its backend, but that doesn't mean they're the same layer.
"Storage format is a detail I'll figure out later"
Choosing raw CSV files over Parquet, or choosing flat files over Iceberg when you need schema evolution, is a Layer 2 decision that cascades to every layer above it. A Spark query over 1 TB of CSV takes 40× longer than the same query over Parquet. Get storage format right before you build compute, transformation, or serving. This track covers formats in Arc B, before any other tool.

✅ Key Takeaways

  • The modern data stack has 5 layers: Ingestion → Storage → Compute → Transformation → Serving. Every pipeline — batch or streaming — passes through all five.
  • This track covers all 5 layers: Kafka (L1), Parquet + Cassandra + MinIO (L2), PySpark + DuckDB (L3), dbt + Airflow (L4), OpenSearch + DuckDB (L5).
  • Batch pipeline: nightly cadence with hours of latency — simpler to operate, cheaper compute, right for most analytical workloads.
  • Streaming pipeline: Kafka + Spark Structured Streaming with seconds of latency — justified when freshness genuinely matters (fraud detection, live rankings), not for convenience.
  • Both capstone projects (Lessons 34 and 35) build end-to-end across all 5 layers — you will have built a complete data platform on your laptop by the end of this track.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Now you know where everything fits — let's build it layer by layer.

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →