All Tracks
๐Ÿ˜

Big Data Engineering with Spark & Kafka

Intermediate ยท 36 lessons ยท PySpark ยท Kafka ยท Cassandra ยท OpenSearch ยท DuckDB ยท Airflow

From docker compose up to Kafka pipelines that scale to a billion events. Every output verified by running the code โ€” no invented outputs, ever.

36 of 36 lessons available100%
A

Foundations

Set up the full local stack (Kafka, Cassandra, OpenSearch), understand when Big Data tools are actually needed, and tour the modern data stack.

B

PySpark + DuckDB + Data Formats + Testing

From your first DataFrame to 100M-row aggregations โ€” PySpark transforms, joins, Spark SQL, Parquet/Avro/ORC, DuckDB, performance tuning, and pytest.

C

Kafka

Topics, partitions, producers, consumers, consumer groups, delivery guarantees โ€” Kafka from first message to production-ready idempotent pipelines.

D

Spark + Kafka + Backfills

Spark Structured Streaming reading Kafka, batch-vs-streaming decisions, handling late data and backfills, tumbling-window aggregations.

E

NoSQL at Scale

Cassandra wide-column data model, OpenSearch indexing and search โ€” with Kafka โ†’ Cassandra and Kafka โ†’ OpenSearch mini-builds.

F

Orchestration & the Rest of the Stack

Airflow DAGs, dbt with DuckDB, and three survey lessons covering cloud warehouses, lakehouses, and data quality.

G

Capstones

Two end-to-end projects: a batch pipeline (Postgres โ†’ Spark โ†’ Parquet โ†’ dbt โ†’ DuckDB) and a streaming pipeline (Kafka โ†’ Structured Streaming โ†’ Cassandra + OpenSearch).

Extras

Ready to build Big Data pipelines?

Our cohort takes you from docker compose up to production Spark + Kafka on day 60 โ€” mentorship, real projects, placement support.

Apply Now โ†’