Big Data Engineering with Spark & Kafka
Intermediate ยท 36 lessons ยท PySpark ยท Kafka ยท Cassandra ยท OpenSearch ยท DuckDB ยท Airflow
From docker compose up to Kafka pipelines that scale to a billion events. Every output verified by running the code โ no invented outputs, ever.
Foundations
Set up the full local stack (Kafka, Cassandra, OpenSearch), understand when Big Data tools are actually needed, and tour the modern data stack.
PySpark + DuckDB + Data Formats + Testing
From your first DataFrame to 100M-row aggregations โ PySpark transforms, joins, Spark SQL, Parquet/Avro/ORC, DuckDB, performance tuning, and pytest.
Kafka
Topics, partitions, producers, consumers, consumer groups, delivery guarantees โ Kafka from first message to production-ready idempotent pipelines.
Spark + Kafka + Backfills
Spark Structured Streaming reading Kafka, batch-vs-streaming decisions, handling late data and backfills, tumbling-window aggregations.
NoSQL at Scale
Cassandra wide-column data model, OpenSearch indexing and search โ with Kafka โ Cassandra and Kafka โ OpenSearch mini-builds.
Orchestration & the Rest of the Stack
Airflow DAGs, dbt with DuckDB, and three survey lessons covering cloud warehouses, lakehouses, and data quality.
Capstones
Two end-to-end projects: a batch pipeline (Postgres โ Spark โ Parquet โ dbt โ DuckDB) and a streaming pipeline (Kafka โ Structured Streaming โ Cassandra + OpenSearch).
Ready to build Big Data pipelines?
Our cohort takes you from docker compose up to production Spark + Kafka on day 60 โ mentorship, real projects, placement support.