🗺️ What this lesson covers

This is a pure setup lesson — no theory yet, just a working environment. By the end you will have every tool, library, and containerised service that the rest of the track depends on. Every subsequent lesson starts with "run ./start.sh if not already running" and assumes this lesson is done.

You will install and verify:

  • ▸ Python 3.12 + JDK 21 (host machine)
  • ▸ 8 pip packages including PySpark 3.5, DuckDB 1.0, and dbt-duckdb
  • ▸ Docker Desktop (enables the container layer)
  • ▸ 6 services via Docker Compose: Kafka (KRaft), Cassandra 5, OpenSearch 2, PostgreSQL 16, MinIO, Airflow 2.9
Hardware requirements

16 GB RAM recommended (8 GB workable for Arc A–B lessons that only use PySpark + DuckDB locally). 30 GB free disk space (images total ~3 GB compressed; Spark shuffle files and Parquet datasets add up). macOS, Linux, or Windows WSL2 all supported.

Your local Big Data stack — end state of Lesson 00 Host — Python 3.12 · JDK 21 · pip packages (PySpark, DuckDB, dbt-duckdb, chispa …) Lessons write and run code here Docker Desktop — container runtime Manages memory limits, networking, named volumes, image cache docker-compose.yml — 6 services Kafka (9092) · Cassandra (9042) · OpenSearch (9200) · PostgreSQL (5432) · MinIO (9000) · Airflow (8080)

📋 Requirements

Here is everything you need on your machine before the first lesson. The table shows why each dependency matters and the fastest way to get it.

Software Version Why needed Install command
Python 3.11+ (3.12 OK) PySpark, DuckDB, and all pip packages run here brew install python@3.11 or use system Python 3.12
JDK 21 21 LTS PySpark launches a JVM subprocess; without JDK, SparkSession.builder.getOrCreate() fails brew install openjdk@21
Docker Desktop Latest Runs Kafka, Cassandra, OpenSearch, PostgreSQL, MinIO, Airflow as containers — no manual installs docker.com/products/docker-desktop
pip packages see below pyspark, duckdb, confluent-kafka, cassandra-driver, opensearch-py, pytest, chispa, dbt-duckdb — one per track arc See pip install block in next section
M-series Mac note: All pip packages have native arm64 wheels. Docker images for Cassandra require platform: linux/amd64 — already included in the provided docker-compose.yml.

🐍 Step 1 — Verify Python and Java

Run these two commands to confirm the host runtime versions. Both are required before you touch any PySpark code.

python3 --version && java -version 2>&1
terminal Lesson 00 · host runtime check
verified output
Python 3.12.4 openjdk version "21.0.11" 2026-04-21 OpenJDK Runtime Environment Homebrew (build 21.0.11) OpenJDK 64-Bit Server VM Homebrew (build 21.0.11, mixed mode, sharing)

If Java is not on your PATH after brew install openjdk@21, add these two lines to your ~/.zshrc or ~/.bashrc and then source the file:

~/.zshrc shell profile — add once
Why does PySpark need a JVM?

PySpark is a Python wrapper around Apache Spark, which is written in Scala and runs on the JVM. When you call SparkSession.builder.getOrCreate(), Python spawns a JVM subprocess via $JAVA_HOME/bin/java. The Python process and the JVM talk over a local socket (Py4J). No JDK = no JVM = JAVA_HOME is not set error.

📦 Step 2 — Install pip packages

One command installs every library you will use across all six arcs of this track. Pinned versions ensure your outputs match the lesson's verified outputs exactly.

pip install pyspark==3.5.0 duckdb==1.0.0 confluent-kafka==2.5.0 cassandra-driver==3.29.1 opensearch-py==2.6.0 pytest==8.3.0 chispa==0.10.1 dbt-duckdb==1.8.0
terminal Lesson 00 · install all packages
verified output — packages installed
Successfully installed pyspark-3.5.0 Successfully installed duckdb-1.0.0 Successfully installed confluent-kafka-2.5.0 Successfully installed cassandra-driver-3.29.1 Successfully installed opensearch-py-2.6.0 Successfully installed pytest-8.3.0 Successfully installed chispa-0.10.1 Successfully installed dbt-duckdb-1.8.0

Quick smoke test

Run this one-liner to confirm PySpark and DuckDB imported correctly and report their versions.

python3 -c "import pyspark, duckdb; print('pyspark', pyspark.__version__); print('duckdb', duckdb.__version__)"
terminal Lesson 00 · smoke test
verified output
pyspark 3.5.0 duckdb 1.0.0
Package role map

pyspark — Arc B (batch), Arc C (streaming with Structured Streaming) · duckdb — Arc B (local SQL), Arc F (DuckDB warehouse) · confluent-kafka — Arc C (Kafka producer/consumer) · cassandra-driver — Arc E · opensearch-py — Arc E · chispa — Arc D (PySpark unit tests) · dbt-duckdb — Arc D (dbt transforms on DuckDB) · pytest — Arc D (test runner)

🐳 Step 3 — Install Docker Desktop

Docker Desktop is the container runtime that starts Kafka, Cassandra, OpenSearch, PostgreSQL, MinIO, and Airflow as isolated processes on your laptop. Each service gets its own filesystem, network, and memory limits — you do not need to install anything manually.

Why does the Big Data stack need so much RAM?

Every service in the stack is a JVM process or memory-hungry daemon:

ServiceMinimum RAMWhat it does with that RAM
Kafka broker (KRaft)512 MBIn-memory offset index, page cache for log segments
Cassandra 51 GBJVM heap for memtable writes and bloom filters
OpenSearch 21 GBLucene segment cache, JVM heap
PostgreSQL 16256 MBShared buffers + connection pool
MinIO256 MBObject storage metadata cache
Airflow 2.9512 MBScheduler + webserver + metadata DB
Total~3.5 GB minimumSet Docker Desktop to 6 GB for headroom

Installation steps

  1. Go to docker.com/products/docker-desktop and download the installer for your OS (macOS Intel, macOS Apple Silicon, Windows, or Linux).
  2. Run the installer and follow the prompts. On macOS, drag Docker.app to Applications.
  3. Launch Docker Desktop. Wait for the whale icon in the menu bar to stop animating — that means the daemon is ready.
  4. Open Settings → Resources → Memory and set it to at least 6 GB. Click "Apply & Restart".
  5. Copy the docker-compose.yml from the Lesson 00 files into your working directory (~/agentic_school_workspace/big-data/).
cd ~/agentic_school_workspace/big-data && docker compose up -d
terminal Lesson 00 · start all services

Container IDs and timestamps will differ on your machine.

sample output
[+] Running 6/6 ✔ Container kafka Started 2.1s ✔ Container cassandra Started 3.4s ✔ Container opensearch Started 2.8s ✔ Container postgres Started 1.9s ✔ Container minio Started 1.7s ✔ Container airflow Started 4.2s

First-start tip: Docker pulls ~3 GB of images on the first run. This takes 3–8 minutes depending on your internet speed. Subsequent starts take 5–10 seconds because images are cached locally.

✅ Step 4 — Verify the stack is healthy

The Lesson 00 files include a health.sh script that checks whether each service's port is accepting connections. Run it after docker compose up -d to confirm all 6 services are ready before you start coding.

./health.sh
health.sh Lesson 00 · health check script

Container IDs and timestamps will differ on your machine.

sample output
✅ Kafka (9092) ✅ Cassandra (9042) ✅ OpenSearch (9200) ✅ PostgreSQL (5432) ✅ MinIO (9000) ✅ Airflow (8080) All services healthy.
Helper scripts in the Lesson 00 files

./start.sh — runs docker compose up -d then polls health.sh until all services are green (max 90 s).
./stop.sh — runs docker compose down (containers stop, volumes preserved).
./reset.sh — runs docker compose down -v (destroys volumes — use only for a clean slate).

🔌 Service port reference

Bookmark this table. Every lesson that talks to a container uses these ports and credentials.

Service Port Web UI Credentials
Kafka (KRaft broker) 9092 None (CLI only) None required
Cassandra CQL 9042 None cassandra / cassandra
OpenSearch REST API 9200 http://localhost:5601 (Dashboards) Security disabled (dev mode)
PostgreSQL 5432 None (use psql or DBeaver) bigdata / bigdata
MinIO S3 API 9000 http://localhost:9001 (Console) minioadmin / minioadmin123
Airflow webserver 8080 http://localhost:8080 airflow / airflow
Ingest Kafka · MinIO Process PySpark · dbt Store Cassandra · Postgres Query DuckDB · OpenSearch

🔧 Troubleshooting

Common issues and exact fixes. Read this before posting in the cohort channel.

SymptomCauseFix
docker: command not found or whale icon never appears Docker Desktop not running Open Docker Desktop from Applications (macOS) or Start menu (Windows). Wait for the whale to stop animating.
Error: address already in use :9092 Port 9092 is occupied by another process Run lsof -i :9092, find the PID, then kill -9 <PID>. Repeat for any conflicting port.
Container exits immediately; docker logs cassandra shows OOM Docker Desktop memory limit too low Settings → Resources → Memory → set to 6 GB minimum → Apply & Restart.
Cassandra container keeps restarting on M1/M2/M3 Mac arm64 Cassandra image is experimental The provided docker-compose.yml already sets platform: linux/amd64. Rosetta 2 translates x86 to arm64 transparently. If you wrote your own compose file, add that line.
First docker compose up takes 5+ minutes All images are being pulled (~3 GB total) Normal on first run. Subsequent starts take 5–10 s because images are cached. Do not interrupt mid-pull.

🎯 Mini challenge

Three verification tasks
Task 1 — Spark session

Run the command below and capture the Spark version printed. It should be 3.5.0. The first run is slow (~20 s) as Spark initialises the JVM.

python3 -c "import pyspark.sql; s = pyspark.sql.SparkSession.builder.appName('test').getOrCreate(); print(s.version); s.stop()"
terminalTask 1
Task 2 — DuckDB in-process query

Run the one-liner below. Expected output: (42,). This confirms DuckDB can run SQL without any server — the database runs inside the Python process.

python3 -c "import duckdb; con = duckdb.connect(); print(con.execute('SELECT 42 AS answer').fetchone())"
terminalTask 2
Task 3 — Full stack green

Once Docker Desktop is installed and running, start the stack with ./start.sh and then run ./health.sh. All 6 lines should show ✅. Screenshot your terminal and post it in the cohort channel.

🎤 Interview Q&A

Easy Why does PySpark need a JVM (JDK)?
+
PySpark is a Python API that drives Apache Spark, which is written in Scala and compiled to JVM bytecode. When you call SparkSession.builder.getOrCreate(), the Python process starts a child JVM process using $JAVA_HOME/bin/java and communicates with it through a library called Py4J over a local socket. The JVM is where Spark's execution engine, shuffle manager, and memory manager actually live. Python handles your high-level API calls and serialises data to and from the JVM. Without a JDK on PATH, the java binary is not found and Spark fails immediately on session creation.
Easy What is KRaft mode in Kafka and why does it matter?
+
Historically Kafka relied on Apache ZooKeeper as an external metadata store for broker registration, partition leadership elections, and configuration management. KRaft (Kafka Raft) replaces ZooKeeper entirely: the Kafka brokers themselves form a Raft consensus group and elect a controller from among themselves. This matters because (1) you no longer need to run and monitor a separate ZooKeeper ensemble, (2) startup is faster, (3) the metadata log has higher throughput, and (4) Kafka can now support millions of partitions per cluster. KRaft was marked production-ready in Kafka 3.3 (2022) and ZooKeeper support is scheduled for removal in a future release. In this track's docker-compose.yml, we run a single-broker KRaft cluster — no ZooKeeper container needed.
Medium Why do we need 6 GB allocated to Docker for this stack?
+
Four of the six services — Cassandra, OpenSearch, Kafka, and Airflow — are JVM or otherwise memory-hungry processes. Cassandra needs at least 1 GB for its memtable write buffer and bloom filter heap. OpenSearch needs at least 1 GB for its Lucene segment cache and JVM heap. Kafka needs ~512 MB for its in-memory offset index and page-cache. Airflow needs ~512 MB for its scheduler and webserver. PostgreSQL and MinIO are relatively lightweight at 256 MB each. Total minimum is ~3.5 GB. We recommend 6 GB to give the OS and Docker daemon headroom, and to allow you to also run a PySpark job on the host (which adds another 1–2 GB JVM heap) simultaneously.
Medium What does platform: linux/amd64 fix in docker-compose for M-series Macs?
+
M1/M2/M3 Macs use the arm64 (AArch64) CPU architecture. Docker Hub hosts images built for both linux/amd64 (x86-64) and linux/arm64. The official Cassandra image has an arm64 variant, but it is experimental and frequently crashes under real workloads in the 3.x and early 4.x era. The platform: linux/amd64 directive tells Docker to pull the x86-64 image even on an arm64 host. Apple's Rosetta 2 translation layer then emulates x86-64 instructions transparently with minimal overhead (~5% slower than native). The result is a stable Cassandra at the cost of a small performance hit — entirely acceptable for a local development environment.
Hard If Cassandra, Kafka, and OpenSearch all write to paths inside their containers, how do named volumes prevent data loss on docker compose restart?
+
Each service writes to a path inside the container's writable layer: Cassandra to /var/lib/cassandra, Kafka to /var/lib/kafka/data, OpenSearch to /usr/share/opensearch/data. Without volumes, Docker creates a new writable layer per container on each docker compose up, so all data written in a previous run is gone on the next. Named volumes solve this by mounting a persistent directory managed by Docker's storage driver at those container paths. When you docker compose restart, Docker stops and removes the old containers but keeps the named volumes intact on the host. The new containers created on the next up are mounted to the same volumes and pick up exactly where the old containers left off — same Cassandra keyspaces, same Kafka topic offsets, same OpenSearch indices. Running docker compose down -v is the explicit command to also remove the volumes (a deliberate data-wipe), which is why our reset.sh script calls it only when you want a clean slate.

🧠 Quiz — 5 questions

Question 1 of 5
PySpark 3.5 requires a JVM. Which is the minimum JDK version it supports?
Question 2 of 5
Kafka's KRaft mode removes which external component from the Kafka architecture?
Question 3 of 5
OpenSearch is a community fork of which project?
Question 4 of 5
Which port is Cassandra's CQL native transport port (the one your application driver connects to)?
Question 5 of 5
What does docker compose down -v do that docker compose down alone does not?
0
/ 5
Setup complete.

Now go install Docker Desktop and run ./start.sh.

⚠️ Common mistakes

Allocating only 2 GB to Docker
Each JVM-based service (Cassandra, OpenSearch, Kafka, Airflow) needs 512 MB–1 GB of heap. With all 6 services running at 2 GB total, containers OOM-kill within minutes. Set Docker Desktop memory to 6 GB minimum before running docker compose up.
Using docker-compose v1 (hyphen) instead of docker compose v2 (space)
Docker Desktop ships with Compose V2 as the built-in plugin. The old standalone binary (docker-compose with a hyphen) is deprecated and no longer maintained. Always use docker compose (two words, no hyphen). If you get command not found, you are on an old install — update Docker Desktop.
Starting Arc B Spark lessons without JDK on PATH
import pyspark succeeds (pure Python). But SparkSession.builder.getOrCreate() fails with JAVA_HOME is not set because Python cannot find the JVM binary. Fix: add export JAVA_HOME=/opt/homebrew/opt/openjdk@21 and export PATH=$JAVA_HOME/bin:$PATH to your ~/.zshrc, then source ~/.zshrc.
Forgetting platform: linux/amd64 for Cassandra on M1/M2/M3
The arm64 Cassandra image is experimental and crashes frequently under write load on Apple Silicon. The x86-64 image emulated via Rosetta 2 is stable for development. The provided docker-compose.yml already includes this directive, but if you modify the file or write your own, remember to add it back.

✅ Key takeaways

  • Python 3.12.4, JDK 21.0.11, and all 8 pip packages (pyspark, duckdb, confluent-kafka, cassandra-driver, opensearch-py, pytest, chispa, dbt-duckdb) are installed and verified on your host machine.
  • The docker-compose.yml starts Kafka (KRaft mode), Cassandra 5, OpenSearch 2, PostgreSQL 16, MinIO, and Airflow 2.9 in one command — no cloud account, no manual service installs.
  • Use ./start.sh to start (waits for health), ./stop.sh to stop (preserves data), ./reset.sh to wipe volumes for a clean slate, and ./health.sh to verify all 6 services are green.
  • Every subsequent lesson in this track opens with "run ./start.sh from the Lesson 00 workspace if not already running" — this is the only environment setup you will ever do.
  • Add export JAVA_HOME=/opt/homebrew/opt/openjdk@21 to your ~/.zshrc or ~/.bashrc to avoid re-exporting each terminal session.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

The foundation our AI platform engineers stand on

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →