🗺️ What this lesson covers
This is a pure setup lesson — no theory yet, just a working environment. By the end you will have every tool, library, and containerised service that the rest of the track depends on. Every subsequent lesson starts with "run ./start.sh if not already running" and assumes this lesson is done.
You will install and verify:
- ▸ Python 3.12 + JDK 21 (host machine)
- ▸ 8 pip packages including PySpark 3.5, DuckDB 1.0, and dbt-duckdb
- ▸ Docker Desktop (enables the container layer)
- ▸ 6 services via Docker Compose: Kafka (KRaft), Cassandra 5, OpenSearch 2, PostgreSQL 16, MinIO, Airflow 2.9
16 GB RAM recommended (8 GB workable for Arc A–B lessons that only use PySpark + DuckDB locally). 30 GB free disk space (images total ~3 GB compressed; Spark shuffle files and Parquet datasets add up). macOS, Linux, or Windows WSL2 all supported.
📋 Requirements
Here is everything you need on your machine before the first lesson. The table shows why each dependency matters and the fastest way to get it.
| Software | Version | Why needed | Install command |
|---|---|---|---|
| Python | 3.11+ (3.12 OK) | PySpark, DuckDB, and all pip packages run here | brew install python@3.11 or use system Python 3.12 |
| JDK 21 | 21 LTS | PySpark launches a JVM subprocess; without JDK, SparkSession.builder.getOrCreate() fails |
brew install openjdk@21 |
| Docker Desktop | Latest | Runs Kafka, Cassandra, OpenSearch, PostgreSQL, MinIO, Airflow as containers — no manual installs | docker.com/products/docker-desktop |
| pip packages | see below | pyspark, duckdb, confluent-kafka, cassandra-driver, opensearch-py, pytest, chispa, dbt-duckdb — one per track arc | See pip install block in next section |
platform: linux/amd64 — already included in the provided docker-compose.yml.
🐍 Step 1 — Verify Python and Java
Run these two commands to confirm the host runtime versions. Both are required before you touch any PySpark code.
If Java is not on your PATH after brew install openjdk@21, add these two lines to your ~/.zshrc or ~/.bashrc and then source the file:
PySpark is a Python wrapper around Apache Spark, which is written in Scala and runs on the JVM. When you call SparkSession.builder.getOrCreate(), Python spawns a JVM subprocess via $JAVA_HOME/bin/java. The Python process and the JVM talk over a local socket (Py4J). No JDK = no JVM = JAVA_HOME is not set error.
📦 Step 2 — Install pip packages
One command installs every library you will use across all six arcs of this track. Pinned versions ensure your outputs match the lesson's verified outputs exactly.
Quick smoke test
Run this one-liner to confirm PySpark and DuckDB imported correctly and report their versions.
pyspark — Arc B (batch), Arc C (streaming with Structured Streaming) · duckdb — Arc B (local SQL), Arc F (DuckDB warehouse) · confluent-kafka — Arc C (Kafka producer/consumer) · cassandra-driver — Arc E · opensearch-py — Arc E · chispa — Arc D (PySpark unit tests) · dbt-duckdb — Arc D (dbt transforms on DuckDB) · pytest — Arc D (test runner)
🐳 Step 3 — Install Docker Desktop
Docker Desktop is the container runtime that starts Kafka, Cassandra, OpenSearch, PostgreSQL, MinIO, and Airflow as isolated processes on your laptop. Each service gets its own filesystem, network, and memory limits — you do not need to install anything manually.
Why does the Big Data stack need so much RAM?
Every service in the stack is a JVM process or memory-hungry daemon:
| Service | Minimum RAM | What it does with that RAM |
|---|---|---|
| Kafka broker (KRaft) | 512 MB | In-memory offset index, page cache for log segments |
| Cassandra 5 | 1 GB | JVM heap for memtable writes and bloom filters |
| OpenSearch 2 | 1 GB | Lucene segment cache, JVM heap |
| PostgreSQL 16 | 256 MB | Shared buffers + connection pool |
| MinIO | 256 MB | Object storage metadata cache |
| Airflow 2.9 | 512 MB | Scheduler + webserver + metadata DB |
| Total | ~3.5 GB minimum | Set Docker Desktop to 6 GB for headroom |
Installation steps
- Go to docker.com/products/docker-desktop and download the installer for your OS (macOS Intel, macOS Apple Silicon, Windows, or Linux).
- Run the installer and follow the prompts. On macOS, drag Docker.app to Applications.
- Launch Docker Desktop. Wait for the whale icon in the menu bar to stop animating — that means the daemon is ready.
- Open Settings → Resources → Memory and set it to at least 6 GB. Click "Apply & Restart".
- Copy the
docker-compose.ymlfrom the Lesson 00 files into your working directory (~/agentic_school_workspace/big-data/).
Container IDs and timestamps will differ on your machine.
First-start tip: Docker pulls ~3 GB of images on the first run. This takes 3–8 minutes depending on your internet speed. Subsequent starts take 5–10 seconds because images are cached locally.
✅ Step 4 — Verify the stack is healthy
The Lesson 00 files include a health.sh script that checks whether each service's port is accepting connections. Run it after docker compose up -d to confirm all 6 services are ready before you start coding.
Container IDs and timestamps will differ on your machine.
./start.sh — runs docker compose up -d then polls health.sh until all services are green (max 90 s).
./stop.sh — runs docker compose down (containers stop, volumes preserved).
./reset.sh — runs docker compose down -v (destroys volumes — use only for a clean slate).
🔌 Service port reference
Bookmark this table. Every lesson that talks to a container uses these ports and credentials.
| Service | Port | Web UI | Credentials |
|---|---|---|---|
| Kafka (KRaft broker) | 9092 | None (CLI only) | None required |
| Cassandra CQL | 9042 | None | cassandra / cassandra |
| OpenSearch REST API | 9200 | http://localhost:5601 (Dashboards) | Security disabled (dev mode) |
| PostgreSQL | 5432 | None (use psql or DBeaver) | bigdata / bigdata |
| MinIO S3 API | 9000 | http://localhost:9001 (Console) | minioadmin / minioadmin123 |
| Airflow webserver | 8080 | http://localhost:8080 | airflow / airflow |
🔧 Troubleshooting
Common issues and exact fixes. Read this before posting in the cohort channel.
| Symptom | Cause | Fix |
|---|---|---|
| docker: command not found or whale icon never appears | Docker Desktop not running | Open Docker Desktop from Applications (macOS) or Start menu (Windows). Wait for the whale to stop animating. |
Error: address already in use :9092 |
Port 9092 is occupied by another process | Run lsof -i :9092, find the PID, then kill -9 <PID>. Repeat for any conflicting port. |
Container exits immediately; docker logs cassandra shows OOM |
Docker Desktop memory limit too low | Settings → Resources → Memory → set to 6 GB minimum → Apply & Restart. |
| Cassandra container keeps restarting on M1/M2/M3 Mac | arm64 Cassandra image is experimental | The provided docker-compose.yml already sets platform: linux/amd64. Rosetta 2 translates x86 to arm64 transparently. If you wrote your own compose file, add that line. |
First docker compose up takes 5+ minutes |
All images are being pulled (~3 GB total) | Normal on first run. Subsequent starts take 5–10 s because images are cached. Do not interrupt mid-pull. |
🎯 Mini challenge
Run the command below and capture the Spark version printed. It should be 3.5.0. The first run is slow (~20 s) as Spark initialises the JVM.
Run the one-liner below. Expected output: (42,). This confirms DuckDB can run SQL without any server — the database runs inside the Python process.
Once Docker Desktop is installed and running, start the stack with ./start.sh and then run ./health.sh. All 6 lines should show ✅. Screenshot your terminal and post it in the cohort channel.
🎤 Interview Q&A
SparkSession.builder.getOrCreate(), the Python process starts a child JVM process using $JAVA_HOME/bin/java and communicates with it through a library called Py4J over a local socket. The JVM is where Spark's execution engine, shuffle manager, and memory manager actually live. Python handles your high-level API calls and serialises data to and from the JVM. Without a JDK on PATH, the java binary is not found and Spark fails immediately on session creation.
platform: linux/amd64 fix in docker-compose for M-series Macs?
linux/amd64 (x86-64) and linux/arm64. The official Cassandra image has an arm64 variant, but it is experimental and frequently crashes under real workloads in the 3.x and early 4.x era. The platform: linux/amd64 directive tells Docker to pull the x86-64 image even on an arm64 host. Apple's Rosetta 2 translation layer then emulates x86-64 instructions transparently with minimal overhead (~5% slower than native). The result is a stable Cassandra at the cost of a small performance hit — entirely acceptable for a local development environment.
docker compose restart?
/var/lib/cassandra, Kafka to /var/lib/kafka/data, OpenSearch to /usr/share/opensearch/data. Without volumes, Docker creates a new writable layer per container on each docker compose up, so all data written in a previous run is gone on the next. Named volumes solve this by mounting a persistent directory managed by Docker's storage driver at those container paths. When you docker compose restart, Docker stops and removes the old containers but keeps the named volumes intact on the host. The new containers created on the next up are mounted to the same volumes and pick up exactly where the old containers left off — same Cassandra keyspaces, same Kafka topic offsets, same OpenSearch indices. Running docker compose down -v is the explicit command to also remove the volumes (a deliberate data-wipe), which is why our reset.sh script calls it only when you want a clean slate.
🧠 Quiz — 5 questions
docker compose down -v do that docker compose down alone does not?Now go install Docker Desktop and run ./start.sh.
⚠️ Common mistakes
docker compose up.docker-compose with a hyphen) is deprecated and no longer maintained. Always use docker compose (two words, no hyphen). If you get command not found, you are on an old install — update Docker Desktop.import pyspark succeeds (pure Python). But SparkSession.builder.getOrCreate() fails with JAVA_HOME is not set because Python cannot find the JVM binary. Fix: add export JAVA_HOME=/opt/homebrew/opt/openjdk@21 and export PATH=$JAVA_HOME/bin:$PATH to your ~/.zshrc, then source ~/.zshrc.platform: linux/amd64 for Cassandra on M1/M2/M3✅ Key takeaways
- Python 3.12.4, JDK 21.0.11, and all 8 pip packages (pyspark, duckdb, confluent-kafka, cassandra-driver, opensearch-py, pytest, chispa, dbt-duckdb) are installed and verified on your host machine.
- The docker-compose.yml starts Kafka (KRaft mode), Cassandra 5, OpenSearch 2, PostgreSQL 16, MinIO, and Airflow 2.9 in one command — no cloud account, no manual service installs.
- Use
./start.shto start (waits for health),./stop.shto stop (preserves data),./reset.shto wipe volumes for a clean slate, and./health.shto verify all 6 services are green. - Every subsequent lesson in this track opens with "run
./start.shfrom the Lesson 00 workspace if not already running" — this is the only environment setup you will ever do. - Add
export JAVA_HOME=/opt/homebrew/opt/openjdk@21to your~/.zshrcor~/.bashrcto avoid re-exporting each terminal session.
The foundation our AI platform engineers stand on
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →