- Ctrl+F / Cmd+F to search for any command by keyword (e.g., "topic", "schema", "parquet").
- Commands inside the Docker stack always assume you've run
docker compose up -dfirst. Check the services card below for ports. - Shell commands that begin with
docker exec -itrun inside the container โ the host machine doesn't have Kafka or cqlsh installed directly. - PySpark code blocks are meant to be pasted into a
pysparkREPL or run withspark-submit. - If a command fails, run
docker psfirst โ the most common cause is a container that isn't running.
๐ณ Quick Docker Compose โ Services & Ports
Every lesson in the track uses the same Docker stack. These are the services and their exposed ports on localhost.
๐ PySpark
Run Spark jobs locally in single-machine mode (local[*] uses all CPU cores) or interactively via the pyspark shell.
SparkSession boilerplate
DataFrame inspection
Transformations (lazy โ no execution until an action)
Reading & writing data
Persistence (caching)
๐ Kafka CLI (inside Docker)
All Kafka CLI tools live inside the Kafka container. Run them with docker exec -it bigdata-kafka <command> or open a shell first with docker exec -it bigdata-kafka bash.
In every command below, --bootstrap-server localhost:9092 points to the Kafka broker. If you're running commands from inside the container, you may need to use the container hostname instead (e.g., kafka:9092). Check docker inspect bigdata-kafka if you're unsure.
Topic management
Produce & consume from the terminal
Consumer groups & lag
๐๏ธ Cassandra (cqlsh)
Connect to cqlsh โ the Cassandra query shell โ via Docker. All CQL commands end with a semicolon.
CQL โ keyspaces, tables, CRUD
๐ OpenSearch (curl REST API)
OpenSearch exposes a REST API on port 9200. Use curl from your host machine (no Docker exec needed). Add ?pretty to any request for formatted JSON output.
๐ฆ DuckDB
DuckDB is an embedded OLAP database โ no server needed. Run it in-process from Python or use the CLI for ad hoc analysis of Parquet and CSV files directly on disk.
๐ง dbt
dbt transforms data in your warehouse using SQL SELECT statements. Run these commands from inside your dbt project directory (where dbt_project.yml lives).
dbt reads your database credentials from ~/.dbt/profiles.yml. Run dbt debug to verify the connection before running any models. The project name in dbt_project.yml must match the profile name in profiles.yml.
โฐ Airflow CLI
Use the Airflow CLI to manage DAGs, trigger runs, and test individual tasks from the terminal. The Airflow webserver UI is at http://localhost:8080 (default credentials: airflow / airflow).
docker exec -it bigdata-airflow-webserver. E.g., docker exec -it bigdata-airflow-webserver airflow dags list.
๐ฆ Python packages
Install all dependencies for the track in one shot, or copy individual packages as needed.
| Package | What it does | Import as |
|---|---|---|
pyspark | Distributed data processing | from pyspark.sql import SparkSession |
confluent-kafka | Kafka producer & consumer (librdkafka-backed) | from confluent_kafka import Producer, Consumer |
cassandra-driver | Cassandra Python client | from cassandra.cluster import Cluster |
opensearch-py | OpenSearch Python client | from opensearchpy import OpenSearch |
duckdb | Embedded OLAP, queries Parquet/CSV | import duckdb |
dbt-duckdb | dbt adapter for DuckDB | CLI only |
great-expectations | Data quality / validation | import great_expectations as gx |
apache-airflow | Workflow orchestration | from airflow import DAG |
chispa | PySpark DataFrame equality assertions | from chispa.dataframe_comparer import assert_df_equality |
pytest | Python test runner | import pytest |
โ Key takeaways
- Bookmark this page โ you'll use it every single day during the track. Open it in a pinned tab alongside your code editor.
- Always start a session by running
docker compose up -dand verifying withdocker psbefore running any commands. - Every Kafka CLI tool requires the
--bootstrap-server localhost:9092flag โ if you forget it, the command hangs. - In Cassandra, always filter on the partition key โ queries that don't include it will trigger a full-cluster scan and time out in production.
- DuckDB can query Parquet and CSV files directly without importing โ this is the fastest path for exploratory analysis.
- Run
dbt debugbefore anydbt runโ it verifies your profile, credentials, and project structure in one shot.
From commands to production pipelines
Join the Best AI Program: The AI-Era Data Engineer โ 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer โ