๐Ÿ’ก How to use this cheatsheet
This page is your daily companion throughout the Big Data Engineering track. Keep it open in a pinned tab.
  • Ctrl+F / Cmd+F to search for any command by keyword (e.g., "topic", "schema", "parquet").
  • Commands inside the Docker stack always assume you've run docker compose up -d first. Check the services card below for ports.
  • Shell commands that begin with docker exec -it run inside the container โ€” the host machine doesn't have Kafka or cqlsh installed directly.
  • PySpark code blocks are meant to be pasted into a pyspark REPL or run with spark-submit.
  • If a command fails, run docker ps first โ€” the most common cause is a container that isn't running.

๐Ÿณ Quick Docker Compose โ€” Services & Ports

Every lesson in the track uses the same Docker stack. These are the services and their exposed ports on localhost.

Apache Kafka
9092
Broker โ€” producers & consumers
Apache Cassandra
9042
CQL native transport
OpenSearch
9200
REST API + Dashboards 5601
PostgreSQL
5432
Relational store / Airflow DB
MinIO
9000
S3-compatible object store; Console 9001
Apache Airflow
8080
Webserver UI
docker โ€” stack managementrun from the project root

๐Ÿ PySpark

PySpark 3.5

Run Spark jobs locally in single-machine mode (local[*] uses all CPU cores) or interactively via the pyspark shell.

# Submit a job spark-submit --master local[*] job.py
# Interactive shell (Python) pyspark
# Submit with extra packages spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.0 job.py
# Set executor memory spark-submit --master local[*] --executor-memory 2g job.py

SparkSession boilerplate

spark_session.pyalways your first few lines

DataFrame inspection

df_inspection.pyunderstand your data

Transformations (lazy โ€” no execution until an action)

transformations.pybuild the plan โ€” nothing runs yet

Reading & writing data

io.pyparquet, csv, json, and spark.sql

Persistence (caching)

caching.pyreuse DataFrames across multiple actions

๐ŸŒŠ Kafka CLI (inside Docker)

Kafka 3.x

All Kafka CLI tools live inside the Kafka container. Run them with docker exec -it bigdata-kafka <command> or open a shell first with docker exec -it bigdata-kafka bash.

Bootstrap server shorthand

In every command below, --bootstrap-server localhost:9092 points to the Kafka broker. If you're running commands from inside the container, you may need to use the container hostname instead (e.g., kafka:9092). Check docker inspect bigdata-kafka if you're unsure.

Topic management

kafka-topics.shlist, create, describe, delete

Produce & consume from the terminal

kafka-console-producer/consumerquick testing without writing code

Consumer groups & lag

kafka-consumer-groups.shmonitor lag โ€” the most important operational metric

๐Ÿ—„๏ธ Cassandra (cqlsh)

Cassandra 4.x

Connect to cqlsh โ€” the Cassandra query shell โ€” via Docker. All CQL commands end with a semicolon.

# Open cqlsh docker exec -it bigdata-cassandra cqlsh
# Node health (run outside container) docker exec bigdata-cassandra nodetool status
# Node info docker exec bigdata-cassandra nodetool info
# Compact a table docker exec bigdata-cassandra nodetool compact payments_ks payments

CQL โ€” keyspaces, tables, CRUD

cassandra.cqlrun inside cqlsh

๐Ÿ” OpenSearch (curl REST API)

OpenSearch 2.x

OpenSearch exposes a REST API on port 9200. Use curl from your host machine (no Docker exec needed). Add ?pretty to any request for formatted JSON output.

opensearch-curl.shcluster, indices, documents, search

๐Ÿฆ† DuckDB

DuckDB 0.10+

DuckDB is an embedded OLAP database โ€” no server needed. Run it in-process from Python or use the CLI for ad hoc analysis of Parquet and CSV files directly on disk.

# Open / create a database file duckdb mydb.duckdb
# In-memory (no persistence) duckdb
# List tables in current DB .tables
# Set output to column mode .mode column
duckdb.sqlparquet, create table, export

๐Ÿ”ง dbt

dbt-core + dbt-duckdb

dbt transforms data in your warehouse using SQL SELECT statements. Run these commands from inside your dbt project directory (where dbt_project.yml lives).

# Create a new project dbt init my_project
# Check connection to DB dbt debug
# Run all models dbt run
# Run a single model dbt run --select stg_payments
# Run + upstream dependencies dbt run --select +stg_payments
# Run + downstream dependents dbt run --select stg_payments+
# Run all tests dbt test
# Test a single model dbt test --select stg_payments
# Build = run + test in order dbt build
# Generate docs + serve locally dbt docs generate && dbt docs serve
# Compile SQL without running dbt compile
# List all models dbt ls
profiles.yml location

dbt reads your database credentials from ~/.dbt/profiles.yml. Run dbt debug to verify the connection before running any models. The project name in dbt_project.yml must match the profile name in profiles.yml.

โฐ Airflow CLI

Airflow 2.x

Use the Airflow CLI to manage DAGs, trigger runs, and test individual tasks from the terminal. The Airflow webserver UI is at http://localhost:8080 (default credentials: airflow / airflow).

# Initialize Airflow DB (first time only) airflow db init
# Migrate DB schema (after upgrade) airflow db migrate
# List all DAGs airflow dags list
# Trigger a DAG manually airflow dags trigger payment_pipeline
# Pause a DAG (stop scheduled runs) airflow dags pause payment_pipeline
# Unpause a DAG airflow dags unpause payment_pipeline
# List DAG runs airflow dags list-runs -d payment_pipeline
# List tasks in a DAG airflow tasks list payment_pipeline
# Test a single task (dry run) airflow tasks test payment_pipeline load_payments 2026-01-01
# Check Airflow config airflow config list
In Docker: prefix every Airflow command with docker exec -it bigdata-airflow-webserver. E.g., docker exec -it bigdata-airflow-webserver airflow dags list.

๐Ÿ“ฆ Python packages

Install all dependencies for the track in one shot, or copy individual packages as needed.

requirements.txt / pip installpaste into your virtual environment
PackageWhat it doesImport as
pysparkDistributed data processingfrom pyspark.sql import SparkSession
confluent-kafkaKafka producer & consumer (librdkafka-backed)from confluent_kafka import Producer, Consumer
cassandra-driverCassandra Python clientfrom cassandra.cluster import Cluster
opensearch-pyOpenSearch Python clientfrom opensearchpy import OpenSearch
duckdbEmbedded OLAP, queries Parquet/CSVimport duckdb
dbt-duckdbdbt adapter for DuckDBCLI only
great-expectationsData quality / validationimport great_expectations as gx
apache-airflowWorkflow orchestrationfrom airflow import DAG
chispaPySpark DataFrame equality assertionsfrom chispa.dataframe_comparer import assert_df_equality
pytestPython test runnerimport pytest

โœ… Key takeaways

  • Bookmark this page โ€” you'll use it every single day during the track. Open it in a pinned tab alongside your code editor.
  • Always start a session by running docker compose up -d and verifying with docker ps before running any commands.
  • Every Kafka CLI tool requires the --bootstrap-server localhost:9092 flag โ€” if you forget it, the command hangs.
  • In Cassandra, always filter on the partition key โ€” queries that don't include it will trigger a full-cluster scan and time out in production.
  • DuckDB can query Parquet and CSV files directly without importing โ€” this is the fastest path for exploratory analysis.
  • Run dbt debug before any dbt run โ€” it verifies your profile, credentials, and project structure in one shot.
Africa & India's Best AI Academy
๐Ÿค– AI-first ๐ŸŽ“ Placement Assistance

From commands to production pipelines

Join the Best AI Program: The AI-Era Data Engineer โ€” 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer โ†’