🔑 What is KRaft?
Before Kafka 3.0, every Kafka cluster required a running Apache ZooKeeper ensemble to manage broker metadata — who is the controller, which partitions have which leaders, what topics exist. ZooKeeper was a separate system you had to install, operate, monitor, and back up alongside Kafka itself.
Kafka 3.0 replaced ZooKeeper with its own internal metadata management based on the Raft consensus algorithm. Brokers themselves form a quorum and elect a controller from among themselves — no external ZooKeeper needed. The result: fewer moving parts, simpler setup, faster leader election, and a single system to monitor and scale.
In this lesson Tunde, NaijaCart's backend engineer, sets up Kafka locally using the KRaft-mode bitnami/kafka:3.7 image. By the end of the lesson he will have a running broker, a 3-partition topic called naijacart.orders, and messages flowing through a console producer and consumer.
| Capability | ZooKeeper era (Kafka < 3.0) | KRaft era (Kafka 3.0+) |
|---|---|---|
| Metadata store | External ZooKeeper cluster | Internal Raft log on each broker |
| Number of processes to start | ZooKeeper + Kafka = 2 systems | Kafka only = 1 system |
| Controller election | ZooKeeper ephemeral nodes (seconds) | Raft leader election (milliseconds) |
| Production-ready since | Kafka 0.8 (2012) | Kafka 3.3 (2022) |
| Max partitions per cluster | ~200 000 (ZooKeeper bottleneck) | Millions (Raft log scales further) |
📋 Prerequisites
You need Docker Desktop (or an equivalent runtime such as Colima or Rancher Desktop) running on your laptop, and the docker-compose.yml file from Lesson 00 located at ~/bigdata-stack/docker-compose.yml. If you have not completed Lesson 00 yet, do that first — this lesson adds only the Kafka-specific steps on top of the stack you already set up.
Confirm Docker is running before proceeding:
Container IDs and timestamps will differ on your machine.
Any version number here means the Docker daemon is healthy. If you get Cannot connect to the Docker daemon, open Docker Desktop and wait for the whale icon to stop animating.
🐳 The Kafka service in docker-compose.yml
The full stack docker-compose.yml defines one service per tool. Below is the Kafka service excerpt. KRaft is configured entirely through environment variables — no separate configuration file needed.
Key environment variables explained:
| Variable | Value | What it does |
|---|---|---|
KAFKA_CFG_PROCESS_ROLES |
controller,broker | This single node acts as both broker (handles produce/consume) and controller (handles leader election). Typical for local dev. |
KAFKA_CFG_NODE_ID |
0 | Unique integer ID for this broker in the KRaft quorum. Must match the voter list. |
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS |
0@kafka:9093 | Defines the KRaft quorum: node ID 0 reachable at hostname kafka on port 9093. For a single-node cluster there is only one voter. |
KAFKA_CFG_ADVERTISED_LISTENERS |
PLAINTEXT://localhost:9092 | Address that clients on the host machine use to connect. localhost:9092 works because the port is mapped. |
ALLOW_PLAINTEXT_LISTENER |
yes | Disables TLS enforcement so dev clients can connect without certificates. |
▶️ Start Kafka
From the ~/bigdata-stack directory, start only the Kafka service. The -d flag runs it in the background (detached mode).
Container IDs and timestamps will differ on your machine.
Docker pulls the bitnami/kafka:3.7 image on first run (about 500 MB). Subsequent starts are instant because the image is cached locally.
✅ Wait for Kafka to be healthy
The healthcheck defined in docker-compose.yml runs kafka-topics.sh --list every 10 seconds. Once Kafka is fully initialised and accepting connections the status changes from starting to running (healthy). Check it with:
Container IDs and timestamps will differ on your machine.
Kafka needs 15–30 seconds to write its KRaft metadata log and open listener ports. Re-run docker compose ps kafka until STATUS shows running (healthy) before proceeding to topic creation.
📦 Create your first topic
Tunde creates a topic called naijacart.orders with 3 partitions. Using 3 partitions means up to 3 consumers in a group can read in parallel — one partition per consumer — matching NaijaCart's 3-region deployment (Lagos, Abuja, Port Harcourt).
Container IDs and timestamps will differ on your machine.
docker exec bigdata-kafka <command> runs any Kafka CLI tool inside the running container. You never need to install Kafka locally — everything runs in the container. The Kafka CLI scripts (kafka-topics.sh, kafka-console-producer.sh, etc.) are all bundled inside the bitnami/kafka:3.7 image.
The --replication-factor 1 flag is correct for a single-broker setup. With only one broker there is nowhere to replicate data. Production clusters typically use replication factor 3 across 3+ brokers.
📋 List topics
Confirm the topic was created by listing all topics on the broker:
Container IDs and timestamps will differ on your machine.
🔍 Describe the topic
The --describe flag shows the full partition layout: how many partitions, which broker is the leader for each, and which brokers are in the in-sync replica (ISR) set.
Container IDs and timestamps will differ on your machine.
All three partitions have Leader 0 — that is our single broker with node ID 0. In a multi-broker cluster the leaders would be distributed across different broker IDs for load balancing.
📤 Console producer — send test messages
The console producer opens an interactive prompt. Each line you type is sent as one Kafka message. Tunde sends two order events for NaijaCart:
Container IDs and timestamps will differ on your machine.
The -it flag in docker exec -it allocates a pseudo-TTY so the interactive prompt works correctly. Without it the producer exits immediately.
📥 Console consumer — read messages
The console consumer reads messages from the topic and prints them to stdout. The --from-beginning flag means "start reading from offset 0, not from the latest offset" — so you will see all messages Tunde just sent.
Container IDs and timestamps will differ on your machine.
Press Ctrl+C to stop the consumer. The messages are not deleted when consumed — Kafka retains them on disk until the configured retention period expires (default 7 days). You can run the consumer again and it will re-read from the beginning.
🩺 health.sh check
Run the health.sh script from Lesson 00 to confirm Kafka's port is accepting connections:
Container IDs and timestamps will differ on your machine.
⚙️ Topic config commands (reference)
These commands are useful day-to-day but produce no interesting output to show here. Keep them as a reference:
You can increase the partition count of a topic with --alter --partitions but you can never decrease it. This is by design — decreasing would require moving data between partitions, which would break message ordering guarantees. Always choose a partition count you can grow from, but that you are comfortable starting with.
⏹️ Stop Kafka
When you are done with the lesson, stop the Kafka container. Data is preserved in the Docker named volume — the topic and its messages will still be there when you restart.
Container IDs and timestamps will differ on your machine.
docker compose down -v removes named volumes, which permanently deletes all Kafka topic data. Use docker compose stop kafka if you want to preserve your topics and messages.
⚠️ Common mistakes
naijacart.orders and NaijaCart.Orders are two completely different topics. Kafka will silently create a new topic rather than erroring — so your producer writes to one topic and your consumer reads from another, both with zero messages. Always double-check the topic name in both producer and consumer commands.kafka-console-consumer.sh starts reading from the latest offset — meaning it only shows messages produced after the consumer started. If you sent messages earlier and see nothing, add --from-beginning to read from offset 0.order_id as a key), increasing the partition count changes which partition each key maps to. Messages produced before the increase went to the old partition; messages after go to a new one. This breaks per-key ordering. Plan your partition count before going to production.--replication-factor 2 fails with an error like Replication factor: 2 larger than available brokers: 1. Use --replication-factor 1 for a single-broker local setup.🧠 Quiz — 3 questions
Next: learn to write a Python producer that sends NaijaCart order events programmatically.
🎤 Interview Q&A
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <group-id>. The output shows LAG per partition. In production, teams use metrics exporters like the JMX Exporter with Prometheus and alert when lag exceeds a threshold (e.g., 100 000 messages or 5 minutes of data). High lag can mean consumers are too slow (scale out the consumer group), a consumer has crashed (check consumer logs), or a message is causing repeated processing failures (check for deserialization errors or dead letter queue usage). For NaijaCart, Tunde would alert if order processing lag exceeds 30 seconds, since orders must be processed before the customer sees a confirmation.
log.retention.hours=168, which is 7 days) or size-based (log.retention.bytes). When a log segment's age exceeds the retention period, or the total topic size exceeds the size limit, Kafka's log cleaner deletes those old segments. The deletion is at segment granularity, not individual message granularity — an entire segment file (typically 1 GB) is removed at once. Consumers that have already committed past those offsets are unaffected. Consumers that have lagged behind and try to read from a deleted offset will get an OffsetOutOfRange exception — this is a lag alert signal. For topics where you want to keep only the latest value per key (e.g., user preferences), you can use log compaction (log.cleanup.policy=compact) instead of deletion: compaction retains the most recent message per key indefinitely, removing older duplicates.
✅ Key takeaways
- KRaft mode eliminates ZooKeeper — Kafka 3.x brokers manage their own metadata using the Raft consensus algorithm. Fewer processes, faster leader election, simpler setup.
- The entire Kafka setup is three environment variables:
KAFKA_CFG_PROCESS_ROLES,KAFKA_CFG_CONTROLLER_QUORUM_VOTERS, andKAFKA_CFG_ADVERTISED_LISTENERS. Everything else is defaults. docker exec bigdata-kafka <command>gives you access to all Kafka CLI tools without installing Kafka on your host machine — use it for topics, producers, consumers, and consumer group management.- Partition count can only increase, never decrease. Choose a partition count that is a multiple of your expected maximum consumer group size and leave room to scale.
Build real streaming pipelines with Kafka
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →