🔑 What is KRaft?

Before Kafka 3.0, every Kafka cluster required a running Apache ZooKeeper ensemble to manage broker metadata — who is the controller, which partitions have which leaders, what topics exist. ZooKeeper was a separate system you had to install, operate, monitor, and back up alongside Kafka itself.

KRaft = Kafka Raft

Kafka 3.0 replaced ZooKeeper with its own internal metadata management based on the Raft consensus algorithm. Brokers themselves form a quorum and elect a controller from among themselves — no external ZooKeeper needed. The result: fewer moving parts, simpler setup, faster leader election, and a single system to monitor and scale.

In this lesson Tunde, NaijaCart's backend engineer, sets up Kafka locally using the KRaft-mode bitnami/kafka:3.7 image. By the end of the lesson he will have a running broker, a 3-partition topic called naijacart.orders, and messages flowing through a console producer and consumer.

CapabilityZooKeeper era (Kafka < 3.0)KRaft era (Kafka 3.0+)
Metadata store External ZooKeeper cluster Internal Raft log on each broker
Number of processes to start ZooKeeper + Kafka = 2 systems Kafka only = 1 system
Controller election ZooKeeper ephemeral nodes (seconds) Raft leader election (milliseconds)
Production-ready since Kafka 0.8 (2012) Kafka 3.3 (2022)
Max partitions per cluster ~200 000 (ZooKeeper bottleneck) Millions (Raft log scales further)

📋 Prerequisites

Before you start this lesson

You need Docker Desktop (or an equivalent runtime such as Colima or Rancher Desktop) running on your laptop, and the docker-compose.yml file from Lesson 00 located at ~/bigdata-stack/docker-compose.yml. If you have not completed Lesson 00 yet, do that first — this lesson adds only the Kafka-specific steps on top of the stack you already set up.

Confirm Docker is running before proceeding:

terminal Lesson 15 · confirm Docker daemon is running

Container IDs and timestamps will differ on your machine.

sample output
26.1.4

Any version number here means the Docker daemon is healthy. If you get Cannot connect to the Docker daemon, open Docker Desktop and wait for the whale icon to stop animating.

🐳 The Kafka service in docker-compose.yml

The full stack docker-compose.yml defines one service per tool. Below is the Kafka service excerpt. KRaft is configured entirely through environment variables — no separate configuration file needed.

docker-compose.yml (Kafka service excerpt) yaml

Key environment variables explained:

VariableValueWhat it does
KAFKA_CFG_PROCESS_ROLES controller,broker This single node acts as both broker (handles produce/consume) and controller (handles leader election). Typical for local dev.
KAFKA_CFG_NODE_ID 0 Unique integer ID for this broker in the KRaft quorum. Must match the voter list.
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS 0@kafka:9093 Defines the KRaft quorum: node ID 0 reachable at hostname kafka on port 9093. For a single-node cluster there is only one voter.
KAFKA_CFG_ADVERTISED_LISTENERS PLAINTEXT://localhost:9092 Address that clients on the host machine use to connect. localhost:9092 works because the port is mapped.
ALLOW_PLAINTEXT_LISTENER yes Disables TLS enforcement so dev clients can connect without certificates.

▶️ Start Kafka

From the ~/bigdata-stack directory, start only the Kafka service. The -d flag runs it in the background (detached mode).

terminal Lesson 15 · start Kafka container

Container IDs and timestamps will differ on your machine.

sample output
[+] Running 2/2 ✔ Network bigdata-stack_bigdata-net Created ✔ Container bigdata-kafka Started

Docker pulls the bitnami/kafka:3.7 image on first run (about 500 MB). Subsequent starts are instant because the image is cached locally.

✅ Wait for Kafka to be healthy

The healthcheck defined in docker-compose.yml runs kafka-topics.sh --list every 10 seconds. Once Kafka is fully initialised and accepting connections the status changes from starting to running (healthy). Check it with:

terminal Lesson 15 · check container health

Container IDs and timestamps will differ on your machine.

sample output
NAME IMAGE COMMAND SERVICE STATUS bigdata-kafka bitnami/kafka:3.7 "/opt/bitnami/script…" kafka running (healthy)
First start may show "starting" briefly

Kafka needs 15–30 seconds to write its KRaft metadata log and open listener ports. Re-run docker compose ps kafka until STATUS shows running (healthy) before proceeding to topic creation.

📦 Create your first topic

Tunde creates a topic called naijacart.orders with 3 partitions. Using 3 partitions means up to 3 consumers in a group can read in parallel — one partition per consumer — matching NaijaCart's 3-region deployment (Lagos, Abuja, Port Harcourt).

terminal Lesson 15 · create naijacart.orders topic

Container IDs and timestamps will differ on your machine.

sample output
Created topic naijacart.orders.
docker exec pattern

docker exec bigdata-kafka <command> runs any Kafka CLI tool inside the running container. You never need to install Kafka locally — everything runs in the container. The Kafka CLI scripts (kafka-topics.sh, kafka-console-producer.sh, etc.) are all bundled inside the bitnami/kafka:3.7 image.

The --replication-factor 1 flag is correct for a single-broker setup. With only one broker there is nowhere to replicate data. Production clusters typically use replication factor 3 across 3+ brokers.

📋 List topics

Confirm the topic was created by listing all topics on the broker:

terminal Lesson 15 · list all topics

Container IDs and timestamps will differ on your machine.

sample output
naijacart.orders

🔍 Describe the topic

The --describe flag shows the full partition layout: how many partitions, which broker is the leader for each, and which brokers are in the in-sync replica (ISR) set.

terminal Lesson 15 · describe naijacart.orders

Container IDs and timestamps will differ on your machine.

sample output
Topic: naijacart.orders TopicId: xYz123... PartitionCount: 3 ReplicationFactor: 1 Topic: naijacart.orders Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Topic: naijacart.orders Partition: 1 Leader: 0 Replicas: 0 Isr: 0 Topic: naijacart.orders Partition: 2 Leader: 0 Replicas: 0 Isr: 0

All three partitions have Leader 0 — that is our single broker with node ID 0. In a multi-broker cluster the leaders would be distributed across different broker IDs for load balancing.

📤 Console producer — send test messages

The console producer opens an interactive prompt. Each line you type is sent as one Kafka message. Tunde sends two order events for NaijaCart:

terminal Lesson 15 · console producer (interactive)

Container IDs and timestamps will differ on your machine.

sample output
>{"order_id":"ORD001","city":"Lagos","amount":45000} >{"order_id":"ORD002","city":"Abuja","amount":8900} >
Exit the producer: press Ctrl+C to exit the interactive producer session. The messages you sent are already written to the Kafka log — they persist in the topic regardless of whether the producer is running.

The -it flag in docker exec -it allocates a pseudo-TTY so the interactive prompt works correctly. Without it the producer exits immediately.

📥 Console consumer — read messages

The console consumer reads messages from the topic and prints them to stdout. The --from-beginning flag means "start reading from offset 0, not from the latest offset" — so you will see all messages Tunde just sent.

terminal Lesson 15 · console consumer

Container IDs and timestamps will differ on your machine.

sample output
{"order_id":"ORD001","city":"Lagos","amount":45000} {"order_id":"ORD002","city":"Abuja","amount":8900} ^CProcessed a total of 2 messages

Press Ctrl+C to stop the consumer. The messages are not deleted when consumed — Kafka retains them on disk until the configured retention period expires (default 7 days). You can run the consumer again and it will re-read from the beginning.

🩺 health.sh check

Run the health.sh script from Lesson 00 to confirm Kafka's port is accepting connections:

terminal Lesson 15 · stack health check

Container IDs and timestamps will differ on your machine.

sample output
✅ Kafka (9092)

⚙️ Topic config commands (reference)

These commands are useful day-to-day but produce no interesting output to show here. Keep them as a reference:

terminal Lesson 15 · topic administration commands
Partition count is a one-way door

You can increase the partition count of a topic with --alter --partitions but you can never decrease it. This is by design — decreasing would require moving data between partitions, which would break message ordering guarantees. Always choose a partition count you can grow from, but that you are comfortable starting with.

⏹️ Stop Kafka

When you are done with the lesson, stop the Kafka container. Data is preserved in the Docker named volume — the topic and its messages will still be there when you restart.

terminal Lesson 15 · stop Kafka

Container IDs and timestamps will differ on your machine.

sample output
[+] Stopping 1/1 ✔ Container bigdata-kafka Stopped
Warning: docker compose down -v removes named volumes, which permanently deletes all Kafka topic data. Use docker compose stop kafka if you want to preserve your topics and messages.

⚠️ Common mistakes

Typos in topic names
Kafka is case-sensitive. naijacart.orders and NaijaCart.Orders are two completely different topics. Kafka will silently create a new topic rather than erroring — so your producer writes to one topic and your consumer reads from another, both with zero messages. Always double-check the topic name in both producer and consumer commands.
Forgetting --from-beginning in consumer
By default, kafka-console-consumer.sh starts reading from the latest offset — meaning it only shows messages produced after the consumer started. If you sent messages earlier and see nothing, add --from-beginning to read from offset 0.
Increasing partitions on a keyed topic
If producers use message keys (e.g., order_id as a key), increasing the partition count changes which partition each key maps to. Messages produced before the increase went to the old partition; messages after go to a new one. This breaks per-key ordering. Plan your partition count before going to production.
Running --replication-factor 2 with only 1 broker
Kafka requires at least as many available brokers as the requested replication factor. With only 1 broker, --replication-factor 2 fails with an error like Replication factor: 2 larger than available brokers: 1. Use --replication-factor 1 for a single-broker local setup.

🧠 Quiz — 3 questions

Question 1 of 3
What does KRaft replace in Kafka 3.x?
Question 2 of 3
How do you see all messages sent to a topic from the very beginning?
Question 3 of 3
Can you decrease the partition count of a Kafka topic?
0
/ 3
Kafka is up and running.

Next: learn to write a Python producer that sends NaijaCart order events programmatically.

🎤 Interview Q&A

Easy What is KRaft and why did Kafka introduce it?
+
KRaft (Kafka Raft) is the internal metadata management system introduced in Kafka 3.0 that replaces the external Apache ZooKeeper dependency. Before KRaft, every Kafka cluster required a separate ZooKeeper ensemble for broker registration, partition leadership election, and topic configuration storage. This meant engineers had to operate and monitor two independent distributed systems. KRaft moves all of this metadata management inside Kafka itself: brokers use the Raft consensus algorithm to elect a controller from among themselves and maintain a replicated metadata log. Benefits include: fewer processes to run and monitor, faster controller failover (milliseconds via Raft instead of seconds via ZooKeeper ephemeral nodes), the ability to scale to millions of partitions, and simpler Docker and Kubernetes deployments because there is only one system to configure.
Medium How do you monitor consumer group lag?
+
Consumer group lag is the difference between the latest offset produced to a partition and the last offset committed by a consumer group for that partition. It tells you how far behind your consumers are. You can monitor it with the Kafka CLI tool: kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <group-id>. The output shows LAG per partition. In production, teams use metrics exporters like the JMX Exporter with Prometheus and alert when lag exceeds a threshold (e.g., 100 000 messages or 5 minutes of data). High lag can mean consumers are too slow (scale out the consumer group), a consumer has crashed (check consumer logs), or a message is causing repeated processing failures (check for deserialization errors or dead letter queue usage). For NaijaCart, Tunde would alert if order processing lag exceeds 30 seconds, since orders must be processed before the customer sees a confirmation.
Medium What happens to messages when the retention period expires?
+
Kafka retains messages on disk in immutable log segments. Each topic has a retention policy — either time-based (default: log.retention.hours=168, which is 7 days) or size-based (log.retention.bytes). When a log segment's age exceeds the retention period, or the total topic size exceeds the size limit, Kafka's log cleaner deletes those old segments. The deletion is at segment granularity, not individual message granularity — an entire segment file (typically 1 GB) is removed at once. Consumers that have already committed past those offsets are unaffected. Consumers that have lagged behind and try to read from a deleted offset will get an OffsetOutOfRange exception — this is a lag alert signal. For topics where you want to keep only the latest value per key (e.g., user preferences), you can use log compaction (log.cleanup.policy=compact) instead of deletion: compaction retains the most recent message per key indefinitely, removing older duplicates.

✅ Key takeaways

  • KRaft mode eliminates ZooKeeper — Kafka 3.x brokers manage their own metadata using the Raft consensus algorithm. Fewer processes, faster leader election, simpler setup.
  • The entire Kafka setup is three environment variables: KAFKA_CFG_PROCESS_ROLES, KAFKA_CFG_CONTROLLER_QUORUM_VOTERS, and KAFKA_CFG_ADVERTISED_LISTENERS. Everything else is defaults.
  • docker exec bigdata-kafka <command> gives you access to all Kafka CLI tools without installing Kafka on your host machine — use it for topics, producers, consumers, and consumer group management.
  • Partition count can only increase, never decrease. Choose a partition count that is a multiple of your expected maximum consumer group size and leave room to scale.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Build real streaming pipelines with Kafka

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →