👥 What is a consumer group?
A consumer group is a set of consumers that cooperate to read from a Kafka topic. Each consumer in the group reads from a disjoint set of partitions — Kafka guarantees that each partition is consumed by exactly one consumer in the group at any time.
A group_id string groups consumers together. Kafka divides the topic's partitions equally among them. Each partition is assigned to exactly one consumer in the group at a time. A consumer can read from multiple partitions; no two consumers in the same group read the same partition simultaneously.
This model gives you horizontal scaling on the read path. If a single consumer is too slow to keep up with producers, you add more consumers to the same group and each gets a smaller slice of the partition space to manage.
Importantly, multiple independent consumer groups can all read from the same topic at the same time. Each group maintains its own set of committed offsets — they do not interfere with each other.
📊 Partition assignment — how it works at scale
NaijaCart's naijacart.orders topic has 6 partitions (P0 through P5). The diagram below shows how Kafka divides those partitions depending on how many consumers are in the group.
💻 Code — two consumers, one group
The code below is the same for consumer_1.py and consumer_2.py. The only thing that makes them a group is the shared group_id. Start them in two separate terminals, then produce messages — Kafka will route partitions to each.
Both files are identical. Kafka's Group Coordinator automatically detects that two consumers share the same group_id='order-processor' and divides the partitions between them.
Kafka broker must be running. Output will vary.
Kafka broker must be running. Output will vary.
Notice that ORD001 and ORD002 never appear in the same terminal. Each partition is owned by exactly one consumer. If you kill consumer 1, Kafka will rebalance and consumer 2 will pick up partitions 0 and 2 in addition to its own.
🔀 Multiple independent consumer groups
You can have many independent consumer groups reading the same topic simultaneously. Each group has its own offset cursor — they do not share state. This is one of Kafka's most powerful features for event-driven architectures.
NaijaCart runs three independent groups on naijacart.orders:
Each group maintains independent offsets. The analytics group can be replaying events from 48 hours ago — perhaps because Tunde pushed a fix to a broken aggregation model. That replay does not affect fraud-detection, which is always reading at the live edge. Each group sees the full topic independently.
🔄 Rebalancing — what it is and when it happens
A rebalance is the process by which Kafka redistributes topic partitions among the consumers in a group. It is triggered automatically when group membership changes.
The Group Coordinator — a special role played by one of the Kafka brokers — monitors heartbeats from each consumer. When it detects a membership change (consumer joins, leaves, or misses a heartbeat deadline), it signals all consumers to stop and initiates the rebalance protocol.
During a rebalance, all consumers in the group pause reading. This is sometimes called a "stop the world" event. After the rebalance completes — typically within a few seconds — each consumer resumes reading from its newly assigned partitions at the last committed offset.
Rebalance triggers
| Trigger | What causes it |
|---|---|
| Consumer joins | A new consumer starts and connects with the same group_id |
| Consumer leaves (graceful) | Consumer calls consumer.close() — sends a LeaveGroup request |
| Consumer leaves (timeout) | Consumer misses heartbeat for longer than session.timeout.ms (default 45 s) |
| Partition count changes | A topic is re-partitioned (partitions added) |
| Heartbeat timeout | Consumer is alive but processing too slowly — fails to call poll() within max.poll.interval.ms |
⚠️ The processing-too-slow problem
This is the most common source of unexpected rebalances in production. The Kafka consumer sends heartbeats in a background thread, so a slow main thread does not immediately trigger a session timeout. However, Kafka also tracks how often your consumer calls poll().
If your processing takes longer than max.poll.interval.ms (default: 300,000 ms = 5 minutes) between consecutive poll() calls, Kafka assumes the consumer is dead and triggers a rebalance. The unprocessed messages are reassigned to another consumer in the group, which will re-deliver them from the last committed offset.
Two fixes are available:
- ▸ Increase max.poll.interval.ms: set it higher than your worst-case processing time.
max.poll.interval.ms=600000gives you 10 minutes. - ▸ Make processing faster: use async I/O, batch external calls, or move heavy work to a separate thread pool and commit offsets after the async work completes.
🚨 Chidi's NaijaCart incident
Black Friday Sale — 8-minute payment API call
During a peak sales event, one of NaijaCart's order processor instances called an external payment verification API. Due to a network hiccup at the payment provider's data centre, the API took 8 minutes to respond — well over the default max.poll.interval.ms of 5 minutes.
Kafka's Group Coordinator removed that consumer from the order-processor group. This triggered a rebalance across all 6 remaining consumers. During the rebalance, all 6 consumers paused for 4 seconds. In those 4 seconds, 12,000 orders queued up in the topic with no active consumer reading them.
When the rebalance completed, the 6 consumers resumed — but also had to re-process the orders that the evicted consumer had fetched but not committed, since those offsets were never committed before it was removed.
max.poll.interval.ms=600000 (10 minutes) and replaced the synchronous payment API call with an async HTTP client (aiohttp). The consumer now fires the API request asynchronously, batches up to 50 responses, then commits offsets in bulk. The external call time no longer blocks the poll loop.
📉 Monitoring consumer lag
Consumer lag is the gap between the latest message offset in a partition and the offset the consumer group has committed. A lag of zero means the consumer is keeping up with producers in real time. Growing lag means consumers are falling behind.
Check lag with the built-in Kafka CLI tool. This command describes the order-processor group across all partitions of naijacart.orders:
Kafka broker must be running. Output will vary.
Here, the order-processor group has a lag of 5, 7, and 2 on partitions 0, 1, and 2 respectively. The LOG-END-OFFSET is where the producer last wrote; CURRENT-OFFSET is what the consumer group has committed. The difference is lag.
A lag of 5–10 on a low-throughput topic may be fine. A lag of 500,000 growing at 10,000/s means your consumers are drowning. Set alerts when lag grows by more than X% over 5 minutes. Funmi's team at NaijaCart pages on-call when lag on any partition exceeds 50,000.
⚠️ Common mistakes
session.timeout.ms (e.g., 6,000 ms) causes false rebalances under load — a momentarily busy consumer misses a heartbeat and gets evicted. The default (45 s) is appropriate for most use cases. Do not reduce it unless you need very fast failure detection.ConsumerRebalanceListener.on_partitions_revoked() callback, you get duplicate processing for every rebalance event.group_id strings (e.g., order-processor and Order-Processor) create two completely independent consumer groups. Both read the full partition set independently. If this is unintentional, you end up with duplicate processing and double the resource usage.🧠 Quiz — 3 questions
Next: Delivery Guarantees — at-least-once and exactly-once.
🎤 Interview Q&A
group_id. Kafka's Group Coordinator (one of the brokers) assigns each partition of the subscribed topic to exactly one member of the group. This means reads are parallelised: if a topic has 6 partitions and your group has 6 consumers, each consumer handles exactly one partition's reads independently and concurrently. Adding a 7th consumer would leave it idle (no partition left to assign). The parallelism ceiling is therefore the number of partitions. Multiple consumer groups can read the same topic simultaneously without any coordination between them — each group maintains its own set of committed offsets.
ConsumerRebalanceListener, and (3) use Kafka's Cooperative Sticky Assignor (Kafka 2.4+) which does incremental rebalancing instead of revoking all partitions upfront.
partition.assignment.strategy=CooperativeStickyAssignor) which only revokes and reassigns the minimum set of partitions during a membership change, rather than revoking all partitions from all consumers simultaneously. (4) Implement ConsumerRebalanceListener.on_partitions_revoked() to commit all pending offsets before partitions are taken away, preventing re-processing of already-completed work. (5) Pre-scale before traffic spikes — adding consumers ahead of a sale event is better than having Kafka repeatedly rebalance under load as consumers are dynamically added.
✅ Key takeaways
- A consumer group (identified by
group_id) lets multiple consumers read a topic in parallel — Kafka assigns each partition to exactly one consumer in the group. Parallelism is capped by partition count. - Multiple independent groups can read the same topic simultaneously with separate offset cursors — analytics replaying old events does not affect fraud detection reading the live edge.
- Rebalancing pauses all consumers in a group while Kafka redistributes partitions. It is triggered by consumers joining, leaving, or failing to poll within
max.poll.interval.ms. - Monitor consumer lag (
kafka-consumer-groups.sh --describe) as your primary operational signal. Growing lag means consumers are falling behind — add consumers, increase partitions, or speed up processing.
The foundation our AI platform engineers stand on
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →