👥 What is a consumer group?

A consumer group is a set of consumers that cooperate to read from a Kafka topic. Each consumer in the group reads from a disjoint set of partitions — Kafka guarantees that each partition is consumed by exactly one consumer in the group at any time.

The key rule

A group_id string groups consumers together. Kafka divides the topic's partitions equally among them. Each partition is assigned to exactly one consumer in the group at a time. A consumer can read from multiple partitions; no two consumers in the same group read the same partition simultaneously.

This model gives you horizontal scaling on the read path. If a single consumer is too slow to keep up with producers, you add more consumers to the same group and each gets a smaller slice of the partition space to manage.

Importantly, multiple independent consumer groups can all read from the same topic at the same time. Each group maintains its own set of committed offsets — they do not interfere with each other.

📊 Partition assignment — how it works at scale

NaijaCart's naijacart.orders topic has 6 partitions (P0 through P5). The diagram below shows how Kafka divides those partitions depending on how many consumers are in the group.

naijacart.orders — 6 partitions, 3 group configurations Group A — 2 consumers P0 P1 P2 P3 P4 P5 Consumer 1 → P0, P1, P2 Consumer 2 → P3, P4, P5 Group B — 3 consumers P0 P1 P2 P3 P4 P5 Consumer 1 → P0,P1 Consumer 2 → P2,P3 Consumer 3 → P4,P5 Group C — 6 consumers (maximum parallelism) C1 → P0 C2 → P1 C3 → P2 C4 → P3 C5 → P4 C6 → P5 More consumers = more parallelism (up to partition count)
Rule of thumb: Set partition count to the peak number of consumers you will ever need. You can add partitions later but only during off-hours, because doing so triggers a rebalance. You cannot reduce partitions without recreating the topic.

💻 Code — two consumers, one group

The code below is the same for consumer_1.py and consumer_2.py. The only thing that makes them a group is the shared group_id. Start them in two separate terminals, then produce messages — Kafka will route partitions to each.

Run in two separate terminals

Both files are identical. Kafka's Group Coordinator automatically detects that two consumers share the same group_id='order-processor' and divides the partitions between them.

consumer_1.py / consumer_2.py Lesson 18 · identical code, same group_id

Kafka broker must be running. Output will vary.

sample output — consumer 1 (terminal 1)
[Consumer] Partition 0 | Order: ORD001 [Consumer] Partition 0 | Order: ORD003 [Consumer] Partition 2 | Order: ORD005

Kafka broker must be running. Output will vary.

sample output — consumer 2 (terminal 2)
[Consumer] Partition 1 | Order: ORD002 [Consumer] Partition 1 | Order: ORD004 [Consumer] Partition 3 | Order: ORD006

Notice that ORD001 and ORD002 never appear in the same terminal. Each partition is owned by exactly one consumer. If you kill consumer 1, Kafka will rebalance and consumer 2 will pick up partitions 0 and 2 in addition to its own.

🔀 Multiple independent consumer groups

You can have many independent consumer groups reading the same topic simultaneously. Each group has its own offset cursor — they do not share state. This is one of Kafka's most powerful features for event-driven architectures.

NaijaCart runs three independent groups on naijacart.orders:

multiple_groups.py Lesson 18 · same topic, three independent groups
Why this matters

Each group maintains independent offsets. The analytics group can be replaying events from 48 hours ago — perhaps because Tunde pushed a fix to a broken aggregation model. That replay does not affect fraud-detection, which is always reading at the live edge. Each group sees the full topic independently.

🔄 Rebalancing — what it is and when it happens

A rebalance is the process by which Kafka redistributes topic partitions among the consumers in a group. It is triggered automatically when group membership changes.

The Group Coordinator — a special role played by one of the Kafka brokers — monitors heartbeats from each consumer. When it detects a membership change (consumer joins, leaves, or misses a heartbeat deadline), it signals all consumers to stop and initiates the rebalance protocol.

During a rebalance, all consumers in the group pause reading. This is sometimes called a "stop the world" event. After the rebalance completes — typically within a few seconds — each consumer resumes reading from its newly assigned partitions at the last committed offset.

Rebalance triggers

Trigger What causes it
Consumer joins A new consumer starts and connects with the same group_id
Consumer leaves (graceful) Consumer calls consumer.close() — sends a LeaveGroup request
Consumer leaves (timeout) Consumer misses heartbeat for longer than session.timeout.ms (default 45 s)
Partition count changes A topic is re-partitioned (partitions added)
Heartbeat timeout Consumer is alive but processing too slowly — fails to call poll() within max.poll.interval.ms

⚠️ The processing-too-slow problem

This is the most common source of unexpected rebalances in production. The Kafka consumer sends heartbeats in a background thread, so a slow main thread does not immediately trigger a session timeout. However, Kafka also tracks how often your consumer calls poll().

max.poll.interval.ms

If your processing takes longer than max.poll.interval.ms (default: 300,000 ms = 5 minutes) between consecutive poll() calls, Kafka assumes the consumer is dead and triggers a rebalance. The unprocessed messages are reassigned to another consumer in the group, which will re-deliver them from the last committed offset.

Two fixes are available:

  • ▸ Increase max.poll.interval.ms: set it higher than your worst-case processing time. max.poll.interval.ms=600000 gives you 10 minutes.
  • ▸ Make processing faster: use async I/O, batch external calls, or move heavy work to a separate thread pool and commit offsets after the async work completes.

🚨 Chidi's NaijaCart incident

Black Friday Sale — 8-minute payment API call

During a peak sales event, one of NaijaCart's order processor instances called an external payment verification API. Due to a network hiccup at the payment provider's data centre, the API took 8 minutes to respond — well over the default max.poll.interval.ms of 5 minutes.

Kafka's Group Coordinator removed that consumer from the order-processor group. This triggered a rebalance across all 6 remaining consumers. During the rebalance, all 6 consumers paused for 4 seconds. In those 4 seconds, 12,000 orders queued up in the topic with no active consumer reading them.

When the rebalance completed, the 6 consumers resumed — but also had to re-process the orders that the evicted consumer had fetched but not committed, since those offsets were never committed before it was removed.

Chidi's fix Set max.poll.interval.ms=600000 (10 minutes) and replaced the synchronous payment API call with an async HTTP client (aiohttp). The consumer now fires the API request asynchronously, batches up to 50 responses, then commits offsets in bulk. The external call time no longer blocks the poll loop.

📉 Monitoring consumer lag

Consumer lag is the gap between the latest message offset in a partition and the offset the consumer group has committed. A lag of zero means the consumer is keeping up with producers in real time. Growing lag means consumers are falling behind.

Check lag with the built-in Kafka CLI tool. This command describes the order-processor group across all partitions of naijacart.orders:

terminal Lesson 18 · check consumer lag

Kafka broker must be running. Output will vary.

sample output
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG order-processor naijacart.orders 0 150 155 5 order-processor naijacart.orders 1 148 155 7 order-processor naijacart.orders 2 153 155 2

Here, the order-processor group has a lag of 5, 7, and 2 on partitions 0, 1, and 2 respectively. The LOG-END-OFFSET is where the producer last wrote; CURRENT-OFFSET is what the consumer group has committed. The difference is lag.

Lag is your most important operational metric

A lag of 5–10 on a low-throughput topic may be fine. A lag of 500,000 growing at 10,000/s means your consumers are drowning. Set alerts when lag grows by more than X% over 5 minutes. Funmi's team at NaijaCart pages on-call when lag on any partition exceeds 50,000.

⚠️ Common mistakes

More consumers than partitions
If you have 6 partitions and 8 consumers in a group, 2 consumers will be permanently idle — they have no partition to read. Extra consumers do not speed things up. Scale consumers up to the partition count, then increase partitions if you need more parallelism.
session.timeout.ms set too short
A very short session.timeout.ms (e.g., 6,000 ms) causes false rebalances under load — a momentarily busy consumer misses a heartbeat and gets evicted. The default (45 s) is appropriate for most use cases. Do not reduce it unless you need very fast failure detection.
Not handling rebalance listeners
When Kafka revokes partitions during a rebalance, uncommitted offsets on those partitions will be re-processed by the new owner. If you do not commit offsets in a ConsumerRebalanceListener.on_partitions_revoked() callback, you get duplicate processing for every rebalance event.
Mixed group_ids by accident
Two microservices with different group_id strings (e.g., order-processor and Order-Processor) create two completely independent consumer groups. Both read the full partition set independently. If this is unintentional, you end up with duplicate processing and double the resource usage.

🧠 Quiz — 3 questions

Question 1 of 3
A topic has 4 partitions and a consumer group has 6 members. How many consumers are idle?
Question 2 of 3
What triggers a Kafka consumer group rebalance?
Question 3 of 3
Consumer lag measures:
0
/ 3
Group dynamics mastered.

Next: Delivery Guarantees — at-least-once and exactly-once.

🎤 Interview Q&A

Easy What is a consumer group and how does it enable parallel reading?
+
A consumer group is a named set of consumers identified by a shared group_id. Kafka's Group Coordinator (one of the brokers) assigns each partition of the subscribed topic to exactly one member of the group. This means reads are parallelised: if a topic has 6 partitions and your group has 6 consumers, each consumer handles exactly one partition's reads independently and concurrently. Adding a 7th consumer would leave it idle (no partition left to assign). The parallelism ceiling is therefore the number of partitions. Multiple consumer groups can read the same topic simultaneously without any coordination between them — each group maintains its own set of committed offsets.
Medium What is rebalancing and what are its performance implications?
+
Rebalancing is the protocol Kafka runs whenever the membership of a consumer group changes — a consumer joins, leaves, or misses a heartbeat/poll deadline. The Group Coordinator pauses all consumers in the group simultaneously (a "stop the world" event), recalculates the partition assignment, and distributes the new assignment to each member. During this pause, no messages are read from the topic. In a high-throughput system, even a 1–2 second rebalance window can cause significant queuing. Additionally, any messages fetched but not committed by the departing consumer will be re-delivered to the new partition owner, leading to potential duplicate processing. The key performance levers are: (1) avoid frequent membership changes in steady state, (2) commit offsets immediately before partition revocation using a ConsumerRebalanceListener, and (3) use Kafka's Cooperative Sticky Assignor (Kafka 2.4+) which does incremental rebalancing instead of revoking all partitions upfront.
Hard How would you minimize rebalancing in a high-throughput consumer?
+
Several complementary strategies reduce rebalancing frequency and impact: (1) Increase max.poll.interval.ms to be larger than your worst-case processing time, so slow processing batches do not appear as dead consumers. (2) Use async I/O for external calls — if your consumer calls a database or HTTP API, do it asynchronously so the poll loop is never blocked longer than the interval allows. (3) Switch to the Cooperative Sticky Assignor (set partition.assignment.strategy=CooperativeStickyAssignor) which only revokes and reassigns the minimum set of partitions during a membership change, rather than revoking all partitions from all consumers simultaneously. (4) Implement ConsumerRebalanceListener.on_partitions_revoked() to commit all pending offsets before partitions are taken away, preventing re-processing of already-completed work. (5) Pre-scale before traffic spikes — adding consumers ahead of a sale event is better than having Kafka repeatedly rebalance under load as consumers are dynamically added.

✅ Key takeaways

  • A consumer group (identified by group_id) lets multiple consumers read a topic in parallel — Kafka assigns each partition to exactly one consumer in the group. Parallelism is capped by partition count.
  • Multiple independent groups can read the same topic simultaneously with separate offset cursors — analytics replaying old events does not affect fraud detection reading the live edge.
  • Rebalancing pauses all consumers in a group while Kafka redistributes partitions. It is triggered by consumers joining, leaving, or failing to poll within max.poll.interval.ms.
  • Monitor consumer lag (kafka-consumer-groups.sh --describe) as your primary operational signal. Growing lag means consumers are falling behind — add consumers, increase partitions, or speed up processing.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

The foundation our AI platform engineers stand on

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →