👂 Simple consumer
A Kafka consumer subscribes to one or more topics and iterates over messages in an infinite loop. Each message carries its partition, offset, key, and value. The consumer in this example uses enable_auto_commit=True — Kafka automatically records how far the consumer has read every 5 seconds.
Make sure your Kafka broker is running and the topic borapay.payments has messages in it (run the producer from Lesson 16 first), then try this:
Kafka broker must be running at localhost:9092. Output will vary. Messages shown reflect what the Lesson 16 producer wrote to the topic.
Three things to notice:
- ▸
group_id='payment-processor'identifies this consumer as part of a consumer group. Kafka tracks committed offsets per group. - ▸
value_deserializerreverses what the producer'svalue_serializerdid — JSON bytes become a Python dict. - ▸ Messages from ACC101 (Partition 2, Offsets 0, 1, 2…) arrive in the exact order the producer wrote them.
📍 auto_offset_reset explained
When a consumer starts up, Kafka checks whether there is a committed offset for this consumer group and topic. The auto_offset_reset setting controls what happens when there is not one:
| Value | Behaviour | When to use |
|---|---|---|
'earliest' |
Start from the very first message in the partition — like --from-beginning in the CLI. Re-reads all historical data if no committed offset exists. |
New consumers that need all historical data. Development. Backfills. |
'latest' |
Start from the next new message produced after the consumer started. Messages produced before the consumer joined are skipped. | Real-time dashboards. Alert services. Only care about what happens from now. |
'none' |
Throw a NoOffsetForPartitionError if no committed offset is found. Forces you to handle this explicitly. |
Strict environments where an absent offset is a bug to be caught, not silently handled. |
If the consumer group has already committed offsets for this topic, auto_offset_reset is completely ignored — the consumer picks up from where it last committed. You only see this setting take effect the first time a new consumer group reads a topic, or after a --reset-offsets operation.
✅ Auto-commit vs manual commit
Committing an offset tells Kafka: "I have successfully processed everything up to this point. If I restart, start me from the next message after this." The question is: who decides when to commit?
Kafka broker must be running at localhost:9092. Output will vary based on the messages in the topic.
With enable_auto_commit=False and consumer.commit() called after processing:
- ▸ If your code crashes while processing TXN003, the offset for TXN003 is never committed.
- ▸ On restart, Kafka re-delivers TXN003 from the last committed offset — this is at-least-once delivery.
- ▸ Your business logic (
check_fraud) must be idempotent to handle re-deliveries safely.
Auto-commit vs manual commit comparison
| Mode | Simplicity | Delivery guarantee | Risk |
|---|---|---|---|
Auto-commit (enable_auto_commit=True) |
Simple — no commit code needed | At-most-once — offset committed on a timer, not after processing | Crash between auto-commit and processing = message lost. Timer fires every 5 seconds by default. |
Manual commit (enable_auto_commit=False) |
More code — must call consumer.commit() |
At-least-once — process first, commit after. Restart re-delivers uncommitted messages. | Re-deliveries require idempotent processing. Acceptable for most data pipelines. |
📊 Offset management diagram
An offset is just an integer — the position of a message in a partition's log. Committed offset is the position Kafka remembers for your consumer group. On restart, the consumer picks up from the committed offset, not the last-read offset.
The gap between the committed offset (2) and the consumer's current read position (3) is what gets re-delivered on restart. This is the core mechanic of at-least-once delivery: commit only after you are done processing, so a crash always gives you a second chance on the uncommitted messages.
🔍 Seeking to a specific offset
Sometimes you need to jump to a specific position in a partition — for replays, debugging, or disaster recovery. Use consumer.assign() with a TopicPartition object, then consumer.seek().
consumer.subscribe(['topic']) lets Kafka assign partitions automatically (used in consumer groups with automatic rebalancing). consumer.assign([TopicPartition(...)]) manually pins the consumer to specific partitions — required before using seek(). You cannot use both assign and subscribe on the same consumer instance.
Kwame uses offset seeking when he needs to replay the last hour of BoraPay transactions through an updated fraud model — he seeks each partition back to the offset that corresponds to one hour ago and lets the consumer re-process everything from there.
⚠️ Common mistakes
enable_auto_commit=True, Kafka commits offsets every 5 seconds on a background timer — regardless of whether your code has finished processing those messages. If processing takes longer than 5 seconds and the consumer crashes, the offset is committed but the processing was incomplete. Use manual commit for any consumer where processing time is variable or where data loss is unacceptable.consumer.close(), it does not send a graceful LeaveGroup request to the broker. The broker waits until the session timeout expires (default 10 seconds) before triggering a consumer group rebalance. This delays the other consumers in the group. Always close the consumer in a try/finally block.group_id=None creates a standalone consumer that is not part of any group. Kafka does not track offsets for it. Every time this consumer starts, it uses auto_offset_reset to decide where to begin. This is useful for ad-hoc inspection but should never be used in a production pipeline — you lose offset tracking and at-least-once guarantees entirely.🧠 Quiz — 3 questions
auto_offset_reset='earliest' means:enable_auto_commit=False) with commit-after-processing gives:Next: learn how consumer groups and rebalancing work together.
🎤 Interview Q&A
Both values only take effect when there is no committed offset for the consumer group on the topic partition. If a committed offset exists, the consumer picks up from there and this setting is irrelevant. When there is no committed offset: 'earliest' positions the consumer at the very first message in the partition log (offset 0 if no compaction has occurred), so it reads all historical data. 'latest' positions the consumer at the tail of the log — only messages produced after the consumer joined will be seen. Use 'earliest' when you need all history (backfill, new service starting for the first time, development). Use 'latest' when you only care about real-time events (dashboards, alerting, live feeds). The CLI equivalent is --from-beginning for 'earliest'; without that flag the CLI defaults to 'latest'.
Auto-commit gives at-most-once delivery semantics in practice, despite the documentation sometimes describing it loosely as "at-least-once." Here is why: with enable_auto_commit=True, the Kafka client commits offsets on a background timer (default: every 5 seconds via auto_commit_interval_ms). The commit happens independently of whether your application code has successfully processed those messages. Scenario: the timer fires at second 5, committing offset 100. Your consumer processes message 101 and crashes at second 6. On restart, Kafka delivers from offset 101 — message 101 is delivered again. That sounds like at-least-once. But consider: the timer fires at second 5, committing offset 100, then your application processes messages 95–100 and crashes before it actually finishes processing 99 and 100. On restart, Kafka delivers from 101 — messages 99 and 100 are never re-processed. That is at-most-once. The race condition depends on when the timer fires relative to your processing speed. For any pipeline where message loss is unacceptable, always use manual commit.
True exactly-once requires that both the business logic execution and the offset commit happen atomically — neither can succeed without the other. There are two main approaches. First, the transactional outbox pattern: store processed results and the Kafka offset together in a single database transaction. If the transaction succeeds, both the result is persisted and the offset is durably recorded in your own database — not in Kafka's __consumer_offsets topic. On restart, you read the last-processed offset from your database and use consumer.seek() to resume from there. This works with any database that supports ACID transactions. Second, Kafka's native exactly-once semantics (EOS): use Kafka Streams or a transactional producer-consumer pair. The consumer reads, processes, and produces results to another Kafka topic. The producer commits the output and the input offset atomically using a Kafka transaction. This requires isolation.level=read_committed on the consumer side. In practice, the transactional outbox is more commonly used when writing to an external store (database, search index). Kafka EOS is used in pure Kafka-to-Kafka pipelines. Both approaches add significant complexity — most BoraPay-style pipelines accept at-least-once with idempotent processing rather than true exactly-once.
✅ Key takeaways
- An offset is an integer position in a partition log. Committing an offset tells Kafka where to restart this consumer group on the next startup. Without a commit, the message will be re-delivered.
auto_offset_resetonly fires when there is no committed offset for the consumer group.'earliest'reads all history;'latest'reads only new messages from the point of joining.- Manual commit (
enable_auto_commit=False+consumer.commit()after processing) gives at-least-once delivery. This is the correct default for production pipelines — accept re-deliveries and make your processing idempotent. - Always call
consumer.close()in atry/finallyblock to send a graceful LeaveGroup request and avoid delayed rebalances for other consumers in the group.
Build real Kafka pipelines, not toy examples
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →