📋 The three delivery guarantees
Every message streaming system must answer one question: what happens to a message when something goes wrong? Network drops, producer crashes, consumer failures — the guarantee you choose determines how the system behaves in each case.
At-most-once
A message is delivered zero or one time. No duplicates, but messages can be lost if a crash occurs before acknowledgement.
At-least-once
Every message is delivered, but the same message may arrive more than once. Retries guarantee no loss but allow duplicates.
Exactly-once
Every message is delivered exactly one time — no loss and no duplicates. The most complex guarantee to implement correctly.
| Guarantee | What it means | Trade-off |
|---|---|---|
| At-most-once | Message might be lost, never duplicated | Fastest; use for analytics where losing a metric is acceptable |
| At-least-once | No messages lost, but might be delivered more than once | Default for most pipelines; requires idempotent consumers |
| Exactly-once | No loss, no duplicates — every message processed exactly one time | Most complex; Kafka 3.x EOS transactions + idempotent producer; adds latency |
🚀 At-most-once — fastest but lossy
At-most-once is achieved by minimising the work done before acknowledging a message. On the producer side, this means acks=0 — the producer does not wait for any broker acknowledgement before moving to the next message. On the consumer side, this means committing offsets before processing.
At-most-once is appropriate for use cases where losing an occasional data point is acceptable — for example, clickstream analytics where a few dropped events do not change the aggregate result meaningfully. It is never appropriate for financial transactions.
🔁 At-least-once — the practical default
At-least-once is achieved by committing offsets only after successfully processing the message. If the consumer crashes between processing and committing, the message is re-delivered and processed again — which is a duplicate, not a loss.
This is the default for most production Kafka pipelines. The critical requirement is that your processing logic must be idempotent — applying it twice must have the same effect as applying it once.
Kafka broker must be running. Output will vary.
If the consumer crashes after process_payment() succeeds but before consumer.commit() runs, TXN will be re-processed on restart. Your process_payment() function must be idempotent — running it twice with the same txn_id must produce the same outcome as running it once.
🔑 Idempotency — the key to at-least-once
An operation is idempotent if applying it multiple times produces the same result as applying it once. This is the contract your consumer must honour to make at-least-once safe in practice.
| Operation | Idempotent? | Why |
|---|---|---|
INSERT OR IGNORE INTO payments WHERE txn_id = ? |
Yes | Second insert is a no-op because the row already exists |
Upsert by txn_id in Cassandra (INSERT ... IF NOT EXISTS) |
Yes | Subsequent writes with the same key update the same row — no duplicates created |
| Check if record exists before inserting | Yes | Explicit guard prevents duplicate creation |
Raw INSERT INTO payments ... (no dedup key) |
No | Each retry creates a new row — duplicate payments appear in the database |
| Sending a confirmation email | No | Each retry sends another email — the customer gets duplicates |
| Debiting a bank account | No | Each retry moves real money — double-charge on the customer |
At BoraPay, Wanjiku's rule is: every database write keyed on txn_id must be an upsert. Raw inserts are banned in the payment consumer services.
🛡️ Idempotent producer — broker-side deduplication
Kafka's built-in idempotent producer solves a different problem: duplicate writes caused by producer retries. When a producer retries a failed send(), the broker might receive the same message twice. With enable_idempotence=True, the broker deduplicates automatically.
Internally, Kafka assigns each producer a Producer ID (PID) and each message a monotonically increasing sequence number per partition. If the broker receives a message with a sequence number it has already committed, it acknowledges the send but discards the duplicate — the consumer never sees it.
Setting enable_idempotence=True automatically forces acks='all' (wait for all in-sync replicas) and retries=5 (retry on transient errors). You do not need to set these manually. The idempotent producer is a safe upgrade to any existing producer — it does not change the consumer API at all.
🔐 Exactly-once semantics (EOS)
Exactly-once semantics (EOS) go one step further than idempotent producers. They guarantee that a produce-consume-produce pipeline — where a consumer reads from one topic, processes, and writes to another — applies its effect exactly once end-to-end, even across crashes.
Kafka implements EOS through transactions: a producer can group multiple writes to multiple partitions and commit offset advances atomically. Either all succeed or none do. This is how Kafka Streams and Spark Structured Streaming implement their exactly-once processing guarantees.
The pattern: begin transaction → produce to output topic → commit consumer offsets → commit transaction. If any step fails, the transaction is aborted and no consumer will see the partial writes.
Kafka broker must be running. Output will vary.
🧭 Wanjiku's decision framework
"At BoraPay we use at-least-once everywhere, but every consumer is idempotent — we upsert by txn_id rather than insert. This is 95% of what you need in practice. We only use EOS for the inter-bank settlement topic where duplicates would move real money. EOS is powerful but it adds latency and operational complexity. If your consumer is idempotent, at-least-once is just as safe and ten times simpler to operate."
Which guarantee for which use case
| Use case | Recommended guarantee |
|---|---|
| Clickstream analytics | At-most-once or at-least-once — a missed click does not change the aggregate |
| Payment processing | At-least-once + idempotent consumer (upsert by txn_id) |
| Inter-bank transfers | Exactly-once (EOS) — duplicates move real money |
| Log aggregation | At-least-once — duplicate log lines are acceptable |
| Financial reporting | Exactly-once — duplicate rows corrupt totals |
⚠️ Common mistakes
INSERT on every message, rebalances and retries will create duplicate database rows. Every at-least-once consumer must be idempotent, or you will get data integrity bugs that only appear under failure conditions.transactional_id without enable_idempotence=True raises a ConfigException at producer creation time. Kafka transactions require idempotent producers as a foundation — the transactional guarantees build on top of the sequence-number deduplication mechanism.🧠 Quiz — 3 questions
Next: Mini Build — Clickstream Pipeline.
🎤 Interview Q&A
enable_idempotence=True is set, the Kafka broker assigns the producer a Producer ID (PID) — a unique 64-bit integer scoped to the broker epoch. The producer then tags every batch of messages with a sequence number that increments monotonically per partition. When a produce request arrives, the broker checks the sequence number for that PID and partition against its in-memory sequence state. If the incoming sequence number is equal to the last committed number (a duplicate caused by a retry), the broker acknowledges the send as if it were new but silently discards the messages — the consumer never sees them. If the sequence number is less than expected (an old retry out of order), the broker returns an error. If it is exactly one higher than expected (normal case), it is committed. This mechanism provides deduplication at the producer-broker interface, eliminating the class of duplicates caused by producer retries after transient network or broker errors. It does not, however, protect against consumer-side duplicates — that requires idempotent consumer logic or full EOS transactions.
enable_auto_commit=False, commit offsets only after successful processing, and ensure every write to the downstream datastore is an upsert keyed on a stable, unique identifier from the message (such as a transaction ID or event UUID). The upsert means that if the same message arrives twice, the second write overwrites the first with identical data — the net result in the datastore is as if the message arrived exactly once. This works well for database writes, cache updates, and any operation where the downstream system supports upsert or conditional write semantics. It does not work for inherently non-idempotent side effects like sending emails, debiting accounts, or making API calls that do not support deduplication — for those, you need true EOS transactions, or you need to implement your own deduplication layer (e.g., storing processed event IDs in Redis with a TTL and checking before processing).
✅ Key takeaways
- Kafka supports three delivery guarantees: at-most-once (fastest, may lose messages), at-least-once (default, no loss but possible duplicates), and exactly-once (EOS, no loss and no duplicates, highest complexity and latency).
- At-least-once is safe when your consumer is idempotent — use upserts keyed on a unique message ID rather than raw inserts. This is the practical choice for 95% of pipelines.
- The idempotent producer (
enable_idempotence=True) eliminates duplicates caused by producer retries, using a PID and per-partition sequence numbers that the broker deduplicates automatically. - Kafka transactions (
transactional_id+enable_idempotence=True) provide true EOS but add 50–200 ms per batch. Use EOS only where duplicates are truly unacceptable — inter-bank transfers, financial reporting — and at-least-once + idempotency everywhere else.
The foundation our AI platform engineers stand on
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →