📋 The three delivery guarantees

Every message streaming system must answer one question: what happens to a message when something goes wrong? Network drops, producer crashes, consumer failures — the guarantee you choose determines how the system behaves in each case.

At-most-once

A message is delivered zero or one time. No duplicates, but messages can be lost if a crash occurs before acknowledgement.

Fastest — fire and forget

At-least-once

Every message is delivered, but the same message may arrive more than once. Retries guarantee no loss but allow duplicates.

Default — requires idempotent consumers

Exactly-once

Every message is delivered exactly one time — no loss and no duplicates. The most complex guarantee to implement correctly.

Most complex — Kafka 3.x EOS transactions
Guarantee What it means Trade-off
At-most-once Message might be lost, never duplicated Fastest; use for analytics where losing a metric is acceptable
At-least-once No messages lost, but might be delivered more than once Default for most pipelines; requires idempotent consumers
Exactly-once No loss, no duplicates — every message processed exactly one time Most complex; Kafka 3.x EOS transactions + idempotent producer; adds latency

🚀 At-most-once — fastest but lossy

At-most-once is achieved by minimising the work done before acknowledging a message. On the producer side, this means acks=0 — the producer does not wait for any broker acknowledgement before moving to the next message. On the consumer side, this means committing offsets before processing.

At-most-once — consumer commits BEFORE processing Producer acks=0 send() no wait Broker has message poll() Consumer commit() first CRASH before process() Message committed but never processed — permanently lost. On restart, consumer reads from committed offset — the lost message is skipped.

At-most-once is appropriate for use cases where losing an occasional data point is acceptable — for example, clickstream analytics where a few dropped events do not change the aggregate result meaningfully. It is never appropriate for financial transactions.

🔁 At-least-once — the practical default

At-least-once is achieved by committing offsets only after successfully processing the message. If the consumer crashes between processing and committing, the message is re-delivered and processed again — which is a duplicate, not a loss.

This is the default for most production Kafka pipelines. The critical requirement is that your processing logic must be idempotent — applying it twice must have the same effect as applying it once.

at_least_once_consumer.py Lesson 19 · commit AFTER processing

Kafka broker must be running. Output will vary.

sample output
Processed TXN001 → OK Processed TXN002 → OK Processed TXN003 → OK
The duplicate risk

If the consumer crashes after process_payment() succeeds but before consumer.commit() runs, TXN will be re-processed on restart. Your process_payment() function must be idempotent — running it twice with the same txn_id must produce the same outcome as running it once.

🔑 Idempotency — the key to at-least-once

An operation is idempotent if applying it multiple times produces the same result as applying it once. This is the contract your consumer must honour to make at-least-once safe in practice.

Operation Idempotent? Why
INSERT OR IGNORE INTO payments WHERE txn_id = ? Yes Second insert is a no-op because the row already exists
Upsert by txn_id in Cassandra (INSERT ... IF NOT EXISTS) Yes Subsequent writes with the same key update the same row — no duplicates created
Check if record exists before inserting Yes Explicit guard prevents duplicate creation
Raw INSERT INTO payments ... (no dedup key) No Each retry creates a new row — duplicate payments appear in the database
Sending a confirmation email No Each retry sends another email — the customer gets duplicates
Debiting a bank account No Each retry moves real money — double-charge on the customer

At BoraPay, Wanjiku's rule is: every database write keyed on txn_id must be an upsert. Raw inserts are banned in the payment consumer services.

🛡️ Idempotent producer — broker-side deduplication

Kafka's built-in idempotent producer solves a different problem: duplicate writes caused by producer retries. When a producer retries a failed send(), the broker might receive the same message twice. With enable_idempotence=True, the broker deduplicates automatically.

Internally, Kafka assigns each producer a Producer ID (PID) and each message a monotonically increasing sequence number per partition. If the broker receives a message with a sequence number it has already committed, it acknowledges the send but discards the duplicate — the consumer never sees it.

idempotent_producer.py Lesson 19 · enable_idempotence=True
What enable_idempotence implies

Setting enable_idempotence=True automatically forces acks='all' (wait for all in-sync replicas) and retries=5 (retry on transient errors). You do not need to set these manually. The idempotent producer is a safe upgrade to any existing producer — it does not change the consumer API at all.

🔐 Exactly-once semantics (EOS)

Exactly-once semantics (EOS) go one step further than idempotent producers. They guarantee that a produce-consume-produce pipeline — where a consumer reads from one topic, processes, and writes to another — applies its effect exactly once end-to-end, even across crashes.

Kafka implements EOS through transactions: a producer can group multiple writes to multiple partitions and commit offset advances atomically. Either all succeed or none do. This is how Kafka Streams and Spark Structured Streaming implement their exactly-once processing guarantees.

The pattern: begin transaction → produce to output topic → commit consumer offsets → commit transaction. If any step fails, the transaction is aborted and no consumer will see the partial writes.

exactly_once_producer.py Lesson 19 · Kafka transactional API

Kafka broker must be running. Output will vary.

sample output
Transaction committed.
EOS latency cost: each transaction requires at least two broker round trips — one for begin and one for commit. In practice this adds 50–200 ms per batch depending on network and broker load. For inter-bank settlement this is acceptable. For a real-time fraud score with a 50 ms SLA it is not.

🧭 Wanjiku's decision framework

Wanjiku — Senior Data Engineer, BoraPay
"At BoraPay we use at-least-once everywhere, but every consumer is idempotent — we upsert by txn_id rather than insert. This is 95% of what you need in practice. We only use EOS for the inter-bank settlement topic where duplicates would move real money. EOS is powerful but it adds latency and operational complexity. If your consumer is idempotent, at-least-once is just as safe and ten times simpler to operate."

Which guarantee for which use case

Use case Recommended guarantee
Clickstream analytics At-most-once or at-least-once — a missed click does not change the aggregate
Payment processing At-least-once + idempotent consumer (upsert by txn_id)
Inter-bank transfers Exactly-once (EOS) — duplicates move real money
Log aggregation At-least-once — duplicate log lines are acceptable
Financial reporting Exactly-once — duplicate rows corrupt totals

⚠️ Common mistakes

Assuming at-least-once "just works" without idempotency
At-least-once guarantees delivery — not correctness. If your consumer does a raw INSERT on every message, rebalances and retries will create duplicate database rows. Every at-least-once consumer must be idempotent, or you will get data integrity bugs that only appear under failure conditions.
Using transactional_id without enable_idempotence
Setting transactional_id without enable_idempotence=True raises a ConfigException at producer creation time. Kafka transactions require idempotent producers as a foundation — the transactional guarantees build on top of the sequence-number deduplication mechanism.
Forgetting the latency cost of EOS
Each Kafka transaction adds at least one broker round trip for the transaction log write. In benchmarks this adds 50–200 ms per committed batch. If you have a 50 ms end-to-end SLA for fraud scoring, EOS will violate it. Profile before adopting EOS for latency-sensitive paths.

🧠 Quiz — 3 questions

Question 1 of 3
Which delivery guarantee is the cheapest (lowest latency) but may lose messages?
Question 2 of 3
An idempotent operation is one that:
Question 3 of 3
Exactly-once semantics in Kafka require:
0
/ 3
Delivery semantics unlocked.

Next: Mini Build — Clickstream Pipeline.

🎤 Interview Q&A

Medium What is the difference between at-least-once and exactly-once?
+
At-least-once guarantees that every message is delivered, but the same message may be delivered more than once — typically when a consumer crashes after processing but before committing its offset. On restart, the message is re-fetched and re-processed. This means duplicate processing is possible. At-least-once is the default for most Kafka pipelines and is safe when your processing logic is idempotent — an operation that produces the same result whether applied once or ten times. Exactly-once guarantees that each message is processed exactly one time, with no loss and no duplicates, even across crashes and retries. It is implemented in Kafka using the transactional API: a producer groups its writes and offset commits atomically. Either the entire transaction commits — the output message is visible and the offset is advanced — or it is aborted and neither side effect happens. EOS is more expensive in terms of latency and operational complexity, so it is appropriate only when duplicate effects would be unacceptable, such as inter-bank money transfers.
Hard How does the idempotent producer work internally?
+
When enable_idempotence=True is set, the Kafka broker assigns the producer a Producer ID (PID) — a unique 64-bit integer scoped to the broker epoch. The producer then tags every batch of messages with a sequence number that increments monotonically per partition. When a produce request arrives, the broker checks the sequence number for that PID and partition against its in-memory sequence state. If the incoming sequence number is equal to the last committed number (a duplicate caused by a retry), the broker acknowledges the send as if it were new but silently discards the messages — the consumer never sees them. If the sequence number is less than expected (an old retry out of order), the broker returns an error. If it is exactly one higher than expected (normal case), it is committed. This mechanism provides deduplication at the producer-broker interface, eliminating the class of duplicates caused by producer retries after transient network or broker errors. It does not, however, protect against consumer-side duplicates — that requires idempotent consumer logic or full EOS transactions.
Medium In practice, how do most teams achieve exactly-once semantics without Kafka transactions?
+
Most production teams achieve effective exactly-once semantics through at-least-once delivery combined with idempotent consumers — sometimes called "at-least-once + idempotency = effectively exactly-once." The approach: use enable_auto_commit=False, commit offsets only after successful processing, and ensure every write to the downstream datastore is an upsert keyed on a stable, unique identifier from the message (such as a transaction ID or event UUID). The upsert means that if the same message arrives twice, the second write overwrites the first with identical data — the net result in the datastore is as if the message arrived exactly once. This works well for database writes, cache updates, and any operation where the downstream system supports upsert or conditional write semantics. It does not work for inherently non-idempotent side effects like sending emails, debiting accounts, or making API calls that do not support deduplication — for those, you need true EOS transactions, or you need to implement your own deduplication layer (e.g., storing processed event IDs in Redis with a TTL and checking before processing).

✅ Key takeaways

  • Kafka supports three delivery guarantees: at-most-once (fastest, may lose messages), at-least-once (default, no loss but possible duplicates), and exactly-once (EOS, no loss and no duplicates, highest complexity and latency).
  • At-least-once is safe when your consumer is idempotent — use upserts keyed on a unique message ID rather than raw inserts. This is the practical choice for 95% of pipelines.
  • The idempotent producer (enable_idempotence=True) eliminates duplicates caused by producer retries, using a PID and per-partition sequence numbers that the broker deduplicates automatically.
  • Kafka transactions (transactional_id + enable_idempotence=True) provide true EOS but add 50–200 ms per batch. Use EOS only where duplicates are truly unacceptable — inter-bank transfers, financial reporting — and at-least-once + idempotency everywhere else.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

The foundation our AI platform engineers stand on

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →