📦 Install kafka-python
This lesson uses kafka-python, a pure-Python Kafka client. Install it with one command. You will also need the Kafka broker running locally — start it with ./start.sh from the Lesson 00 workspace before running any producer code.
pip install kafka-python
All examples in this lesson use the BoraPay scenario — a Kenyan mobile payments platform built by engineers Wanjiku and Kwame. The topic borapay.payments carries payment events. You will learn how to produce those events reliably and efficiently.
🚀 Simple producer
The simplest working producer: create a KafkaProducer with a JSON serializer, send one payment event with a key, wait for the acknowledgement, then close the producer cleanly.
Kafka broker must be running at localhost:9092. Output will vary. The partition number depends on your topic configuration.
Three things to notice:
- ▸
value_serializerconverts the Python dict to JSON bytes automatically before sending. - ▸
key='ACC101'sets the routing key — all events from account ACC101 will go to the same partition. - ▸
future.get(timeout=10)blocks until the broker acknowledges the message — this is synchronous mode.
🔑 Key-based partitioning
Kafka uses the message key to determine which partition a message goes to. The formula is:
partition = hash(key) % num_partitions
The same key always hashes to the same partition — as long as the number of partitions does not change. This guarantees ordering: all messages with the same key are written to the same partition in the order they are sent.
For BoraPay, using the sender account as the key means all transactions from ACC101 always land on Partition 2. Kwame's fraud detection consumer can then read Partition 2 and see every ACC101 transaction in chronological order — no shuffling, no gaps.
⚡ Async vs sync sending
Kafka producers buffer messages in memory and send them in batches. You choose how aggressively you wait for acknowledgements:
Kafka broker must be running at localhost:9092. Output will vary.
The critical difference is what happens between sending and acknowledging:
| Mode | How it works | Throughput | Risk |
|---|---|---|---|
| Async (fire-and-forget) | Send returns immediately. Messages queue in an internal buffer. The producer flushes the buffer when it fills or flush() is called. |
Very high — batch network I/O | Buffered messages lost if process crashes before flush |
| Sync (wait for ack) | future.get() blocks the thread until the broker returns an acknowledgement. One message at a time. |
Low — one RTT per message | Slowest throughput; safe for critical single events |
| Async with callbacks | future.add_callback() and future.add_errback() called after broker responds. Non-blocking. |
High | Errors handled in callback — production best practice |
Without flush(), any messages still sitting in the internal buffer will be dropped when the Python process exits. Even in async mode, finish with producer.flush() to drain all buffered messages to the broker, then producer.close().
⚙️ Producer config reference
The most important KafkaProducer constructor arguments and what they control. Wanjiku tunes these to balance throughput against durability on the BoraPay platform.
| Config | Description | Recommended value |
|---|---|---|
acks |
0 = no ack (fire-and-forget, no durability). 1 = leader broker only. 'all' = all in-sync replicas must acknowledge. |
'all' for durability |
retries |
Number of automatic retries on transient errors (e.g. leader election in progress). Retries preserve ordering when max_in_flight_requests_per_connection=1. |
3 |
linger_ms |
How many milliseconds the producer waits before sending a batch. A non-zero value lets more messages accumulate in the buffer, increasing batch size and reducing per-message overhead. | 5 (ms) |
batch_size |
Maximum bytes per batch per partition. When the batch fills before linger_ms expires, it is sent immediately. |
16384 (16 KB) |
compression_type |
Compress message batches before sending. 'snappy' is fast with moderate compression. 'lz4' is faster. 'gzip' gives best ratio but highest CPU cost. |
'snappy' |
🏭 Wanjiku's production producer
This is the pattern Wanjiku uses in BoraPay's payment ingestion service. It combines all the config knobs from above with async callbacks so errors are logged without blocking the send loop.
Key design decisions in this pattern:
- ▸
acks='all'withretries=3means no message is lost even if one broker replica fails during a send. - ▸
linger_ms=5gives the buffer 5 ms to accumulate messages before flushing — dramatically improves throughput at BoraPay's transaction volumes. - ▸ Callbacks log success or failure asynchronously — the send loop never blocks waiting for a per-message ack.
- ▸ The producer instance is reused across calls (not recreated per payment) because creating a producer is expensive.
⚠️ Common mistakes
producer.flush() before producer.close() — especially in batch scripts that produce a fixed number of events and then exit.acks=0 means the broker does not acknowledge messages at all. The producer assumes every send succeeded. If the broker crashes between the send and the write to disk, those messages are silently lost. Use acks='all' for financial data like BoraPay payments.max.message.bytes is 1 MB per message. Payloads larger than this are rejected by the broker with a MessageSizeTooLargeError. Compress data, strip unnecessary fields, or split large payloads into smaller events before producing.🧠 Quiz — 3 questions
producer.flush() do?acks='all' means:Next: learn how consumers read what you just produced.
🎤 Interview Q&A
Kafka guarantees ordering within a single partition, not across partitions. When you assign a key to a message, the producer hashes the key (using the default murmur2 algorithm) and takes the result modulo the number of partitions: partition = hash(key) % num_partitions. All messages with the same key always land on the same partition. Since a partition is an ordered, append-only log, consumers reading that partition see messages in the exact order they were written. This is why BoraPay uses the sender account ID as the key — all transactions from the same sender arrive on the same partition in chronological order, which the fraud detection service depends on. If you change the partition count of a topic, the hash formula returns different partition numbers for the same keys, breaking the ordering guarantee for the transition period. Never resize a live production topic without a migration plan.
linger_ms is the time the producer waits before sending a batch, even if the batch has not yet filled to batch_size. By default it is 0, which means the producer sends messages as soon as they arrive — maximising latency but minimising batching efficiency. Setting linger_ms=5 tells the producer: "wait up to 5 milliseconds to accumulate more messages before sending a batch." This dramatically reduces the number of network round-trips and allows compression to work more effectively across a larger batch. The trade-off is up to 5 ms of added latency per message. For a high-volume system like BoraPay that sends thousands of payment events per second, this is almost always the right call — the throughput gain far outweighs the tiny added latency. For a low-volume topic where latency matters (e.g., a real-time alert), keep linger_ms=0.
An idempotent producer guarantees that retries do not create duplicate messages in the Kafka topic. Without idempotence: if the broker writes a message but the network drops before it can send an ack back to the producer, the producer retries and the message is written a second time — a duplicate. With idempotence enabled (enable_idempotence=True in the config), each producer instance is assigned a unique Producer ID (PID) by the broker, and every message is tagged with a monotonically increasing sequence number. The broker deduplicates: if it receives a message with a PID and sequence number it has already committed, it acknowledges the send but discards the duplicate. Idempotent producers require acks='all', retries > 0, and max_in_flight_requests_per_connection <= 5. This gives you exactly-once delivery at the producer-to-broker leg. For end-to-end exactly-once (including the consumer side), you additionally need Kafka transactions — a more advanced topic.
✅ Key takeaways
- The message key determines partition assignment via
hash(key) % num_partitions. Same key always routes to the same partition, guaranteeing per-key ordering as long as partition count is stable. - Async sending with
producer.flush()at the end gives the highest throughput. Synchronous sending withfuture.get()is safe but slow — use it only for critical single events. - Production producers should use
acks='all'andretries=3for durability, pluslinger_ms=5andcompression_type='snappy'for throughput. - Always call
producer.flush()beforeproducer.close()— messages still in the internal buffer are silently dropped on process exit without it.
Build real Kafka pipelines, not toy examples
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →