📦 Install kafka-python

This lesson uses kafka-python, a pure-Python Kafka client. Install it with one command. You will also need the Kafka broker running locally — start it with ./start.sh from the Lesson 00 workspace before running any producer code.

pip install kafka-python

pip install kafka-python

All examples in this lesson use the BoraPay scenario — a Kenyan mobile payments platform built by engineers Wanjiku and Kwame. The topic borapay.payments carries payment events. You will learn how to produce those events reliably and efficiently.

🚀 Simple producer

The simplest working producer: create a KafkaProducer with a JSON serializer, send one payment event with a key, wait for the acknowledgement, then close the producer cleanly.

simple_producer.py Lesson 16 · basic producer

Kafka broker must be running at localhost:9092. Output will vary. The partition number depends on your topic configuration.

sample output
Topic : borapay.payments Partition : 2 Offset : 0

Three things to notice:

  • ▸ value_serializer converts the Python dict to JSON bytes automatically before sending.
  • ▸ key='ACC101' sets the routing key — all events from account ACC101 will go to the same partition.
  • ▸ future.get(timeout=10) blocks until the broker acknowledges the message — this is synchronous mode.

🔑 Key-based partitioning

Kafka uses the message key to determine which partition a message goes to. The formula is:

Partition assignment formula

partition = hash(key) % num_partitions

The same key always hashes to the same partition — as long as the number of partitions does not change. This guarantees ordering: all messages with the same key are written to the same partition in the order they are sent.

For BoraPay, using the sender account as the key means all transactions from ACC101 always land on Partition 2. Kwame's fraud detection consumer can then read Partition 2 and see every ACC101 transaction in chronological order — no shuffling, no gaps.

Key-based partitioning — borapay.payments topic key='ACC101' hash → Partition 2 key='ACC202' hash → Partition 0 key='ACC303' hash → Partition 1 Partition 2 (ACC101 msgs) Partition 0 (ACC202 msgs) Partition 1 (ACC303 msgs) borapay.payments 3 partitions · ordering per key
Important: If you change the number of partitions on a topic after data is already flowing, the hash formula produces different results. Messages with the same key can land on a different partition than before. Never change partition count on a live production topic without a migration plan.

⚡ Async vs sync sending

Kafka producers buffer messages in memory and send them in batches. You choose how aggressively you wait for acknowledgements:

async_vs_sync.py Lesson 16 · sending modes

Kafka broker must be running at localhost:9092. Output will vary.

sample output
Sent 100 messages asynchronously Sent synchronously — ack received

The critical difference is what happens between sending and acknowledging:

Mode How it works Throughput Risk
Async (fire-and-forget) Send returns immediately. Messages queue in an internal buffer. The producer flushes the buffer when it fills or flush() is called. Very high — batch network I/O Buffered messages lost if process crashes before flush
Sync (wait for ack) future.get() blocks the thread until the broker returns an acknowledgement. One message at a time. Low — one RTT per message Slowest throughput; safe for critical single events
Async with callbacks future.add_callback() and future.add_errback() called after broker responds. Non-blocking. High Errors handled in callback — production best practice
Always call producer.flush() before exit

Without flush(), any messages still sitting in the internal buffer will be dropped when the Python process exits. Even in async mode, finish with producer.flush() to drain all buffered messages to the broker, then producer.close().

⚙️ Producer config reference

The most important KafkaProducer constructor arguments and what they control. Wanjiku tunes these to balance throughput against durability on the BoraPay platform.

Config Description Recommended value
acks 0 = no ack (fire-and-forget, no durability). 1 = leader broker only. 'all' = all in-sync replicas must acknowledge. 'all' for durability
retries Number of automatic retries on transient errors (e.g. leader election in progress). Retries preserve ordering when max_in_flight_requests_per_connection=1. 3
linger_ms How many milliseconds the producer waits before sending a batch. A non-zero value lets more messages accumulate in the buffer, increasing batch size and reducing per-message overhead. 5 (ms)
batch_size Maximum bytes per batch per partition. When the batch fills before linger_ms expires, it is sent immediately. 16384 (16 KB)
compression_type Compress message batches before sending. 'snappy' is fast with moderate compression. 'lz4' is faster. 'gzip' gives best ratio but highest CPU cost. 'snappy'

🏭 Wanjiku's production producer

This is the pattern Wanjiku uses in BoraPay's payment ingestion service. It combines all the config knobs from above with async callbacks so errors are logged without blocking the send loop.

borapay_producer.py Lesson 16 · production pattern

Key design decisions in this pattern:

  • ▸ acks='all' with retries=3 means no message is lost even if one broker replica fails during a send.
  • ▸ linger_ms=5 gives the buffer 5 ms to accumulate messages before flushing — dramatically improves throughput at BoraPay's transaction volumes.
  • ▸ Callbacks log success or failure asynchronously — the send loop never blocks waiting for a per-message ack.
  • ▸ The producer instance is reused across calls (not recreated per payment) because creating a producer is expensive.

⚠️ Common mistakes

Not calling producer.flush()
Messages in the internal send buffer are dropped when the Python process exits. Always call producer.flush() before producer.close() — especially in batch scripts that produce a fixed number of events and then exit.
Using acks=0 in production
acks=0 means the broker does not acknowledge messages at all. The producer assumes every send succeeded. If the broker crashes between the send and the write to disk, those messages are silently lost. Use acks='all' for financial data like BoraPay payments.
Sending very large messages
Kafka's default max.message.bytes is 1 MB per message. Payloads larger than this are rejected by the broker with a MessageSizeTooLargeError. Compress data, strip unnecessary fields, or split large payloads into smaller events before producing.

🧠 Quiz — 3 questions

Question 1 of 3
Same key always goes to the same partition. True or False?
Question 2 of 3
What does producer.flush() do?
Question 3 of 3
acks='all' means:
0
/ 3
Producer basics locked in.

Next: learn how consumers read what you just produced.

🎤 Interview Q&A

Medium How does Kafka guarantee per-key ordering?
+

Kafka guarantees ordering within a single partition, not across partitions. When you assign a key to a message, the producer hashes the key (using the default murmur2 algorithm) and takes the result modulo the number of partitions: partition = hash(key) % num_partitions. All messages with the same key always land on the same partition. Since a partition is an ordered, append-only log, consumers reading that partition see messages in the exact order they were written. This is why BoraPay uses the sender account ID as the key — all transactions from the same sender arrive on the same partition in chronological order, which the fraud detection service depends on. If you change the partition count of a topic, the hash formula returns different partition numbers for the same keys, breaking the ordering guarantee for the transition period. Never resize a live production topic without a migration plan.

Medium What is linger_ms and why would you set it greater than 0?
+

linger_ms is the time the producer waits before sending a batch, even if the batch has not yet filled to batch_size. By default it is 0, which means the producer sends messages as soon as they arrive — maximising latency but minimising batching efficiency. Setting linger_ms=5 tells the producer: "wait up to 5 milliseconds to accumulate more messages before sending a batch." This dramatically reduces the number of network round-trips and allows compression to work more effectively across a larger batch. The trade-off is up to 5 ms of added latency per message. For a high-volume system like BoraPay that sends thousands of payment events per second, this is almost always the right call — the throughput gain far outweighs the tiny added latency. For a low-volume topic where latency matters (e.g., a real-time alert), keep linger_ms=0.

Hard What is an idempotent producer?
+

An idempotent producer guarantees that retries do not create duplicate messages in the Kafka topic. Without idempotence: if the broker writes a message but the network drops before it can send an ack back to the producer, the producer retries and the message is written a second time — a duplicate. With idempotence enabled (enable_idempotence=True in the config), each producer instance is assigned a unique Producer ID (PID) by the broker, and every message is tagged with a monotonically increasing sequence number. The broker deduplicates: if it receives a message with a PID and sequence number it has already committed, it acknowledges the send but discards the duplicate. Idempotent producers require acks='all', retries > 0, and max_in_flight_requests_per_connection <= 5. This gives you exactly-once delivery at the producer-to-broker leg. For end-to-end exactly-once (including the consumer side), you additionally need Kafka transactions — a more advanced topic.

✅ Key takeaways

  • The message key determines partition assignment via hash(key) % num_partitions. Same key always routes to the same partition, guaranteeing per-key ordering as long as partition count is stable.
  • Async sending with producer.flush() at the end gives the highest throughput. Synchronous sending with future.get() is safe but slow — use it only for critical single events.
  • Production producers should use acks='all' and retries=3 for durability, plus linger_ms=5 and compression_type='snappy' for throughput.
  • Always call producer.flush() before producer.close() — messages still in the internal buffer are silently dropped on process exit without it.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Build real Kafka pipelines, not toy examples

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →