🛠️ What you're building
NaijaCart's engineering team — Tunde, Funmi, and Chidi — needs to track page-click behaviour across their Nigerian e-commerce platform. This mini-build takes you from zero to a complete Kafka pipeline: a producer that simulates 100 click events and a consumer that counts clicks per page, deduplicated so that replays never double-count.
- Topic:
naijacart.clicks— 3 partitions, keyed byuser_idso the same user always lands on the same partition - Producer: simulates 100 click events spread across 5 pages (
/home,/product,/cart,/checkout,/confirm) and 5 cities (Lagos, Abuja, Kano, Port Harcourt, Ibadan) - Consumer: reads events, counts clicks per page, writes totals into a local
click_countsdict - Idempotency: every event carries a unique
event_id(UUID); the consumer maintains aseen_event_idsset and skips duplicates, so replaying the topic from the beginning never double-counts
The Kafka broker from Lesson 00 must be running (./start.sh). Install kafka-python if you haven't yet: pip install kafka-python==2.0.2. This lesson uses kafka-python (not confluent-kafka) for its simpler producer/consumer API — both talk to the same broker at localhost:9092.
🗺️ Architecture
Four stages, all running locally. The arrow from consumer to click_counts represents an in-memory write — in Arc E we replace this with a Cassandra upsert.
Create naijacart.clicks with 3 partitions. The --replication-factor 1 flag is correct for a single-broker local setup.
Kafka broker must be running at localhost:9092. Output will vary.
Verify the topic was created with the right partition count:
Kafka hashes the message key (user_id) to select a partition. Using the same user as the key guarantees ordering per user — all events for U01 always land in the same partition, so a per-user state machine could process them in arrival order. Three partitions lets up to three consumer instances in the same group share the load in Arc D.
Save this as producer.py. It uses enable_idempotence=True on the producer side — this prevents the broker from storing duplicate messages if the producer retries a failed send.
Kafka broker must be running at localhost:9092. Output will vary.
enable_idempotence=True does on the producer sideWhen a producer send fails and retries, the broker could receive the same message twice without idempotence. With enable_idempotence=True, the producer stamps each batch with a sequence number and the broker de-duplicates at the log level — exactly-once delivery per partition, even under retries. Requires acks='all' (already set above).
Save this as consumer.py. The seen_event_ids set is the deduplication guard — if an event's event_id is already in the set, it is skipped without updating the counts. Manual commit() after each event ensures offsets are only advanced for successfully processed messages.
Kafka broker must be running at localhost:9092. Output will vary.
random.seed(42) — the same seed produces the same pseudo-random sequence every time. In production, the distribution would be skewed: product pages and the cart typically see 3–5x more traffic than confirm.
To test idempotency, reset the consumer group offset back to the earliest position and run consumer.py again. Kafka re-delivers all 100 events. Because the seen_event_ids set is populated in memory from the first run — if the process is still running — all 100 are detected as duplicates and the counts stay at zero.
Reset the consumer group offset
Kafka broker must be running at localhost:9092. Output will vary.
Rerun consumer.py (same process session)
Run python consumer.py again immediately without restarting the process. All 100 events are re-delivered, but the in-memory seen_event_ids set still holds all 100 IDs from the first run — so every event is a duplicate.
The seen_event_ids set lives in Python heap memory. If the consumer process crashes and restarts, the set is empty — replaying the topic now processes all events again as if they are new. In a real pipeline, Funmi would persist the seen IDs in Redis (fast, TTL-based) or Cassandra (durable, keyed on event_id) before committing the Kafka offset. We cover persistent idempotent writes in Arc E.
🏭 Full pipeline in production at NaijaCart
In production, Funmi runs this pipeline continuously. The producer is a FastAPI endpoint that publishes a click event to naijacart.clicks on every page load — one HTTP request in, one Kafka message out. The consumer is a long-running Python process on a Kubernetes pod in the NaijaCart cluster. The click_counts dict is replaced by a Cassandra upsert keyed on (date, page) — because Cassandra upserts are idempotent by design, the consumer can restart at any time, replay from its last committed offset, and upsert without double-counting. The seen_event_ids set is backed by a Redis SADD call with a 24-hour TTL.
| Component | Local (this lesson) | NaijaCart production |
|---|---|---|
| Producer | producer.py script, runs once |
FastAPI endpoint, publishes on every page load |
| Consumer | consumer.py script, exits after 3 s idle |
Long-running Kubernetes pod, never exits |
| Deduplication store | In-memory Python set() |
Redis SADD with 24-hour TTL |
| Output | click_counts dict, printed to stdout |
Cassandra upsert on (date, page) |
| Offset commit | Manual consumer.commit() per event |
Manual commit after Cassandra upsert succeeds |
| Scale | 1 producer, 1 consumer, 3 partitions | N producers, 3 consumer pods (one per partition) |
▶️ Running both together
Open two terminal windows side by side. Start the consumer first so it is ready when the producer begins sending. The & runs it in the background — you can also open a true second terminal tab.
The consumer uses consumer_timeout_ms=3000 — it exits automatically 3 seconds after the last message. The producer sends all 100 events in under a second. You should see the click summary print in Terminal 1 within 5–6 seconds of starting both scripts.
consumer_timeout_ms to 8000. On slow laptops with Docker under load, the broker may buffer messages for a second or two before the consumer polls them.
⚠️ Common mistakes
set() lives in heap memory and disappears when the process exits. On restart, the consumer sees all messages as new and double-counts on replay. Use Redis SADD or a Cassandra row keyed on event_id for persistence across restarts.enable_idempotence=True on the producerenable_idempotence=True with acks='all'.event_idts (millisecond timestamp) alone is not safe — two events from the same user at the same millisecond produce identical timestamps. Always generate a UUID at event creation time and embed it in the payload.consumer_timeout_ms too short🧠 Quiz — 3 questions
event_id field in each NaijaCart click event enables what behaviour in the consumer?seen_event_ids set insufficient for production use?enable_idempotence=True on the Kafka producer prevents which specific problem?You understand idempotent writes — now apply them with a persistent store in Arc E.
🎤 Interview Q&A
A Kafka pipeline has three roles: producer, broker, and consumer. The producer serialises an event (here, a Python dict to JSON bytes), stamps it with an optional key, and calls producer.send(topic, key, value). The broker appends the message to the partition determined by the key hash and acknowledges the write. The consumer polls the broker, deserialises the bytes, and processes the event — in this lesson, incrementing a page counter and committing the offset. The topic acts as a durable, ordered log between the two sides. Producers and consumers are decoupled: the producer does not know how many consumers exist, and the consumer can replay the topic from any offset independently of the producer.
Idempotent processing requires three things: (1) a stable, unique identifier on every event — we use a UUID generated at produce time and embedded as event_id; (2) a persistent store that survives process restarts — Redis SADD with a TTL or a Cassandra row keyed on event_id; (3) correct ordering of operations — check the store before writing the result, add to the store before committing the Kafka offset, so that a crash between write and commit causes a safe replay rather than data loss. On the producer side, enable_idempotence=True prevents duplicates from broker retries, but this is separate from consumer-side deduplication — both are needed for end-to-end exactly-once semantics without Kafka Transactions.
Several levers, applied in order of cost: (1) Increase partitions — more partitions allow more parallel consumer instances. At 1M events/s, start with 30–60 partitions and benchmark. (2) Scale consumer group — deploy one consumer pod per partition on Kubernetes; each pod handles its own subset. (3) Batch the deduplication check — instead of one Redis lookup per event, buffer 1000 events, pipeline the SADD calls, and process the batch atomically. Redis pipelines cut round-trips by 100x. (4) Replace Redis with a bloom filter — for at-most-once dedup with controlled false-positive rate, a distributed bloom filter (Redis module or ScyllaDB) is O(1) per lookup with no storage growth. (5) Use Kafka Streams or Flink — at this scale, Python consumers become a bottleneck; Kafka Streams (JVM) or Apache Flink handle stateful deduplication natively with exactly-once guarantees via distributed state stores. (6) Multi-broker cluster — 3-broker KRaft cluster with rack-aware replication for fault tolerance and higher write throughput per partition.
✅ Key takeaways
- A Kafka topic with 3 partitions and a
user_idkey guarantees per-user ordering — all events for the same user land in the same partition, enabling future stateful processing per user. - Producer-side idempotence (
enable_idempotence=True,acks='all') prevents duplicate messages at the broker level when the producer retries a failed send — this is not the same as consumer-side deduplication. - Consumer-side idempotency requires a stable
event_idon every event; without a unique identifier, deduplication is impossible regardless of how the consumer is implemented. - In-memory deduplication sets are sufficient for testing but not for production — a process restart clears the set, causing a full replay to be treated as all-new events; use Redis or Cassandra for persistence.
- The correct operation order for safe idempotent processing is: check the persistent store → write the result → add to the persistent store → commit the Kafka offset. Any crash before the commit triggers a safe replay, not data loss.
Build pipelines like Tunde, Funmi, and Chidi
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →