🛠️ What you're building

NaijaCart's engineering team — Tunde, Funmi, and Chidi — needs to track page-click behaviour across their Nigerian e-commerce platform. This mini-build takes you from zero to a complete Kafka pipeline: a producer that simulates 100 click events and a consumer that counts clicks per page, deduplicated so that replays never double-count.

Pipeline spec
  • Topic: naijacart.clicks — 3 partitions, keyed by user_id so the same user always lands on the same partition
  • Producer: simulates 100 click events spread across 5 pages (/home, /product, /cart, /checkout, /confirm) and 5 cities (Lagos, Abuja, Kano, Port Harcourt, Ibadan)
  • Consumer: reads events, counts clicks per page, writes totals into a local click_counts dict
  • Idempotency: every event carries a unique event_id (UUID); the consumer maintains a seen_event_ids set and skips duplicates, so replaying the topic from the beginning never double-counts
Prerequisites

The Kafka broker from Lesson 00 must be running (./start.sh). Install kafka-python if you haven't yet: pip install kafka-python==2.0.2. This lesson uses kafka-python (not confluent-kafka) for its simpler producer/consumer API — both talk to the same broker at localhost:9092.

🗺️ Architecture

Four stages, all running locally. The arrow from consumer to click_counts represents an in-memory write — in Arc E we replace this with a Cassandra upsert.

Producer producer.py 100 events naijacart.clicks 3 partitions keyed by user_id Kafka topic Consumer consumer.py dedup + count click_counts dict print summary
1
Create the topic

Create naijacart.clicks with 3 partitions. The --replication-factor 1 flag is correct for a single-broker local setup.

Kafka broker must be running at localhost:9092. Output will vary.

terminal Lesson 20 · create topic
sample output
Created topic naijacart.clicks.

Verify the topic was created with the right partition count:

terminal Lesson 20 · describe topic
sample output
Topic: naijacart.clicks PartitionCount: 3 ReplicationFactor: 1 Topic: naijacart.clicks Partition: 0 Leader: 1 Replicas: 1 Isr: 1 Topic: naijacart.clicks Partition: 1 Leader: 1 Replicas: 1 Isr: 1 Topic: naijacart.clicks Partition: 2 Leader: 1 Replicas: 1 Isr: 1
Why 3 partitions keyed by user_id?

Kafka hashes the message key (user_id) to select a partition. Using the same user as the key guarantees ordering per user — all events for U01 always land in the same partition, so a per-user state machine could process them in arrival order. Three partitions lets up to three consumer instances in the same group share the load in Arc D.

2
The producer

Save this as producer.py. It uses enable_idempotence=True on the producer side — this prevents the broker from storing duplicate messages if the producer retries a failed send.

Kafka broker must be running at localhost:9092. Output will vary.

producer.py Lesson 20 · clickstream producer
sample output
Producing 100 NaijaCart click events... Sent 20/100 events Sent 40/100 events Sent 60/100 events Sent 80/100 events Sent 100/100 events Done.
What enable_idempotence=True does on the producer side

When a producer send fails and retries, the broker could receive the same message twice without idempotence. With enable_idempotence=True, the producer stamps each batch with a sequence number and the broker de-duplicates at the log level — exactly-once delivery per partition, even under retries. Requires acks='all' (already set above).

3
The consumer with idempotent writes

Save this as consumer.py. The seen_event_ids set is the deduplication guard — if an event's event_id is already in the set, it is skipped without updating the counts. Manual commit() after each event ensures offsets are only advanced for successfully processed messages.

Kafka broker must be running at localhost:9092. Output will vary.

consumer.py Lesson 20 · idempotent consumer
sample output
=== NaijaCart Click Summary === /product 23 ███████████ /home 21 ██████████ /cart 19 █████████ /checkout 19 █████████ /confirm 18 █████████ Processed : 100 Duplicates: 0
Tunde's note: The click distribution across 5 pages is roughly even because we used random.seed(42) — the same seed produces the same pseudo-random sequence every time. In production, the distribution would be skewed: product pages and the cart typically see 3–5x more traffic than confirm.
4
Simulate a replay — idempotency test

To test idempotency, reset the consumer group offset back to the earliest position and run consumer.py again. Kafka re-delivers all 100 events. Because the seen_event_ids set is populated in memory from the first run — if the process is still running — all 100 are detected as duplicates and the counts stay at zero.

Reset the consumer group offset

Kafka broker must be running at localhost:9092. Output will vary.

terminal Lesson 20 · reset offsets to earliest
sample output
GROUP TOPIC PARTITION NEW-OFFSET click-counter naijacart.clicks 0 0 click-counter naijacart.clicks 1 0 click-counter naijacart.clicks 2 0

Rerun consumer.py (same process session)

Run python consumer.py again immediately without restarting the process. All 100 events are re-delivered, but the in-memory seen_event_ids set still holds all 100 IDs from the first run — so every event is a duplicate.

sample output — after replay
=== NaijaCart Click Summary === /product 0 /home 0 /cart 0 /checkout 0 /confirm 0 Processed : 0 Duplicates: 100
The limitation: in-memory sets reset on restart

The seen_event_ids set lives in Python heap memory. If the consumer process crashes and restarts, the set is empty — replaying the topic now processes all events again as if they are new. In a real pipeline, Funmi would persist the seen IDs in Redis (fast, TTL-based) or Cassandra (durable, keyed on event_id) before committing the Kafka offset. We cover persistent idempotent writes in Arc E.

🏭 Full pipeline in production at NaijaCart

NaijaCart production setup (Funmi's pipeline)

In production, Funmi runs this pipeline continuously. The producer is a FastAPI endpoint that publishes a click event to naijacart.clicks on every page load — one HTTP request in, one Kafka message out. The consumer is a long-running Python process on a Kubernetes pod in the NaijaCart cluster. The click_counts dict is replaced by a Cassandra upsert keyed on (date, page) — because Cassandra upserts are idempotent by design, the consumer can restart at any time, replay from its last committed offset, and upsert without double-counting. The seen_event_ids set is backed by a Redis SADD call with a 24-hour TTL.

Component Local (this lesson) NaijaCart production
Producer producer.py script, runs once FastAPI endpoint, publishes on every page load
Consumer consumer.py script, exits after 3 s idle Long-running Kubernetes pod, never exits
Deduplication store In-memory Python set() Redis SADD with 24-hour TTL
Output click_counts dict, printed to stdout Cassandra upsert on (date, page)
Offset commit Manual consumer.commit() per event Manual commit after Cassandra upsert succeeds
Scale 1 producer, 1 consumer, 3 partitions N producers, 3 consumer pods (one per partition)

▶️ Running both together

Open two terminal windows side by side. Start the consumer first so it is ready when the producer begins sending. The & runs it in the background — you can also open a true second terminal tab.

Terminal 1 — start consumer first
python consumer.py
Terminal 2 — then run producer
python producer.py

The consumer uses consumer_timeout_ms=3000 — it exits automatically 3 seconds after the last message. The producer sends all 100 events in under a second. You should see the click summary print in Terminal 1 within 5–6 seconds of starting both scripts.

Chidi's tip: If the consumer exits before the producer finishes sending, increase consumer_timeout_ms to 8000. On slow laptops with Docker under load, the broker may buffer messages for a second or two before the consumer polls them.

⚠️ Common mistakes

In-memory dedup set — lost on restart
A Python set() lives in heap memory and disappears when the process exits. On restart, the consumer sees all messages as new and double-counts on replay. Use Redis SADD or a Cassandra row keyed on event_id for persistence across restarts.
Not setting enable_idempotence=True on the producer
Without producer-side idempotence, a retry after a network hiccup sends the same batch twice. The broker stores both copies as distinct messages. The consumer then sees duplicates even with deduplication logic. Always pair enable_idempotence=True with acks='all'.
Not using a unique event_id
Deduplication is impossible without a stable identifier that survives retries and replays. Using ts (millisecond timestamp) alone is not safe — two events from the same user at the same millisecond produce identical timestamps. Always generate a UUID at event creation time and embed it in the payload.
consumer_timeout_ms too short
Set too low (e.g. 500 ms), the consumer exits before the broker has delivered all messages, especially under Docker load. The remaining messages stay in the topic and are processed on the next run — which looks like a deduplication miss. Set to at least 3000 ms for local dev; remove the timeout entirely for production long-running consumers.

🧠 Quiz — 3 questions

Question 1 of 3
The event_id field in each NaijaCart click event enables what behaviour in the consumer?
Question 2 of 3
Why is an in-memory seen_event_ids set insufficient for production use?
Question 3 of 3
enable_idempotence=True on the Kafka producer prevents which specific problem?
0
/ 3
Pipeline built.

You understand idempotent writes — now apply them with a persistent store in Arc E.

🎤 Interview Q&A

Easy Walk me through a basic Kafka producer-consumer pipeline.
+

A Kafka pipeline has three roles: producer, broker, and consumer. The producer serialises an event (here, a Python dict to JSON bytes), stamps it with an optional key, and calls producer.send(topic, key, value). The broker appends the message to the partition determined by the key hash and acknowledges the write. The consumer polls the broker, deserialises the bytes, and processes the event — in this lesson, incrementing a page counter and committing the offset. The topic acts as a durable, ordered log between the two sides. Producers and consumers are decoupled: the producer does not know how many consumers exist, and the consumer can replay the topic from any offset independently of the producer.

Medium How do you implement idempotent message processing?
+

Idempotent processing requires three things: (1) a stable, unique identifier on every event — we use a UUID generated at produce time and embedded as event_id; (2) a persistent store that survives process restarts — Redis SADD with a TTL or a Cassandra row keyed on event_id; (3) correct ordering of operations — check the store before writing the result, add to the store before committing the Kafka offset, so that a crash between write and commit causes a safe replay rather than data loss. On the producer side, enable_idempotence=True prevents duplicates from broker retries, but this is separate from consumer-side deduplication — both are needed for end-to-end exactly-once semantics without Kafka Transactions.

Hard How would you scale this pipeline to process 1 million events per second?
+

Several levers, applied in order of cost: (1) Increase partitions — more partitions allow more parallel consumer instances. At 1M events/s, start with 30–60 partitions and benchmark. (2) Scale consumer group — deploy one consumer pod per partition on Kubernetes; each pod handles its own subset. (3) Batch the deduplication check — instead of one Redis lookup per event, buffer 1000 events, pipeline the SADD calls, and process the batch atomically. Redis pipelines cut round-trips by 100x. (4) Replace Redis with a bloom filter — for at-most-once dedup with controlled false-positive rate, a distributed bloom filter (Redis module or ScyllaDB) is O(1) per lookup with no storage growth. (5) Use Kafka Streams or Flink — at this scale, Python consumers become a bottleneck; Kafka Streams (JVM) or Apache Flink handle stateful deduplication natively with exactly-once guarantees via distributed state stores. (6) Multi-broker cluster — 3-broker KRaft cluster with rack-aware replication for fault tolerance and higher write throughput per partition.

✅ Key takeaways

  • A Kafka topic with 3 partitions and a user_id key guarantees per-user ordering — all events for the same user land in the same partition, enabling future stateful processing per user.
  • Producer-side idempotence (enable_idempotence=True, acks='all') prevents duplicate messages at the broker level when the producer retries a failed send — this is not the same as consumer-side deduplication.
  • Consumer-side idempotency requires a stable event_id on every event; without a unique identifier, deduplication is impossible regardless of how the consumer is implemented.
  • In-memory deduplication sets are sufficient for testing but not for production — a process restart clears the set, causing a full replay to be treated as all-new events; use Redis or Cassandra for persistence.
  • The correct operation order for safe idempotent processing is: check the persistent store → write the result → add to the persistent store → commit the Kafka offset. Any crash before the commit triggers a safe replay, not data loss.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Build pipelines like Tunde, Funmi, and Chidi

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →