🧠 The core mental model

Most engineers come to Kafka having used RabbitMQ, SQS, or a similar queue. They think of a message broker as a post office: a message goes in, a consumer picks it up, and the message is gone. Kafka works nothing like this, and if you approach it with the queue mental model you will design your system incorrectly from day one.

Kafka is a distributed, durable, ordered log.

Producers append records to the end of that log. Consumers read records from any position in the log — and can replay from any earlier point at any time. Kafka does not delete a message when a consumer reads it. Messages are retained based on a time or size policy (e.g., 7 days), not based on acknowledgment.

Think of it like a distributed, append-only database table — not a mailbox.

This distinction has enormous practical consequences. It means:

  • ▸ Ten independent systems can all read the same stream of events — each at its own pace, without interfering.
  • ▸ If your analytics pipeline breaks for 3 hours, it can catch up by re-reading from where it left off — no data loss.
  • ▸ You can replay all events from yesterday through a new fraud detection model you deployed today, as if it had been running live.

⚖️ Kafka vs traditional message queues

Here is how Kafka differs from classic queues such as RabbitMQ or Amazon SQS on the dimensions that matter most in production pipeline design.

Dimension RabbitMQ / SQS Kafka
Deletion on consume Message is gone once consumed — no replay possible from the broker Message stays on disk until retention.ms or retention.bytes expires
Replay Not possible — once a consumer ACKs, the message is deleted Any consumer can seek to any offset and re-read the full history within the retention window
Consumer groups Limited or absent — competing consumers share the queue Native — multiple independent consumer groups each get a full copy of every message
Throughput scale Queue-per-service; scaling requires more queues and routing logic Partitions provide horizontal parallelism; millions of messages/sec on a single topic
Retention Message lives until consumed or TTL expires — short-lived by design Configurable from minutes to forever (compacted topics retain latest value per key indefinitely)
Rule of thumb: Use a traditional queue when you need simple task dispatch (one producer, one worker). Use Kafka when multiple systems need the same event stream, when you need replay, or when throughput exceeds 10K messages/sec.

📂 Topics

A topic is a named category for a stream of related events — conceptually similar to a database table, but append-only. There is no UPDATE or DELETE: every event becomes a new row at the end of the log.

Topics are named by convention using dot or hyphen delimiters that encode the owning system and event type:

Illustrative — topic naming conventions
naijacart.orders ← NaijaCart order lifecycle events naijacart.payment-events ← payment authorizations and refunds jengatel.call-events ← call detail records from cell towers jengatel.sms-events ← SMS delivery and failure events southbank.transactions ← core banking transaction log

Each topic can have one or more partitions. The number of partitions is a key design decision — it determines how much parallelism the topic supports.

Topic vs table: the key difference

A database table holds the current state of each entity. A Kafka topic holds the full history of events. In NaijaCart's naijacart.orders topic, every state change — placed, confirmed, shipped, delivered, returned — appears as a separate event. A downstream consumer can materialize the current order state by replaying all events for a given order ID.

🗂️ Partitions

A topic is divided into one or more partitions. Each partition is an independent, ordered, append-only log. Messages are written to a specific partition and never move between partitions. Ordering is guaranteed within a partition, but not across partitions.

Partitions are what make Kafka horizontally scalable: each partition can be hosted on a different broker, and each partition can be consumed by a different consumer thread in parallel.

Topic: naijacart.orders (3 partitions) Partition 0 offset 0 offset 1 offset 2 offset 3 ← newest Partition 1 offset 0 offset 1 offset 2 ← newest Partition 2 offset 0 offset 1 offset 2 offset 3 offset 4 ← newest

Each partition is an independent ordered log. Offsets restart from 0 in every partition. Partition 2 has received more messages than the others — that is normal.

Parallelism = partition count.

A topic with 3 partitions can be consumed by at most 3 consumer threads simultaneously (within a single consumer group). A topic with 12 partitions can be consumed by up to 12 parallel consumers. More partitions = more throughput capacity, but also more overhead. Start small (3–6) and increase when you have a measured bottleneck.

🔢 Offsets

Every message written to a partition is assigned a sequential integer offset, starting from 0. The offset uniquely identifies a message within its partition. It is immutable — offset 7 in Partition 2 always refers to the same message, forever.

Offsets serve three critical purposes:

  • ▸Position tracking. A consumer records which offset it has read up to. This is called "committing the offset." On restart, the consumer asks Kafka: "What offset did I last commit for this partition?" and resumes exactly there — no messages lost or duplicated (assuming at-least-once or exactly-once semantics).
  • ▸Replay. Any consumer can seek to offset 0 and re-read all messages from the beginning, or seek to a specific offset (e.g., the offset corresponding to midnight last night) to replay a time window.
  • ▸Independent consumer groups. Each consumer group maintains its own offset for each partition. Group A at offset 10,000 and Group B at offset 500 can both be reading the same partition at different speeds — Kafka tracks both separately in an internal topic called __consumer_offsets.
Offset is not a timestamp. Offsets are sequential integers — they say nothing about time. To find messages from a specific time window, use Kafka's offsetsForTimes() API, which maps a timestamp to the nearest offset.

📤 Producers

A producer is any application that writes messages to a Kafka topic. The producer decides which partition to write to, and that decision determines whether message ordering is preserved for related events.

Kafka provides three partition selection strategies:

Strategy How it works When to use it
Key-based (hash) hash(key) % num_partitions — the same key always routes to the same partition When you need all events for a given entity (e.g., all orders for Lagos) to be ordered. This is the most common choice in production.
Round-robin Messages are distributed evenly across all partitions in rotation When you have no ordering requirement and want maximum throughput. Used when no message key is specified.
Custom partitioner A user-supplied function that returns a partition number based on any message attribute When business logic dictates routing — e.g., VIP customer orders to Partition 0 for priority processing.
Key selection is a one-time architectural decision.

Once you choose the partition key for a topic, changing it means a potential rebalance of which messages go where — downstream consumers relying on ordering guarantees by key will break. Choose your key based on the entity whose events must be ordered: order_id, customer_id, city, cell_tower_id, etc.

👥 Consumers and consumer groups

A consumer reads messages from one or more topic partitions. Consumers are almost always organized into consumer groups. The group ID is a string that identifies a set of consumers that collectively read a topic — each partition is assigned to exactly one consumer within the group.

naijacart.order-events (3 partitions) Partition 0 Partition 1 Partition 2 Group A warehouse-svc Consumer A-1 ← P0 Consumer A-2 ← P1 Consumer A-3 ← P2 Group B fraud-detection Consumer B-1 (all) Each partition → exactly one consumer within a group. Multiple groups read independently. Group A has 3 consumers (1 per partition — full parallelism). Group B has 1 consumer (reads all 3 partitions sequentially). Both groups receive every message. Neither knows the other exists.

Two independent consumer groups reading the same topic. Adding consumers beyond the partition count does not increase throughput — extra consumers sit idle.

The key rules of consumer groups:

  • ▸Within a group: each partition is owned by exactly one consumer. Two consumers in the same group cannot both read Partition 0 at the same time.
  • ▸Across groups: every group gets a full independent copy of the stream. A message written to Partition 0 will be read by Consumer A-1 (Group A) AND Consumer B-1 (Group B) — Kafka tracks each group's offset separately.
  • ▸Idle consumers: if a group has more consumers than partitions, the extra consumers sit idle. They become active only if another consumer in the group leaves (triggering a rebalance).

🛒 NaijaCart: a real design story

Tunde is a senior data engineer at NaijaCart, a fast-growing Nigerian e-commerce platform. During the 2025 Black Friday sale, NaijaCart processed over 400,000 orders in a single hour — 110 orders per second at peak. A monolithic PostgreSQL-backed event bus could not keep up. Tunde migrated NaijaCart's order pipeline to Kafka.

His design for the naijacart.order-events topic:

  • ▸6 partitions. NaijaCart serves 6 major regions (Lagos, Abuja, Kano, Port Harcourt, Ibadan, Enugu). One partition per region gives regional ordering without overcomplicating the partition count.
  • ▸Partition key: city. All Lagos orders route to Partition 0, all Abuja orders to Partition 1, and so on. This guarantees that for any given city, orders are processed in the sequence they were placed — critical for inventory reservation logic.
  • ▸Three consumer groups reading the same topic:
Consumer Group ID What it does Processing speed
warehouse-svc Reserves inventory and dispatches pick-and-pack tasks to warehouse workers in real time Keeps up with live throughput — offset near head
analytics-pipeline Aggregates hourly GMV, conversion funnels, and city-level dashboards ~30 seconds behind live; catches up between spikes
fraud-detection Scores each order for risk signals (velocity, location mismatch, payment anomaly) Keeps up with live throughput; must act before warehouse-svc ships

None of these three consumer groups interfere with each other. Each has its own committed offset, its own lag metric, and its own scaling strategy. Tunde can restart or redeploy any one of them without affecting the others, and each can replay from the beginning of the retention window independently.

⚙️ Key Kafka configurations

These are the configurations you will set most often when creating topics or tuning a Kafka cluster. Understanding what each controls prevents costly mistakes in production.

Config What it controls Typical value
num.partitions Default number of partitions per topic when created without an explicit count 3–12 (start small; you can increase but not easily decrease)
retention.ms How long messages are retained on disk, regardless of whether they have been consumed 604800000 (7 days) for most event streams; longer for audit logs
replication.factor How many broker copies each partition's data is written to (durability) 3 in production (survives 2 broker failures); 1 in local dev
compression.type Algorithm used to compress message batches before writing to disk and sending over the network snappy (balanced) or lz4 (fastest decompression); none only for tiny topics
max.message.bytes Maximum size of a single message (in bytes) that the broker will accept 1 MB (default); increase to 10 MB for large payloads like image metadata
Topic-level vs broker-level configs: retention.ms and max.message.bytes can be set per-topic and override the broker default. replication.factor is set at topic creation time and cannot be changed without reassigning partitions.

📜 The commit log concept

Kafka's performance comes from a fundamental architectural choice: it writes to disk sequentially rather than using random-access updates. Sequential disk writes on a modern HDD are 150 MB/s. Random writes on the same disk might be 0.5 MB/s. Kafka is designed around this physical reality.

Each partition is stored as a set of segment files on the broker's disk. New messages are always appended to the end of the active segment. Old segments are purged based on:

  • ▸retention.ms — delete segments older than N milliseconds (time-based retention).
  • ▸retention.bytes — delete oldest segments when the total partition size exceeds N bytes (size-based retention).
  • ▸Log compaction — instead of time or size, Kafka retains the most recent message for each key indefinitely. Used for topics that represent the current state of entities (like a changelog from a database).
Consumer acknowledgment does not trigger deletion.

This is one of the most important things to internalize. In RabbitMQ, a consumer ACK tells the broker to delete the message. In Kafka, committing an offset only tells Kafka "I have read up to here" — the message stays on disk until the retention policy says otherwise. This is what enables multiple consumer groups to read the same data independently.

📡 Wanjiku's design at JengaTel

JengaTel: one topic, two completely independent pipelines

Wanjiku is the lead data engineer at JengaTel, a Kenyan telecommunications company handling ~12 million mobile calls per day. She designed the jengatel.call-events topic with 12 partitions, using cell_tower_id as the partition key. This guarantees that all call events from a given cell tower arrive in the order they occurred — essential for detecting tower overload patterns.

Two consumer groups read the same topic with completely independent purposes:

  • ▸Group fraud-rt — the real-time fraud detection system. It processes each call event within 200 milliseconds, looking for SIM swap fraud, international revenue share fraud, and abnormal call velocity. It keeps its offset near the head of the log at all times.
  • ▸Group cdr-batch — the daily CDR (Call Detail Record) batch aggregation pipeline. It runs at 01:00 EAT each night, catching up to the previous 24 hours of events and writing summary rows to JengaTel's data warehouse. It deliberately lags 23 hours behind and is not concerned with latency.

Same Kafka topic. Same 12 partitions. Two completely independent pipelines — neither knows the other exists. Wanjiku can redeploy, rebalance, or replay either group without touching the other. This is Kafka's consumer group model at its most powerful.

⚠️ Common mistakes

Too few partitions
Starting with 1 or 2 partitions means you cannot add more consumers to a group to scale throughput. You can increase partition count later, but that changes key-to-partition routing — downstream systems relying on per-key ordering may break. Design for 2–3x your expected peak consumer count from the start.
Too many partitions
Each partition requires open file handles, memory for the offset index, and replication overhead on the broker. The practical limit is around 100,000 partitions per broker cluster. A topic with 1,000 partitions for a 5 message/sec stream is unnecessary and creates measurable overhead. Match partition count to your actual consumer concurrency need.
Not choosing a partition key
Without a message key, Kafka uses round-robin — messages for the same entity (same order ID, same customer) go to different partitions. This destroys per-entity ordering. Downstream consumers that assume "all events for order #1234 arrive in order" will see them interleaved with other orders. Always choose a key when ordering matters.
Retention too low for replay needs
Setting retention.ms to 1 hour (3600000) means any consumer group that falls behind by more than an hour loses messages permanently. Backfill and replay — two of Kafka's greatest strengths — become impossible. Use at least 7 days (604800000 ms) for most event streams, and longer for compliance-critical data.

🧠 Quiz — 3 questions

Question 1 of 3
What is a Kafka offset?
Question 2 of 3
A topic has 4 partitions. A single consumer group has 6 consumers. How many consumers are idle?
Question 3 of 3
True or False: Kafka deletes a message from a partition as soon as a consumer reads and acknowledges it.
0
/ 3
Kafka fundamentals locked in.

Next up: local Kafka setup with Docker and writing your first producer.

🎤 Interview Q&A

Easy What is the difference between a Kafka topic and a partition?
+

A topic is the logical namespace for a stream of related events — it is what producers write to and consumers subscribe to. You give it a name like naijacart.order-events and it represents a category of events.

A partition is the physical unit of storage and parallelism within a topic. Every topic is divided into one or more partitions. Each partition is an independent, ordered, append-only log stored on a specific broker. When a producer writes to a topic, Kafka routes the message to one specific partition. Ordering is guaranteed only within a partition, not across the topic as a whole.

The relationship: one topic has many partitions. A topic with 6 partitions supports up to 6 parallel consumers within a single consumer group, and each partition is independently replicated across brokers.

Medium How does Kafka guarantee message ordering?
+

Kafka guarantees ordering within a partition only. Messages written to Partition 2 will always be read in the order they were written — the offset monotonically increases. Kafka makes no guarantee about the relative ordering of messages across different partitions.

In practice, this means ordering guarantees are tied to how you choose your partition key. If you need all events for a specific entity (e.g., all events for a single order) to arrive in order, you must partition by an identifier for that entity (e.g., order_id). The hash of the key always routes the same entity's events to the same partition, where they are appended sequentially.

If you use round-robin (no key), events for the same entity can end up in different partitions and will not have guaranteed ordering relative to each other. For global ordering across an entire topic, you would need exactly 1 partition — which eliminates horizontal scalability. The practical design is: choose your key to match the ordering granularity you need.

Medium What is consumer group rebalancing?
+

Consumer group rebalancing is the process by which Kafka reassigns partitions among the consumers in a group. It is triggered by:

  • ▸A new consumer joining the group
  • ▸An existing consumer leaving the group (crash, shutdown, or missing heartbeat)
  • ▸New partitions being added to a topic the group is subscribed to

During a rebalance, all consumers in the group stop consuming (the "stop the world" phase in older versions of Kafka). The group coordinator (a Kafka broker) runs the partition assignment algorithm and sends each consumer its new set of assigned partitions. Consumers then resume from the last committed offset for each assigned partition.

Rebalancing is necessary for self-healing (if a consumer crashes, its partitions are automatically handed off to another consumer), but frequent rebalancing — caused by consumers with slow processing or short max.poll.interval.ms — can cause significant consumer lag. Modern Kafka (2.4+) supports "incremental cooperative rebalancing," which allows consumers to keep partitions they are not being reassigned, greatly reducing disruption.

✅ Key takeaways

  • Kafka is a distributed, durable, ordered commit log — not a message queue. Messages are not deleted when consumers read them; they are retained based on a time or size policy. This enables replay, backfill, and multiple independent consumer groups reading the same stream.
  • A topic is the logical category; partitions are the physical units of parallelism and ordering. Ordering is guaranteed per partition. Use a meaningful partition key so that all events for a given entity always land in the same partition.
  • Each message in a partition gets a monotonically increasing offset. Consumers commit offsets to track progress. On restart, they resume from the last committed offset — enabling fault tolerance without data loss.
  • Multiple consumer groups can read the same topic independently, each at its own pace, each with its own committed offsets. This is how NaijaCart runs warehouse, analytics, and fraud detection off the same order-events stream with zero coupling between the three systems.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Build real streaming pipelines with Kafka

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →