🗺️ What we are building

In this mini-build you will construct a complete streaming analytics pipeline from scratch. By the time you finish, you will have a working system that generates clickstream events, aggregates them into 1-minute tumbling windows, and writes the results to both the console (for debugging) and Parquet files (for storage).

kafka_producer.py

Generates synthetic clicks at 5 events/sec

→

Kafka Topic

clickstream (3 partitions)

→

Spark Structured Streaming

readStream → parse JSON → window → groupBy → count

→

Console + Parquet

Debug output + durable storage

The full pipeline uses three files: a docker-compose.yml to run Kafka, a kafka_producer.py that sends events, and a spark_streaming.py that reads, aggregates, and writes. We will go through each in detail.

Prerequisites

Docker Desktop installed and running. Python 3.9+ with kafka-python installed (pip install kafka-python). Apache Spark 3.4+ with the Kafka connector JAR. The easiest way to get the connector is to use spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.4.0 which downloads it automatically.

🐳 Step 1 — Start Kafka with Docker Compose

We use KRaft mode (Kafka without ZooKeeper, available from Kafka 3.3+) for simplicity. The compose file starts a single-broker Kafka instance with a topic called clickstream automatically created on startup.

docker-compose.yml Lesson 24 · Kafka KRaft single-broker
docker compose up -d
shell wait ~10 seconds for Kafka to be ready

⚡ Step 2 — The Kafka producer

The producer generates synthetic clickstream events and sends them to the clickstream topic at 5 events per second. Each event is a JSON object with a URL, a user ID, and an ISO 8601 timestamp — the timestamp is the event time that Spark will use for windowing.

kafka_producer.py Lesson 24 · 5 events/sec · event-time timestamps
producer output (updates every 50 events)
Producer started. Sending 5 events/sec to topic 'clickstream'. Ctrl+C to stop. Sent 50 events. Last: {'url': '/home', 'user_id': 'u0042', 'ts': '2026-04-27T09:00:09.812341+00:00'} Sent 100 events. Last: {'url': '/product', 'user_id': 'u0117', 'ts': '2026-04-27T09:00:19.831022+00:00'} Sent 150 events. Last: {'url': '/search', 'user_id': 'u0083', 'ts': '2026-04-27T09:00:29.854799+00:00'}
Why ISO 8601 timestamps? Spark Structured Streaming's window() function operates on event time — the timestamp embedded in the event — not on the time Spark processes the event. Using ISO 8601 strings lets us cast to TimestampType cleanly. The cast handles UTC offset notation automatically.

⚙️ Step 3 — The Spark Structured Streaming job

The streaming job reads from the Kafka topic, parses the JSON events, applies a 1-minute tumbling window keyed on the url field, counts page views per window, and writes results to both the console sink and a Parquet sink.

Understanding tumbling windows

A tumbling window is a fixed-size, non-overlapping time bucket. A 1-minute tumbling window groups events by which 60-second interval they fall into, based on their event timestamp:

  • ▸ Window [09:00:00 – 09:01:00) contains all events with ts in that minute
  • ▸ Window [09:01:00 – 09:02:00) contains the next minute's events
  • ▸ Each event belongs to exactly one window — no overlap
spark_streaming.py Lesson 24 · tumbling window · console + Parquet sink
Two queries, two checkpoints: each writeStream is an independent streaming query with its own checkpoint location. They can run at different trigger intervals. The console query triggers every 30 s for fast feedback; the Parquet query triggers every 60 s to reduce small-file write overhead.

💧 Watermarking — handling late data

In streaming systems, events do not always arrive in the order they were generated. A click at 09:01:55 might arrive at the Spark driver at 09:02:05 — after the [09:01:00 – 09:02:00) window has nominally closed. Without watermarking, Spark keeps the state for every past window alive indefinitely, which causes unbounded memory growth.

Watermarking tells Spark how long to wait for late events before closing a window and discarding its state. The statement:

watermark definition

means: "Track the maximum event timestamp seen so far. Once the current maximum minus 10 seconds exceeds a window's end time, that window is finalised and its state is evicted from memory." Events arriving after that threshold are silently dropped.

Watermark delay Effect Use when
"0 seconds" Windows close immediately — no late data tolerated Events always arrive in order (rare)
"10 seconds" Wait 10 s for late events; good for same-datacenter sources Kafka on the same network with low producer lag
"2 minutes" Tolerate up to 2 min of out-of-order events Mobile apps that buffer events and flush in batches
"1 hour" Large state store; costly on memory Edge devices with intermittent connectivity
Event time vs processing time

The window() function in Spark operates on event time — the timestamp inside the event payload. This is almost always what you want: you care about when the user clicked, not when Spark received the message. If you use processing time (current_timestamp()) for windowing, your windows will shift every time the pipeline has backpressure or restarts, making results non-reproducible. Always window on event time; use watermarks to manage the late-data trade-off.

📟 Expected console output

After the producer has been running for about 90 seconds, the console sink will begin printing micro-batch results. With 5 events/sec over 60 seconds, you will see roughly 300 events distributed across the URLs according to the producer's weights.

spark console output — batch 1 (after first full window closes)
------------------------------------------- Batch: 1 Trigger: 2026-04-27 09:01:30 ------------------------------------------- +-------------------+-------------------+----------+-----+ |window_start |window_end |url |views| +-------------------+-------------------+----------+-----+ |2026-04-27 09:00:00|2026-04-27 09:01:00|/home | 90| |2026-04-27 09:00:00|2026-04-27 09:01:00|/product | 75| |2026-04-27 09:00:00|2026-04-27 09:01:00|/search | 60| |2026-04-27 09:00:00|2026-04-27 09:01:00|/checkout | 30| |2026-04-27 09:00:00|2026-04-27 09:01:00|/cart | 24| |2026-04-27 09:00:00|2026-04-27 09:01:00|/account | 15| |2026-04-27 09:00:00|2026-04-27 09:01:00|/help | 6| +-------------------+-------------------+----------+-----+ ------------------------------------------- Batch: 2 Trigger: 2026-04-27 09:02:00 ------------------------------------------- +-------------------+-------------------+----------+-----+ |window_start |window_end |url |views| +-------------------+-------------------+----------+-----+ |2026-04-27 09:01:00|2026-04-27 09:02:00|/home | 88| |2026-04-27 09:01:00|2026-04-27 09:02:00|/product | 73| |2026-04-27 09:01:00|2026-04-27 09:02:00|/search | 63| |2026-04-27 09:01:00|2026-04-27 09:02:00|/checkout | 28| |2026-04-27 09:01:00|2026-04-27 09:02:00|/cart | 26| |2026-04-27 09:01:00|2026-04-27 09:02:00|/account | 16| |2026-04-27 09:01:00|2026-04-27 09:02:00|/help | 6| +-------------------+-------------------+----------+-----+

The counts match the URL weight distribution: /home at 30% of 300 events = ~90 views per minute. Minor variation is expected due to random sampling.

💾 Checkpointing — surviving restarts

Checkpointing is mandatory for any production streaming job. The checkpoint stores three things: the Kafka offsets last processed, the streaming query plan, and the operator state (window counts in our case). If the job crashes, Spark restarts from the last checkpoint and replays from the saved Kafka offsets — no data loss.

checkpoint_config.py Lesson 24 · checkpoint options explained
checkpoint output
Query ID: 7a3f1c2e-9b4d-4a88-8f3e-1c2b3d4e5f6a Status: {'message': 'Waiting for data to arrive', 'isDataAvailable': True, 'isTriggerActive': False}
Production checkpoint storage

In production, always place checkpoints on durable distributed storage — S3, Azure Blob, or HDFS. A local filesystem checkpoint is lost when the container or VM restarts. On Databricks, use DBFS or a mounted S3 bucket. On EMR, use s3://your-bucket/checkpoints/job-name. Never reuse a checkpoint directory for a different job or after changing the query schema — delete the checkpoint and restart from latest offsets instead.

🚀 How to run the full pipeline

1

Start Kafka

In terminal 1, from the directory containing docker-compose.yml:

terminal 1
2

Start the Kafka producer

In terminal 2, start the event generator. Leave it running — it will produce 5 events/sec continuously.

terminal 2
3

Submit the Spark Structured Streaming job

In terminal 3. The --packages flag downloads the Kafka connector JAR automatically (~3 MB, cached after first run).

terminal 3
4

Inspect the Parquet output

After the first 60-second trigger, you can read the output with a Spark batch job or with pandas + pyarrow:

inspect_output.py
parquet inspection output
window_start window_end url views 0 2026-04-27 09:00:00 2026-04-27 09:01:00 /home 90 1 2026-04-27 09:00:00 2026-04-27 09:01:00 /product 75 2 2026-04-27 09:00:00 2026-04-27 09:01:00 /search 60 3 2026-04-27 09:00:00 2026-04-27 09:01:00 /checkout 30 4 2026-04-27 09:00:00 2026-04-27 09:01:00 /cart 24 5 2026-04-27 09:00:00 2026-04-27 09:01:00 /account 15 6 2026-04-27 09:00:00 2026-04-27 09:01:00 /help 6 Total rows: 14

🔧 Troubleshooting common issues

"Why is my window empty?" — event time vs processing time
The most common issue. If to_timestamp(col("e.ts")) returns null (check with .filter(col("ts").isNull()).count()), the timestamp cast is failing. Verify your producer is writing valid ISO 8601 strings. Also check that you are windowing on col("ts") (event time from the payload) — not on col("timestamp") (Kafka's ingest time, which is a different column in the raw Kafka DataFrame).
Kafka offset errors — "offsets out of range"
This happens when startingOffsets: "earliest" is set but the topic has been compacted or the retention period has expired, deleting earlier offsets. Set failOnDataLoss: "false" to allow Spark to skip missing offsets and continue from the nearest available offset. For checkpointed jobs that have been down longer than the topic's retention window, reset by deleting the checkpoint and restarting with startingOffsets: "latest".
No output after several minutes
The watermark must be exceeded before windows emit results. With a 10-second watermark on a 1-minute window, the first output appears roughly 70 seconds after the first events arrive (60 s window + 10 s watermark). If output never appears, verify the producer is running (kafka-console-consumer --topic clickstream --from-beginning --max-messages 5 should print events) and confirm the Spark job is consuming (query.recentProgress shows processed row counts).
Parquet small-files problem
Each micro-batch writes multiple small Parquet files (one per partition). After hours of running, you may have thousands of tiny files, which degrades read performance. In production, add a compaction job (a Spark batch job that reads the output directory and rewrites it as a smaller set of larger files) triggered daily. On Databricks, Delta Lake's OPTIMIZE command handles this automatically.

🏆 Challenge — extend the pipeline

Two extension tasks — try both
Challenge 1: Add a 5-minute hopping window

A hopping window (also called a sliding window) has both a window size and a slide interval. A 5-minute window that hops every 1 minute means each event appears in 5 windows. This gives you a rolling 5-minute view updated every minute.

Hint: change window(col("ts"), "1 minute") to window(col("ts"), "5 minutes", "1 minute"). The third argument is the slide interval. Adjust the watermark to at least "5 minutes" to handle the longer window lifecycle.

Challenge 2: Add a Kafka sink instead of Parquet

Instead of (or in addition to) writing to Parquet, write the windowed aggregation results back to a Kafka topic called clickstream_aggregated. This enables downstream consumers (a dashboard, an alerting service) to subscribe to pre-aggregated results.

Hint: use .format("kafka") with .option("kafka.bootstrap.servers", "localhost:9092") and .option("topic", "clickstream_aggregated"). The Kafka sink requires the output DataFrame to have a value column of type StringType or BinaryType — use to_json(struct(*)) to serialise all columns as a JSON string into the value column. The update output mode is supported for Kafka sinks.

🧠 Quiz — 4 questions

Question 1 of 4
In Spark Structured Streaming, what does withWatermark("ts", "10 seconds") do?
Question 2 of 4
What is the difference between a tumbling window and a hopping (sliding) window?
Question 3 of 4
Why should a streaming job's checkpoint directory be on durable distributed storage (S3, HDFS) rather than local disk?
Question 4 of 4
In the Kafka DataFrame produced by spark.readStream.format("kafka"), which column contains the actual event payload?
0
/ 4
Streaming aggregations shipped!

Next: Cassandra Fundamentals — learn the wide-column store that powers sub-millisecond reads at scale.

🎤 Interview Q&A

Medium What is the purpose of a watermark in Spark Structured Streaming and what happens to events that arrive after the watermark threshold?
+
A watermark in Spark Structured Streaming defines the maximum amount of time the engine will wait for late-arriving events before closing a window and evicting its state from memory. Internally, Spark tracks the maximum event timestamp it has seen across all partitions. The current watermark is defined as: max_event_time_seen − watermark_delay. Any window whose end time is less than or equal to the current watermark is considered complete and its state is eligible for eviction. Events that arrive after the watermark threshold — meaning their event timestamp falls within a window that has already been closed — are silently dropped and do not affect the output. This is the deliberate trade-off: watermarking bounds state store size (and thus memory cost) at the cost of discarding a small fraction of very late events. The watermark delay should be set to the maximum expected event latency in your pipeline. For a Kafka topic in the same datacenter, 10–30 seconds is typical. For mobile apps that buffer events offline, a watermark of 5–15 minutes may be needed.
Hard Explain what happens when a Spark Structured Streaming job restarts from a checkpoint. What exactly is recovered?
+
When a Spark Structured Streaming job restarts from a checkpoint, it recovers three things: Kafka offsets — the exact partition offsets last committed in the previous run, ensuring processing resumes from where it left off with no duplicate consumption of committed offsets; query plan — the logical and physical streaming query plan is stored and validated on restart to detect if code changes break compatibility; and operator state — for stateful operations like windowed aggregations, the in-memory state (the running counts for each window bucket) is stored in a RocksDB state store that is checkpointed to durable storage. On restart, this state is restored, so a window that was partially computed (e.g., received 150 of an expected 300 events) continues accumulating without losing what was already counted. The recovery guarantee is exactly-once processing: each Kafka offset is processed exactly once across runs. If the job crashes mid-micro-batch after writing some output but before committing the checkpoint, Spark reprocesses that entire micro-batch on restart. This is safe only if your output sinks are idempotent (Parquet with a unique path structure is idempotent; raw INSERT into a database is not).
Easy What output mode should you use for streaming aggregations and why?
+
For streaming aggregations (like windowed counts), use update mode in most cases. Update mode emits only the rows that have changed since the last micro-batch — if a window received new events, that window's row is emitted with its updated count; windows with no new events are not re-emitted. This minimises output volume and is the standard choice for window aggregations. Append mode is not supported for aggregations without a watermark, because without a watermark Spark cannot know when a window is final and safe to append — it might still receive late events that would change the count. With a watermark, append mode emits a window's final result only once, when the watermark has passed the window's end time. Append mode is appropriate when you need exactly-once writes to a sink that does not support upserts (like a Parquet table where each window row must appear only once). Complete mode re-emits the entire result set every micro-batch — useful for global aggregations that don't use a watermark, but it grows without bound for windowed queries and is rarely appropriate in production.

✅ Key takeaways

  • A complete Kafka + Spark Structured Streaming pipeline has three components: a producer writing events to a topic, a readStream source reading from that topic, and a writeStream sink writing aggregated results — all using the same Spark DataFrame API as batch jobs.
  • withWatermark("ts", "10 seconds") bounds state store size by telling Spark how long to wait for late events. Without it, Spark keeps all window state in memory forever, causing OOM errors on long-running jobs. Set the delay to the maximum expected event delivery lag in your network.
  • Always window on event time (the timestamp in the payload), not processing time. Event-time windowing is reproducible: replaying the same Kafka offsets always produces the same window counts. Processing-time windowing produces different results on replay depending on when Spark processes the events.
  • Checkpoints must be on durable distributed storage (S3, HDFS) for production jobs. The checkpoint stores Kafka offsets, operator state (window counts), and the query plan — together they enable exactly-once recovery after any crash.
  • The Parquet small-file problem is real: each micro-batch writes multiple small files. For production workloads, add a periodic compaction job (or use Delta Lake's OPTIMIZE) to consolidate small files for efficient downstream reads.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ship streaming pipelines like a senior engineer

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →