🗺️ What we are building
In this mini-build you will construct a complete streaming analytics pipeline from scratch. By the time you finish, you will have a working system that generates clickstream events, aggregates them into 1-minute tumbling windows, and writes the results to both the console (for debugging) and Parquet files (for storage).
kafka_producer.py
Generates synthetic clicks at 5 events/sec
Kafka Topic
clickstream (3 partitions)
Spark Structured Streaming
readStream → parse JSON → window → groupBy → count
Console + Parquet
Debug output + durable storage
The full pipeline uses three files: a docker-compose.yml to run Kafka, a kafka_producer.py that sends events, and a spark_streaming.py that reads, aggregates, and writes. We will go through each in detail.
Docker Desktop installed and running. Python 3.9+ with kafka-python installed (pip install kafka-python). Apache Spark 3.4+ with the Kafka connector JAR. The easiest way to get the connector is to use spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.4.0 which downloads it automatically.
🐳 Step 1 — Start Kafka with Docker Compose
We use KRaft mode (Kafka without ZooKeeper, available from Kafka 3.3+) for simplicity. The compose file starts a single-broker Kafka instance with a topic called clickstream automatically created on startup.
⚡ Step 2 — The Kafka producer
The producer generates synthetic clickstream events and sends them to the clickstream topic at 5 events per second. Each event is a JSON object with a URL, a user ID, and an ISO 8601 timestamp — the timestamp is the event time that Spark will use for windowing.
window() function operates on event time — the timestamp embedded in the event — not on the time Spark processes the event. Using ISO 8601 strings lets us cast to TimestampType cleanly. The cast handles UTC offset notation automatically.
⚙️ Step 3 — The Spark Structured Streaming job
The streaming job reads from the Kafka topic, parses the JSON events, applies a 1-minute tumbling window keyed on the url field, counts page views per window, and writes results to both the console sink and a Parquet sink.
Understanding tumbling windows
A tumbling window is a fixed-size, non-overlapping time bucket. A 1-minute tumbling window groups events by which 60-second interval they fall into, based on their event timestamp:
- ▸ Window [09:00:00 – 09:01:00) contains all events with
tsin that minute - ▸ Window [09:01:00 – 09:02:00) contains the next minute's events
- ▸ Each event belongs to exactly one window — no overlap
writeStream is an independent streaming query with its own checkpoint location. They can run at different trigger intervals. The console query triggers every 30 s for fast feedback; the Parquet query triggers every 60 s to reduce small-file write overhead.
💧 Watermarking — handling late data
In streaming systems, events do not always arrive in the order they were generated. A click at 09:01:55 might arrive at the Spark driver at 09:02:05 — after the [09:01:00 – 09:02:00) window has nominally closed. Without watermarking, Spark keeps the state for every past window alive indefinitely, which causes unbounded memory growth.
Watermarking tells Spark how long to wait for late events before closing a window and discarding its state. The statement:
means: "Track the maximum event timestamp seen so far. Once the current maximum minus 10 seconds exceeds a window's end time, that window is finalised and its state is evicted from memory." Events arriving after that threshold are silently dropped.
| Watermark delay | Effect | Use when |
|---|---|---|
"0 seconds" |
Windows close immediately — no late data tolerated | Events always arrive in order (rare) |
"10 seconds" |
Wait 10 s for late events; good for same-datacenter sources | Kafka on the same network with low producer lag |
"2 minutes" |
Tolerate up to 2 min of out-of-order events | Mobile apps that buffer events and flush in batches |
"1 hour" |
Large state store; costly on memory | Edge devices with intermittent connectivity |
The window() function in Spark operates on event time — the timestamp inside the event payload. This is almost always what you want: you care about when the user clicked, not when Spark received the message. If you use processing time (current_timestamp()) for windowing, your windows will shift every time the pipeline has backpressure or restarts, making results non-reproducible. Always window on event time; use watermarks to manage the late-data trade-off.
📟 Expected console output
After the producer has been running for about 90 seconds, the console sink will begin printing micro-batch results. With 5 events/sec over 60 seconds, you will see roughly 300 events distributed across the URLs according to the producer's weights.
The counts match the URL weight distribution: /home at 30% of 300 events = ~90 views per minute. Minor variation is expected due to random sampling.
💾 Checkpointing — surviving restarts
Checkpointing is mandatory for any production streaming job. The checkpoint stores three things: the Kafka offsets last processed, the streaming query plan, and the operator state (window counts in our case). If the job crashes, Spark restarts from the last checkpoint and replays from the saved Kafka offsets — no data loss.
In production, always place checkpoints on durable distributed storage — S3, Azure Blob, or HDFS. A local filesystem checkpoint is lost when the container or VM restarts. On Databricks, use DBFS or a mounted S3 bucket. On EMR, use s3://your-bucket/checkpoints/job-name. Never reuse a checkpoint directory for a different job or after changing the query schema — delete the checkpoint and restart from latest offsets instead.
🚀 How to run the full pipeline
Start Kafka
In terminal 1, from the directory containing docker-compose.yml:
Start the Kafka producer
In terminal 2, start the event generator. Leave it running — it will produce 5 events/sec continuously.
Submit the Spark Structured Streaming job
In terminal 3. The --packages flag downloads the Kafka connector JAR automatically (~3 MB, cached after first run).
Inspect the Parquet output
After the first 60-second trigger, you can read the output with a Spark batch job or with pandas + pyarrow:
🔧 Troubleshooting common issues
to_timestamp(col("e.ts")) returns null (check with .filter(col("ts").isNull()).count()), the timestamp cast is failing. Verify your producer is writing valid ISO 8601 strings. Also check that you are windowing on col("ts") (event time from the payload) — not on col("timestamp") (Kafka's ingest time, which is a different column in the raw Kafka DataFrame).
startingOffsets: "earliest" is set but the topic has been compacted or the retention period has expired, deleting earlier offsets. Set failOnDataLoss: "false" to allow Spark to skip missing offsets and continue from the nearest available offset. For checkpointed jobs that have been down longer than the topic's retention window, reset by deleting the checkpoint and restarting with startingOffsets: "latest".
kafka-console-consumer --topic clickstream --from-beginning --max-messages 5 should print events) and confirm the Spark job is consuming (query.recentProgress shows processed row counts).
OPTIMIZE command handles this automatically.
🏆 Challenge — extend the pipeline
A hopping window (also called a sliding window) has both a window size and a slide interval. A 5-minute window that hops every 1 minute means each event appears in 5 windows. This gives you a rolling 5-minute view updated every minute.
Hint: change window(col("ts"), "1 minute") to window(col("ts"), "5 minutes", "1 minute"). The third argument is the slide interval. Adjust the watermark to at least "5 minutes" to handle the longer window lifecycle.
Instead of (or in addition to) writing to Parquet, write the windowed aggregation results back to a Kafka topic called clickstream_aggregated. This enables downstream consumers (a dashboard, an alerting service) to subscribe to pre-aggregated results.
Hint: use .format("kafka") with .option("kafka.bootstrap.servers", "localhost:9092") and .option("topic", "clickstream_aggregated"). The Kafka sink requires the output DataFrame to have a value column of type StringType or BinaryType — use to_json(struct(*)) to serialise all columns as a JSON string into the value column. The update output mode is supported for Kafka sinks.
🧠 Quiz — 4 questions
withWatermark("ts", "10 seconds") do?spark.readStream.format("kafka"), which column contains the actual event payload?Next: Cassandra Fundamentals — learn the wide-column store that powers sub-millisecond reads at scale.
🎤 Interview Q&A
✅ Key takeaways
- A complete Kafka + Spark Structured Streaming pipeline has three components: a producer writing events to a topic, a
readStreamsource reading from that topic, and awriteStreamsink writing aggregated results — all using the same Spark DataFrame API as batch jobs. withWatermark("ts", "10 seconds")bounds state store size by telling Spark how long to wait for late events. Without it, Spark keeps all window state in memory forever, causing OOM errors on long-running jobs. Set the delay to the maximum expected event delivery lag in your network.- Always window on event time (the timestamp in the payload), not processing time. Event-time windowing is reproducible: replaying the same Kafka offsets always produces the same window counts. Processing-time windowing produces different results on replay depending on when Spark processes the events.
- Checkpoints must be on durable distributed storage (S3, HDFS) for production jobs. The checkpoint stores Kafka offsets, operator state (window counts), and the query plan — together they enable exactly-once recovery after any crash.
- The Parquet small-file problem is real: each micro-batch writes multiple small files. For production workloads, add a periodic compaction job (or use Delta Lake's
OPTIMIZE) to consolidate small files for efficient downstream reads.
Ship streaming pipelines like a senior engineer
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →