What is Structured Streaming
Spark Structured Streaming treats a stream as an unbounded DataFrame. The same API you use for batch — filter, groupBy, agg, join — works identically on a live stream. Spark handles state management, checkpointing, and exactly-once semantics (with the Kafka source) automatically. You write DataFrame code; Spark handles the rest.
Before Structured Streaming, Spark had a lower-level API called DStream (Discretised Streams). DStream required thinking in RDD batches, had limited support for event-time processing, and made exactly-once guarantees difficult. Structured Streaming replaced it entirely with a high-level DataFrame model introduced in Spark 2.0 and made production-ready in Spark 2.3+.
The model is simple: imagine a table that keeps growing as new data arrives. Your query runs continuously against this table, emitting updated results at each trigger interval.
Micro-batch vs Continuous mode
Structured Streaming offers two processing modes. The vast majority of production workloads use micro-batch.
| Mode | How it works | Characteristics |
|---|---|---|
| Micro-batch (default) | Processes data in small batches every trigger interval. Each trigger runs a full Spark job over the new records since the last trigger. | Throughput-optimized. ~100ms latency minimum. Supports all operators including stateful aggregations. |
| Continuous | Processes records row by row as they arrive, with an asynchronous checkpoint mechanism. | Latency under 1ms. Supports limited operators (no stateful aggregations). Experimental in Spark 3.5. |
Reading from Kafka
The Kafka source is added via a Maven JAR package. Spark downloads it automatically on first run using the spark.jars.packages configuration. The raw Kafka DataFrame has fixed columns: key, value (both binary), topic, partition, offset, and timestamp.
Kafka broker and Spark must be running. Output will vary.
"latest" — only process new messages arriving after the job starts. Use in production when you don't need historical data.
"earliest" — read from the beginning of the topic's retention window. Use for backfills or initial loads.
A JSON offset map — read from a specific partition offset, e.g. {"naijacart.orders":{"0":1500}}.
Parsing the JSON payload
Kafka messages arrive as raw bytes. You must cast the value column to a string and then parse it using from_json with a defined schema. Spark never infers schemas on streams — you must declare the schema explicitly.
Kafka broker and Spark must be running. Output will vary.
kafka_ts (the Kafka broker timestamp) rather than event_ts (the producer-set timestamp) for windowing. The broker timestamp is monotonically reliable; the producer timestamp can be skewed by device clock drift.
Streaming aggregation with tumbling window
Structured Streaming supports time-based windows using window(). A tumbling window is a non-overlapping, fixed-size window — every record belongs to exactly one window. Below, NaijaCart aggregates orders per city every 1 minute.
Kafka broker and Spark must be running. Output will vary.
Batch 1 shows Lagos updated from 3 to 4 orders — a late message arrived for the 10:00–10:01 window and was accepted because it fell within the 30-second watermark. Once the watermark passes, that window closes and no further late data is accepted.
Output modes
The output mode controls which rows are written to the sink at each trigger. Choose the wrong one and you will either blow memory or lose updates.
| Mode | When to use |
|---|---|
| append | New rows only — works for non-aggregated streams or aggregations after watermark closes the window. Safe for file sinks (Parquet, Delta). Cannot update a row once written. |
| update | Changed rows only — best for aggregations with large state. Sends only windows that changed in this trigger. Efficient for database sinks (Cassandra, PostgreSQL). Not supported by file sinks. |
| complete | Full result table every trigger — only for small aggregations where you need the entire result set. State grows unbounded. Use only when the result set is bounded by design (e.g., aggregating by a fixed set of 10 categories). |
Checkpointing
Structured Streaming writes its state — consumer offsets, aggregation state, and operator state — to a checkpoint location on durable storage (HDFS, S3, or local disk in development). If the job crashes, Spark reads the checkpoint and restarts from exactly where it stopped. Without a checkpoint, a crash forces a full re-read from the configured startingOffsets.
Spark writes three things: (1) offset log — the Kafka offsets consumed in each batch, so Spark knows where to resume; (2) commit log — which batches have been committed to the sink; (3) state store — the in-memory aggregation state (e.g., running sums per window per city). Recovering all three gives exactly-once end-to-end semantics when paired with an idempotent sink.
Watermarks
Late data is inevitable in distributed systems. A mobile app may queue orders while offline and flush them minutes later. Without a watermark, Spark must hold all historical state open indefinitely — eventually causing out-of-memory errors.
.withWatermark("kafka_ts", "30 seconds") tells Spark: "wait up to 30 seconds for late messages before closing a window." After the watermark threshold passes, late messages for closed windows are silently dropped. The watermark advances as Spark observes the maximum event timestamp across all partitions.
| Scenario | Behaviour |
|---|---|
| Message arrives 20 seconds late (within watermark) | Accepted. Window is updated. Result emitted on next trigger. |
| Message arrives 45 seconds late (beyond watermark) | Dropped silently. The window has already been finalised and cleaned from state. |
| No watermark set, stateful aggregation | State grows unbounded. OOM in production after hours or days. |
Writing to a file sink
The console sink is for development only. In production, NaijaCart writes streaming output to Parquet files on MinIO (S3-compatible). With outputMode("append") and a watermark, Spark writes a window's results once the watermark confirms the window is closed — guaranteeing no partial writes.
Parquet files are immutable — you cannot update a row already written. So you must use outputMode("append") with a file sink. This means Spark will only write a window after the watermark confirms it is closed. For windowed aggregations this works well: each window's final result is written exactly once. For non-aggregated streams, append writes every new record immediately.
Tunde's streaming principle
At NaijaCart, Tunde's streaming jobs run as Kubernetes pods. If a pod dies, Kubernetes restarts it and Spark recovers from the checkpoint. The Kafka consumer group offset + Spark checkpoint = exactly-once semantics end-to-end. The checkpoint is stored on MinIO so it survives pod restarts, node failures, and even cluster upgrades. The job has been running continuously for 14 months with zero data loss.
This combination — Kubernetes for process recovery, Spark checkpoints for state recovery, Kafka offsets for source replay — is the standard production pattern for Structured Streaming at scale. Each piece plays a distinct role:
- ▸ Kubernetes — detects pod failure and restarts within seconds
- ▸ Spark checkpoint — restores aggregation state and knows which offsets were already processed
- ▸ Kafka offset management — Spark commits the offset to Kafka after checkpointing, so no message is processed twice
- ▸ Idempotent sink — even if a batch is replayed, writing the same Parquet file twice to MinIO is safe
Common mistakes
complete output mode for large aggregationswithWatermark on stateful queriescheckpointLocation setstartingOffsets and reprocesses all data from the beginning of Kafka's retention window — potentially millions of duplicate records downstream.StreamingContext, dstream.foreachRDD) is the legacy low-level API. It lacks event-time support, requires thinking in RDD batches, and has weaker exactly-once guarantees. Structured Streaming replaced it. Any tutorial or Stack Overflow answer that references StreamingContext is outdated.Quiz — 3 questions
.withWatermark("ts", "30 seconds") means:Now add a watermark and a checkpoint to your streaming job.
Interview Q&A
Micro-batch (default) collects new records arriving between trigger intervals and processes them as a single Spark job. The minimum latency is ~100ms (one trigger interval). It supports all operators including stateful aggregations with watermarks, and achieves exactly-once semantics by combining Kafka offset tracking with Spark's commit log. The execution model is identical to a batch Spark job — the same DAG, the same shuffle mechanisms — just run repeatedly on new data.
Continuous processing eliminates the trigger interval: records are processed row by row as they arrive, using a persistent asynchronous checkpoint mechanism (not the batch commit log). Latency drops below 1ms. However, continuous mode supports only a subset of operators — no stateful aggregations, no window functions — and is still experimental in Spark 3.5. In practice, nearly all production streaming jobs at NaijaCart use micro-batch because the ~100ms minimum latency is far below any real-time requirement they have.
Exactly-once end-to-end requires three things working together: (1) the source is replayable, (2) the engine tracks what has been processed, and (3) the sink is idempotent.
Kafka satisfies (1): messages are retained and identified by partition + offset. Spark satisfies (2) through the checkpoint mechanism — before processing a micro-batch, Spark writes the Kafka offsets it will consume to the offset log. After the batch is committed to the sink, Spark writes to the commit log. If the job crashes mid-batch, on restart Spark reads the offset log, sees the batch was not committed, and replays it from the same offsets. If the job crashes after commit, Spark's commit log shows the batch is done and advances to the next offset. The sink must satisfy (3): for file sinks like Parquet, writing the same data twice is idempotent because Spark uses unique task IDs in file names and uses atomic rename. For database sinks, idempotency requires upsert semantics or partition-level overwrite.
A watermark is Spark's mechanism for handling late data while bounding state size. When you write .withWatermark("kafka_ts", "2 minutes"), you are telling Spark: "the maximum expected delay for data arrival is 2 minutes." Spark tracks the maximum event timestamp seen across all partitions. The current watermark is: max observed timestamp - 2 minutes. Any event with a timestamp below the watermark is considered late and is dropped. Any window whose end time is below the watermark is considered closed — Spark finalises the result and evicts the window's state from the state store.
Without a watermark, Spark cannot evict any state. Every open window must be kept in memory indefinitely because a late message could theoretically arrive at any time. For a job aggregating per city per minute, running for 7 days without a watermark means holding 10,080 window states open. This reliably causes executor OOM in production. The watermark is what allows Spark to release state and write append-mode results to file sinks.
Key takeaways
- Structured Streaming treats a Kafka topic as an unbounded DataFrame — the same
filter,groupBy, andaggAPI you use for batch works unchanged on a live stream. - Always set
.withWatermark()on stateful aggregations. Without it, the state store grows unbounded and the job will OOM in production within days. - Always set a
checkpointLocation. Without it, a crash forces a full re-read fromstartingOffsets, producing duplicate records downstream. - Choose output mode based on your sink:
updatefor database sinks where rows can change,appendfor file sinks (Parquet, Delta) which require watermark-closed windows.
Build streaming pipelines that handle millions of events
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →