📋 The choice that shapes everything
The single most important architecture decision in a data pipeline is whether to process data in batch or in streams. Get it wrong and you either over-engineer a simple problem with expensive Kafka infrastructure, or you disappoint a business that needed near-real-time answers and got a stale hourly report instead.
This lesson gives you a repeatable decision framework. By the end you will be able to look at any pipeline requirement and choose the right processing model in under five minutes.
Batch processing
- Processes a fixed, bounded dataset
- Runs on a schedule (hourly, daily, weekly)
- High throughput — can read terabytes in a single job
- Latency measured in minutes to hours
- Simple to reason about — no state management
- Cheap at scale — runs overnight on spot instances
Stream processing
- Processes an unbounded, continuous flow of events
- Runs continuously — always on
- Lower throughput per cost, but latency in seconds
- Latency measured in milliseconds to seconds
- Complex: watermarks, state, checkpoints required
- More expensive — requires persistent compute
⚖️ Latency vs throughput — the core trade-off
Every data processing decision is, at its heart, a trade-off between latency and throughput. You can have very low latency (process each event the moment it arrives) or very high throughput (process millions of events per second in one big pass) — but not both at the same cost.
| Dimension | Batch | Micro-batch (Spark SS) | True streaming (Flink) |
|---|---|---|---|
| Latency | Minutes to hours | Seconds to ~1 minute | Milliseconds to seconds |
| Throughput | Very high (petabyte-scale) | High | Moderate to high |
| Infrastructure cost | Low — runs on spot instances, scales down to zero | Medium — cluster always running | Higher — requires persistent, low-GC JVM tuning |
| Operational complexity | Low | Medium | High |
| State management | None needed | RocksDB-backed state store | Distributed keyed state |
| Fault recovery | Re-run the job | Checkpoint + WAL | Savepoints + changelog |
📦 Bounded vs unbounded data
The most fundamental distinction is whether your data source is bounded (finite, has an end) or unbounded (continuous, never ends).
| Data type | Description | Natural processing model | Examples |
|---|---|---|---|
| Bounded | A fixed dataset that has a beginning and an end | Batch — process the whole thing at once | Yesterday's sales CSV, a database snapshot, a completed log file |
| Unbounded | A continuous, infinite stream of events with no defined end | Stream — process each event or micro-batch as it arrives | Kafka topic, clickstream, sensor readings, WebSocket feed |
Most "batch" jobs against a Kafka topic or database table are actually reading an unbounded source with an artificial time boundary — "give me events from 00:00 to 23:59 yesterday." This is perfectly fine, but it means your latency is at least one full day. Streaming removes that artificial boundary and processes each event as it arrives.
🎯 Canonical use cases — which model fits?
| Use case | Processing model | Why |
|---|---|---|
| Hourly sales dashboard | Batch | Data is queried once per hour; freshness of 5–10 minutes adds zero business value; batch is cheaper and simpler |
| Nightly ETL into data warehouse | Batch | Full historical loads, complex transformations, idempotent re-runs — batch with Airflow is the right tool |
| Daily analytics report (finance, ops) | Batch | Report consumed once a day; stakeholders do not need intra-day freshness; simple Spark job with S3 output |
| Fraud detection scoring | Streaming | A fraud score 60 minutes after the transaction is worthless; must score within seconds of the event |
| Real-time chat / content moderation | Streaming | Harmful content flagged minutes later is already seen; moderation must happen in <1 s from post time |
| Clickstream sessionisation | Micro-batch | 30-second windows are acceptable; Spark Structured Streaming micro-batch is simpler than Flink and hits the latency SLA |
| IoT sensor anomaly detection | Streaming | Equipment failure must trigger an alert within seconds; batch detects the anomaly an hour after the machine broke |
| E-commerce order reconciliation (daily) | Batch | Finance runs reconciliation at EOD; all orders for the day are bounded; a Spark job reading from Postgres is ideal |
🧭 The decision flowchart
Run every new pipeline through this flowchart before writing a single line of code. The questions are ordered from most-decisive to least-decisive.
Schedule with Airflow; Spark or dbt for transforms
Spark Structured Streaming; trigger every 30 s
Flink or Kafka Streams for sub-second latency
Spark Structured Streaming runs as a micro-batch engine by default. Each micro-batch trigger interval (configurable from 1 second to many minutes) reads available Kafka offsets, processes them as a bounded DataFrame, and commits results. You get most of the streaming benefit — data in seconds, not hours — while keeping the familiar Spark API and the lower operational overhead compared to Flink. The sweet spot is any pipeline with a latency SLA between 10 seconds and 5 minutes.
⚠️ The Lambda Architecture trap
In 2011, Nathan Marz proposed the Lambda Architecture: maintain both a batch layer (accurate historical data) and a speed layer (real-time approximations), with a serving layer that merges both. It was influential — and it is, in most cases, a trap.
The Kappa Architecture (Jay Kreps, 2014) simplifies this: keep only the streaming layer. For backfills and historical reprocessing, replay Kafka topics from the beginning. The same streaming job handles both real-time and historical data. One codebase, one team, one pipeline. If your streaming framework (Spark, Flink, Kafka Streams) is mature enough — and in 2026 they are — you rarely need Lambda.
The exception: if you have a truly massive historical dataset (multi-petabyte) that is expensive to store in Kafka for replay, a hybrid approach with a batch historical layer and a streaming incremental layer is justified. But default to Kappa and only add batch complexity when you have proven you need it.
💻 The same job: batch mode vs streaming mode
Below is a PySpark job that aggregates page views by URL. The batch version reads a static Parquet file. The streaming version reads the same logic from a Kafka topic. The transformation code is nearly identical — only the source and sink differ.
Batch version — reads a bounded Parquet file
Streaming version — reads from an unbounded Kafka topic
groupBy and count logic is identical in both versions. The only differences are spark.read vs spark.readStream, the Kafka source configuration, the withWatermark call (needed for streaming state management), and the writeStream sink. This is why Spark Structured Streaming is the go-to for teams that already know Spark batch.
🏗️ Three real decision scenarios — worked examples
Key questions: Does business need data in < 5 min? No — the report is reviewed once per day. Is the source unbounded? No — the day's orders are a bounded dataset once the cutoff time passes. Is latency SLA defined? Yes — 08:00 the next morning.
Analysis: This is a textbook batch problem. A Spark job triggered by Airflow at 02:00, reading from Postgres and the payment gateway API, producing a reconciliation Parquet file, is the right answer. Adding Kafka and Spark Structured Streaming would add cost and complexity with zero business benefit.
Key questions: Does business need data in < 5 min? Yes — 3 seconds. Is source unbounded? Yes — applications arrive continuously. Is latency SLA defined? Yes — 3 s hard SLA.
Analysis: This requires streaming. The 3-second SLA rules out batch entirely. Spark Structured Streaming micro-batch (minimum trigger interval ~1 s) may be acceptable; Kafka Streams or a simple microservice reading directly from Kafka with a pre-built feature store (Redis) is more reliable for a 3 s SLA. The key is pre-materialising the 30-day feature window in a low-latency store — computing it at request time from raw events would be too slow regardless of the processing model.
Key questions: Does business need data in < 5 min? No — hourly is explicitly acceptable. Is source unbounded? The clickstream is unbounded, but we're only looking at the last hour's worth. Is latency SLA defined? Yes — 60-minute freshness window.
Analysis: Hourly batch is the right answer. An Airflow DAG that triggers a Spark job at :55 past each hour, reading the last 65 minutes of events from S3 or a Kafka topic compacted to S3, then writing aggregates to Redshift or BigQuery. The 65-minute read window provides overlap to handle any late-arriving events. Streaming would cost 10x more per month to run a continuously-running Spark cluster for a dashboard refreshed once per hour.
💰 Cost comparison
Streaming infrastructure is fundamentally more expensive than batch because compute resources cannot scale to zero between runs. Below are plausible representative costs for a mid-scale pipeline processing 10 GB/hour of event data.
| Processing model | Compute pattern | Approximate monthly cost | When justified |
|---|---|---|---|
| Nightly batch (Spark on EMR/spot) | 2 hr/day, then shuts down | ~$60–120/month | Always acceptable as a baseline |
| Hourly batch (Spark on EMR/spot) | 15 min/hour, scales down | ~$200–350/month | When stakeholders check dashboards > twice daily |
| Micro-batch (Spark SS, Databricks) | Always-on, small cluster (4 workers) | ~$600–900/month | Latency SLA 30 s – 5 min |
| True streaming (Flink on Kubernetes) | Always-on, tuned cluster | ~$1,200–2,000/month | Latency SLA < 30 s, or complex stateful logic |
A common mistake junior engineers make is streaming everything because it "sounds more impressive" or because Kafka is already in the stack. Streaming a daily report that no one checks until morning costs 10x more than a batch job and provides zero additional value. Always ask: "What decision does this data power, and how quickly does that decision need to be made?" If the answer is "once a day," use batch.
✅ The key question checklist
Before committing to a processing model, answer these questions. The answers drive the decision.
Batch vs Streaming Decision Checklist
YES → Streaming (or micro-batch) NO → Batch
YES → Natural fit for streaming NO → Natural fit for batch
YES → Use the SLA to drive the trigger interval NO → Default to batch until an SLA is defined
YES → Streaming requires watermarks + idempotent sinks NO → Batch re-runs handle this naturally
YES → Streaming is viable; ensure checkpointing and monitoring are in place NO → Start with batch; migrate to streaming after building the expertise
YES → Streaming is approved NO → Use batch or hourly micro-batch to reduce cost
⚠️ Common mistakes
🧠 Quiz — 4 questions
Next: Backfills and Idempotency — learn how to safely reprocess historical data.
🎤 Interview Q&A
✅ Key takeaways
- The primary decision driver is not the data arrival rate but the business decision latency — how quickly does the consumer of the data need to act on it? Default to batch unless a concrete sub-5-minute SLA exists.
- Batch excels at bounded, high-volume historical data with high throughput and low cost; streaming excels at unbounded, continuous data where latency is a genuine business requirement.
- Spark Structured Streaming's micro-batch mode (trigger every 10–60 seconds) hits the sweet spot for most pipelines that need "fast but not millisecond" — it uses the familiar Spark DataFrame API at roughly one-third the infrastructure cost of a true streaming engine.
- The Lambda Architecture (batch layer + speed layer) is usually a trap: it requires two codebases for the same logic. Prefer the Kappa Architecture — one streaming pipeline that can replay Kafka for backfills — unless your historical scale makes Kafka replay economically impractical.
- Streaming everything because it "sounds impressive" is one of the most expensive engineering mistakes. Always ask: "What is the cost of adding 30 minutes of latency?" If the answer is "nothing," use batch.
Build pipelines that run at BoraPay scale
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →