📋 The choice that shapes everything

The single most important architecture decision in a data pipeline is whether to process data in batch or in streams. Get it wrong and you either over-engineer a simple problem with expensive Kafka infrastructure, or you disappoint a business that needed near-real-time answers and got a stale hourly report instead.

This lesson gives you a repeatable decision framework. By the end you will be able to look at any pipeline requirement and choose the right processing model in under five minutes.

Batch processing

  • Processes a fixed, bounded dataset
  • Runs on a schedule (hourly, daily, weekly)
  • High throughput — can read terabytes in a single job
  • Latency measured in minutes to hours
  • Simple to reason about — no state management
  • Cheap at scale — runs overnight on spot instances
Tools: Spark, dbt, Airflow

Stream processing

  • Processes an unbounded, continuous flow of events
  • Runs continuously — always on
  • Lower throughput per cost, but latency in seconds
  • Latency measured in milliseconds to seconds
  • Complex: watermarks, state, checkpoints required
  • More expensive — requires persistent compute
Tools: Spark Structured Streaming, Flink, Kafka Streams

⚖️ Latency vs throughput — the core trade-off

Every data processing decision is, at its heart, a trade-off between latency and throughput. You can have very low latency (process each event the moment it arrives) or very high throughput (process millions of events per second in one big pass) — but not both at the same cost.

Dimension Batch Micro-batch (Spark SS) True streaming (Flink)
Latency Minutes to hours Seconds to ~1 minute Milliseconds to seconds
Throughput Very high (petabyte-scale) High Moderate to high
Infrastructure cost Low — runs on spot instances, scales down to zero Medium — cluster always running Higher — requires persistent, low-GC JVM tuning
Operational complexity Low Medium High
State management None needed RocksDB-backed state store Distributed keyed state
Fault recovery Re-run the job Checkpoint + WAL Savepoints + changelog
Rule of thumb: if your business stakeholder asks for data "as soon as possible" but actually checks the dashboard once an hour, you need batch. Streaming is expensive. Deploy it only when the business makes decisions in real time, not just when data engineers think it sounds cool.

📦 Bounded vs unbounded data

The most fundamental distinction is whether your data source is bounded (finite, has an end) or unbounded (continuous, never ends).

Data type Description Natural processing model Examples
Bounded A fixed dataset that has a beginning and an end Batch — process the whole thing at once Yesterday's sales CSV, a database snapshot, a completed log file
Unbounded A continuous, infinite stream of events with no defined end Stream — process each event or micro-batch as it arrives Kafka topic, clickstream, sensor readings, WebSocket feed
The classic trick: batch reads unbounded data with time bounds

Most "batch" jobs against a Kafka topic or database table are actually reading an unbounded source with an artificial time boundary — "give me events from 00:00 to 23:59 yesterday." This is perfectly fine, but it means your latency is at least one full day. Streaming removes that artificial boundary and processes each event as it arrives.

🎯 Canonical use cases — which model fits?

Use case Processing model Why
Hourly sales dashboard Batch Data is queried once per hour; freshness of 5–10 minutes adds zero business value; batch is cheaper and simpler
Nightly ETL into data warehouse Batch Full historical loads, complex transformations, idempotent re-runs — batch with Airflow is the right tool
Daily analytics report (finance, ops) Batch Report consumed once a day; stakeholders do not need intra-day freshness; simple Spark job with S3 output
Fraud detection scoring Streaming A fraud score 60 minutes after the transaction is worthless; must score within seconds of the event
Real-time chat / content moderation Streaming Harmful content flagged minutes later is already seen; moderation must happen in <1 s from post time
Clickstream sessionisation Micro-batch 30-second windows are acceptable; Spark Structured Streaming micro-batch is simpler than Flink and hits the latency SLA
IoT sensor anomaly detection Streaming Equipment failure must trigger an alert within seconds; batch detects the anomaly an hour after the machine broke
E-commerce order reconciliation (daily) Batch Finance runs reconciliation at EOD; all orders for the day are bounded; a Spark job reading from Postgres is ideal

🧭 The decision flowchart

Run every new pipeline through this flowchart before writing a single line of code. The questions are ordered from most-decisive to least-decisive.

Does the business need results in less than 5 minutes?
↓
No
↓
Use Batch
Schedule with Airflow; Spark or dbt for transforms
Yes
↓
Is latency SLA < 30 seconds?
↓
No (30 s – 5 min)
↓
Use Micro-batch
Spark Structured Streaming; trigger every 30 s
Yes (< 30 s)
↓
Use Streaming
Flink or Kafka Streams for sub-second latency
When Spark Structured Streaming hits the sweet spot

Spark Structured Streaming runs as a micro-batch engine by default. Each micro-batch trigger interval (configurable from 1 second to many minutes) reads available Kafka offsets, processes them as a bounded DataFrame, and commits results. You get most of the streaming benefit — data in seconds, not hours — while keeping the familiar Spark API and the lower operational overhead compared to Flink. The sweet spot is any pipeline with a latency SLA between 10 seconds and 5 minutes.

⚠️ The Lambda Architecture trap

In 2011, Nathan Marz proposed the Lambda Architecture: maintain both a batch layer (accurate historical data) and a speed layer (real-time approximations), with a serving layer that merges both. It was influential — and it is, in most cases, a trap.

The Lambda problem: two codebases for the same logic
Lambda means you write the same transformation twice — once as a Spark batch job and once as a Flink streaming job — and hope they produce identical results. In practice they diverge over time. Bug fixes applied to batch are missed in streaming. Subtle differences in windowing semantics produce inconsistencies. You end up with two separate oncall rotations and a serving layer that must reconcile two potentially contradictory answers.
The modern alternative: Kappa Architecture

The Kappa Architecture (Jay Kreps, 2014) simplifies this: keep only the streaming layer. For backfills and historical reprocessing, replay Kafka topics from the beginning. The same streaming job handles both real-time and historical data. One codebase, one team, one pipeline. If your streaming framework (Spark, Flink, Kafka Streams) is mature enough — and in 2026 they are — you rarely need Lambda.

The exception: if you have a truly massive historical dataset (multi-petabyte) that is expensive to store in Kafka for replay, a hybrid approach with a batch historical layer and a streaming incremental layer is justified. But default to Kappa and only add batch complexity when you have proven you need it.

💻 The same job: batch mode vs streaming mode

Below is a PySpark job that aggregates page views by URL. The batch version reads a static Parquet file. The streaming version reads the same logic from a Kafka topic. The transformation code is nearly identical — only the source and sink differ.

Batch version — reads a bounded Parquet file

pageviews_batch.py Lesson 22 · SparkSession · batch read
batch output (runs once and exits)
Wrote 4,217 URL rows to Parquet. Job complete. Elapsed: 3 min 12 s.

Streaming version — reads from an unbounded Kafka topic

pageviews_streaming.py Lesson 22 · readStream · Kafka source
streaming output (updates every 30 seconds)
------------------------------------------- Batch: 1 Time: 2026-04-27 09:00:30 ------------------------------------------- +------------------------------------------+----------+-----+ |window |url |views| +------------------------------------------+----------+-----+ |{2026-04-27 09:00:00, 2026-04-27 09:01:00}|/checkout | 42 | |{2026-04-27 09:00:00, 2026-04-27 09:01:00}|/home | 118 | |{2026-04-27 09:00:00, 2026-04-27 09:01:00}|/product | 87 | +------------------------------------------+----------+-----+
Notice: the groupBy and count logic is identical in both versions. The only differences are spark.read vs spark.readStream, the Kafka source configuration, the withWatermark call (needed for streaming state management), and the writeStream sink. This is why Spark Structured Streaming is the go-to for teams that already know Spark batch.

🏗️ Three real decision scenarios — worked examples

1
E-commerce order reconciliation
Requirement: Every night, finance needs to reconcile all orders placed during the day against the payment gateway records. The reconciliation report is reviewed at 08:00 the next morning. Data volume: ~200,000 orders per day.

Key questions: Does business need data in < 5 min? No — the report is reviewed once per day. Is the source unbounded? No — the day's orders are a bounded dataset once the cutoff time passes. Is latency SLA defined? Yes — 08:00 the next morning.

Analysis: This is a textbook batch problem. A Spark job triggered by Airflow at 02:00, reading from Postgres and the payment gateway API, producing a reconciliation Parquet file, is the right answer. Adding Kafka and Spark Structured Streaming would add cost and complexity with zero business benefit.
Verdict: Batch (Spark + Airflow, nightly)
2
Real-time credit risk scoring
Requirement: When a customer submits a loan application, the system must return a risk score within 3 seconds. The ML model needs the customer's last 30 days of transaction history as features. A score returned 10 minutes later will lose the customer and violate the product SLA.

Key questions: Does business need data in < 5 min? Yes — 3 seconds. Is source unbounded? Yes — applications arrive continuously. Is latency SLA defined? Yes — 3 s hard SLA.

Analysis: This requires streaming. The 3-second SLA rules out batch entirely. Spark Structured Streaming micro-batch (minimum trigger interval ~1 s) may be acceptable; Kafka Streams or a simple microservice reading directly from Kafka with a pre-built feature store (Redis) is more reliable for a 3 s SLA. The key is pre-materialising the 30-day feature window in a low-latency store — computing it at request time from raw events would be too slow regardless of the processing model.
Verdict: Streaming (Kafka Streams + Redis feature store)
3
Daily business analytics report
Requirement: The marketing team wants a dashboard showing page views, conversion rates, and session counts. The dashboard is refreshed once per hour by the analyst team. Current data freshness tolerance: "within the last hour is fine."

Key questions: Does business need data in < 5 min? No — hourly is explicitly acceptable. Is source unbounded? The clickstream is unbounded, but we're only looking at the last hour's worth. Is latency SLA defined? Yes — 60-minute freshness window.

Analysis: Hourly batch is the right answer. An Airflow DAG that triggers a Spark job at :55 past each hour, reading the last 65 minutes of events from S3 or a Kafka topic compacted to S3, then writing aggregates to Redshift or BigQuery. The 65-minute read window provides overlap to handle any late-arriving events. Streaming would cost 10x more per month to run a continuously-running Spark cluster for a dashboard refreshed once per hour.
Verdict: Batch (hourly Spark job via Airflow)

💰 Cost comparison

Streaming infrastructure is fundamentally more expensive than batch because compute resources cannot scale to zero between runs. Below are plausible representative costs for a mid-scale pipeline processing 10 GB/hour of event data.

Processing model Compute pattern Approximate monthly cost When justified
Nightly batch (Spark on EMR/spot) 2 hr/day, then shuts down ~$60–120/month Always acceptable as a baseline
Hourly batch (Spark on EMR/spot) 15 min/hour, scales down ~$200–350/month When stakeholders check dashboards > twice daily
Micro-batch (Spark SS, Databricks) Always-on, small cluster (4 workers) ~$600–900/month Latency SLA 30 s – 5 min
True streaming (Flink on Kubernetes) Always-on, tuned cluster ~$1,200–2,000/month Latency SLA < 30 s, or complex stateful logic
The streaming-everything anti-pattern

A common mistake junior engineers make is streaming everything because it "sounds more impressive" or because Kafka is already in the stack. Streaming a daily report that no one checks until morning costs 10x more than a batch job and provides zero additional value. Always ask: "What decision does this data power, and how quickly does that decision need to be made?" If the answer is "once a day," use batch.

✅ The key question checklist

Before committing to a processing model, answer these questions. The answers drive the decision.

Batch vs Streaming Decision Checklist

Does the business need data in less than 5 minutes?
YES → Streaming (or micro-batch) NO → Batch
Is the data source unbounded (no defined end)?
YES → Natural fit for streaming NO → Natural fit for batch
Is a latency SLA explicitly defined by the business?
YES → Use the SLA to drive the trigger interval NO → Default to batch until an SLA is defined
Will duplicate or late-arriving events cause significant harm?
YES → Streaming requires watermarks + idempotent sinks NO → Batch re-runs handle this naturally
Does the team have operational experience running always-on streaming jobs?
YES → Streaming is viable; ensure checkpointing and monitoring are in place NO → Start with batch; migrate to streaming after building the expertise
Is the additional monthly cost of streaming justified by the business value?
YES → Streaming is approved NO → Use batch or hourly micro-batch to reduce cost

⚠️ Common mistakes

Streaming everything because it sounds cool
The most expensive data engineering mistake. A report read once per day does not need a Kafka cluster and a continuously running Spark driver. Streaming everything inflates infrastructure costs, adds operational complexity, and increases on-call burden — with no business benefit. Start with batch; upgrade when a real latency SLA demands it.
Building Lambda Architecture instead of Kappa
Maintaining two separate pipelines (batch for accuracy, streaming for speed) means two codebases, two sets of bugs, and a serving layer that must merge two potentially inconsistent answers. In 2026, Spark Structured Streaming and Flink can replay from Kafka for backfills. You almost never need a separate batch layer.
Ignoring micro-batch as the middle path
Engineers often see "batch vs streaming" as a binary choice. Spark Structured Streaming micro-batch provides latency in the range of 10 seconds to 2 minutes with the familiar Spark DataFrame API and no Flink operational overhead. Many pipelines that claim to "need streaming" actually need micro-batch — which is 3x cheaper and 5x simpler to operate.

🧠 Quiz — 4 questions

Question 1 of 4
A fraud detection system must score a transaction within 3 seconds of it occurring. Which processing model is most appropriate?
Question 2 of 4
What is the primary advantage of batch processing over streaming for large historical datasets?
Question 3 of 4
Which statement best describes the Lambda Architecture's main drawback?
Question 4 of 4
A marketing dashboard is refreshed once per hour and stakeholders view it once in the morning. The data volume is 5 GB/day. Which is the most cost-effective architecture?
0
/ 4
Decision framework mastered.

Next: Backfills and Idempotency — learn how to safely reprocess historical data.

🎤 Interview Q&A

Medium When would you choose batch over streaming, even if your data arrives continuously?
+
I would choose batch over streaming when the business's consumption pattern does not require low latency. The key question is not "how often does the data arrive?" but "how often does the business make a decision based on this data?" If a dashboard is checked once per hour, or a report is reviewed once per day, streaming is wasteful — it keeps compute running around the clock to produce answers that will not be consumed for hours. Batch also has a significant cost advantage: Spark batch jobs on spot instances can process terabytes of data for a fraction of the cost of an always-on streaming cluster. I default to batch unless a stakeholder provides a concrete latency SLA below five minutes, the source is truly unbounded with no natural time boundary, or a real-time reaction is required (fraud, moderation, alerting).
Hard Explain the Lambda Architecture and why many teams are moving away from it.
+
Lambda Architecture, proposed by Nathan Marz, combines three layers: a batch layer that processes historical data for accuracy, a speed layer that processes recent data for low latency, and a serving layer that merges both views for queries. The idea was sound in 2011 when streaming frameworks were immature and could not produce accurate results — you needed the batch layer to "correct" the approximations from the speed layer. The problem is that maintaining two codebases for the same business logic is extremely expensive in engineering time. The batch and streaming jobs express the same transformations in different frameworks (say, PySpark batch and Flink) and diverge over time — bug fixes, new business rules, and schema changes must be applied twice, and subtle differences in windowing semantics cause the two layers to produce different answers. Modern teams prefer the Kappa Architecture: a single streaming pipeline that handles both real-time processing and historical backfills by replaying Kafka topics from offset 0. Tools like Spark Structured Streaming and Flink support this pattern well. Lambda is still justified when the historical dataset is so large it cannot be economically stored in Kafka for replay (multi-petabyte scale) and the cost of separate batch processing is lower than expanded Kafka storage — but this is the exception, not the rule.
Easy What is "bounded" vs "unbounded" data and why does it matter for pipeline design?
+
Bounded data is a finite dataset with a defined beginning and end — yesterday's sales records, a completed log file, a database snapshot exported to S3. Unbounded data is a continuous, infinite stream of events with no defined end — a Kafka topic receiving events as long as users interact with the application, a WebSocket feed, a sensor publishing temperature every second indefinitely. The distinction matters because bounded data is naturally processed by batch engines: you can read the whole thing, apply your transformation, and exit. Unbounded data cannot be fully read — you must decide when to trigger computation, how to handle late-arriving events, and how to manage the state that accumulates over time. These concerns — watermarks, trigger intervals, checkpointing, state TTLs — are unique to stream processing. Many batch pipelines read unbounded sources (like a Kafka topic) by imposing artificial time bounds ("give me everything from midnight to midnight"), which is a valid pattern but means your freshness is bounded by the batch interval, not by the event arrival time.

✅ Key takeaways

  • The primary decision driver is not the data arrival rate but the business decision latency — how quickly does the consumer of the data need to act on it? Default to batch unless a concrete sub-5-minute SLA exists.
  • Batch excels at bounded, high-volume historical data with high throughput and low cost; streaming excels at unbounded, continuous data where latency is a genuine business requirement.
  • Spark Structured Streaming's micro-batch mode (trigger every 10–60 seconds) hits the sweet spot for most pipelines that need "fast but not millisecond" — it uses the familiar Spark DataFrame API at roughly one-third the infrastructure cost of a true streaming engine.
  • The Lambda Architecture (batch layer + speed layer) is usually a trap: it requires two codebases for the same logic. Prefer the Kappa Architecture — one streaming pipeline that can replay Kafka for backfills — unless your historical scale makes Kafka replay economically impractical.
  • Streaming everything because it "sounds impressive" is one of the most expensive engineering mistakes. Always ask: "What is the cost of adding 30 minutes of latency?" If the answer is "nothing," use batch.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Build pipelines that run at BoraPay scale

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →