🤔 You probably don't need Spark

Here is the most valuable thing this track will teach you, and it is in Lesson 01 for a reason: most data problems don't need Spark or Kafka.

The single biggest mistake data teams make is reaching for distributed compute when a well-indexed Postgres table on an appropriately sized server would have solved the problem in a weekend — without the three engineers-months of cluster ops, Kafka tuning, and YAML sprawl that Spark pipelines demand.

The honest framing of this track

This track teaches you Spark and Kafka because they are genuinely essential at scale. But it also teaches you exactly when that scale threshold is crossed — and when you are nowhere near it yet. The engineers who understand when not to use Spark are the ones who build simple systems that work, rather than complex systems that impress nobody.

In this lesson you will learn:

  • ▸ The Rule of 100 GB — which tool to use at which data scale
  • ▸ The 5 questions to ask before choosing any Big Data tool
  • ▸ When Postgres wins vs when Spark is genuinely the right answer
  • ▸ Three genuine Kafka use cases — and three signs that Kafka is overkill

📏 The Rule of 100 GB

Data scale dictates tool choice more than anything else. This single heuristic will save your team months of unnecessary infrastructure work:

< 1 GB 🐼 Pandas on your laptop. Done.
1 GB – 100 GB 🦆 DuckDB or Postgres with proper indexing. Fast, simple, cheap.
100 GB – 10 TB 🐘 PySpark on a managed cluster. Worth the ops complexity.
> 10 TB ❄️ Cloud warehouse (Snowflake / BigQuery) or Iceberg + Spark.
Continuous events 🌊 Kafka + Spark Structured Streaming. Sub-second latency.
Reality check: Most Indian startups and mid-size product companies sit firmly in the 1–100 GB zone. The majority of data teams in IT services are solving batch problems on data that fits comfortably in DuckDB. If you are not sure where you sit, measure first.

Tool selection table

Scale Tool Reason to pick Reason to skip
< 1 GB Pandas Fast iteration, zero setup, every data scientist knows it Crashes on 10 M+ rows in memory; no query optimizer
1 – 100 GB DuckDB + Postgres SQL you already know, zero infra, DuckDB reads Parquet directly Not distributed; single node — scale-up only, not scale-out
100 GB – 10 TB PySpark (EMR, Dataproc, HDInsight) Horizontal scaling, handles data that overflows any single machine Cluster ops overhead; slow startup; overkill for anything smaller
> 10 TB Cloud warehouse (Snowflake / BigQuery) Auto-scale, separation of compute and storage, managed infra Cost spikes fast; vendor lock-in is real
Continuous events Kafka + Spark Streaming Sub-second latency, decoupled producers and consumers, replay built in Complex state management; significant ops burden; overkill for batch

🔎 The 5 questions before reaching for Big Data

Before your team spins up a Kafka cluster or migrates to Spark, every data engineer should work through these five questions in order. If you can answer "no" to any one of them, there is a simpler solution available.

  • 1
    Does the data fit on a single machine?
    If yes → Postgres or DuckDB first. A 32 GB r6g.2xlarge handles 500 GB of data with proper indexing. Buy a bigger VM before building a cluster.
  • 2
    Does speed matter more than simplicity?
    If no → batch overnight. A nightly Postgres aggregation that runs in 20 minutes is far simpler than a real-time Spark pipeline that runs in 20 seconds but requires 3 engineers to maintain.
  • 3
    Is the data growing faster than your hardware can grow?
    If no → buy a bigger VM. Cloud VMs now go up to 192 vCPUs and 1.5 TB of RAM. Scaling up is almost always faster and cheaper than migrating to Spark — until you hit the ceiling.
  • 4
    Do multiple teams or services need to produce or consume the same events in real time?
    If no → no Kafka. Kafka's value is in decoupling multiple independent producers from multiple independent consumers. A single producer writing to a single consumer is just a database queue in disguise.
  • 5
    Is the existing SQL tool actually slow? Have you tried indexing?
    If you haven't added the right indexes yet → add indexes first. A missing index on a 50 M-row table turns a 2-second query into a 4-minute one. Adding the index takes 10 minutes and costs nothing.
Interview reality

Indian product-company data engineering interviews frequently include a scenario question: "Your team's Postgres queries are timing out on a 200 GB table. What do you do?" The expected answer is not "migrate to Spark" — it is: check the query plan, identify missing indexes, consider partitioning by date, and only then evaluate whether the data volume is the actual bottleneck.

🏦 Scenario: SouthBank India

🏦
Real-world scenario
Arjun prevents 6 months of unnecessary Kafka ops at SouthBank India

Arjun is a data engineer at SouthBank India in Mumbai. His team is tasked with building a "real-time fraud alert" system to flag suspicious activity across 2 million daily transactions. Management has proposed a Kafka + Spark Streaming architecture after reading a case study from a large American payments company.

Before Arjun's team spins up a Kafka cluster, he works through the 5 questions. The math is straightforward:

2,000,000 transactions/day × 200 bytes each = 400 MB/day
400 MB/day × 30 days = 12 GB/month

12 GB per month is well within the Postgres + DuckDB tier. A single CREATE INDEX ON transactions(account_id, created_at) brings query time from minutes down to milliseconds. Kafka would add 3 engineer-months of ops for zero performance gain at this scale. The fraud detection model runs as a nightly batch job on Postgres — simple, auditable, and easy to debug.

Result: Arjun saves the team 6 months of Kafka cluster ops. The system ships in 3 weeks instead of 4 months.

The lesson is not "never use Kafka." It is "match the tool to the actual scale." If SouthBank India grows to 200 million transactions per day — that is 40 GB per day, 1.2 TB per month — then Kafka becomes justified. Build for your current scale; migrate when you actually hit the limit.

🗄️ When Postgres wins vs when Spark wins

Postgres is not "small data only." A properly indexed Postgres instance on a large cloud VM handles hundreds of gigabytes of transactional data with excellent performance. The following table shows where the boundary actually lies:

Scenario Postgres wins Spark wins
Query 100K rows for a dashboard ✓ Postgres (milliseconds with index) Spark (way too much overhead — cluster cold start alone is 30+ seconds)
Nightly rollup of 1 GB of logs ✓ Postgres with a date-range index (runs in minutes) Spark (overkill — cluster startup costs more time than the query saves)
Daily rollup of 500 GB of events Postgres (too slow — full table scans on 500 GB are painful even with indexes) ✓ Spark (horizontal partitioning distributes the work)
Real-time fraud check on a single transaction ✓ Postgres + Redis (point lookup in <1ms) Spark Streaming (acceptable, but far more complex; Spark doesn't do point lookups)
ML feature engineering on 10 TB of data Postgres (impossible — data doesn't fit; joins would take days) ✓ Spark (partitioned reads across a cluster; each executor works independently)
Ad-hoc exploratory SQL query on 5 GB Parquet file ✓ DuckDB (reads Parquet directly, no import step, columnar execution, results in seconds) Spark (slower to start than to query at this scale; DuckDB wins easily)

The pattern is clear: Spark's value kicks in when the data physically cannot be processed by a single machine in a reasonable time window. Below that threshold, Spark's cluster startup cost and shuffle overhead make it consistently slower than a well-tuned single-node database.

🌊 When Kafka is justified — and when it's overkill

Kafka is one of the most over-used tools in the data engineering ecosystem. It is extraordinary when you genuinely need it, and expensive dead weight when you don't.

Kafka is justified when...

  • 10,000+ events/second from multiple producers. JengaTel's CDR records arrive at 460/sec on average and 2,000/sec at peak across 200 towers — that is a genuine streaming problem.
  • Multiple independent consumers need the same event stream. A fraud team, a BI team, and a regulatory reporting team all need the same payment events but process them differently. Kafka's consumer groups decouple them cleanly.
  • Events must be replayed or audited. Kafka's log retention is an audit trail by design. If you need to reprocess 30 days of events after fixing a bug in your pipeline, Kafka lets you seek back to any offset.

Kafka is overkill when...

  • Single producer, single consumer. You are building a message queue, not an event bus. A Postgres table with a processed flag, or a Redis List, handles this with zero ops burden.
  • Under 100 events/second. A Postgres LISTEN/NOTIFY or Redis Pub/Sub handles this trivially. Kafka's minimum footprint (brokers, ZooKeeper historically, or KRaft coordinators, topic management, consumer group coordination) is massive relative to the throughput.
  • You don't have ops capacity. Kafka requires continuous tuning — partition counts, replication factors, consumer lag monitoring, disk management. An understaffed team with a Kafka cluster is a production incident waiting to happen.
Rule of thumb: If you can draw your pipeline as "one service sends data to one other service," you don't need Kafka. If you need to draw a fan-out diagram with multiple consumers each doing different things with the same events, Kafka starts to make sense.

🗺️ Where each tool fits in the pipeline

The modern data stack is a pipeline from sources to serving. Each tool occupies a distinct position. Understanding where a tool lives helps you decide whether it belongs in your architecture at all.

Source Postgres / Files / APIs Compute DuckDB or PySpark Storage Parquet / Iceberg Serve DuckDB / Dashboard

Kafka lives between Source and Compute — but only when events are continuous and multiple consumers need them

Data > 100 GB? No Yes DuckDB or Postgres + add indexes first Continuous events? (streaming needed?) No (batch) Yes PySpark on cluster EMR / Dataproc / HDInsight Kafka + Spark Structured Streaming Start with the simplest option. Migrate only when you actually hit the limit — not before.

💪 Mini challenge — 3 decision exercises

For each scenario below, choose the right tool and write one sentence justifying your choice. Expand the hint when you're ready to check your reasoning.

Tool selection decisions
Scenario 1

A fintech startup has 500,000 users averaging 1 transaction per user per day. The data team has 2 engineers. They want to build a nightly dashboard showing daily spend by category across all users. Which tool for the nightly analytics pipeline, and why?

Show answer hint
Postgres or DuckDB. 500K transactions/day × ~200 bytes = 100 MB/day. This is solidly in the 1–100 GB zone. With a date + category index on Postgres, nightly aggregations run in seconds. A 2-person team has zero capacity to operate Spark clusters. DuckDB on Parquet exports is an equally valid choice and adds analytical speed without infrastructure complexity.
Scenario 2

A telecom has 50 million CDR (Call Detail Records) per day from 200 towers. They need a fraud detection system that must flag suspicious patterns within 5 seconds of a call ending. The fraud patterns require comparing a caller's recent history against population-level baselines. Which architecture?

Show answer hint
Kafka + Spark Structured Streaming. 50M records/day = 578/sec average, likely 2,000+/sec at peak. This crosses the 10,000 events/second threshold at peak. The sub-5-second requirement demands streaming, not batch. Multiple consumers (fraud detection, BI, regulatory) justify Kafka's fan-out architecture. This is a genuine Kafka + Spark use case.
Scenario 3

A retail analytics team has a 2 TB Parquet data lake containing 3 years of order history. A BI analyst needs to run ad-hoc SQL queries — things like "revenue by region by week for the last 6 months" or "top 100 products by return rate." There is no engineering support available to manage infrastructure. Which tool?

Show answer hint
DuckDB directly on Parquet. DuckDB can query Parquet files with standard SQL without any import step. Its columnar execution means it only reads the columns each query needs, making 2 TB queries practical on a laptop or a single large VM. No infrastructure to manage, no cluster to spin up — the BI analyst installs DuckDB and queries the Parquet files directly. This is DuckDB's sweet spot.

🎤 Interview Q&A

These questions appear regularly in data engineering interviews at Indian product companies. Click each one to see the answer.

Easy What is the "Rule of 100 GB" in data engineering?
+

The Rule of 100 GB is a practical heuristic for tool selection: data under 100 GB should be handled with DuckDB or Postgres (possibly with an upgraded VM and proper indexing); data between 100 GB and 10 TB is where PySpark on a managed cluster starts to earn its complexity; above 10 TB or for continuous event streams, cloud warehouses or Kafka + Spark Streaming become justified. The rule prevents teams from over-engineering solutions for data that a single well-tuned machine can handle.

Easy Name two situations where adding a Postgres index is better than migrating to Spark.
+

First: when slow queries are caused by full table scans on a column that is frequently filtered — a B-tree index on that column reduces the query from minutes to milliseconds at no infrastructure cost. Second: when the data volume is under 100 GB — Spark's cluster startup overhead and shuffle cost make it consistently slower than an indexed Postgres table at this scale. Adding an index takes 10 minutes; migrating to Spark takes months.

Medium Why is Kafka overkill for a single-producer, single-consumer pipeline?
+

Kafka's architectural value comes from fan-out: one event stream consumed independently by multiple consumer groups, with log retention for replay. When there is only one producer and one consumer, the entire fan-out architecture is unused. You pay for Kafka's ops complexity — broker management, partition tuning, consumer offset tracking, monitoring consumer lag — without gaining the decoupling benefits. A Postgres table with a processed status column, a Redis List, or even a simple REST webhook achieves the same outcome with a fraction of the operational burden.

Medium Arjun's SouthBank team now has 3 TB of daily transaction logs. Their Postgres queries are timing out. What are the two options to try before choosing Spark?
+

Option 1: Table partitioning by date. Postgres native partitioning splits the 3 TB table into daily or monthly partitions. Queries filtered by date range only scan the relevant partition — a query for "last 7 days" reads 7 partitions instead of the entire 3 TB. Combined with partition-local indexes, this can recover query performance dramatically. Option 2: Move to Columnar storage or DuckDB. Export the log data to Parquet files and query with DuckDB. DuckDB's columnar execution skips irrelevant columns entirely, making aggregation queries on 3 TB Parquet dramatically faster than row-oriented Postgres scans. Only if both options prove insufficient at the actual query latency targets should the team consider PySpark — at 3 TB per day, Spark may well be justified, but the simpler paths should be validated first.

Hard Explain how Spark's shuffle cost makes it slower than Postgres for queries on less than 1 GB of data.
+

Spark's execution model assumes data is too large to fit in memory on a single node, so it distributes work across a cluster. When you run a groupBy or join on data that spans multiple partitions, Spark must perform a shuffle — each executor sorts its local data by the grouping key, writes it to disk, and then sends it across the network to the executor responsible for that key's bucket. This involves disk I/O, network transfer, serialization, deserialization, and a second sort on the receiving side. On 1 GB of data, the shuffle easily takes longer than the entire Postgres query on an indexed table. Additionally, Spark's SparkSession startup — launching the JVM, negotiating resources with the cluster manager, loading configuration — adds 15–30 seconds of overhead that a Postgres connection does not have. For small data, Spark's distributed overhead dominates the actual compute time.

🧠 Quiz — 5 questions

Question 1 of 5
Which tool handles 5 GB of Parquet data for ad-hoc SQL queries with no setup or infrastructure?
Question 2 of 5
At what data scale does PySpark typically start to outperform Postgres for aggregation workloads?
Question 3 of 5
What is the primary reason NOT to use Kafka for a single producer → single consumer pipeline?
Question 4 of 5
SouthBank India processes 2 million transactions per day at 200 bytes each. What is the approximate daily data volume?
Question 5 of 5
DuckDB is described as "fast local analytics." What is its key architectural advantage over Pandas for analytical queries?
0 / 5
Good work on the decision framework!

The skill of choosing the right tool for the scale is what separates strong data engineers from those who over-engineer everything.

⚠️ Common mistakes

"Choosing Spark because the data might grow"
Premature scale is the most expensive kind. Build for your current data volume; migrate when you actually hit the limit. Spark adds months of engineering work and ongoing ops cost — pay that cost when you need it, not before. Data that "might grow" to 100 GB rarely does so on schedule.
"Running Kafka for under 100 events/second"
A Redis Pub/Sub, a Postgres LISTEN/NOTIFY, or even a REST webhook handles this trivially. Kafka's minimum viable deployment involves multiple brokers, partition management, consumer group coordination, and continuous lag monitoring. The ops cost is enormous relative to what you get at low throughput.
"Treating Postgres as 'small data only'"
A properly indexed Postgres on an r6g.2xlarge (64 vCPUs, 512 GB RAM) handles 500 GB tables with excellent performance. Before migrating to Spark, always profile the slow queries, add missing indexes, consider table partitioning by date, and upgrade the VM. Spark should be a last resort — not a first response to a slow query.
"Conflating Big Data tools with production-readiness"
A Kafka cluster managed by three engineers who don't know Kafka is far riskier than a Postgres solution your whole team understands. Tool familiarity is a legitimate engineering constraint. A simple system your team can debug at 2 AM is always more production-ready than a complex system nobody fully understands.

✅ Key takeaways

  • The Rule of 100 GB: under 100 GB, reach for DuckDB or Postgres first; above 100 GB, consider PySpark; above 10 TB or for continuous streams, cloud warehouse or Kafka + Spark Streaming.
  • Always work through the 5 questions before choosing a Big Data tool — especially "have you tried indexing?" Missing indexes are the most common cause of avoidable Spark migrations.
  • Kafka is justified only when: you have 10,000+ events/second, multiple independent consumers need the same stream, or events must be replayed or audited. Anything less, and simpler tools cost a fraction of the ops burden.
  • Adding a Postgres index or upgrading to a larger VM is often 10 times faster to implement than migrating to Spark — and the business benefit is identical if the data fits.
  • This track teaches Spark and Kafka because they are essential at genuine scale — not because they are always the right tool. The engineers who know when not to use them are the ones who ship reliable systems.