🤔 You probably don't need Spark
Here is the most valuable thing this track will teach you, and it is in Lesson 01 for a reason: most data problems don't need Spark or Kafka.
The single biggest mistake data teams make is reaching for distributed compute when a well-indexed Postgres table on an appropriately sized server would have solved the problem in a weekend — without the three engineers-months of cluster ops, Kafka tuning, and YAML sprawl that Spark pipelines demand.
This track teaches you Spark and Kafka because they are genuinely essential at scale. But it also teaches you exactly when that scale threshold is crossed — and when you are nowhere near it yet. The engineers who understand when not to use Spark are the ones who build simple systems that work, rather than complex systems that impress nobody.
In this lesson you will learn:
- ▸ The Rule of 100 GB — which tool to use at which data scale
- ▸ The 5 questions to ask before choosing any Big Data tool
- ▸ When Postgres wins vs when Spark is genuinely the right answer
- ▸ Three genuine Kafka use cases — and three signs that Kafka is overkill
📏 The Rule of 100 GB
Data scale dictates tool choice more than anything else. This single heuristic will save your team months of unnecessary infrastructure work:
Tool selection table
| Scale | Tool | Reason to pick | Reason to skip |
|---|---|---|---|
< 1 GB |
Pandas | Fast iteration, zero setup, every data scientist knows it | Crashes on 10 M+ rows in memory; no query optimizer |
1 – 100 GB |
DuckDB + Postgres | SQL you already know, zero infra, DuckDB reads Parquet directly | Not distributed; single node — scale-up only, not scale-out |
100 GB – 10 TB |
PySpark (EMR, Dataproc, HDInsight) | Horizontal scaling, handles data that overflows any single machine | Cluster ops overhead; slow startup; overkill for anything smaller |
> 10 TB |
Cloud warehouse (Snowflake / BigQuery) | Auto-scale, separation of compute and storage, managed infra | Cost spikes fast; vendor lock-in is real |
Continuous events |
Kafka + Spark Streaming | Sub-second latency, decoupled producers and consumers, replay built in | Complex state management; significant ops burden; overkill for batch |
🔎 The 5 questions before reaching for Big Data
Before your team spins up a Kafka cluster or migrates to Spark, every data engineer should work through these five questions in order. If you can answer "no" to any one of them, there is a simpler solution available.
-
1
Does the data fit on a single machine?If yes → Postgres or DuckDB first. A 32 GB r6g.2xlarge handles 500 GB of data with proper indexing. Buy a bigger VM before building a cluster.
-
2
Does speed matter more than simplicity?If no → batch overnight. A nightly Postgres aggregation that runs in 20 minutes is far simpler than a real-time Spark pipeline that runs in 20 seconds but requires 3 engineers to maintain.
-
3
Is the data growing faster than your hardware can grow?If no → buy a bigger VM. Cloud VMs now go up to 192 vCPUs and 1.5 TB of RAM. Scaling up is almost always faster and cheaper than migrating to Spark — until you hit the ceiling.
-
4
Do multiple teams or services need to produce or consume the same events in real time?If no → no Kafka. Kafka's value is in decoupling multiple independent producers from multiple independent consumers. A single producer writing to a single consumer is just a database queue in disguise.
-
5
Is the existing SQL tool actually slow? Have you tried indexing?If you haven't added the right indexes yet → add indexes first. A missing index on a 50 M-row table turns a 2-second query into a 4-minute one. Adding the index takes 10 minutes and costs nothing.
Indian product-company data engineering interviews frequently include a scenario question: "Your team's Postgres queries are timing out on a 200 GB table. What do you do?" The expected answer is not "migrate to Spark" — it is: check the query plan, identify missing indexes, consider partitioning by date, and only then evaluate whether the data volume is the actual bottleneck.
🏦 Scenario: SouthBank India
Arjun is a data engineer at SouthBank India in Mumbai. His team is tasked with building a "real-time fraud alert" system to flag suspicious activity across 2 million daily transactions. Management has proposed a Kafka + Spark Streaming architecture after reading a case study from a large American payments company.
Before Arjun's team spins up a Kafka cluster, he works through the 5 questions. The math is straightforward:
400 MB/day × 30 days = 12 GB/month
12 GB per month is well within the Postgres + DuckDB tier. A single CREATE INDEX ON transactions(account_id, created_at) brings query time from minutes down to milliseconds. Kafka would add 3 engineer-months of ops for zero performance gain at this scale. The fraud detection model runs as a nightly batch job on Postgres — simple, auditable, and easy to debug.
Result: Arjun saves the team 6 months of Kafka cluster ops. The system ships in 3 weeks instead of 4 months.
The lesson is not "never use Kafka." It is "match the tool to the actual scale." If SouthBank India grows to 200 million transactions per day — that is 40 GB per day, 1.2 TB per month — then Kafka becomes justified. Build for your current scale; migrate when you actually hit the limit.
🗄️ When Postgres wins vs when Spark wins
Postgres is not "small data only." A properly indexed Postgres instance on a large cloud VM handles hundreds of gigabytes of transactional data with excellent performance. The following table shows where the boundary actually lies:
| Scenario | Postgres wins | Spark wins |
|---|---|---|
| Query 100K rows for a dashboard | ✓ Postgres (milliseconds with index) | Spark (way too much overhead — cluster cold start alone is 30+ seconds) |
| Nightly rollup of 1 GB of logs | ✓ Postgres with a date-range index (runs in minutes) | Spark (overkill — cluster startup costs more time than the query saves) |
| Daily rollup of 500 GB of events | Postgres (too slow — full table scans on 500 GB are painful even with indexes) | ✓ Spark (horizontal partitioning distributes the work) |
| Real-time fraud check on a single transaction | ✓ Postgres + Redis (point lookup in <1ms) | Spark Streaming (acceptable, but far more complex; Spark doesn't do point lookups) |
| ML feature engineering on 10 TB of data | Postgres (impossible — data doesn't fit; joins would take days) | ✓ Spark (partitioned reads across a cluster; each executor works independently) |
| Ad-hoc exploratory SQL query on 5 GB Parquet file | ✓ DuckDB (reads Parquet directly, no import step, columnar execution, results in seconds) | Spark (slower to start than to query at this scale; DuckDB wins easily) |
The pattern is clear: Spark's value kicks in when the data physically cannot be processed by a single machine in a reasonable time window. Below that threshold, Spark's cluster startup cost and shuffle overhead make it consistently slower than a well-tuned single-node database.
🌊 When Kafka is justified — and when it's overkill
Kafka is one of the most over-used tools in the data engineering ecosystem. It is extraordinary when you genuinely need it, and expensive dead weight when you don't.
Kafka is justified when...
- 10,000+ events/second from multiple producers. JengaTel's CDR records arrive at 460/sec on average and 2,000/sec at peak across 200 towers — that is a genuine streaming problem.
- Multiple independent consumers need the same event stream. A fraud team, a BI team, and a regulatory reporting team all need the same payment events but process them differently. Kafka's consumer groups decouple them cleanly.
- Events must be replayed or audited. Kafka's log retention is an audit trail by design. If you need to reprocess 30 days of events after fixing a bug in your pipeline, Kafka lets you seek back to any offset.
Kafka is overkill when...
- Single producer, single consumer. You are building a message queue, not an event bus. A Postgres table with a
processedflag, or a Redis List, handles this with zero ops burden. - Under 100 events/second. A Postgres
LISTEN/NOTIFYor Redis Pub/Sub handles this trivially. Kafka's minimum footprint (brokers, ZooKeeper historically, or KRaft coordinators, topic management, consumer group coordination) is massive relative to the throughput. - You don't have ops capacity. Kafka requires continuous tuning — partition counts, replication factors, consumer lag monitoring, disk management. An understaffed team with a Kafka cluster is a production incident waiting to happen.
🗺️ Where each tool fits in the pipeline
The modern data stack is a pipeline from sources to serving. Each tool occupies a distinct position. Understanding where a tool lives helps you decide whether it belongs in your architecture at all.
Kafka lives between Source and Compute — but only when events are continuous and multiple consumers need them
💪 Mini challenge — 3 decision exercises
For each scenario below, choose the right tool and write one sentence justifying your choice. Expand the hint when you're ready to check your reasoning.
A fintech startup has 500,000 users averaging 1 transaction per user per day. The data team has 2 engineers. They want to build a nightly dashboard showing daily spend by category across all users. Which tool for the nightly analytics pipeline, and why?
Show answer hint
A telecom has 50 million CDR (Call Detail Records) per day from 200 towers. They need a fraud detection system that must flag suspicious patterns within 5 seconds of a call ending. The fraud patterns require comparing a caller's recent history against population-level baselines. Which architecture?
Show answer hint
A retail analytics team has a 2 TB Parquet data lake containing 3 years of order history. A BI analyst needs to run ad-hoc SQL queries — things like "revenue by region by week for the last 6 months" or "top 100 products by return rate." There is no engineering support available to manage infrastructure. Which tool?
Show answer hint
🎤 Interview Q&A
These questions appear regularly in data engineering interviews at Indian product companies. Click each one to see the answer.
The Rule of 100 GB is a practical heuristic for tool selection: data under 100 GB should be handled with DuckDB or Postgres (possibly with an upgraded VM and proper indexing); data between 100 GB and 10 TB is where PySpark on a managed cluster starts to earn its complexity; above 10 TB or for continuous event streams, cloud warehouses or Kafka + Spark Streaming become justified. The rule prevents teams from over-engineering solutions for data that a single well-tuned machine can handle.
First: when slow queries are caused by full table scans on a column that is frequently filtered — a B-tree index on that column reduces the query from minutes to milliseconds at no infrastructure cost. Second: when the data volume is under 100 GB — Spark's cluster startup overhead and shuffle cost make it consistently slower than an indexed Postgres table at this scale. Adding an index takes 10 minutes; migrating to Spark takes months.
Kafka's architectural value comes from fan-out: one event stream consumed independently by multiple consumer groups, with log retention for replay. When there is only one producer and one consumer, the entire fan-out architecture is unused. You pay for Kafka's ops complexity — broker management, partition tuning, consumer offset tracking, monitoring consumer lag — without gaining the decoupling benefits. A Postgres table with a processed status column, a Redis List, or even a simple REST webhook achieves the same outcome with a fraction of the operational burden.
Option 1: Table partitioning by date. Postgres native partitioning splits the 3 TB table into daily or monthly partitions. Queries filtered by date range only scan the relevant partition — a query for "last 7 days" reads 7 partitions instead of the entire 3 TB. Combined with partition-local indexes, this can recover query performance dramatically. Option 2: Move to Columnar storage or DuckDB. Export the log data to Parquet files and query with DuckDB. DuckDB's columnar execution skips irrelevant columns entirely, making aggregation queries on 3 TB Parquet dramatically faster than row-oriented Postgres scans. Only if both options prove insufficient at the actual query latency targets should the team consider PySpark — at 3 TB per day, Spark may well be justified, but the simpler paths should be validated first.
Spark's execution model assumes data is too large to fit in memory on a single node, so it distributes work across a cluster. When you run a groupBy or join on data that spans multiple partitions, Spark must perform a shuffle — each executor sorts its local data by the grouping key, writes it to disk, and then sends it across the network to the executor responsible for that key's bucket. This involves disk I/O, network transfer, serialization, deserialization, and a second sort on the receiving side. On 1 GB of data, the shuffle easily takes longer than the entire Postgres query on an indexed table. Additionally, Spark's SparkSession startup — launching the JVM, negotiating resources with the cluster manager, loading configuration — adds 15–30 seconds of overhead that a Postgres connection does not have. For small data, Spark's distributed overhead dominates the actual compute time.
🧠 Quiz — 5 questions
The skill of choosing the right tool for the scale is what separates strong data engineers from those who over-engineer everything.
⚠️ Common mistakes
✅ Key takeaways
- The Rule of 100 GB: under 100 GB, reach for DuckDB or Postgres first; above 100 GB, consider PySpark; above 10 TB or for continuous streams, cloud warehouse or Kafka + Spark Streaming.
- Always work through the 5 questions before choosing a Big Data tool — especially "have you tried indexing?" Missing indexes are the most common cause of avoidable Spark migrations.
- Kafka is justified only when: you have 10,000+ events/second, multiple independent consumers need the same stream, or events must be replayed or audited. Anything less, and simpler tools cost a fraction of the ops burden.
- Adding a Postgres index or upgrading to a larger VM is often 10 times faster to implement than migrating to Spark — and the business benefit is identical if the data fits.
- This track teaches Spark and Kafka because they are essential at genuine scale — not because they are always the right tool. The engineers who know when not to use them are the ones who ship reliable systems.