🗺️ What this lesson covers
Tunde joined NaijaCart's data engineering team six months ago. His first week he noticed that a nightly pipeline that analysed 500,000 orders was taking 40 minutes. After applying the three techniques in this lesson, it ran in under 4 minutes. No infrastructure change — just smarter use of Spark's built-in levers.
This lesson walks through those same three levers with verified benchmarks you can run on your local machine. By the end you will know exactly when and why to use each one.
NaijaCart is a Nigerian e-commerce platform. Engineers Tunde and Funmi work on the data platform team. The dataset in this lesson: 500,000 synthetic order records spread across 5 cities (Lagos, Abuja, Kano, Port Harcourt, Ibadan) and 5 product categories.
💾 Caching — avoid re-scanning
Spark is lazy: every time you call an action (count(), collect(), write()), it replays the full computation from scratch — including reading source data and applying all transformations. If you use the same DataFrame in two or more actions, you pay that cost twice.
cache() breaks that pattern. After the first action materialises the result into executor memory, every subsequent action reads from the in-memory snapshot instead of re-scanning the source.
The benchmark below runs two actions on a filtered view of 500,000 NaijaCart orders — once without caching, once with.
Without caching, Spark scans all 500,000 rows twice — once per action. With caching, the filtered result (orders where amount_ngn > 50,000 NGN) is stored in executor memory after the first count(). Both subsequent actions read from that in-memory snapshot. Total time drops from 1.93s to 0.25s — a 7.7× speedup for just two actions. With 10 or 20 downstream actions on the same filtered set, the multiplier is proportionally larger.
cache() is lazy — calling it does not immediately store anything. The first action that runs against the cached DataFrame triggers materialisation. Every subsequent action on that same DataFrame object reads from the stored result.
📋 cache() vs persist() — storage levels
cache() is shorthand for persist(StorageLevel.MEMORY_AND_DISK). When you need finer control — for example, to trade RAM for reliability or to store only on disk — you use persist() with an explicit storage level.
| Method | Storage Level | Behaviour | Example call |
|---|---|---|---|
cache() |
MEMORY_AND_DISK | Stores in RAM; spills partitions to local disk if RAM is full. Safe default for most use cases. | df.cache() |
persist(MEMORY_ONLY) |
MEMORY_ONLY | Stores only in RAM. Partitions that do not fit are recomputed on demand — faster reads if all fits, but risky on large DataFrames. | df.persist(StorageLevel.MEMORY_ONLY) |
persist(DISK_ONLY) |
DISK_ONLY | Writes to local executor disk. Slower than RAM reads, but always works regardless of available heap space. | df.persist(StorageLevel.DISK_ONLY) |
unpersist() |
— | Releases the cached data from executor memory and disk. Call this when you are done with a cached DataFrame to free resources for the next stage. | df.unpersist() |
- ▸ Single-use DataFrames — if a DataFrame is only consumed by one action, caching adds overhead with no benefit. Spark already computes it once.
- ▸ DataFrames too large for RAM — excessive spill to disk triggers GC pressure and can make things slower than re-reading from a fast source like Parquet on SSD.
- ▸ DataFrames written immediately to storage — if you write a DataFrame to Parquet right after creating it and never read it again in the same job, caching it first wastes RAM.
📡 Broadcast joins — avoid the shuffle
Funmi noticed NaijaCart's city-to-zone lookup was joining 500,000 order rows against a 5-row city table. Spark defaulted to a sort-merge join — both sides sorted, shuffled across the network, then merged. The lookup table is tiny; it should never leave the driver.
The broadcast() hint tells Spark to replicate the small DataFrame to every executor. Each executor then does a local hash join against its own copy of the lookup. Zero shuffle, zero network transfer for the lookup side.
The broadcast join runs in 0.42s vs 0.80s — 1.9× faster on a 5-row lookup. On production at NaijaCart, where the city lookup was joined with a 50 million-row orders table, the speedup was over 10× because the shuffle cost scales with the large side.
Spark automatically broadcasts tables under spark.sql.autoBroadcastJoinThreshold (default: 10 MB). Add an explicit broadcast() hint for tables you know are small even if Spark does not automatically detect them — for example, when the table is read from a source whose size Spark cannot statistically estimate at plan time.
🔀 Partitioning — repartition vs coalesce
Every Spark DataFrame is divided into partitions — chunks of data that executors process in parallel. The default number of partitions after a shuffle is controlled by spark.sql.shuffle.partitions (default: 200). Getting partition count right matters: too few partitions under-utilise cores, too many create excessive task overhead.
repartition(N)
Causes a full shuffle. Spark redistributes all rows across N new partitions using a hash function. Use when you need to increase the number of partitions (going up) or want perfectly balanced partition sizes regardless of current layout. Typical use case: before a wide aggregation on a large dataset where default 200 partitions are too coarse or too fine.
coalesce(N)
No shuffle. Spark merges adjacent partitions on the same executor. Use when you need to decrease the number of partitions (going down), especially just before writing output files. Writing 200 tiny Parquet files is inefficient — coalesce(4) before the write produces 4 well-sized files without the cost of a full shuffle.
| Operation | Shuffle? | When to use | Cost |
|---|---|---|---|
repartition(N) |
Yes — full shuffle | Increasing partitions, or balancing skewed data evenly across executors | Network I/O proportional to data size |
coalesce(N) |
No | Decreasing partitions before writing output — avoids many small files | CPU only; some tasks handle more data than others |
repartition() immediately before coalesce(). That is two operations — a full shuffle then a merge — where coalesce alone would have sufficed. Always check which direction you are going: up = repartition, down = coalesce.
🔄 Adaptive Query Execution (AQE)
Spark's query planner works ahead of time — it generates an execution plan before it has seen the actual data. This creates a mismatch: the plan assumes estimated statistics (row counts, data sizes), which may be wildly off for skewed or unevenly distributed datasets.
Adaptive Query Execution (AQE), introduced in Spark 3.0 and stabilised in 3.2, fixes this by collecting real runtime statistics at each shuffle boundary and using them to rewrite the remainder of the plan on the fly. Three concrete optimisations AQE can apply:
autoBroadcastJoinThreshold, it switches to a broadcast join even if the planner estimated the table was large.spark.sql.adaptive.enabled = true
This is the default in Spark 3.2+. You only need to set it explicitly if you are on Spark 3.0 or 3.1, or if something in your config has disabled it. The verified output above confirms it is on: AQE enabled: true.
✅ Funmi's tuning checklist — NaijaCart
Funmi runs through this checklist before declaring any Spark job ready for production at NaijaCart. Five items, in order of impact.
- Cache DataFrames used in 2 or more actions in the same job
- Add
broadcast()hint for lookup tables smaller than 50 MB — even if Spark might auto-broadcast, the hint makes the intent explicit in code review - Check
spark.sql.shuffle.partitions— default 200 is almost always too high for jobs processing under 10 GB; set it to 2× the number of cores for small-to-medium jobs - Confirm AQE is enabled (
spark.sql.adaptive.enabled=true) — it is on by default in Spark 3.2+ but worth verifying in legacy job configs - Profile with the Spark UI (localhost:4040) before and after any optimisation — never tune blind; the Stage tab shows shuffle bytes and GC time per stage
🖥️ Reading the Spark UI
The Spark UI is available at http://localhost:4040 while a Spark session is running. It is the ground truth for performance diagnosis — tuning without it is guesswork.
The two most important signals
Wide stage with large shuffle bytes. In the Stages tab, each stage shows a "Shuffle Read" and "Shuffle Write" column. If a stage is moving hundreds of megabytes or gigabytes of shuffle data, that is a signal to: (1) add a broadcast hint if the join is against a small table, or (2) increase spark.sql.shuffle.partitions if partitions are too large and causing memory pressure.
Long GC time. The Tasks tab shows GC time per task. If GC time is more than 10% of task duration, executors are spending too much time managing memory. Common causes: too much data cached (cache is eating into execution memory), or executor heap is too small for the data volume. Fix: unpersist() caches you no longer need, or increase executor memory via spark.executor.memory.
⚠️ Common mistakes
df.unpersist() as soon as you are done with a cached DataFrame.repartition(4) already does a full shuffle. Adding .coalesce(4) afterward adds a second unnecessary operation. Use whichever one is appropriate for the direction: repartition going up, coalesce going down.spark.sql.shuffle.partitions to 4 because it "feels right" for a small job, then shipping the same config to a production job on 500 GB of data, will cause OOM. Always profile with the Spark UI. AQE can also auto-coalesce partitions — let it, then adjust if needed.🧠 Quiz — 3 questions
Apply these three levers to your next PySpark job and open the Spark UI to verify the improvement.
🎤 Interview Q&A
Use cache() when a DataFrame is consumed by two or more actions in the same job. Classic cases: a filtered or joined DataFrame that feeds both an aggregation and a write; a DataFrame used as training input in multiple ML model evaluations; an intermediate result shared across multiple report queries in a notebook. The benefit is proportional to how expensive the upstream computation is and how many times it is reused.
Avoid cache() when: (1) the DataFrame is only used once — caching adds overhead with zero benefit; (2) the DataFrame is too large for executor memory — excessive spill causes GC pressure that can make the job slower than re-reading from a fast Parquet source; (3) the DataFrame is immediately written to storage and discarded. Always call unpersist() when done to free memory for subsequent stages.
Adaptive Query Execution (AQE) is a Spark 3.x feature that makes the query planner adaptive rather than static. The traditional planner generates a physical plan before execution based on estimated statistics (row counts, data sizes). AQE augments this by collecting real runtime statistics at shuffle boundaries — after each shuffle completes, Spark knows the actual partition sizes and row counts. It uses this to rewrite the remainder of the plan.
The three optimisations AQE can apply:
- Convert sort-merge joins to broadcast joins — if at runtime one join side is found to be under the broadcast threshold, AQE switches strategy.
- Coalesce shuffle partitions — if many shuffle partitions are empty or near-empty, AQE merges them to reduce task overhead.
- Handle data skew — if one partition is dramatically larger, AQE splits it and replicates the matching rows from the other side, turning a straggler task into several balanced ones.
AQE is on by default from Spark 3.2. Enable it explicitly on older configs: spark.sql.adaptive.enabled=true.
The configuration key is spark.sql.shuffle.partitions (default: 200). There is no universally "right" value — it depends on data volume, cluster size, and the nature of the query. The 200 default was chosen for large clusters; for small-to-medium jobs it is almost always too high, creating hundreds of tiny tasks that waste scheduler overhead.
A practical starting heuristic: target partition sizes of 100 MB–200 MB each. If your shuffled dataset is 2 GB, set partitions to 10–20. If it is 200 GB, set partitions to 1000–2000. More precisely: use 2–4× the total number of CPU cores available across all executors.
With AQE enabled (spark.sql.adaptive.enabled=true), Spark will automatically coalesce small shuffle partitions down toward the actual data volume, so setting a high initial value (e.g., 400) and letting AQE reduce it is a valid production approach. Check the Spark UI's Stages tab after a representative run: if "Shuffle Read" per task is consistently under 10 MB, your partition count is too high; if tasks are regularly spilling to disk, it is too low.
✅ Key takeaways
- Use
cache()(shorthand for MEMORY_AND_DISK) when a DataFrame is consumed by two or more actions. Callunpersist()when done. In the NaijaCart benchmark, two actions on a cached filtered DataFrame ran in 0.25s vs 1.93s uncached — a 7.7× speedup. - Add a
broadcast()hint for any lookup table under 50 MB. This replaces a network shuffle with a local hash join on every executor. The NaijaCart city-zone join ran 1.9× faster with the hint, and the advantage scales dramatically with the size of the large side. - Use
repartition(N)when increasing partitions or balancing skewed data — it causes a full shuffle. Usecoalesce(N)when decreasing partitions (e.g., before writing output) — no shuffle required. Never chain repartition then coalesce. - Adaptive Query Execution (on by default in Spark 3.2+) rewrites the plan mid-execution using real shuffle statistics. It can auto-convert joins to broadcast, coalesce tiny partitions, and split skewed partitions. Always verify it is enabled in legacy job configs.
Build the pipelines that power NaijaCart-scale platforms
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →