💡 The core idea

The most important rule in Spark:

Transformations build a plan (called a DAG — Directed Acyclic Graph). Actions execute the plan. Nothing moves until an action fires. You can write 20 transformations in a row and Spark will not touch a single row of data until you call an action like show() or count().

When you call df.filter(...), Spark does not immediately scan your data. It adds a "filter" node to an internal execution plan. When you then call df.withColumn(...), it adds another node. This continues until you fire an action — at which point Spark hands the entire plan to the Catalyst optimizer, which rewrites it into the most efficient physical execution strategy, and only then does data flow through your transformations.

This design is not an accident. It is the fundamental reason why Spark can process petabytes efficiently: by seeing the whole plan before running, the optimizer can push filters down to storage, prune unused columns, reorder joins, and pipeline stages — none of which would be possible if Spark ran each line immediately.

🗺️ Lazy evaluation — visualized

The diagram below shows exactly what happens when you write transformations and then fire an action. The plan is stored silently until show() triggers the run.

Spark Lazy Evaluation — BoraPay Payments Pipeline df 6-row DataFrame lazy .filter( status==success plan node added lazy .withColumn( amount_usd plan node added Plan stored in memory — zero data movement ACTION .show() triggers run Catalyst optimizes Data flows filter → withColumn pipelined, no temp copy

Notice the critical point: Spark pipelines the filter and withColumn stages together. A row that passes the filter immediately has amount_usd computed — there is no intermediate materialized DataFrame sitting in memory between the two steps. This pipelining is only possible because Spark saw the entire plan before executing it.

📋 Transformations vs Actions — the definitive list

Every operation in PySpark is either a transformation (returns a new DataFrame, lazy) or an action (triggers execution, returns a result). Learn to distinguish them instantly — it is the single most important skill for debugging slow Spark jobs.

Type Returns Triggers execution? Common operations
Transformation A new DataFrame (lazy plan) No — adds a node to the DAG filter, select, withColumn, groupBy, join, orderBy, distinct, drop, union
Action A value, list, or side-effect (write) Yes — sends plan to Catalyst, executes show, count, collect, write, take, first, foreach
Quick rule of thumb: If the result is a DataFrame, it is a transformation. If it returns a Python value (integer, list, None) or writes to disk, it is an action. When in doubt, check whether calling it triggers a Spark job in the UI — transformations never do.

💻 Code demo — lazy evaluation in practice

This example builds a BoraPay payment pipeline step by step. Watch the print statement after the transformations — it runs immediately, confirming that no Spark job has fired yet. Only show() triggers execution.

transformations_and_actions.py Lesson 04 · BoraPay lazy evaluation demo
verified output
Transformations registered — no data moved yet === BoraPay Payments (status=success) === +--------+----------+----------+ |event_id|amount_ksh|amount_usd| +--------+----------+----------+ | EV0002| 120.0| 0.92| | EV0003| 151.0| 1.16| | EV0006| 202.0| 1.55| +--------+----------+----------+ Count (triggers another action): 3

The print before show() executes instantly — Python code runs line by line. But neither the filter nor the withColumn touches data until show() fires. The second action (count()) re-runs the full plan from scratch because we have not cached the DataFrame. This is the re-run problem — covered in the next section.

✨ Why lazy evaluation is brilliant

Lazy evaluation is not just a quirk — it is the engineering decision that makes Spark competitive with hand-tuned C++ at petabyte scale. Three reasons it matters:

  • ▸ Catalyst optimizer sees the whole plan before running. When you fire an action, Spark's Catalyst optimizer analyzes your entire transformation chain. It can reorder operations, combine filters, eliminate redundant steps, and choose between different physical join algorithms — optimizations impossible if Spark ran each transformation immediately.
  • ▸ Predicate pushdown — only the columns you need get read from disk. When Spark reads Parquet files, it can push your filter conditions down to the file reader, skipping entire row groups and columns that your query does not need. This can reduce I/O by 90%+ on wide tables. This is called predicate pushdown, and it only works because Catalyst knows your filter before it opens the file.
  • ▸ Multiple transformations are pipelined — no intermediate materialization. In a non-lazy system, filter would produce a full intermediate DataFrame in memory before withColumn started. In Spark, both operations are compiled into a single stage. Each record passes through filter and withColumn in one pass, with no temporary copy sitting in memory between them.

⚠️ The re-run problem

Lazy evaluation has one sharp edge: every action re-executes the full plan from the beginning.

If you call count() on a DataFrame and then call show() on the same DataFrame, Spark runs your filter and withColumn (and any preceding transformations) twice — once per action. On a 500-million-row dataset reading from S3, this can double your job time and double your cloud bill.

The fix: df.cache()

Call df.cache() before your first action. Spark will materialize the DataFrame in executor memory after the first action completes, and subsequent actions will read from the in-memory cache instead of re-executing the full plan. Use df.unpersist() when you are done with it to free the memory.

cache_example.py Lesson 04 · solving the re-run problem

🐛 Wanjiku's bug — 40 minutes instead of 2

Real scenario from BoraPay engineering

Wanjiku was building the end-of-month reconciliation job for BoraPay's 500-million-row transaction ledger. Her pipeline read raw Parquet files from storage, filtered to the current month, converted KES amounts to USD, and computed aggregate totals by account.

She ran the job and stared at the terminal as the job crept past 40 minutes. Her colleague Kwame estimated it should take about 2 minutes. What went wrong?

Root cause: no cache, multiple actions. Wanjiku's pipeline called count() to validate row counts after filtering, then called agg() for the summary totals, then called write() to save the output. That is three actions on the same DataFrame with no cache() — meaning the filter and USD conversion ran three times across 500 million rows.

Fix:

filtered_df.cache() inserted before the first count() call. The plan ran once, materialized in executor memory, and the subsequent agg() and write() served from cache. Job time dropped from 40 minutes to 2 minutes 20 seconds.

Rule of thumb from Wanjiku's postmortem: Any time you call more than one action on the same DataFrame, ask yourself: "Should I cache this?" If the DataFrame took more than 30 seconds to compute the first time, almost always yes.

🔍 What the Spark UI shows you

When you run PySpark with spark.ui.enabled set to true (or just remove that config), Spark starts a web UI at http://localhost:4040. This is one of the most powerful debugging tools in the ecosystem.

The three views you will use every day

Jobs tab

  • Each action creates one job
  • Jobs are numbered in order of execution
  • Click a job to see its stages
  • Failed jobs show the exception

Stages tab

  • Narrow transformations (filter, map) = one stage
  • Wide transformations (groupBy, join) = shuffle = new stage
  • Stage boundary = data crosses the network between executors
  • Skew shows up here as one task taking 10x longer

DAG visualization

  • Visual graph of your transformation plan
  • Boxes = stages, arrows = data dependencies
  • Exchange = shuffle (expensive — minimize these)
  • WholeStageCodegen = pipelined stages (good)

Storage tab

  • Shows cached DataFrames
  • Memory used vs total available
  • Fraction cached (partial cache on large datasets)
  • Use this to confirm .cache() actually materialized

A key thing to look for: if the DAG shows the same set of transformations repeated across multiple jobs (e.g., you see "Filter" and "Project" in Job 0, then the exact same nodes in Job 1), that is the re-run problem. Add .cache() between the repeated plan and the first action.

⚠️ Common mistakes

Assuming transformations run immediately
Writing df.filter(...) and then checking logs for execution — nothing runs until an action fires. If you need to verify a transformation worked, call .show(5) to trigger a small action. Never assume data has been filtered just because you wrote the transformation.
Calling collect() to "see" intermediate data
collect() is an action that pulls the entire DataFrame to the driver as a Python list. On a 100M-row DataFrame this will crash your driver with an OOM error. Use .show(20) or .take(10) for inspection — these fetch a small number of rows without moving the full dataset.
Using Python loops with .filter() calls inside
Writing a Python for loop that calls df.filter(col("date") == d).count() for each date creates a separate Spark job per iteration. With 365 dates that is 365 jobs. Instead, use a single groupBy("date").count() — one job, all dates at once.

🧠 Quiz — test your understanding

Question 1 of 3
Which of the following is a transformation (not an action)?
Question 2 of 3
Why does Spark use lazy evaluation?
Question 3 of 3
If you call count() then show() on the same DataFrame with no cache(), how many times does the filter() transformation run?
0
/ 3
Lazy evaluation mastered.

Move on to Filters, selects, and withColumn — the most-used transformations in every pipeline.

🎤 Interview Q&A

Medium What is the Catalyst optimizer?
+

The Catalyst optimizer is Spark's query optimization framework, written in Scala. When you fire an action, Spark passes your logical plan (the chain of transformations you wrote) to Catalyst, which applies a series of rule-based and cost-based transformations to produce an optimized physical plan.

Rule-based optimizations include predicate pushdown (move filters as early as possible in the plan), constant folding (evaluate constant expressions at planning time rather than per-row), and projection pruning (drop columns not needed by later steps). Cost-based optimizations use table statistics to choose between, for example, a broadcast hash join vs a sort-merge join — whichever has the lower estimated cost given the data sizes involved.

The key interview point: Catalyst enables Spark SQL and the DataFrame API to generate execution plans that are often faster than hand-written RDD code, because a human writing low-level RDD transformations cannot apply all these optimizations as systematically as the optimizer can.

Medium What is the difference between narrow and wide transformations?
+

Narrow transformations are operations where each output partition depends on at most one input partition. No data moves between executors (no shuffle). Examples: filter, map, select, withColumn, drop, union. These are fast because they require no network I/O between nodes.

Wide transformations are operations where each output partition can depend on multiple input partitions. Data must be shuffled across the network between executors. Examples: groupBy, join, distinct, orderBy, repartition. Each wide transformation creates a stage boundary in the DAG — data is written to disk on the executor (shuffle write), sent over the network, and then read by the receiving executor (shuffle read).

Why it matters: shuffles are the most expensive operations in Spark. Minimizing wide transformations — or batching them together — is the primary lever for improving Spark job performance. If the Spark UI shows many stage boundaries, look for opportunities to eliminate or consolidate wide transformations.

Hard When would you use cache() vs persist()?
+

cache() is a shorthand for persist(StorageLevel.MEMORY_AND_DISK). It stores the DataFrame in executor memory if there is space, and spills to disk if memory is insufficient. It is the right choice for most use cases where you just want to avoid re-computation.

persist() accepts an explicit StorageLevel, giving you fine-grained control:

  • ▸ MEMORY_ONLY — store in memory only; if data does not fit, partitions are recomputed on demand. Fastest but risky on large datasets.
  • ▸ MEMORY_AND_DISK (what cache() uses) — spills to local disk if memory is full. Safer default.
  • ▸ DISK_ONLY — always write to disk; use when the dataset is too large for memory and recompute cost is very high (e.g., after a complex join).
  • ▸ MEMORY_AND_DISK_SER — serialized storage (Java serialization, not Java objects). Uses 2–5x less memory than deserialized but adds CPU overhead for serialization/deserialization on each read.

Use cache() by default. Switch to persist() with a specific level when you have a concrete reason — e.g., the dataset is bigger than executor heap, or you need to survive executor failures across iterations of an ML training loop.

✅ Key takeaways

  • Transformations (filter, select, withColumn, groupBy, join) are lazy — they build a plan (DAG) but move zero data. Actions (show, count, collect, write) fire the plan.
  • Spark's Catalyst optimizer sees your full transformation chain before executing. This enables predicate pushdown, column pruning, join reordering, and stage pipelining — none of which are possible in an eager system.
  • Every action re-executes the full plan from scratch. If you call multiple actions on the same DataFrame, use df.cache() before the first action to avoid re-running expensive transformations.
  • Narrow transformations (filter, map) stay in one stage — no shuffle. Wide transformations (groupBy, join) cause a shuffle and create a stage boundary. Minimize shuffles to minimize job time.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

The foundation our AI platform engineers stand on

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →