Why test Spark code?

Spark transformations are lazy

A bug in withColumn doesn't surface until an action — like .collect(), .write(), or .count() — is called. In production that action may be triggered hours after the faulty transformation was scheduled. Tests catch these problems before they reach the cluster.

At TaxSetu, Arjun's team processes over 800,000 ITR filings per filing season. A single off-by-one in a tax bracket calculation would silently miscategorise thousands of taxpayers. The bugs that reach production are rarely syntax errors — they are logic errors that pass linting and type checks but produce wrong values under specific input conditions. That is exactly what unit tests are designed to catch.

The four categories of PySpark bugs that tests catch most reliably:

  • ▸ Wrong column names — a transform expects income_inr but the upstream DataFrame delivers income. The job starts, runs for 40 minutes, then fails with an AnalysisException at action time.
  • ▸ Null handling — col("income_inr") <= 300_000 evaluates to null for null inputs, not false. Rows with null income silently fall through all when branches and hit otherwise — often the wrong bucket.
  • ▸ Schema drift — a new column added upstream breaks a downstream transform that assumed a fixed column count. Tests that check len(result.columns) catch this immediately.
  • ▸ Off-by-one in window functions — boundary conditions in when().otherwise() chains and rank-based windows are where the numbers go subtly wrong.

The testing stack

Three tools. Each has a specific role and they compose naturally:

Tool Role Install
pytest Test runner, fixtures, assertions. Discovers test files automatically, reports pass/fail with timing, and supports parametrize for data-driven tests. pip install pytest
chispa DataFrame equality assertions purpose-built for PySpark. Handles row-order independence, schema diff messages, and nullable mismatch warnings that plain assert misses. pip install chispa
pyspark SparkSession is shared across all tests via a scope="session" pytest fixture. No Spark cluster needed — local[*] mode runs everything in-process on your laptop. pip install pyspark
Lesson 00 note: All three packages are already installed if you followed the Lesson 00 setup. Verify with pip show pytest chispa pyspark.

Project structure

Keep business logic and tests in separate files. Functions that take a DataFrame and return a DataFrame — with no side effects — are the easiest to test. Arjun's rule at TaxSetu: if a function talks to Kafka, writes to S3, or touches a database, it is not a transform — it is an I/O handler and is tested at the integration level, not the unit level.

file tree taxsetu-pipeline project layout

The key design principle: transforms.py knows nothing about pytest. It imports PySpark and nothing else. Tests live entirely in test_transforms.py. When the test suite grows, shared fixtures move to conftest.py — pytest discovers it automatically in the same directory.

transforms.py — the module under test

Two pure transformation functions. Each takes a DataFrame and returns a DataFrame. No SparkSession argument, no side effects — the caller provides the DataFrame, the function applies the transform and hands it back. This pattern makes every function independently testable.

transforms.py Lesson 12 · business logic

add_tax_bracket implements India's new-regime income tax slabs as a column derivation. The when(...).when(...).otherwise(...) chain is evaluated lazily — Spark builds a logical plan and only executes it when an action is called. clean_nulls filters out rows where income_inr is null, which would otherwise corrupt aggregations downstream.

Why stateless, pure functions?

A function that only transforms data — no global state, no side effects, same output for the same input — can be called with any DataFrame in any test. You don't need to mock a database connection, patch an environment variable, or spin up a service. The test is: "give this function this DataFrame, get that DataFrame back." That simplicity is the entire value of the pure-transform architecture.

test_transforms.py

Five tests covering the two functions. Read top to bottom: fixture first, then assertions, then the chispa full-DataFrame equality check at the end.

test_transforms.py Lesson 12 · pytest tests

Walking through each test

Fixture

scope="session" SparkSession

The @pytest.fixture(scope="session") decorator tells pytest to create this fixture once for the entire test session and share it across every test function that requests it. SparkSession startup involves JVM initialisation, Py4J socket setup, and Spark context creation — typically 10–20 seconds. Creating it once and reusing it keeps a 5-test suite under 10 seconds instead of over 2 minutes. yield s hands the session to the test, and s.stop() runs after all tests in the session complete.

Tests 1 & 2

Simple boundary assertions

test_tax_bracket_zero_percent creates a one-row DataFrame with income 250,000 INR (below the 300,000 threshold) and asserts the derived column equals "0%". test_tax_bracket_30_percent does the same for 2,000,000 INR (above 1,500,000 — the 30% bracket). The pattern df.first()["column"] calls the collect action and reads the first row. For single-row DataFrames this is clear and fast.

Test 3

Null handling test

test_clean_nulls_removes_bad_rows feeds three rows to clean_nulls: incomes 500,000, None, and 800,000. The assertion clean_nulls(df).count() == 2 confirms the null row was dropped and the two valid rows survived. Note that None in a Python list passed to createDataFrame becomes a Spark null — this is the correct way to inject null values in test data.

Test 4

Schema preservation test

test_add_tax_bracket_schema_preserved checks that the transform adds exactly one column and that the new column is named tax_bracket. This protects against a refactor that accidentally renames a column or adds an extra intermediate column. Schema tests are especially valuable in CI: if an upstream dependency changes a column type or name, this test fails before any data is processed.

Test 5

chispa full-DataFrame equality

test_chispa_exact_match builds an expected DataFrame with the final schema (filing_id, income_inr, tax_bracket) and runs assert_df_equality(result, expected, ignore_nullable=True). This asserts row values, column names, column order, and data types in one call. The ignore_nullable=True flag is required here because the expected DataFrame was created from a Python list (all columns nullable=True by default), while the result DataFrame's tax_bracket column comes from a when().otherwise() chain which Spark marks nullable=False.

Running the tests

From the taxsetu-pipeline/ directory, run pytest with the verbose flag to see each test name and result individually:

pytest test_transforms.py -v
terminal Lesson 12 · run the test suite
verified output
============================= test session starts ============================== platform darwin -- Python 3.12.4, pytest-8.3.0, pluggy-1.6.0 collecting ... collected 5 items test_transforms.py::test_tax_bracket_zero_percent PASSED [ 20%] test_transforms.py::test_tax_bracket_30_percent PASSED [ 40%] test_transforms.py::test_clean_nulls_removes_bad_rows PASSED [ 60%] test_transforms.py::test_add_tax_bracket_schema_preserved PASSED [ 80%] test_transforms.py::test_chispa_exact_match PASSED [100%] ============================== 5 passed in 4.68s ===============================

4.68 seconds for 5 tests — the bulk of that time is the single SparkSession startup. Because scope="session" shares the session, adding 20 more tests adds roughly 0.1–0.3 seconds each, not another 4 seconds per test.

Why ignore_nullable=True is required in test_chispa_exact_match

When you build a DataFrame from a Python list via createDataFrame([(1, 250_000, "0%"), ...], ["a","b","c"]), Spark infers all columns as nullable=True. When you derive a column from when(...).otherwise(...), Spark marks the result column as nullable=False (because otherwise guarantees a non-null value on every row). These two DataFrames hold identical data but differ in the nullable flag on tax_bracket. Without ignore_nullable=True, chispa reports a schema mismatch. The flag tells chispa: "the data must match exactly, but nullable differences are acceptable."

chispa vs manual assertions

A common first attempt at DataFrame equality is assert result.collect() == expected.collect(). This breaks in three ways:

  • ✗ Row order is non-deterministic — Spark distributes data across partitions and the order of rows in .collect() is not guaranteed across runs or machines.
  • ✗ No diff message — when the assertion fails, Python prints the raw list of Row objects. With 1,000 rows in a test DataFrame this is unreadable.
  • ✗ Nullable mismatch is invisible — the data is correct but the nullable flag differs. A plain list comparison passes; a schema comparison would fail — but .collect() doesn't check schemas.

chispa's assert_df_equality handles all three. It sorts both DataFrames before comparing, reports exact row-level diffs, and shows a schema diff table when the schemas differ.

Here is what a chispa failure message looks like when the expected tax bracket is wrong:

chispa failure output — schema or data mismatch
E chispa.exceptions.DataFramesAreNotEqualError: E E Rows are not equal E Actual row: Row(filing_id=2, income_inr=700000, tax_bracket='10%') E Expected row: Row(filing_id=2, income_inr=700000, tax_bracket='5%') E E Schema comparison: E +----------------+-----------+-----------+----------+ E | Column | ActualType| ExpType | Nullable | E +----------------+-----------+-----------+----------+ E | filing_id | LongType | LongType | OK | E | income_inr | LongType | LongType | OK | E | tax_bracket | StringType| StringType| MISMATCH | E +----------------+-----------+-----------+----------+

The failure message tells you exactly which row differed, what the actual and expected values were, and whether the schema also has a mismatch. Compare this to a raw AssertionError: False is not True from a manual assert — chispa saves debugging time on every failed test.

conftest.py — shared fixtures across multiple test files

When the test suite grows beyond one file, move the SparkSession fixture to conftest.py. pytest discovers this file automatically in the same directory and makes its fixtures available to every test file without an explicit import.

conftest.py Lesson 12 · shared fixture across all test files

With conftest.py in place, you can delete the spark fixture from test_transforms.py and from any future test file — they all inherit it automatically. The SparkSession is still created once and shared across all test files in the session.

local[2] vs local[*]: local[*] uses all available CPU cores. In CI environments, this can cause resource contention with other running jobs. local[2] limits Spark to 2 threads — fast enough for unit tests and predictable in shared CI runners.

CI integration

Adding tests to CI ensures every pull request is validated before merge. Meera's team at TaxSetu runs the full test suite on every push to any branch. The key environment requirement is JAVA_HOME — without it, SparkSession creation fails with JAVA_HOME is not set.

Add this step to your GitHub Actions workflow or GitLab CI stage:

.github/workflows/test.yml Lesson 12 · CI step

Notes on the CI configuration:

  • ▸ JAVA_HOME=/opt/java/openjdk — this path is correct for the GitHub Actions ubuntu-latest runner which ships with OpenJDK via Temurin. Verify the path on your CI runner with which java or echo $JAVA_HOME.
  • ▸ Pin package versions in CI (pip install pyspark==3.5.0 chispa==0.10.1 pytest==8.3.0) to avoid test failures caused by a dependency upgrading between runs.
  • ▸ test_*.py discovers all test files matching the pattern — useful when you split tests by module (e.g., test_transforms.py, test_aggregations.py, test_joins.py).
  • ▸ SparkSession startup in CI is slightly slower than locally (12–25 seconds) because CI runners have fewer cores and no JVM warm cache. A 5-test suite will take 20–30 seconds in CI — expected and acceptable.

Meera's testing principles

Three non-negotiable rules at TaxSetu

"At TaxSetu, Meera enforces three rules across every data engineering pull request:"

  1. Every transform function has at least one happy-path test. If the function exists, there is a test that calls it with valid input and asserts the expected output. No exceptions for "simple" functions.
  2. Every nullable column has a null-input test. If a column can be null in production data — and it can, because source systems are imperfect — there must be a test that passes a null value and asserts the correct behaviour (either filtering the row, defaulting to a sentinel value, or propagating null intentionally).
  3. If a bug reaches production, a test for that exact case is added before the fix. First write the failing test that reproduces the bug. Then fix the code. The test turns green. This is the red-green cycle applied to data pipelines and it prevents the same bug from recurring silently in a future refactor.

Common mistakes

Creating a new SparkSession per test
Without scope="session", pytest creates a new SparkSession for every test function. SparkSession startup takes 10–20 seconds. Five tests become 60–120 seconds. Use scope="session" and share one session across all tests.
Comparing .collect() lists directly
Spark does not guarantee row order across partitions. assert result.collect() == expected.collect() is non-deterministic — it passes on your laptop and fails in CI. Use chispa's assert_df_equality which sorts both DataFrames before comparing.
Not using ignore_nullable=True with chispa
DataFrames created from Python lists have nullable=True for all columns. DataFrames produced by when().otherwise() have nullable=False for the derived column. Without the flag, chispa reports false failures even when the data is correct.
Testing only happy paths
Happy-path tests confirm the function works when input is clean. But production data is not clean. Null inputs, empty DataFrames, strings instead of numbers, and extreme values (zero income, negative income) are where bugs hide. Always add at least one null test per nullable column.

Quiz — 3 questions

Question 1 of 3
Why use scope="session" for the SparkSession fixture?
Question 2 of 3
chispa's assert_df_equality is better than comparing .collect() lists directly because:
Question 3 of 3
When should you add ignore_nullable=True to a chispa assertion?
0
/ 3
Tests written. Pipeline protected.

Add a null test to every nullable column in your next transform.

Interview Q&A

Medium How do you test PySpark transformations?
+

The standard pattern is to write transformation functions as pure functions — each takes a DataFrame and returns a DataFrame, with no side effects and no global state. These functions are easy to unit test because you can create a small in-memory DataFrame with spark.createDataFrame(...), call the function, and assert on the result.

For the test infrastructure, we use pytest with a shared SparkSession fixture at scope="session" so the JVM is started once and shared across all tests. For assertions, we use chispa's assert_df_equality instead of comparing .collect() lists directly, because collect() order is non-deterministic and the raw list comparison gives no useful error message when it fails.

We test at least: a happy-path case, a null-input case for each nullable column, a schema preservation case (correct column count and names), and a boundary condition case for range-based logic like tax brackets.

Easy What is chispa and why is it better than asserting on .collect()?
+

chispa is a Python library purpose-built for asserting DataFrame equality in PySpark unit tests. Its primary function is assert_df_equality(actual, expected).

It is better than assert result.collect() == expected.collect() for three reasons: First, Spark does not guarantee row order across partitions — a direct list comparison is non-deterministic and can fail in CI even when the data is correct. chispa sorts both DataFrames before comparing. Second, when the assertion fails, chispa shows exactly which row differed (actual vs expected values) and whether the schema also has a mismatch. A plain AssertionError on a list of Row objects gives you no actionable information. Third, chispa exposes the ignore_nullable flag to handle the common case where one DataFrame was built from a Python list (nullable=True) and the other from a transformation (nullable=False) — the data is correct but the metadata differs.

Install with pip install chispa. Import with from chispa.dataframe_comparer import assert_df_equality.

Medium How would you set up CI for a PySpark project?
+

The core requirement is a JDK on the CI runner — without it, SparkSession.builder.getOrCreate() fails immediately because PySpark spawns a JVM subprocess. GitHub Actions' ubuntu-latest runner ships with OpenJDK, but you need to set JAVA_HOME explicitly because PySpark reads that environment variable to locate the JVM binary.

The CI step is straightforward: export JAVA_HOME, install pyspark, chispa, and pytest (pinned to exact versions for reproducibility), then run pytest test_*.py -v. SparkSession startup in CI takes 12–25 seconds depending on the runner, so a 10-test suite typically completes in under 40 seconds.

For larger suites, you can cache the pip download artifacts between runs using GitHub Actions' cache action to avoid re-downloading ~200 MB of PySpark wheels on every push. This cuts CI time by 1–2 minutes on slow runners.

Key takeaways

  • Write transformation functions as pure functions — DataFrame in, DataFrame out, no side effects. This makes every function independently testable without mocking infrastructure.
  • Use a scope="session" pytest fixture to share one SparkSession across all tests. SparkSession startup is 10–20 seconds; sharing it keeps a large test suite fast.
  • Use chispa's assert_df_equality(result, expected, ignore_nullable=True) instead of comparing .collect() lists — row order is non-deterministic in Spark, and chispa gives readable diff messages on failure.
  • Test three cases for every transform: happy path, null input for each nullable column, and schema preservation (column count and names). These three patterns catch the majority of production PySpark bugs.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Build pipelines that ship — and stay green

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →