Why test Spark code?
A bug in withColumn doesn't surface until an action — like .collect(), .write(), or .count() — is called. In production that action may be triggered hours after the faulty transformation was scheduled. Tests catch these problems before they reach the cluster.
At TaxSetu, Arjun's team processes over 800,000 ITR filings per filing season. A single off-by-one in a tax bracket calculation would silently miscategorise thousands of taxpayers. The bugs that reach production are rarely syntax errors — they are logic errors that pass linting and type checks but produce wrong values under specific input conditions. That is exactly what unit tests are designed to catch.
The four categories of PySpark bugs that tests catch most reliably:
- ▸ Wrong column names — a transform expects
income_inrbut the upstream DataFrame deliversincome. The job starts, runs for 40 minutes, then fails with anAnalysisExceptionat action time. - ▸ Null handling —
col("income_inr") <= 300_000evaluates tonullfor null inputs, notfalse. Rows with null income silently fall through allwhenbranches and hitotherwise— often the wrong bucket. - ▸ Schema drift — a new column added upstream breaks a downstream transform that assumed a fixed column count. Tests that check
len(result.columns)catch this immediately. - ▸ Off-by-one in window functions — boundary conditions in
when().otherwise()chains and rank-based windows are where the numbers go subtly wrong.
The testing stack
Three tools. Each has a specific role and they compose naturally:
| Tool | Role | Install |
|---|---|---|
| pytest | Test runner, fixtures, assertions. Discovers test files automatically, reports pass/fail with timing, and supports parametrize for data-driven tests. | pip install pytest |
| chispa | DataFrame equality assertions purpose-built for PySpark. Handles row-order independence, schema diff messages, and nullable mismatch warnings that plain assert misses. |
pip install chispa |
| pyspark | SparkSession is shared across all tests via a scope="session" pytest fixture. No Spark cluster needed — local[*] mode runs everything in-process on your laptop. |
pip install pyspark |
pip show pytest chispa pyspark.
Project structure
Keep business logic and tests in separate files. Functions that take a DataFrame and return a DataFrame — with no side effects — are the easiest to test. Arjun's rule at TaxSetu: if a function talks to Kafka, writes to S3, or touches a database, it is not a transform — it is an I/O handler and is tested at the integration level, not the unit level.
The key design principle: transforms.py knows nothing about pytest. It imports PySpark and nothing else. Tests live entirely in test_transforms.py. When the test suite grows, shared fixtures move to conftest.py — pytest discovers it automatically in the same directory.
transforms.py — the module under test
Two pure transformation functions. Each takes a DataFrame and returns a DataFrame. No SparkSession argument, no side effects — the caller provides the DataFrame, the function applies the transform and hands it back. This pattern makes every function independently testable.
add_tax_bracket implements India's new-regime income tax slabs as a column derivation. The when(...).when(...).otherwise(...) chain is evaluated lazily — Spark builds a logical plan and only executes it when an action is called. clean_nulls filters out rows where income_inr is null, which would otherwise corrupt aggregations downstream.
A function that only transforms data — no global state, no side effects, same output for the same input — can be called with any DataFrame in any test. You don't need to mock a database connection, patch an environment variable, or spin up a service. The test is: "give this function this DataFrame, get that DataFrame back." That simplicity is the entire value of the pure-transform architecture.
test_transforms.py
Five tests covering the two functions. Read top to bottom: fixture first, then assertions, then the chispa full-DataFrame equality check at the end.
Walking through each test
scope="session" SparkSession
The @pytest.fixture(scope="session") decorator tells pytest to create this fixture once for the entire test session and share it across every test function that requests it. SparkSession startup involves JVM initialisation, Py4J socket setup, and Spark context creation — typically 10–20 seconds. Creating it once and reusing it keeps a 5-test suite under 10 seconds instead of over 2 minutes. yield s hands the session to the test, and s.stop() runs after all tests in the session complete.
Simple boundary assertions
test_tax_bracket_zero_percent creates a one-row DataFrame with income 250,000 INR (below the 300,000 threshold) and asserts the derived column equals "0%". test_tax_bracket_30_percent does the same for 2,000,000 INR (above 1,500,000 — the 30% bracket). The pattern df.first()["column"] calls the collect action and reads the first row. For single-row DataFrames this is clear and fast.
Null handling test
test_clean_nulls_removes_bad_rows feeds three rows to clean_nulls: incomes 500,000, None, and 800,000. The assertion clean_nulls(df).count() == 2 confirms the null row was dropped and the two valid rows survived. Note that None in a Python list passed to createDataFrame becomes a Spark null — this is the correct way to inject null values in test data.
Schema preservation test
test_add_tax_bracket_schema_preserved checks that the transform adds exactly one column and that the new column is named tax_bracket. This protects against a refactor that accidentally renames a column or adds an extra intermediate column. Schema tests are especially valuable in CI: if an upstream dependency changes a column type or name, this test fails before any data is processed.
chispa full-DataFrame equality
test_chispa_exact_match builds an expected DataFrame with the final schema (filing_id, income_inr, tax_bracket) and runs assert_df_equality(result, expected, ignore_nullable=True). This asserts row values, column names, column order, and data types in one call. The ignore_nullable=True flag is required here because the expected DataFrame was created from a Python list (all columns nullable=True by default), while the result DataFrame's tax_bracket column comes from a when().otherwise() chain which Spark marks nullable=False.
Running the tests
From the taxsetu-pipeline/ directory, run pytest with the verbose flag to see each test name and result individually:
4.68 seconds for 5 tests — the bulk of that time is the single SparkSession startup. Because scope="session" shares the session, adding 20 more tests adds roughly 0.1–0.3 seconds each, not another 4 seconds per test.
When you build a DataFrame from a Python list via createDataFrame([(1, 250_000, "0%"), ...], ["a","b","c"]), Spark infers all columns as nullable=True. When you derive a column from when(...).otherwise(...), Spark marks the result column as nullable=False (because otherwise guarantees a non-null value on every row). These two DataFrames hold identical data but differ in the nullable flag on tax_bracket. Without ignore_nullable=True, chispa reports a schema mismatch. The flag tells chispa: "the data must match exactly, but nullable differences are acceptable."
chispa vs manual assertions
A common first attempt at DataFrame equality is assert result.collect() == expected.collect(). This breaks in three ways:
- ✗ Row order is non-deterministic — Spark distributes data across partitions and the order of rows in
.collect()is not guaranteed across runs or machines. - ✗ No diff message — when the assertion fails, Python prints the raw list of Row objects. With 1,000 rows in a test DataFrame this is unreadable.
- ✗ Nullable mismatch is invisible — the data is correct but the nullable flag differs. A plain list comparison passes; a schema comparison would fail — but
.collect()doesn't check schemas.
chispa's assert_df_equality handles all three. It sorts both DataFrames before comparing, reports exact row-level diffs, and shows a schema diff table when the schemas differ.
Here is what a chispa failure message looks like when the expected tax bracket is wrong:
The failure message tells you exactly which row differed, what the actual and expected values were, and whether the schema also has a mismatch. Compare this to a raw AssertionError: False is not True from a manual assert — chispa saves debugging time on every failed test.
conftest.py — shared fixtures across multiple test files
When the test suite grows beyond one file, move the SparkSession fixture to conftest.py. pytest discovers this file automatically in the same directory and makes its fixtures available to every test file without an explicit import.
With conftest.py in place, you can delete the spark fixture from test_transforms.py and from any future test file — they all inherit it automatically. The SparkSession is still created once and shared across all test files in the session.
local[*] uses all available CPU cores. In CI environments, this can cause resource contention with other running jobs. local[2] limits Spark to 2 threads — fast enough for unit tests and predictable in shared CI runners.
CI integration
Adding tests to CI ensures every pull request is validated before merge. Meera's team at TaxSetu runs the full test suite on every push to any branch. The key environment requirement is JAVA_HOME — without it, SparkSession creation fails with JAVA_HOME is not set.
Add this step to your GitHub Actions workflow or GitLab CI stage:
Notes on the CI configuration:
- ▸
JAVA_HOME=/opt/java/openjdk— this path is correct for the GitHub Actionsubuntu-latestrunner which ships with OpenJDK via Temurin. Verify the path on your CI runner withwhich javaorecho $JAVA_HOME. - ▸ Pin package versions in CI (
pip install pyspark==3.5.0 chispa==0.10.1 pytest==8.3.0) to avoid test failures caused by a dependency upgrading between runs. - ▸
test_*.pydiscovers all test files matching the pattern — useful when you split tests by module (e.g.,test_transforms.py,test_aggregations.py,test_joins.py). - ▸ SparkSession startup in CI is slightly slower than locally (12–25 seconds) because CI runners have fewer cores and no JVM warm cache. A 5-test suite will take 20–30 seconds in CI — expected and acceptable.
Meera's testing principles
Three non-negotiable rules at TaxSetu
"At TaxSetu, Meera enforces three rules across every data engineering pull request:"
- Every transform function has at least one happy-path test. If the function exists, there is a test that calls it with valid input and asserts the expected output. No exceptions for "simple" functions.
- Every nullable column has a null-input test. If a column can be null in production data — and it can, because source systems are imperfect — there must be a test that passes a null value and asserts the correct behaviour (either filtering the row, defaulting to a sentinel value, or propagating null intentionally).
- If a bug reaches production, a test for that exact case is added before the fix. First write the failing test that reproduces the bug. Then fix the code. The test turns green. This is the red-green cycle applied to data pipelines and it prevents the same bug from recurring silently in a future refactor.
Common mistakes
scope="session", pytest creates a new SparkSession for every test function. SparkSession startup takes 10–20 seconds. Five tests become 60–120 seconds. Use scope="session" and share one session across all tests.assert result.collect() == expected.collect() is non-deterministic — it passes on your laptop and fails in CI. Use chispa's assert_df_equality which sorts both DataFrames before comparing.nullable=True for all columns. DataFrames produced by when().otherwise() have nullable=False for the derived column. Without the flag, chispa reports false failures even when the data is correct.Quiz — 3 questions
scope="session" for the SparkSession fixture?assert_df_equality is better than comparing .collect() lists directly because:ignore_nullable=True to a chispa assertion?Add a null test to every nullable column in your next transform.
Interview Q&A
The standard pattern is to write transformation functions as pure functions — each takes a DataFrame and returns a DataFrame, with no side effects and no global state. These functions are easy to unit test because you can create a small in-memory DataFrame with spark.createDataFrame(...), call the function, and assert on the result.
For the test infrastructure, we use pytest with a shared SparkSession fixture at scope="session" so the JVM is started once and shared across all tests. For assertions, we use chispa's assert_df_equality instead of comparing .collect() lists directly, because collect() order is non-deterministic and the raw list comparison gives no useful error message when it fails.
We test at least: a happy-path case, a null-input case for each nullable column, a schema preservation case (correct column count and names), and a boundary condition case for range-based logic like tax brackets.
chispa is a Python library purpose-built for asserting DataFrame equality in PySpark unit tests. Its primary function is assert_df_equality(actual, expected).
It is better than assert result.collect() == expected.collect() for three reasons: First, Spark does not guarantee row order across partitions — a direct list comparison is non-deterministic and can fail in CI even when the data is correct. chispa sorts both DataFrames before comparing. Second, when the assertion fails, chispa shows exactly which row differed (actual vs expected values) and whether the schema also has a mismatch. A plain AssertionError on a list of Row objects gives you no actionable information. Third, chispa exposes the ignore_nullable flag to handle the common case where one DataFrame was built from a Python list (nullable=True) and the other from a transformation (nullable=False) — the data is correct but the metadata differs.
Install with pip install chispa. Import with from chispa.dataframe_comparer import assert_df_equality.
The core requirement is a JDK on the CI runner — without it, SparkSession.builder.getOrCreate() fails immediately because PySpark spawns a JVM subprocess. GitHub Actions' ubuntu-latest runner ships with OpenJDK, but you need to set JAVA_HOME explicitly because PySpark reads that environment variable to locate the JVM binary.
The CI step is straightforward: export JAVA_HOME, install pyspark, chispa, and pytest (pinned to exact versions for reproducibility), then run pytest test_*.py -v. SparkSession startup in CI takes 12–25 seconds depending on the runner, so a 10-test suite typically completes in under 40 seconds.
For larger suites, you can cache the pip download artifacts between runs using GitHub Actions' cache action to avoid re-downloading ~200 MB of PySpark wheels on every push. This cuts CI time by 1–2 minutes on slow runners.
Key takeaways
- Write transformation functions as pure functions — DataFrame in, DataFrame out, no side effects. This makes every function independently testable without mocking infrastructure.
- Use a
scope="session"pytest fixture to share one SparkSession across all tests. SparkSession startup is 10–20 seconds; sharing it keeps a large test suite fast. - Use chispa's
assert_df_equality(result, expected, ignore_nullable=True)instead of comparing.collect()lists — row order is non-deterministic in Spark, and chispa gives readable diff messages on failure. - Test three cases for every transform: happy path, null input for each nullable column, and schema preservation (column count and names). These three patterns catch the majority of production PySpark bugs.
Build pipelines that ship — and stay green
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →