🗂️ Three operations, one mental model
Every PySpark job reads a DataFrame and applies transformations before writing results. Three transformations cover the vast majority of day-to-day work at TaxSetu's data platform: filter() trims rows, select() trims columns, and withColumn() computes new columns. If you know these three, most pipeline logic clicks immediately.
| Operation | What it does | Returns | pandas equivalent |
|---|---|---|---|
| filter() | Keeps rows matching a condition | DataFrame with fewer rows | df[df.col > x] |
| select() | Keeps only named columns | DataFrame with fewer columns | df[['col1','col2']] |
| withColumn() | Adds or replaces a column | DataFrame with same rows, +1 col | df.assign(new=...) |
None of these three operations triggers actual computation. PySpark builds a logical plan and waits. Only a terminal action — show(), count(), write — kicks off execution. You can chain dozens of filters, selects, and withColumns before Spark reads a single byte of data.
🔍 filter() — keeping the rows you need
Arjun, TaxSetu's lead data engineer, often starts every pipeline with a filter(). Filtering early means every downstream transformation — withColumn, aggregations, joins — works on a smaller dataset. The code below loads TaxSetu's filing data and isolates high-income filers (above ₹20 lakh).
Three things to know about filter()
col() vs string column names. You can write either df.filter(col("income_inr") > 2_000_000) or df.filter("income_inr > 2000000") (SQL string). The col() version is preferred when you need to compose expressions — multiply, alias, nest functions — because col() returns a Column object that supports Python operators. The SQL string version is handy for quick one-liners.
Multiple conditions. Chain conditions using & (AND) and | (OR), always wrapping each condition in parentheses. For example, to keep only Delhi salaried filers:
filter() is a narrow transformation. Each output row depends on exactly one input row (no data needs to move between partitions). This means filter() never triggers a shuffle — it is extremely cheap even on hundreds of millions of rows. This is one reason Arjun always pushes filters as early in the plan as possible.
df.filter(...) and df.where(...) compile to the same logical plan. Use whichever reads more naturally to your team. Most TaxSetu engineers default to filter() to match the rest of their Python codebase.
📋 select() — projecting the columns you need
Priya uses select() at two places in every pipeline: right after reading raw data (to drop irrelevant columns early) and at the very end (to produce a clean output schema). Three patterns cover 95% of cases.
Pattern 1 (strings) is the fastest to type and works whenever you only need to pick existing columns without any arithmetic. Internally Spark resolves the string to a Column reference during analysis, so there is no performance difference between strings and col() for plain column selection.
Pattern 2 (expression without alias) is useful in interactive exploration where you just want to eyeball a computed value, but the auto-generated column name is ugly in production output — always prefer Pattern 3 for data you will write to storage.
Pattern 3 (expression with alias) is the production standard. You can put as many computed columns as you want inside a single select() call, making it more efficient than chaining multiple withColumn() calls when you need many new columns at once.
You can freely mix: df.select("filing_id", col("income_inr") * 1.18) is valid. PySpark coerces plain strings to Column references internally. The only rule: if you want to apply an operator or function, you must use col() — a plain string is just a name reference, not a Column object.
🏷️ withColumn() and when/otherwise
Every TaxSetu filing needs a tax bracket tag before the data is loaded into the reporting database. withColumn() adds that column without touching the rest of the DataFrame. The when().when().otherwise() chain is PySpark's equivalent of a Python if/elif/else — but it runs distributed across all partitions simultaneously.
How the when/otherwise chain works
Spark evaluates each when(condition, value) clause from top to bottom and returns the value of the first condition that is true — just like elif in Python. The .otherwise(value) at the end is the catch-all: if no when clause matched, the row gets that value. If you omit .otherwise(), rows that fall through every condition get null — a silent bug that causes downstream aggregation errors.
withColumn() always produces a new DataFrame. If the column name you pass already exists in the DataFrame, Spark replaces that column in the output schema. If the name is new, a new column is appended. Use drop("column_name") on the result if you want to remove an old column after replacing it.
Each withColumn() call adds one column and creates a new logical plan node. If you need to add five computed columns, chaining five withColumn() calls works but creates five plan nodes. Spark's Catalyst optimizer usually collapses them, but for very wide pipelines (>50 columns) Priya's team switches to a single select() that lists all original columns plus all new computed ones — it produces the same result with a leaner plan.
🔗 Combining all three
In practice you rarely use these operations in isolation. A typical TaxSetu pipeline reads raw filings, filters to the population of interest, enriches with computed columns, and then selects only the fields the downstream table needs. All three operations chain naturally because each returns a DataFrame.
Notice that the entire chain — filter, withColumn, select, orderBy — is wrapped in a single set of parentheses. This is a PySpark convention for readable chaining. Every step is still a lazy transformation; Spark builds the full plan and only starts computing when show() is called at the end.
orderBy() is a wide transformation (it requires a full sort across partitions) — but all the filtering and column work before it happens lazily and cheaply first. Spark's Catalyst optimizer pushes the filter as close to the data source as possible, so by the time the sort runs, it is sorting a much smaller dataset.
"At TaxSetu we always filter() before withColumn(). Fewer rows means withColumn() touches less data. If you filter 90% of rows out first, the when().otherwise() expression only executes on the remaining 10%. On a 500-million-row filing dataset that difference is minutes of compute time."
⚠️ Common mistakes
df.filter(if income > x: ...) does not work — Python's if is not a Column expression and PySpark cannot distribute it across partitions. Use col() expressions for simple conditions and when().otherwise() for conditional values. The driver (your Python process) cannot evaluate per-row logic; Spark workers can.when().when() chain without a final .otherwise() returns null for any row that does not match any condition. This is silent — no error, no warning. Downstream groupBy() and count() calls happily count nulls, producing wrong numbers. Always close every when chain with .otherwise(some_default).df.select("income_inr" * 2) raises a TypeError — Python tries to multiply a string by an integer. The Column object is what supports math operators. The fix is always df.select(col("income_inr") * 2). Use strings only for plain column references with no arithmetic; wrap in col() the moment you apply any operator or function.🧠 Quiz — 3 questions
Next up: aggregations and groupBy — where filter + select lay the groundwork.
🎤 Interview Q&A
There is no functional difference. filter() and where() are exact aliases in PySpark — they are both defined in the DataFrame API and compile to the same logical plan node (Filter). The two names exist because PySpark aims to feel natural to both Python programmers (who know filter() from Python builtins and pandas) and SQL users (who know WHERE clauses). Pick whichever your team uses consistently. Most PySpark-first teams default to filter(); teams that write a lot of SQL alongside PySpark often prefer where() for visual consistency with their SQL queries.
You have two options. The first is chaining five withColumn() calls. This works and Spark's Catalyst optimizer usually collapses adjacent withColumn nodes into a single projection in the physical plan, so the performance difference is often negligible for modest numbers of columns. The second option — preferred for wide DataFrames or many computed columns — is a single select() that lists every original column you want to keep (by name or col("*") is not valid; you must list them) plus each new computed column with an alias. A single select() with all projections in one call creates exactly one plan node regardless of how many new columns are added. For adding more than ~10 computed columns, the select() approach produces a cleaner plan and avoids a known Spark issue where very long chains of withColumn() calls cause stack overflows during plan analysis in older Spark versions (pre-3.4).
col("name") returns a Column object — a Spark expression node that understands Python operators and PySpark function calls. A plain string like "income_inr" is just a Python string; it carries no Spark semantics. When you pass a plain string to select() or filter(), Spark internally converts it to a Column reference during plan analysis — so for plain column selection there is no visible difference. The critical distinction appears the moment you apply any operator: "income_inr" * 1.18 raises a Python TypeError (string times float), while col("income_inr") * 1.18 returns a new Column expression that Spark can push down to executors. Use col() whenever you need to do arithmetic, comparisons, function applications, or aliasing; use plain strings only when you just want to reference a column by name with no transformation.
✅ Key takeaways
filter()is a narrow transformation — it drops rows locally on each partition with no shuffle, making it very cheap. Apply it as early as possible to shrink the dataset before more expensive operations.select()projects columns: use plain strings for name-only references,col()wrapped expressions for arithmetic, and always finish with.alias()to give computed columns clean names in the output schema.withColumn()adds or replaces exactly one column per call. Close everywhen().when()chain with.otherwise(default)to avoid silent nulls in rows that match no condition.- All three operations are lazy transformations — they build Spark's logical plan but execute nothing until an action (
show(),write,count()) is called. Chain them freely and Spark's Catalyst optimizer will merge projections and push filters toward the data source.
The foundation our data platform engineers stand on
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →