📋 What this lesson covers
This is your first hands-on PySpark lesson. You will write real code, get real output, and understand exactly what Spark is doing at each step. By the end you will have a working SparkSession and a DataFrame you built yourself from Nigerian e-commerce data.
Five things you will master
- SparkSession — the single entry point to every Spark operation
- createDataFrame — build a distributed DataFrame from Python data
- show() — print a human-readable table to the driver console
- printSchema() — inspect column names, types, and nullability
- count() — trigger a full scan and return the row count
🧠 Why Spark's DataFrame API?
PySpark's DataFrame is not just a table — it is a distributed, lazy, column-oriented abstraction that spans as many machines as you have in your cluster. Understanding these three words is the key to understanding everything else in this track.
- ▸ Distributed: When Spark reads a 500 GB file, it splits the data into partitions — chunks typically 128 MB each. Each partition lives on a different executor (a JVM process, possibly on a different machine). Your code never loops over rows; Spark does.
- ▸ Column-oriented: Spark stores and processes each column independently (using Apache Arrow internally). Queries that touch 3 of 40 columns skip 37 columns entirely — no wasted I/O.
- ▸ Lazy: When you call
df.filter(...)ordf.select(...), Spark records the intent but does nothing. The computation only starts when you call an action —show(),count(),collect(), orwrite. This allows Spark to optimise the full plan before executing.
pandas vs PySpark — when to use which
| Dimension | pandas | PySpark |
|---|---|---|
| Machine scope | Single machine only | Single machine or cluster of hundreds |
| Execution model | Eager — runs immediately on each line | Lazy — builds a plan, runs on action |
| Data scale | Limited to RAM (typically < 32 GB) | TB-scale (data lives on disk/distributed storage) |
| Setup | pip install pandas |
PySpark + JDK 11+ required |
| API feel | Familiar Python, intuitive | Similar to pandas — most methods have pandas equivalents |
| When to use | Datasets that fit comfortably in RAM | Data too large for one machine, or streaming pipelines |
🚀 SparkSession — the entry point
Every PySpark program starts with exactly one SparkSession. It is the gateway to the Spark execution engine — you cannot create a DataFrame, read a file, or run SQL without one.
Breaking down the builder options
| Option | What it means |
|---|---|
.master("local[*]") |
Run Spark on the local machine, using all available CPU cores. On a 10-core laptop, Spark spawns 10 executor threads. In production, this is replaced with a cluster URL like yarn or k8s://.... |
.appName("NaijaCartOrders") |
A human-readable name shown in Spark's web UI (port 4040) and logs. Useful for debugging — especially when multiple jobs run on the same cluster. |
.getOrCreate() |
Returns an existing session if one already exists in the same JVM, or creates a new one. Always use this instead of a raw constructor — it prevents accidentally spawning duplicate sessions in scripts and notebooks. |
.config("spark.ui.enabled", "false") |
Disables the Spark UI web server (port 4040) — keeps output clean in scripts and CI environments. |
Now let us create a SparkSession and build a DataFrame from five NaijaCart orders — a fictional Nigerian e-commerce platform.
local[*]?local[N] means "run on the local machine using N threads". local[*] means "use all CPU cores". local[1] means single-threaded (useful for debugging shuffle issues). local[4] forces exactly 4 threads. In any cloud or cluster environment, you replace local[*] with a cluster master URL — the rest of your code stays identical.
📖 Reading the output
The three lines of output tell you three different things about what Spark did.
show() — collect to driver, print a table
df.show() is an action. It triggers Spark to execute the plan, collect up to 20 rows onto the driver (your local Python process), and print them as a formatted ASCII table. By default, strings are truncated at 20 characters — pass truncate=False to see the full value. For large DataFrames, show() scans only as many partitions as needed to fill 20 rows, so it is safe to call on a 100 GB dataset.
count() — full scan of every partition
df.count() is also an action. Unlike show(), it must visit every row in every partition to produce an exact count. On a dataset with billions of rows spread across hundreds of partitions, this triggers a full distributed scan. Use it intentionally — not in tight loops.
spark.version
spark.version is a property on the SparkSession object. It returns the Spark version string — useful for debugging and for confirming your environment matches the lesson's pinned version (3.5.0).
🗂️ printSchema() — inspect your column types
Before you filter, aggregate, or join a DataFrame, always check its schema. printSchema() is a lightweight action that prints the column tree without touching your actual data rows.
A few things worth noting about this output:
- ▸ string — Spark mapped Python
strto Spark'sStringType. - ▸ long — Python integers become
LongType(64-bit signed integer). This is always the default inference for Pythonint. - ▸ nullable = true — Spark defaults all inferred columns to nullable. Spark runs on the JVM, and the JVM allows any reference type to be
null. Even though our Python data has noNonevalues, Spark marks every column nullable unless you pass an explicit schema that disables it.
Schema inference from Python data works for small datasets and exploration. In production pipelines, pass a StructType schema explicitly: spark.createDataFrame(data, schema=StructType([...])). This prevents type surprises when data contains mixed types, and disables nullable where you need strict non-null guarantees.
📚 DataFrame concepts — quick reference
These four terms appear constantly in PySpark documentation, error messages, and interviews. Learn them now.
| Concept | What it means | Example |
|---|---|---|
| Partition | A chunk of a DataFrame's rows assigned to one executor. Spark splits data into partitions automatically. More partitions = more parallelism (up to the number of cores). | A 1 GB Parquet file becomes ~8 partitions of ~128 MB each, processed in parallel across 8 cores. |
| RDD | Resilient Distributed Dataset — the low-level distributed data structure that DataFrames are built on. You rarely use RDDs directly in modern PySpark; the DataFrame API is faster and safer. | df.rdd exposes the underlying RDD if you need row-level Python functions. |
| Action | An operation that triggers computation and returns a result to the driver (or writes to storage). Actions end the lazy evaluation chain. | show(), count(), collect(), df.write.parquet(...) |
| Transformation | An operation that defines a new DataFrame from an existing one — but does not run any computation yet. Spark records these as steps in a logical plan. | df.filter(...), df.select(...), df.groupBy(...), df.withColumn(...) |
💡 Key insight
Arjun at NaijaCart processes 8 million orders per day. He creates one SparkSession, one DataFrame from a Parquet file on MinIO, and lets Spark split the work across 32 cores. The code looks almost identical to what you'd write for 5 rows — df.show(), df.count(), df.printSchema(). The only difference is the data source and cluster size. Spark's abstraction makes 5 rows and 8 million rows feel exactly the same to write.
⚠️ Common mistakes
collect() pulls every row from every partition onto the driver's memory. On a 50 GB DataFrame, this crashes your Python process with an out-of-memory error. Use show() for inspection, or write to storage — never collect() on production-scale data.SparkSession.builder... .getOrCreate() is safe and cheap — if a session already exists it returns the existing one. But never call spark.stop() mid-workflow and rebuild. Each session start spins up a JVM (~10 s overhead) and discards all cached DataFrames. Use getOrCreate() everywhere and stop only at program exit.$JAVA_HOME/bin/java. If JAVA_HOME is not set, Spark raises JAVA_HOME is not set or No Java runtime present and exits immediately. Fix: set os.environ["JAVA_HOME"] at the top of your script, or add it permanently to ~/.zshrc. JDK 11+ is required; JDK 21 LTS is recommended.🧠 Quiz — 3 questions
.master("local[*]") mean in a SparkSession?printSchema() show?Next up: transformations and actions in depth.
🎤 Interview Q&A
SparkSession is the unified entry point to Spark's execution engine introduced in Spark 2.0. It wraps the older SparkContext, SQLContext, and HiveContext into a single object. You use it to create DataFrames, run SQL queries, read files, and configure Spark properties.
You use getOrCreate() instead of a direct constructor because Spark allows only one active SparkContext per JVM. If you call SparkSession.builder...getOrCreate() and a session already exists (e.g., from a previous cell in a notebook, or from a shared cluster), it returns the existing session without spinning up a new JVM. Creating a second SparkContext in the same JVM raises an error. getOrCreate() is therefore the safe, idiomatic pattern for every PySpark program.
An RDD (Resilient Distributed Dataset) is Spark's low-level abstraction introduced in Spark 1.0. It is an immutable distributed collection of JVM objects, processed row by row in parallel. RDDs give you full control — you write Python functions that Spark serialises and ships to executors — but they are slow because Spark cannot optimise opaque Python functions, and every object is serialised/deserialised over the wire.
A DataFrame is a higher-level abstraction built on top of RDDs. It organises data into named columns with schema information, and uses Spark's Catalyst query optimiser to generate an optimised physical execution plan before running. The optimiser can reorder filters, push predicates into data sources, and choose efficient join strategies — none of which is possible with raw RDDs. In practice, DataFrames are 2–10x faster than equivalent RDD code for structured data, and the API is much easier to write and read. Modern PySpark code almost always uses DataFrames or the SQL API, never raw RDDs.
Apache Spark is written in Scala, a JVM language. The execution engine — scheduling, memory management, shuffle, Catalyst optimiser — all run inside the JVM. When you start a SparkSession in Python, PySpark launches a child JVM process using the binary at $JAVA_HOME/bin/java. Python communicates with the JVM through Py4J, a bridge library that translates Python object calls into JVM method invocations over a local socket.
No, you cannot run PySpark without Java. Even if you install the pyspark pip package, calling SparkSession.builder.getOrCreate() will immediately raise an error if the JVM binary is not found. You need JDK 11 at minimum; JDK 17 or 21 LTS are the recommended versions for PySpark 3.5. The pyspark pip package does not bundle a JDK — you must install one separately.
✅ Key takeaways
- Every PySpark program starts with exactly one
SparkSession. UsegetOrCreate()— not a raw constructor — to avoid spawning duplicate JVMs. local[*]runs Spark on your laptop using all CPU cores. Replace it with a cluster URL in production — the rest of your code is identical.- Transformations (
filter,select,withColumn) are lazy. Actions (show,count,collect,write) trigger computation. Knowing the difference prevents expensive surprises. printSchema()is always safe and cheap — call it before any transformation to confirm your column types are what you expect.
🎯 Challenge
Using df.withColumn() and pyspark.sql.functions.when(), add a new column discount_pct that is 10 for rows where category == "Electronics" and 5 for all other categories. Call show() on the result to verify.
Hint: from pyspark.sql.functions import when, col — then df.withColumn("discount_pct", when(col("category") == "Electronics", 10).otherwise(5))
Apply df.filter(col("city") == "Lagos") to the original DataFrame and call show(). Confirm the result has exactly 2 rows (NaijaCart-001 and NaijaCart-003). Then call count() on the filtered DataFrame to verify programmatically.
From your first DataFrame to a production Spark pipeline
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →