📋 What this lesson covers

This is your first hands-on PySpark lesson. You will write real code, get real output, and understand exactly what Spark is doing at each step. By the end you will have a working SparkSession and a DataFrame you built yourself from Nigerian e-commerce data.

Five things you will master

  • SparkSession — the single entry point to every Spark operation
  • createDataFrame — build a distributed DataFrame from Python data
  • show() — print a human-readable table to the driver console
  • printSchema() — inspect column names, types, and nullability
  • count() — trigger a full scan and return the row count

🧠 Why Spark's DataFrame API?

PySpark's DataFrame is not just a table — it is a distributed, lazy, column-oriented abstraction that spans as many machines as you have in your cluster. Understanding these three words is the key to understanding everything else in this track.

  • ▸ Distributed: When Spark reads a 500 GB file, it splits the data into partitions — chunks typically 128 MB each. Each partition lives on a different executor (a JVM process, possibly on a different machine). Your code never loops over rows; Spark does.
  • ▸ Column-oriented: Spark stores and processes each column independently (using Apache Arrow internally). Queries that touch 3 of 40 columns skip 37 columns entirely — no wasted I/O.
  • ▸ Lazy: When you call df.filter(...) or df.select(...), Spark records the intent but does nothing. The computation only starts when you call an action — show(), count(), collect(), or write. This allows Spark to optimise the full plan before executing.

pandas vs PySpark — when to use which

Dimension pandas PySpark
Machine scope Single machine only Single machine or cluster of hundreds
Execution model Eager — runs immediately on each line Lazy — builds a plan, runs on action
Data scale Limited to RAM (typically < 32 GB) TB-scale (data lives on disk/distributed storage)
Setup pip install pandas PySpark + JDK 11+ required
API feel Familiar Python, intuitive Similar to pandas — most methods have pandas equivalents
When to use Datasets that fit comfortably in RAM Data too large for one machine, or streaming pipelines
Rule of thumb: If your data fits in pandas, use pandas. Switch to PySpark when you hit memory limits, need streaming, or need to schedule jobs on a cluster.

🚀 SparkSession — the entry point

Every PySpark program starts with exactly one SparkSession. It is the gateway to the Spark execution engine — you cannot create a DataFrame, read a file, or run SQL without one.

Breaking down the builder options

Option What it means
.master("local[*]") Run Spark on the local machine, using all available CPU cores. On a 10-core laptop, Spark spawns 10 executor threads. In production, this is replaced with a cluster URL like yarn or k8s://....
.appName("NaijaCartOrders") A human-readable name shown in Spark's web UI (port 4040) and logs. Useful for debugging — especially when multiple jobs run on the same cluster.
.getOrCreate() Returns an existing session if one already exists in the same JVM, or creates a new one. Always use this instead of a raw constructor — it prevents accidentally spawning duplicate sessions in scripts and notebooks.
.config("spark.ui.enabled", "false") Disables the Spark UI web server (port 4040) — keeps output clean in scripts and CI environments.

Now let us create a SparkSession and build a DataFrame from five NaijaCart orders — a fictional Nigerian e-commerce platform.

python Lesson 03 · NaijaCart orders DataFrame
output
=== NaijaCart Orders DataFrame === +-------------+-------------+-----------+----------+ | order_id| city| category|amount_ngn| +-------------+-------------+-----------+----------+ |NaijaCart-001| Lagos|Electronics| 45000| |NaijaCart-002| Abuja| Fashion| 8900| |NaijaCart-003| Lagos|Electronics| 12000| |NaijaCart-004| Kano| Food| 2500| |NaijaCart-005|Port Harcourt| Home| 31000| +-------------+-------------+-----------+----------+ Row count : 5 Spark ver : 3.5.0
What is local[*]?

local[N] means "run on the local machine using N threads". local[*] means "use all CPU cores". local[1] means single-threaded (useful for debugging shuffle issues). local[4] forces exactly 4 threads. In any cloud or cluster environment, you replace local[*] with a cluster master URL — the rest of your code stays identical.

📖 Reading the output

The three lines of output tell you three different things about what Spark did.

show() — collect to driver, print a table

df.show() is an action. It triggers Spark to execute the plan, collect up to 20 rows onto the driver (your local Python process), and print them as a formatted ASCII table. By default, strings are truncated at 20 characters — pass truncate=False to see the full value. For large DataFrames, show() scans only as many partitions as needed to fill 20 rows, so it is safe to call on a 100 GB dataset.

count() — full scan of every partition

df.count() is also an action. Unlike show(), it must visit every row in every partition to produce an exact count. On a dataset with billions of rows spread across hundreds of partitions, this triggers a full distributed scan. Use it intentionally — not in tight loops.

spark.version

spark.version is a property on the SparkSession object. It returns the Spark version string — useful for debugging and for confirming your environment matches the lesson's pinned version (3.5.0).

🗂️ printSchema() — inspect your column types

Before you filter, aggregate, or join a DataFrame, always check its schema. printSchema() is a lightweight action that prints the column tree without touching your actual data rows.

python Lesson 03 · inspect the schema
output
root |-- order_id: string (nullable = true) |-- city: string (nullable = true) |-- category: string (nullable = true) |-- amount_ngn: long (nullable = true)

A few things worth noting about this output:

  • ▸ string — Spark mapped Python str to Spark's StringType.
  • ▸ long — Python integers become LongType (64-bit signed integer). This is always the default inference for Python int.
  • ▸ nullable = true — Spark defaults all inferred columns to nullable. Spark runs on the JVM, and the JVM allows any reference type to be null. Even though our Python data has no None values, Spark marks every column nullable unless you pass an explicit schema that disables it.
Pro tip: pass an explicit schema for production code

Schema inference from Python data works for small datasets and exploration. In production pipelines, pass a StructType schema explicitly: spark.createDataFrame(data, schema=StructType([...])). This prevents type surprises when data contains mixed types, and disables nullable where you need strict non-null guarantees.

📚 DataFrame concepts — quick reference

These four terms appear constantly in PySpark documentation, error messages, and interviews. Learn them now.

Concept What it means Example
Partition A chunk of a DataFrame's rows assigned to one executor. Spark splits data into partitions automatically. More partitions = more parallelism (up to the number of cores). A 1 GB Parquet file becomes ~8 partitions of ~128 MB each, processed in parallel across 8 cores.
RDD Resilient Distributed Dataset — the low-level distributed data structure that DataFrames are built on. You rarely use RDDs directly in modern PySpark; the DataFrame API is faster and safer. df.rdd exposes the underlying RDD if you need row-level Python functions.
Action An operation that triggers computation and returns a result to the driver (or writes to storage). Actions end the lazy evaluation chain. show(), count(), collect(), df.write.parquet(...)
Transformation An operation that defines a new DataFrame from an existing one — but does not run any computation yet. Spark records these as steps in a logical plan. df.filter(...), df.select(...), df.groupBy(...), df.withColumn(...)

💡 Key insight

Real-world scale

Arjun at NaijaCart processes 8 million orders per day. He creates one SparkSession, one DataFrame from a Parquet file on MinIO, and lets Spark split the work across 32 cores. The code looks almost identical to what you'd write for 5 rows — df.show(), df.count(), df.printSchema(). The only difference is the data source and cluster size. Spark's abstraction makes 5 rows and 8 million rows feel exactly the same to write.

⚠️ Common mistakes

Calling collect() on a huge DataFrame
collect() pulls every row from every partition onto the driver's memory. On a 50 GB DataFrame, this crashes your Python process with an out-of-memory error. Use show() for inspection, or write to storage — never collect() on production-scale data.
Creating a new SparkSession per script
Calling SparkSession.builder... .getOrCreate() is safe and cheap — if a session already exists it returns the existing one. But never call spark.stop() mid-workflow and rebuild. Each session start spins up a JVM (~10 s overhead) and discards all cached DataFrames. Use getOrCreate() everywhere and stop only at program exit.
Not setting JAVA_HOME
PySpark spawns a JVM subprocess using the binary at $JAVA_HOME/bin/java. If JAVA_HOME is not set, Spark raises JAVA_HOME is not set or No Java runtime present and exits immediately. Fix: set os.environ["JAVA_HOME"] at the top of your script, or add it permanently to ~/.zshrc. JDK 11+ is required; JDK 21 LTS is recommended.

🧠 Quiz — 3 questions

Question 1 of 3
What does .master("local[*]") mean in a SparkSession?
Question 2 of 3
Which of these is an action (not a transformation)?
Question 3 of 3
What does printSchema() show?
0
/ 3
DataFrame basics locked in.

Next up: transformations and actions in depth.

🎤 Interview Q&A

Easy What is a SparkSession and why do you use getOrCreate()?
+

SparkSession is the unified entry point to Spark's execution engine introduced in Spark 2.0. It wraps the older SparkContext, SQLContext, and HiveContext into a single object. You use it to create DataFrames, run SQL queries, read files, and configure Spark properties.

You use getOrCreate() instead of a direct constructor because Spark allows only one active SparkContext per JVM. If you call SparkSession.builder...getOrCreate() and a session already exists (e.g., from a previous cell in a notebook, or from a shared cluster), it returns the existing session without spinning up a new JVM. Creating a second SparkContext in the same JVM raises an error. getOrCreate() is therefore the safe, idiomatic pattern for every PySpark program.

Medium What is the difference between an RDD and a DataFrame?
+

An RDD (Resilient Distributed Dataset) is Spark's low-level abstraction introduced in Spark 1.0. It is an immutable distributed collection of JVM objects, processed row by row in parallel. RDDs give you full control — you write Python functions that Spark serialises and ships to executors — but they are slow because Spark cannot optimise opaque Python functions, and every object is serialised/deserialised over the wire.

A DataFrame is a higher-level abstraction built on top of RDDs. It organises data into named columns with schema information, and uses Spark's Catalyst query optimiser to generate an optimised physical execution plan before running. The optimiser can reorder filters, push predicates into data sources, and choose efficient join strategies — none of which is possible with raw RDDs. In practice, DataFrames are 2–10x faster than equivalent RDD code for structured data, and the API is much easier to write and read. Modern PySpark code almost always uses DataFrames or the SQL API, never raw RDDs.

Medium Why does Spark need a JVM? Can you run PySpark without Java?
+

Apache Spark is written in Scala, a JVM language. The execution engine — scheduling, memory management, shuffle, Catalyst optimiser — all run inside the JVM. When you start a SparkSession in Python, PySpark launches a child JVM process using the binary at $JAVA_HOME/bin/java. Python communicates with the JVM through Py4J, a bridge library that translates Python object calls into JVM method invocations over a local socket.

No, you cannot run PySpark without Java. Even if you install the pyspark pip package, calling SparkSession.builder.getOrCreate() will immediately raise an error if the JVM binary is not found. You need JDK 11 at minimum; JDK 17 or 21 LTS are the recommended versions for PySpark 3.5. The pyspark pip package does not bundle a JDK — you must install one separately.

✅ Key takeaways

  • Every PySpark program starts with exactly one SparkSession. Use getOrCreate() — not a raw constructor — to avoid spawning duplicate JVMs.
  • local[*] runs Spark on your laptop using all CPU cores. Replace it with a cluster URL in production — the rest of your code is identical.
  • Transformations (filter, select, withColumn) are lazy. Actions (show, count, collect, write) trigger computation. Knowing the difference prevents expensive surprises.
  • printSchema() is always safe and cheap — call it before any transformation to confirm your column types are what you expect.

🎯 Challenge

Extend the NaijaCart DataFrame — two tasks
Task 1 — Add a discount_pct column

Using df.withColumn() and pyspark.sql.functions.when(), add a new column discount_pct that is 10 for rows where category == "Electronics" and 5 for all other categories. Call show() on the result to verify.

Hint: from pyspark.sql.functions import when, col — then df.withColumn("discount_pct", when(col("category") == "Electronics", 10).otherwise(5))

Task 2 — Filter to Lagos orders only

Apply df.filter(col("city") == "Lagos") to the original DataFrame and call show(). Confirm the result has exactly 2 rows (NaijaCart-001 and NaijaCart-003). Then call count() on the filtered DataFrame to verify programmatically.

Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

From your first DataFrame to a production Spark pipeline

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →