📐 Row vs Columnar Storage
The single most important difference between file formats is not which tool writes them — it is how data is laid out on disk. Row-based formats (CSV, JSON, Avro) write one complete record at a time. Columnar formats (Parquet, ORC) write all values for a single column together, across many rows.
SELECT AVG(amount_ngn) FROM orders WHERE status = 'delivered', a columnar format reads only two columns out of six. A row format reads all six for every row. At 100 million rows that difference is measured in minutes, not milliseconds.
📊 Format Overview
Five formats cover the vast majority of production data engineering work. The table below shows how they differ on storage model, schema handling, compression, and ideal use case.
| Format | Storage | Schema | Compression | Best for |
|---|---|---|---|---|
| CSV | Row | No (inferred) | None | Human-readable export, one-time interchange |
| JSON | Row | No | None | API responses, logging |
| Parquet | Columnar | Embedded | Snappy / ZSTD | Analytics, data lakes, Spark pipelines |
| Avro | Row | Embedded | Deflate | Kafka serialization, event streaming |
| ORC | Columnar | Embedded | ZLIB / Snappy | Hive-heavy pipelines, older Hadoop stacks |
Chidi's batch pipeline writes Parquet to the data lake (MinIO). Kafka topics carry Avro messages from the mobile app. The legacy Hive reporting cluster still reads ORC. CSV only appears at the very edge — when Finance exports a report to send to a partner by email.
🔥 Parquet Benchmark — 100k NaijaCart Orders
This benchmark writes 100,000 synthetic NaijaCart orders in both CSV and Parquet, then reads and filters both. Run this on your machine — the output will be close to the verified output below.
When filtering by status == 'delivered', Parquet reads only the status column from disk — skipping order_id, city, category, amount_ngn, and order_date entirely. That is 5 out of 6 columns ignored. CSV has no concept of columns on disk: every byte of every row must be parsed before filtering can happen.
Parquet also compresses each column independently using Snappy. String columns with only 3 distinct values like status compress to near-nothing. Numeric columns stored as doubles use delta encoding. The result is 4.2× smaller on disk and 4.5× faster to query.
🏆 Why Parquet Wins at Analytics
Three Spark optimisations are only possible because Parquet stores data in a columnar, structured binary format. CSV cannot support any of them.
-
1
Column pruning — Spark reads only the columns referenced in your
SELECT. If your query touches 3 of 50 columns, Parquet serves 3 columns' worth of bytes. For wide tables (50–200 columns), this alone produces 10–20× faster scans. -
2
Predicate pushdown — Parquet stores column statistics (min, max, null count) per row-group (typically 128 MB chunks). When Spark applies a filter like
amount_ngn > 100000, it reads the footer statistics first. If a row-group'samount_ngnmax is 90000, Spark skips the entire row-group without reading it. No equivalent is possible with CSV. -
3
Schema embedded in footer — The Parquet file footer contains the full schema.
spark.read.parquet(...)reads the schema in milliseconds without scanning data. CSV requires either an expensiveinferSchema=Truepass (reads the whole file to guess types) or a brittle manual schema declaration that can silently miscast columns.
📨 Avro in Streaming Pipelines
Avro is row-based — not columnar — which makes it a poor choice for analytics queries. So why does NaijaCart's Kafka pipeline use it for every event message? Because Avro was designed for exactly this problem: fast serialization of individual records with guaranteed schema compatibility.
Every Avro file (and every Kafka Avro message) carries the writer schema embedded in the payload or as a compact schema ID that points to a schema registry. This means:
- ▸ The consumer always knows the exact field names and types. No guessing, no
inferSchema. - ▸ Schema evolution is controlled: adding a field with a default does not break existing consumers. Removing a required field breaks them — and the schema registry enforces compatibility rules to prevent that.
- ▸ When the mobile app team changes the
order_createdevent to add apromo_codefield, they register the new Avro schema. Old consumers (that do not know aboutpromo_code) continue working without redeployment.
Avro + Kafka Schema Registry — how it works
NaijaCart's mobile app publishes order_created events to Kafka. Each message is Avro-serialized. The first 5 bytes of every message are a magic byte (0x00) and a 4-byte schema ID. The Kafka consumer reads the schema ID, looks up the full Avro schema from the schema registry over HTTP, and deserializes the rest of the message using that schema. The schema fetch is cached — subsequent messages with the same schema ID deserialize without any network call.
For replay scenarios (re-consuming old Kafka events), Avro is the gold standard. The schema stored alongside the data guarantees that events written 18 months ago can still be decoded exactly — even if the current schema has evolved.
🏛️ ORC — When You Will See It
ORC (Optimised Row Columnar) is Parquet's older sibling, developed at Hortonworks for Hive. Like Parquet it is columnar with embedded schema, but it uses ZLIB compression by default instead of Snappy.
| Dimension | Parquet | ORC |
|---|---|---|
| Default compression | Snappy (fast, moderate ratio) | ZLIB (slower, better ratio) |
| Primary ecosystem | Spark, Flink, Arrow, Hive | Hive, older Hadoop stacks |
| Predicate pushdown | Yes — row-group statistics | Yes — stripe statistics |
| ACID support | No (handled outside format) | Yes — Hive ACID transactions use ORC |
| When to choose | New Spark pipelines, data lakes | Joining Spark to existing Hive tables |
If Chidi is building a new pipeline today he reaches for Parquet. If he is reading from an existing Hive warehouse that predates the Spark migration, those tables are almost certainly ORC. Spark reads both with identical syntax — just swap .parquet(...) for .orc(...).
📝 Writing and Reading Patterns
The PySpark API for reading and writing formats is consistent. Swap the format method; the rest stays the same.
inferSchema=True on CSV?Spark reads the entire CSV file twice when inferSchema=True: once to sample types, once to actually load data. On a 50 GB CSV this doubles your read time and your cluster costs. Always declare the schema explicitly for CSV in production pipelines. Parquet and ORC never have this problem — the schema is in the file footer and read in a single metadata fetch.
🗂️ Tunde's Partition Strategy
"At NaijaCart we partition Parquet by year and month on write:"
df.write.partitionBy('year', 'month').parquet('s3a://bucket/orders/')
"Queries scoped to one month — like the end-of-month finance report — skip 11/12 of the data entirely. Spark reads the directory listing, sees year=2024/month=01/ partitions, and touches only those folders. The other 11 months of Parquet files are never opened. At 200 GB per month, skipping 11 months means reading 200 GB instead of 2.4 TB. That is the real 10× win — not the compression."
Partition columns should be low-cardinality fields used frequently in WHERE clauses. Year/month is the classic choice for time-series data. City or category also work if your queries are always region-scoped. Avoid partitioning on high-cardinality columns like order_id — you would create millions of tiny files, which causes a different class of performance problem.
⚠️ Common Mistakes
spark-avro JAR or use --packages org.apache.spark:spark-avro_2.12:3.5.0 when submitting. Forgetting this gives a confusing runtime error: Failed to find data source: avro. Add the package at session build time in production configs..option("parquet.writer.version", "v1") in cross-team pipelines, and test schema evolution scenarios explicitly before shipping.🧠 Quiz — 3 questions
Now partition your next Parquet table by date and feel the speedup.
🎤 Interview Q&A
Predicate pushdown is an optimisation where the query engine evaluates filter conditions as early as possible in the data access pipeline — ideally before reading any row data. In Parquet and ORC, each file is split into row-groups (Parquet) or stripes (ORC). The file footer stores column-level statistics for each row-group: minimum value, maximum value, and null count. When Spark applies a filter like amount_ngn > 100000, it first reads the footer statistics. If a row-group's maximum amount_ngn is 90000, that entire row-group is skipped — no bytes from that row-group are read from disk or object storage.
CSV and JSON do not support predicate pushdown because they have no block-level statistics — every byte must be read and parsed before filtering can begin. Avro is row-based and also does not benefit from predicate pushdown in the same way. For analytics workloads with selective filters, predicate pushdown on Parquet can reduce IO by 80–95% when filters are highly selective and data is well-ordered within row-groups.
Parquet supports schema evolution through column merging. Spark's mergeSchema option reads schemas from all Parquet files in a directory and produces the union schema. New columns added in later files appear as null for older files. Columns removed from later files still appear — populated from older files.
To enable this: spark.read.option("mergeSchema", "true").parquet("s3a://bucket/orders/"). You can also set it globally with spark.sql.parquet.mergeSchema = true.
Best practices: (1) Only add nullable columns — never remove or rename existing columns, as this breaks readers. (2) Store schema separately in a schema registry or metadata table and enforce it on write. (3) Use partition-based evolution: new partitions (e.g. year=2025) can introduce new columns without touching old partitions. The cost of mergeSchema=true is listing all file footers, which is slow on large tables — avoid it in production hot paths and instead enforce a stable schema on write.
Choose Avro over Parquet in two scenarios:
1. Event streaming with Kafka. Parquet is a columnar batch format — it is designed to batch many rows together to achieve compression gains. A single Kafka message is one event (one row). Writing one-row Parquet files is wasteful and produces enormous file count. Avro serializes a single record efficiently, embeds its schema (or references a schema ID), and deserializes fast. The schema registry integration for forward and backward compatibility is a first-class Avro feature with no Parquet equivalent.
2. Write-heavy pipelines where you need full row access. Because Avro is row-based, writing and reading entire rows is efficient. If your pipeline always reads every column (e.g. a replication pipeline, a changelog feed), Avro avoids the overhead of reassembling rows from column stripes. In practice this is rare in analytics — most analytical queries benefit from column pruning, so Parquet wins for analytics even when you access many columns.
✅ Key Takeaways
- Columnar formats (Parquet, ORC) read only the columns your query needs. At 100k rows, Parquet is 4.2× smaller than CSV and 4.5× faster to filter — the gains scale with data volume.
- Parquet is the default choice for Spark analytics and data lakes: Snappy compression, embedded schema, predicate pushdown, and column pruning all come for free. No extra configuration needed.
- Avro is the right format for Kafka: row-based serialization of individual events, embedded schema, and schema registry integration for safe schema evolution across independent producers and consumers.
- ORC is Parquet's columnar cousin — favoured by Hive-heavy stacks and slightly smaller with ZLIB compression. If you are reading existing Hive tables in Spark, they are probably ORC. New pipelines should default to Parquet.
Build pipelines that actually run in production
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →