📐 Row vs Columnar Storage

The single most important difference between file formats is not which tool writes them — it is how data is laid out on disk. Row-based formats (CSV, JSON, Avro) write one complete record at a time. Columnar formats (Parquet, ORC) write all values for a single column together, across many rows.

CSV / JSON / Avro — Row Storage
Disk blocks (each row together) ORD001 | Lagos | Electronics | 45000 ORD002 | Abuja | Fashion | 12000 ORD003 | Kano | Food | 3500 Query on "amount" → reads EVERY column
Read entire row even for a 1-column query. No compression across rows. Every disk read pulls columns you do not need.
Parquet / ORC — Columnar Storage
Disk blocks (each column together) order_id ORD001 ORD002 ORD003 city Lagos Abuja Kano amount 45000 12000 3500 status delivered pending cancelled Query on "amount" → reads ONLY amount column
Read only the columns in your SELECT. Same-type values compress heavily together. Spark skips entire row-groups via predicate pushdown.
The key insight: When Tunde runs SELECT AVG(amount_ngn) FROM orders WHERE status = 'delivered', a columnar format reads only two columns out of six. A row format reads all six for every row. At 100 million rows that difference is measured in minutes, not milliseconds.

📊 Format Overview

Five formats cover the vast majority of production data engineering work. The table below shows how they differ on storage model, schema handling, compression, and ideal use case.

Format Storage Schema Compression Best for
CSV Row No (inferred) None Human-readable export, one-time interchange
JSON Row No None API responses, logging
Parquet Columnar Embedded Snappy / ZSTD Analytics, data lakes, Spark pipelines
Avro Row Embedded Deflate Kafka serialization, event streaming
ORC Columnar Embedded ZLIB / Snappy Hive-heavy pipelines, older Hadoop stacks
Which formats does NaijaCart actually use?

Chidi's batch pipeline writes Parquet to the data lake (MinIO). Kafka topics carry Avro messages from the mobile app. The legacy Hive reporting cluster still reads ORC. CSV only appears at the very edge — when Finance exports a report to send to a partner by email.

🔥 Parquet Benchmark — 100k NaijaCart Orders

This benchmark writes 100,000 synthetic NaijaCart orders in both CSV and Parquet, then reads and filters both. Run this on your machine — the output will be close to the verified output below.

python Lesson 09 · CSV vs Parquet size and speed
verified output
Dataset: 100,000 NaijaCart orders === Format Comparison: 100k NaijaCart Orders === Format Size (KB) Write (s) --------------------------------- CSV 5,420 0.65 Parquet 1,288 0.91 Parquet is 4.2x smaller than CSV === Read + Filter Benchmark === CSV read+filter : 0.72s Parquet read+filter : 0.16s Parquet column pruning wins on repeated queries === Schema Round-Trip === root |-- order_id: string (nullable = true) |-- city: string (nullable = true) |-- category: string (nullable = true) |-- amount_ngn: double (nullable = true) |-- status: string (nullable = true) |-- order_date: string (nullable = true)
Why is Parquet 4.2× smaller and 4.5× faster on reads?

When filtering by status == 'delivered', Parquet reads only the status column from disk — skipping order_id, city, category, amount_ngn, and order_date entirely. That is 5 out of 6 columns ignored. CSV has no concept of columns on disk: every byte of every row must be parsed before filtering can happen.

Parquet also compresses each column independently using Snappy. String columns with only 3 distinct values like status compress to near-nothing. Numeric columns stored as doubles use delta encoding. The result is 4.2× smaller on disk and 4.5× faster to query.

🏆 Why Parquet Wins at Analytics

Three Spark optimisations are only possible because Parquet stores data in a columnar, structured binary format. CSV cannot support any of them.

  • 1 Column pruning — Spark reads only the columns referenced in your SELECT. If your query touches 3 of 50 columns, Parquet serves 3 columns' worth of bytes. For wide tables (50–200 columns), this alone produces 10–20× faster scans.
  • 2 Predicate pushdown — Parquet stores column statistics (min, max, null count) per row-group (typically 128 MB chunks). When Spark applies a filter like amount_ngn > 100000, it reads the footer statistics first. If a row-group's amount_ngn max is 90000, Spark skips the entire row-group without reading it. No equivalent is possible with CSV.
  • 3 Schema embedded in footer — The Parquet file footer contains the full schema. spark.read.parquet(...) reads the schema in milliseconds without scanning data. CSV requires either an expensive inferSchema=True pass (reads the whole file to guess types) or a brittle manual schema declaration that can silently miscast columns.

📨 Avro in Streaming Pipelines

Avro is row-based — not columnar — which makes it a poor choice for analytics queries. So why does NaijaCart's Kafka pipeline use it for every event message? Because Avro was designed for exactly this problem: fast serialization of individual records with guaranteed schema compatibility.

Every Avro file (and every Kafka Avro message) carries the writer schema embedded in the payload or as a compact schema ID that points to a schema registry. This means:

  • ▸ The consumer always knows the exact field names and types. No guessing, no inferSchema.
  • ▸ Schema evolution is controlled: adding a field with a default does not break existing consumers. Removing a required field breaks them — and the schema registry enforces compatibility rules to prevent that.
  • ▸ When the mobile app team changes the order_created event to add a promo_code field, they register the new Avro schema. Old consumers (that do not know about promo_code) continue working without redeployment.

Avro + Kafka Schema Registry — how it works

NaijaCart's mobile app publishes order_created events to Kafka. Each message is Avro-serialized. The first 5 bytes of every message are a magic byte (0x00) and a 4-byte schema ID. The Kafka consumer reads the schema ID, looks up the full Avro schema from the schema registry over HTTP, and deserializes the rest of the message using that schema. The schema fetch is cached — subsequent messages with the same schema ID deserialize without any network call.

Schema registry Schema evolution Forward compatibility Backward compatibility

For replay scenarios (re-consuming old Kafka events), Avro is the gold standard. The schema stored alongside the data guarantees that events written 18 months ago can still be decoded exactly — even if the current schema has evolved.

🏛️ ORC — When You Will See It

ORC (Optimised Row Columnar) is Parquet's older sibling, developed at Hortonworks for Hive. Like Parquet it is columnar with embedded schema, but it uses ZLIB compression by default instead of Snappy.

DimensionParquetORC
Default compression Snappy (fast, moderate ratio) ZLIB (slower, better ratio)
Primary ecosystem Spark, Flink, Arrow, Hive Hive, older Hadoop stacks
Predicate pushdown Yes — row-group statistics Yes — stripe statistics
ACID support No (handled outside format) Yes — Hive ACID transactions use ORC
When to choose New Spark pipelines, data lakes Joining Spark to existing Hive tables

If Chidi is building a new pipeline today he reaches for Parquet. If he is reading from an existing Hive warehouse that predates the Spark migration, those tables are almost certainly ORC. Spark reads both with identical syntax — just swap .parquet(...) for .orc(...).

Rule of thumb: ORC's ZLIB compression is 10–15% smaller than Parquet's Snappy, but decompression is slower. On CPU-constrained clusters this matters. On modern hardware with fast SSDs or object storage, the difference is negligible and Parquet's ecosystem depth (Arrow, DuckDB, pandas) makes it the better default.

📝 Writing and Reading Patterns

The PySpark API for reading and writing formats is consistent. Swap the format method; the rest stays the same.

python Lesson 09 · write and read patterns
Why avoid inferSchema=True on CSV?

Spark reads the entire CSV file twice when inferSchema=True: once to sample types, once to actually load data. On a 50 GB CSV this doubles your read time and your cluster costs. Always declare the schema explicitly for CSV in production pipelines. Parquet and ORC never have this problem — the schema is in the file footer and read in a single metadata fetch.

🗂️ Tunde's Partition Strategy

From Tunde, NaijaCart Senior Data Engineer

"At NaijaCart we partition Parquet by year and month on write:"

df.write.partitionBy('year', 'month').parquet('s3a://bucket/orders/')

"Queries scoped to one month — like the end-of-month finance report — skip 11/12 of the data entirely. Spark reads the directory listing, sees year=2024/month=01/ partitions, and touches only those folders. The other 11 months of Parquet files are never opened. At 200 GB per month, skipping 11 months means reading 200 GB instead of 2.4 TB. That is the real 10× win — not the compression."

Partition columns should be low-cardinality fields used frequently in WHERE clauses. Year/month is the classic choice for time-series data. City or category also work if your queries are always region-scoped. Avoid partitioning on high-cardinality columns like order_id — you would create millions of tiny files, which causes a different class of performance problem.

⚠️ Common Mistakes

Using CSV for TB-scale pipelines
CSV is 10× slower to query and 5× larger than Parquet at terabyte scale. No predicate pushdown, no column pruning, no embedded schema. If you inherited a CSV pipeline, your first sprint should be converting it to Parquet. The performance gain pays for the migration effort within weeks.
Reading Avro without the spark-avro package
Spark does not include the Avro data source by default. You must add the spark-avro JAR or use --packages org.apache.spark:spark-avro_2.12:3.5.0 when submitting. Forgetting this gives a confusing runtime error: Failed to find data source: avro. Add the package at session build time in production configs.
Not partitioning large Parquet tables
Writing all data to a single unpartitioned Parquet directory forces a full table scan on every query. Parquet's row-group statistics still help within each file, but without directory partitioning Spark must open every file to check statistics. Partition by a common filter column (date, region) to get directory-level pruning on top of row-group pruning.
Mixing Parquet writer versions
Parquet has evolved through multiple format versions (v1, v2) and encoding changes. A Parquet file written by an old Hive version may not be readable by a new Spark version if new encoding types are involved. Always pin the writer version with .option("parquet.writer.version", "v1") in cross-team pipelines, and test schema evolution scenarios explicitly before shipping.

🧠 Quiz — 3 questions

Question 1 of 3
Parquet is better than CSV for analytics because…
Question 2 of 3
Avro is preferred for Kafka over Parquet because…
Question 3 of 3
Parquet column pruning means…
0
/ 3
Format knowledge locked in.

Now partition your next Parquet table by date and feel the speedup.

🎤 Interview Q&A

Medium What is predicate pushdown and which formats support it?
+

Predicate pushdown is an optimisation where the query engine evaluates filter conditions as early as possible in the data access pipeline — ideally before reading any row data. In Parquet and ORC, each file is split into row-groups (Parquet) or stripes (ORC). The file footer stores column-level statistics for each row-group: minimum value, maximum value, and null count. When Spark applies a filter like amount_ngn > 100000, it first reads the footer statistics. If a row-group's maximum amount_ngn is 90000, that entire row-group is skipped — no bytes from that row-group are read from disk or object storage.

CSV and JSON do not support predicate pushdown because they have no block-level statistics — every byte must be read and parsed before filtering can begin. Avro is row-based and also does not benefit from predicate pushdown in the same way. For analytics workloads with selective filters, predicate pushdown on Parquet can reduce IO by 80–95% when filters are highly selective and data is well-ordered within row-groups.

Medium How would you handle schema evolution in Parquet?
+

Parquet supports schema evolution through column merging. Spark's mergeSchema option reads schemas from all Parquet files in a directory and produces the union schema. New columns added in later files appear as null for older files. Columns removed from later files still appear — populated from older files.

To enable this: spark.read.option("mergeSchema", "true").parquet("s3a://bucket/orders/"). You can also set it globally with spark.sql.parquet.mergeSchema = true.

Best practices: (1) Only add nullable columns — never remove or rename existing columns, as this breaks readers. (2) Store schema separately in a schema registry or metadata table and enforce it on write. (3) Use partition-based evolution: new partitions (e.g. year=2025) can introduce new columns without touching old partitions. The cost of mergeSchema=true is listing all file footers, which is slow on large tables — avoid it in production hot paths and instead enforce a stable schema on write.

Medium When would you choose Avro over Parquet?
+

Choose Avro over Parquet in two scenarios:

1. Event streaming with Kafka. Parquet is a columnar batch format — it is designed to batch many rows together to achieve compression gains. A single Kafka message is one event (one row). Writing one-row Parquet files is wasteful and produces enormous file count. Avro serializes a single record efficiently, embeds its schema (or references a schema ID), and deserializes fast. The schema registry integration for forward and backward compatibility is a first-class Avro feature with no Parquet equivalent.

2. Write-heavy pipelines where you need full row access. Because Avro is row-based, writing and reading entire rows is efficient. If your pipeline always reads every column (e.g. a replication pipeline, a changelog feed), Avro avoids the overhead of reassembling rows from column stripes. In practice this is rare in analytics — most analytical queries benefit from column pruning, so Parquet wins for analytics even when you access many columns.

✅ Key Takeaways

  • Columnar formats (Parquet, ORC) read only the columns your query needs. At 100k rows, Parquet is 4.2× smaller than CSV and 4.5× faster to filter — the gains scale with data volume.
  • Parquet is the default choice for Spark analytics and data lakes: Snappy compression, embedded schema, predicate pushdown, and column pruning all come for free. No extra configuration needed.
  • Avro is the right format for Kafka: row-based serialization of individual events, embedded schema, and schema registry integration for safe schema evolution across independent producers and consumers.
  • ORC is Parquet's columnar cousin — favoured by Hive-heavy stacks and slightly smaller with ZLIB compression. If you are reading existing Hive tables in Spark, they are probably ORC. New pipelines should default to Parquet.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Build pipelines that actually run in production

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →