πŸ’³
Real-world scenario
Rebecca (Razorpay): 24 months of UPI data won't fit in memory

Rebecca is a junior analyst at Razorpay in Mumbai. Her team analyses payment flows across Indiaβ€”every UPI transaction, every failed retry, every merchant dispute.

This month, she's asked to analyse 24 months of UPI transaction data: 40 million rows, each with timestamp, amount, VPA, merchant category, and status. The CSV export is 1.2 GB. Her 8 GB laptop crashes when she opens it.

Her senior colleague says: "Stop using CSV. Switch to Parquet." Rebecca learns why in 18 minutes, and her queries now run 15x faster on 1/6 the file size.

40M
UPI rows (24 mo)
1.2 GB
CSV file size
180 MB
Parquet size

What makes Parquet different

CSV stores data row-by-row. Parquet stores it column-by-column. This one difference changes everything.

File size, load speed, and dtype preservation: CSV vs Parquet vs Feather
Bar chart comparing file sizes for 40M rows: CSV 1.2 GB, Parquet 0.18 GB (15% of CSV), Feather 0.25 GB (21% of CSV)
What this shows: Parquet cuts file size to 1/6 of CSV because columnar storage + compression skip redundancy. You're storing β‚Ή50–25,000 once, not repeated in every row.

Row format vs columnar format

CSV (Row-based)
txn_id | timestamp | amount | status
RZP001 | 2026-03-15 | 850 | success
RZP002 | 2026-03-15 | 2400 | success
RZP003 | 2026-03-15 | 350 | failed
...40M rows
ProCon
βœ“ Human-readableβœ— Redundant
βœ“ Universalβœ— No compression
βœ— Slow queries
Parquet (Columnar)
txn_id
RZP001
RZP002
RZP003
timestamp
2026-03-15
2026-03-15
2026-03-15
amount
850
2400
350
ProCon
βœ“ Compressibleβœ— Binary format
βœ“ Skip columnsβœ— Bigger libraries
βœ“ Typed dtypes

The key insight: When you ask for just amount from 40M rows, CSV reads all 1.2 GB. Parquet reads only the amount column chunk (~180 MB for that one column). That's why columnar is faster for analytics.

Reading & writing Parquet: 3 lines you need

Step 1: Install pyarrow

Pandas uses PyArrow or FastParquet as a backend. PyArrow is faster and more stable.

bash

Note: FastParquet also works and is lighter-weight, but PyArrow has better compression and performance. If you're on slow internet, pip install fastparquet is 50 MB smaller.

Step 2: Read Parquet (vs CSV speed)

Rebecca's first insight: same file, massive speed difference.

Loading 40M UPI rows: CSV vs Parquet
Bar chart comparing load times: CSV takes 48 seconds versus Parquet's 3.2 seconds, providing 15x speedup
What this shows: Parquet loads the same 40M rows 15x faster. CSV decodes every field as text, then casts to dtype. Parquet already typed.
python

Step 3: Write Parquet with compression

Parquet compression trades CPU for storage. Choose based on your workflow.

python

In production at Razorpay: They use snappy for daily pipelines (speed matters) and gzip for long-term archive (storage matters). You choose based on your read/write ratio.

Rebecca's query: read only columns you need

Columnar storage's superpower: skip columns you don't ask for.

Memory footprint: full read vs columnar read
Bar chart comparing memory usage: Full read of all 24 columns uses 2.5 GB, whereas columnar read of only amount column uses 0.18 GB (92% reduction)
What this shows: Parquet's columns parameter only reads the columns you ask for. 40M rows Γ— 24 columns β†’ 40M rows Γ— 3 columns. Memory drops 8x.

Sample query: find high-value UPI transactions

Rebecca wants to find which merchant categories process the largest transactions. She only needs 3 columns from the 24-column file.

txn_idamount_inrmerchant_category
RZP001β‚Ή850grocery
RZP002β‚Ή2,400fuel
RZP003β‚Ή350food-delivery
RZP004β‚Ή1,200utilities
RZP005β‚Ή450entertainment
...40M rows
python

Why this matters: At scale, you rarely need all columns. Razorpay's UPI data has 24 columns (timestamp, payer VPA, payee VPA, merchant ID, merchant category, transaction ID, status code, error code, retry count, gateway, fee, settled amount, settlement date, refund status, chargeback flag, customer tenure, device type, location, etc.). Reading all 24 just to analyse 3 wastes RAM and CPU. Parquet skips the 21 you don't need.

Partitioning: read a subset without loading everything

For truly huge datasets, partition the Parquet file by date. Then read only the months you need.

Partitioned directory structure

data/
β”œβ”€β”€ year=2026/
β”‚ β”œβ”€β”€ month=01/part-0000.parquet
β”‚ β”œβ”€β”€ month=02/part-0000.parquet
β”‚ β”œβ”€β”€ month=03/part-0000.parquet
β”‚ β”œβ”€β”€ month=04/part-0000.parquet
β”‚ └── month=12/part-0000.parquet
β”œβ”€β”€ year=2025/
β”‚ β”œβ”€β”€ month=01/part-0000.parquet
β”‚ └── ...

Each month's data lives in its own Parquet file. Now read Q1 2026 without loading Q2–Q4.

python
Query speed: all partitions vs filtered partitions
Bar chart comparing query times: Manual partition takes 12 minutes while auto partition takes 45 seconds (16x faster)
What this shows: Filtering by partition (year/month) skips irrelevant Parquet files entirely. PyArrow's dataset API only touches the files you ask for.

Production pattern: Razorpay writes daily Parquet files, partitioned by date. When analysts query "last 7 days," the system reads only 7 files, not 365.

When to stay on CSV (and when to switch)

ScenarioPick CSVPick Parquet
File size < 100 MB > 500 MB
Number of rows < 100k rows > 10M rows
Use case One-off analysis, data sharing Production pipelines, repeated queries
Team access Non-technical stakeholders (Excel, Google Sheets) Engineers & data teams
Storage S3 occasional, cold storage S3 production, frequent reads
Sharing format Email, Slack, downloads Data warehouse, analytical warehouse
Columns per query Always read all columns Subset of columns regularly
Read frequency One time, infrequent Many times, dashboards, reports

Rebecca's rule of thumb: If your CSV is over 500 MB OR you're reading it more than once a week, convert it to Parquet. Spend 2 minutes converting; save 10 minutes per query.

Sample data to follow along

πŸ“„
Sample Razorpay UPI transaction data
50 rows of realistic fintech transaction data. Save next to your .py file so pd.read_csv('razorpay_sample.csv') works.
python

Cheatsheet: Parquet essentials

TaskCode
Install backendpip install pyarrow
Read Parquet filedf = pd.read_parquet('file.parquet')
Read specific columnsdf = pd.read_parquet('file.parquet', columns=['col1', 'col2'])
Write with snappydf.to_parquet('out.parquet', compression='snappy')
Write with gzipdf.to_parquet('out.parquet', compression='gzip')
Write with brotlidf.to_parquet('out.parquet', compression='brotli')
Read partitioned datasetdf = pd.read_parquet('data/', filters=[('year', '==', 2026)])
Inspect file metadataimport pyarrow.parquet as pq; pq.read_table('file.parquet').schema
Write with indexdf.to_parquet('out.parquet', index=True)
Read without enginedf = pd.read_parquet('file.parquet', engine='pyarrow')

Try it: Convert and query

Task: Download the sample CSV above. Convert it to Parquet. Then read only the txn_id, amount_inr, and merchant_category columns. Print the top 5 merchant categories by total transaction amount.
Hint: Use df.groupby('merchant_category')['amount_inr'].sum().sort_values(ascending=False).head() after reading with the columns parameter.

Interview questions

Parquet uses columnar storage + compression. Identical values (e.g., status='success' repeated 10M times) are stored once, not repeated. CSV stores them row-by-row, so 'success' appears 10M times as text.
Both are Parquet backends for pandas. PyArrow is faster (native C++), has better compression, and is more actively maintained. FastParquet is lighter (50 MB vs 150 MB install). For performance, use PyArrow. For minimal dependencies, use FastParquet.
Partition large datasets by a low-cardinality column (date, region, product category). Write each partition as a separate Parquet file in a folder hierarchy (year/month). Then filter at read time to skip irrelevant files. Use when your file is > 2 GB or queries are typically date-ranged.
Snappy: 3x compression, <100ms write time. Gzip: 6x compression, 1–2s write time. Brotli: 4.5x compression, 0.5s write time. Use snappy for daily pipelines (speed). Use gzip for archival (storage). Use brotli for the middle ground. Test your own data.
Use partitioning + column selection. (1) Write the source as partitioned Parquet by date (year/month). (2) Use filters to skip date partitions. (3) Use columns to read only needed fields. (4) If still large, use PyArrow's `filters` at the row group level to skip chunks. Result: 100 GB file, 5 MB query time.

Common pitfalls

Forgot to install PyArrow
Error: ImportError: pyarrow not found. Fix: pip install pyarrow. Then restart your Jupyter kernel.
Schema mismatch on append
If you append data with different dtypes (e.g., amount as int in file 1, float in file 2), Parquet will raise an error. Cast both to the same dtype before writing: df['amount'] = df['amount'].astype('float32').
Over-partitioning creates 10k tiny files
Partition by date (month/year), not by user_id or txn_id. Each partition should be > 100 MB. Too many tiny files = slow reads (file I/O overhead).
Parquet is not human-readable
You can't cat or grep a Parquet file. To inspect: df = pd.read_parquet('file.parquet'); print(df.head()). Or use parquet-tools cat file.parquet (pip install parquet-tools).
Reading a Parquet directory without PyArrow's dataset API
If you have partitioned data in data/year=2026/month=03/, use pd.read_parquet('data/', filters=...), not pd.read_parquet('data/year=2026/month=03/part-0000.parquet'). Let pandas discover the partition structure.

Fintech Β· E-commerce Β· Telecom contexts

πŸ’³
Razorpay: UPI & card payments
40M+ daily transactions. CSV exports stop working at scale. Fintech teams partition by date and query last 30 days.
πŸ›’
Jumia Kenya: order data
1M+ orders/day. Analysts read only seller_id, product_id, amount to find top products. Parquet's column selection saves 80% memory.
πŸ“ž
Safaricom: call detail records (CDRs)
500M CDRs/month. Partitioned by date. Business analysts query "calls in Nairobi last week" without touching Lagos/Mombasa data.

Why AI & ML teams use Parquet

Feature engineering pipelines read Parquet. A typical ML training loop reads 50 GB of features. With CSV, that's 5 minutes per epoch. With Parquet, it's 20 seconds. columnar selection compression partitioning fast I/O

Quiz: test your knowledge

Question 1 / 5
A Parquet file is 180 MB. The same data in CSV is 1.2 GB. Why?
Question 2 / 5
You need to install Parquet support. What command?
Question 3 / 5
To read only 3 columns from a 24-column Parquet file, you use:
Question 4 / 5
Snappy compression is best for:
Question 5 / 5
You have 12 months of transaction data (100 GB). You usually query a single month. Best strategy?
5
/ 5
Perfect!
Africa & India's Best AI Academy
πŸ€– AI-first πŸŽ“ Placement Assistance

Ready to scale your analysis?

Join the Best AI Program: The AI-Era Analyst β€” 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst β†’