📋 Data lake + warehouse = Lakehouse

Traditional data lakes store raw files on cheap object storage (S3, GCS, ADLS) — great for cost and flexibility, terrible for correctness. No ACID. No time travel. No way to delete a row without rewriting the entire file. Traditional warehouses gave you ACID but locked data into a proprietary format that only their engine could read.

The lakehouse is the middle path: an open table format that sits on top of Parquet files in object storage and adds a transaction layer. Your data stays on cheap S3, but you get:

CapabilityPlain data lakeLakehouse table format
ACID transactionsNoYes — concurrent writers, atomic commits
Time travelNo (you'd have to archive versions yourself)Yes — query any table at any previous snapshot
Row-level deletes and updatesNo (must rewrite Parquet files)Yes — MERGE/UPDATE/DELETE via format's transaction log
Schema evolutionManual and error-proneSafe add/rename/promote columns without rewriting data
Engine compatibilityAny engine can read ParquetAny engine with the format library (Spark, Trino, Flink, Snowflake, DuckDB)

🧊 The three formats at a glance

🧊 Apache Iceberg

Created at Netflix. Now the Apache Software Foundation's fastest-growing project. The leading choice for multi-engine environments.

  • Hidden partitioning — partition on expressions, not raw columns
  • Time travel via snapshot IDs or timestamps
  • Safe schema evolution (add, rename, promote types)
  • Row-level deletes with positional delete files
  • Engines: Spark, Trino, Flink, Dremio, Snowflake, DuckDB

△ Delta Lake

Created at Databricks. Open-sourced as a Linux Foundation project. Native to the Databricks ecosystem; strong Python API via delta-spark.

  • Transaction log (_delta_log/) — JSON commit files
  • OPTIMIZE — compacts small files into larger ones
  • VACUUM — removes old Parquet files beyond retention
  • Z-ORDER — multidimensional clustering for scan efficiency
  • DeltaTable.forPath() Python API for merges and updates

🌊 Apache Hudi

Created at Uber. Optimised for high-frequency upserts and incremental queries. Default format for AWS EMR managed tables.

  • CoW (Copy-on-Write): rewrite Parquet on every upsert — optimised for read-heavy
  • MoR (Merge-on-Read): write delta logs, merge at read time — optimised for write-heavy
  • Incremental queries: read only records changed since a timestamp
  • Native Kafka → Hudi via DeltaStreamer utility
  • Best: CDC pipelines, high-frequency upserts from operational DB

🧊 Apache Iceberg — Spark table + time travel

Iceberg's hidden partitioning means you define a partition transform (like days(created_at)) rather than a raw column. Iceberg stores the actual partition values internally — the table user never needs to know the partition scheme to write correct queries.

iceberg_spark.py Lesson 32 · create table, insert, time travel
spark output
+-------------------+------------------------+-----------+ |snapshot_id |committed_at |operation | +-------------------+------------------------+-----------+ |3421567890123456789|2024-01-15 10:01:23.456 |append | +-------------------+------------------------+-----------+ Rows at snapshot 3421567890123456789: 3 Deleted customer 1002 — compliant with GDPR right to erasure.
Why time travel matters for ML

ML training needs point-in-time features — the state of data at the moment a label was generated. With a plain data lake you cannot go back. With Iceberg time travel you can reproduce any training dataset exactly: spark.read.option("as-of-timestamp", "2024-01-01").table("local.borapay.payments"). This makes model reproducibility dramatically easier.

△ Delta Lake — MERGE upsert + OPTIMIZE

Delta Lake's MERGE is its most powerful feature: it handles insert/update/delete in one atomic SQL statement. This is the pattern for CDC (Change Data Capture) pipelines where a source database streams row-level changes and you want them reflected in your lake table correctly.

delta_merge.py Lesson 32 · MERGE upsert and OPTIMIZE Z-ORDER
after MERGE — spark.read.format("delta").load().show()
+----------+-----------+----------+-------+-------+ |payment_id|customer_id|amount_kes|region |status | +----------+-----------+----------+-------+-------+ |TXN001 |1001 |5000.0 |nairobi|settled| |TXN002 |1002 |12000.0 |mumbai |settled| ← updated |TXN003 |1003 |3500.0 |lagos |pending| |TXN004 |1004 |8000.0 |kampala|pending| ← inserted +----------+-----------+----------+-------+-------+
VACUUM caution: Delta Lake keeps old Parquet files to support time travel. VACUUM removes files older than the retention period — which disables time travel beyond that point. Never set retention below 7 days, and never run VACUUM with RETAIN 0 HOURS in production unless you are certain no concurrent readers are accessing old snapshots.

🌊 Apache Hudi — CoW vs MoR upserts

Hudi is designed for frequent, high-volume upserts — CDC pipelines from operational databases, or IoT event streams where device state updates frequently. The key choice is the table type: Copy-on-Write or Merge-on-Read.

hudi_upsert.py Lesson 32 · Hudi CoW upsert via Spark
hudi snapshot read
+----------+-------+--------------------+----------------------+ |payment_id|status |updated_at |_hoodie_commit_time | +----------+-------+--------------------+----------------------+ |TXN001 |settled|2024-01-15 08:00:00 |20240115080001 | |TXN002 |settled|2024-01-15 10:00:00 |20240115100001 | ← upserted |TXN003 |pending|2024-01-15 10:30:00 |20240115100001 | ← inserted +----------+-------+--------------------+----------------------+ Records changed since 09:00: 2

⚖️ Iceberg vs Delta Lake vs Hudi

Dimension 🧊 Iceberg △ Delta Lake 🌊 Hudi
ACID Yes (snapshot isolation) Yes (optimistic concurrency via delta log) Yes (commit timeline)
Time travel Snapshot ID or as-of timestamp VERSION AS OF n or TIMESTAMP AS OF Commit timeline queries
Schema evolution Best — add, rename, promote columns safely Good — add columns; rename requires rewrite Good — add columns; schema-on-read for MoR
Row-level deletes Positional delete files (efficient) Rewrites affected Parquet files CoW rewrites files; MoR logs deletes at read time
Engine support Widest — Spark, Trino, Flink, Snowflake, DuckDB, Athena, BigQuery Strong — Spark, Trino, Flink, Databricks (native) Good — Spark, Flink, Presto, Athena, Hive
Best for Multi-engine, open ecosystem, regulated data (GDPR row delete) Databricks-primary stacks, CDC pipelines, data quality enforcement High-frequency CDC upserts, incremental ETL, AWS EMR shops
Community ASF — largest vendor-neutral community; Snowflake, Apple, Netflix, LinkedIn all contributing Linux Foundation — Databricks-led; strong Databricks + Microsoft momentum ASF — Uber-originated; strong at Uber, Amazon, Alibaba
Why lakehouses are eating traditional warehouses for ML

A traditional warehouse stores data in a proprietary format. Your ML training code (Python, PySpark) cannot read it without going through the warehouse's query engine — adding latency and cost. With a lakehouse, your ML training code reads the same Parquet files directly from S3. No extra copy, no extra cost. Feature stores, training pipelines, and batch inference all read from the same physical files that power your BI dashboards. This unification is why Databricks (lakehouse-native) has grown so quickly in companies that do both analytics and ML.

🧠 Quiz — 4 questions

Question 1 of 4
What is the key difference between a traditional data lake and a lakehouse table format?
Question 2 of 4
What does Iceberg's "hidden partitioning" mean?
Question 3 of 4
In Apache Hudi, what is the difference between Copy-on-Write (CoW) and Merge-on-Read (MoR)?
Question 4 of 4
Why should you never run VACUUM delta.`path` RETAIN 0 HOURS in production?
0
/ 4
Lakehouse architecture unlocked.

Next: Data Quality and Lineage — Great Expectations, dbt Tests, OpenLineage.

🎤 Interview Q&A

Easy What problem does the lakehouse pattern solve that neither a data lake nor a data warehouse solves alone?
+
A data lake is cheap and flexible but lacks correctness guarantees: no ACID transactions, no way to update or delete individual rows without rewriting entire files, no schema enforcement, no time travel. A data warehouse gives you all of those things but locks data in a proprietary format that only its own engine can read efficiently — your ML training code must go through the warehouse's query engine, adding cost and latency, and you cannot share raw data with external tools cheaply. The lakehouse solves both problems simultaneously: it stores data as open Parquet files on cheap object storage (solving the cost and compatibility problem), while the table format layer (Iceberg, Delta Lake, or Hudi) adds ACID, time travel, schema evolution, and row-level updates on top (solving the correctness problem). The result: your data engineers, analysts, ML engineers, and external partners all read the same physical files — no copying, no proprietary lock-in.
Medium How does Delta Lake's transaction log (_delta_log) work?
+
Every write to a Delta Lake table creates a new JSON file in the _delta_log/ directory alongside the Parquet data files. Each JSON commit file is numbered sequentially (000000.json, 000001.json, ...) and records exactly what changed in that transaction: which Parquet files were added, which were removed (for updates/deletes), and what statistics were computed (min/max per column for data skipping). When you read a Delta table, Spark reads the transaction log from the highest committed version downward, rebuilding the current table state by determining which Parquet files are "live." This log is the source of truth for ACID semantics — concurrent writers each try to commit a new numbered JSON file; if two writers try to commit the same version number, one wins and the other must retry (optimistic concurrency). Periodic checkpointing converts all log entries up to a version into a Parquet checkpoint file for faster reads. VACUUM then removes Parquet files that are no longer referenced by any log entry within the retention window.
Hard When would you choose Apache Iceberg over Delta Lake, and vice versa?
+
Choose Iceberg when: (1) you need multi-engine access — your data must be readable by Spark, Trino, Flink, Snowflake, and DuckDB simultaneously without format conversion; (2) you have GDPR or CCPA row-deletion requirements — Iceberg's positional delete files make row-level deletes efficient without full file rewrites; (3) you need advanced schema evolution — renaming columns, changing partition schemes, or promoting data types without rewriting all historical data; (4) you are building on a cloud-neutral or multi-cloud architecture and want to avoid any single-vendor dependency. Choose Delta Lake when: (1) your team is primarily on Databricks — Delta is the native format, tightly integrated with the Photon engine, Unity Catalog, and Databricks ML; (2) you need the Python DeltaTable API for programmatic merges and table management in PySpark scripts; (3) you want OPTIMIZE and Z-ORDER for query acceleration without configuring Iceberg's hidden partitioning; (4) you are already running Delta in production and the migration cost outweighs the format advantages. In 2025, the industry is converging toward Iceberg for new multi-engine architectures, while Delta remains dominant within the Databricks ecosystem.

✅ Key takeaways

  • The lakehouse pattern = open Parquet files on S3/GCS + a table format layer that adds ACID, time travel, schema evolution, and row-level deletes. No proprietary lock-in — any engine with the format library can read the data.
  • Apache Iceberg has the widest engine support (Spark, Trino, Flink, Snowflake, DuckDB, BigQuery) and the most advanced schema evolution. Hidden partitioning means users never write wrong partition filter predicates. First choice for multi-engine, cloud-neutral architectures.
  • Delta Lake is native to Databricks and the strongest choice for Databricks-primary stacks. OPTIMIZE + Z-ORDER compact small files and cluster data for fast scans. VACUUM removes old Parquet files — never set retention below 7 days.
  • Apache Hudi is optimised for high-frequency upserts and incremental queries. CoW is faster for reads; MoR is faster for writes. Incremental queries let downstream jobs read only changed records — very efficient for CDC pipelines.
  • Lakehouses are eating traditional warehouses for ML workloads because training code can read the same Parquet files directly from S3, eliminating the need to copy data out of the warehouse into a separate storage system for model training.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Build on open lakehouse standards

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →