📋 Data lake + warehouse = Lakehouse
Traditional data lakes store raw files on cheap object storage (S3, GCS, ADLS) — great for cost and flexibility, terrible for correctness. No ACID. No time travel. No way to delete a row without rewriting the entire file. Traditional warehouses gave you ACID but locked data into a proprietary format that only their engine could read.
The lakehouse is the middle path: an open table format that sits on top of Parquet files in object storage and adds a transaction layer. Your data stays on cheap S3, but you get:
| Capability | Plain data lake | Lakehouse table format |
|---|---|---|
| ACID transactions | No | Yes — concurrent writers, atomic commits |
| Time travel | No (you'd have to archive versions yourself) | Yes — query any table at any previous snapshot |
| Row-level deletes and updates | No (must rewrite Parquet files) | Yes — MERGE/UPDATE/DELETE via format's transaction log |
| Schema evolution | Manual and error-prone | Safe add/rename/promote columns without rewriting data |
| Engine compatibility | Any engine can read Parquet | Any engine with the format library (Spark, Trino, Flink, Snowflake, DuckDB) |
🧊 The three formats at a glance
🧊 Apache Iceberg
Created at Netflix. Now the Apache Software Foundation's fastest-growing project. The leading choice for multi-engine environments.
- Hidden partitioning — partition on expressions, not raw columns
- Time travel via snapshot IDs or timestamps
- Safe schema evolution (add, rename, promote types)
- Row-level deletes with positional delete files
- Engines: Spark, Trino, Flink, Dremio, Snowflake, DuckDB
△ Delta Lake
Created at Databricks. Open-sourced as a Linux Foundation project. Native to the Databricks ecosystem; strong Python API via delta-spark.
- Transaction log (_delta_log/) — JSON commit files
- OPTIMIZE — compacts small files into larger ones
- VACUUM — removes old Parquet files beyond retention
- Z-ORDER — multidimensional clustering for scan efficiency
- DeltaTable.forPath() Python API for merges and updates
🌊 Apache Hudi
Created at Uber. Optimised for high-frequency upserts and incremental queries. Default format for AWS EMR managed tables.
- CoW (Copy-on-Write): rewrite Parquet on every upsert — optimised for read-heavy
- MoR (Merge-on-Read): write delta logs, merge at read time — optimised for write-heavy
- Incremental queries: read only records changed since a timestamp
- Native Kafka → Hudi via DeltaStreamer utility
- Best: CDC pipelines, high-frequency upserts from operational DB
🧊 Apache Iceberg — Spark table + time travel
Iceberg's hidden partitioning means you define a partition transform (like days(created_at)) rather than a raw column. Iceberg stores the actual partition values internally — the table user never needs to know the partition scheme to write correct queries.
ML training needs point-in-time features — the state of data at the moment a label was generated. With a plain data lake you cannot go back. With Iceberg time travel you can reproduce any training dataset exactly: spark.read.option("as-of-timestamp", "2024-01-01").table("local.borapay.payments"). This makes model reproducibility dramatically easier.
△ Delta Lake — MERGE upsert + OPTIMIZE
Delta Lake's MERGE is its most powerful feature: it handles insert/update/delete in one atomic SQL statement. This is the pattern for CDC (Change Data Capture) pipelines where a source database streams row-level changes and you want them reflected in your lake table correctly.
RETAIN 0 HOURS in production unless you are certain no concurrent readers are accessing old snapshots.
🌊 Apache Hudi — CoW vs MoR upserts
Hudi is designed for frequent, high-volume upserts — CDC pipelines from operational databases, or IoT event streams where device state updates frequently. The key choice is the table type: Copy-on-Write or Merge-on-Read.
⚖️ Iceberg vs Delta Lake vs Hudi
| Dimension | 🧊 Iceberg | △ Delta Lake | 🌊 Hudi |
|---|---|---|---|
| ACID | Yes (snapshot isolation) | Yes (optimistic concurrency via delta log) | Yes (commit timeline) |
| Time travel | Snapshot ID or as-of timestamp | VERSION AS OF n or TIMESTAMP AS OF | Commit timeline queries |
| Schema evolution | Best — add, rename, promote columns safely | Good — add columns; rename requires rewrite | Good — add columns; schema-on-read for MoR |
| Row-level deletes | Positional delete files (efficient) | Rewrites affected Parquet files | CoW rewrites files; MoR logs deletes at read time |
| Engine support | Widest — Spark, Trino, Flink, Snowflake, DuckDB, Athena, BigQuery | Strong — Spark, Trino, Flink, Databricks (native) | Good — Spark, Flink, Presto, Athena, Hive |
| Best for | Multi-engine, open ecosystem, regulated data (GDPR row delete) | Databricks-primary stacks, CDC pipelines, data quality enforcement | High-frequency CDC upserts, incremental ETL, AWS EMR shops |
| Community | ASF — largest vendor-neutral community; Snowflake, Apple, Netflix, LinkedIn all contributing | Linux Foundation — Databricks-led; strong Databricks + Microsoft momentum | ASF — Uber-originated; strong at Uber, Amazon, Alibaba |
A traditional warehouse stores data in a proprietary format. Your ML training code (Python, PySpark) cannot read it without going through the warehouse's query engine — adding latency and cost. With a lakehouse, your ML training code reads the same Parquet files directly from S3. No extra copy, no extra cost. Feature stores, training pipelines, and batch inference all read from the same physical files that power your BI dashboards. This unification is why Databricks (lakehouse-native) has grown so quickly in companies that do both analytics and ML.
🧠 Quiz — 4 questions
VACUUM delta.`path` RETAIN 0 HOURS in production?Next: Data Quality and Lineage — Great Expectations, dbt Tests, OpenLineage.
🎤 Interview Q&A
_delta_log/ directory alongside the Parquet data files. Each JSON commit file is numbered sequentially (000000.json, 000001.json, ...) and records exactly what changed in that transaction: which Parquet files were added, which were removed (for updates/deletes), and what statistics were computed (min/max per column for data skipping). When you read a Delta table, Spark reads the transaction log from the highest committed version downward, rebuilding the current table state by determining which Parquet files are "live." This log is the source of truth for ACID semantics — concurrent writers each try to commit a new numbered JSON file; if two writers try to commit the same version number, one wins and the other must retry (optimistic concurrency). Periodic checkpointing converts all log entries up to a version into a Parquet checkpoint file for faster reads. VACUUM then removes Parquet files that are no longer referenced by any log entry within the retention window.
✅ Key takeaways
- The lakehouse pattern = open Parquet files on S3/GCS + a table format layer that adds ACID, time travel, schema evolution, and row-level deletes. No proprietary lock-in — any engine with the format library can read the data.
- Apache Iceberg has the widest engine support (Spark, Trino, Flink, Snowflake, DuckDB, BigQuery) and the most advanced schema evolution. Hidden partitioning means users never write wrong partition filter predicates. First choice for multi-engine, cloud-neutral architectures.
- Delta Lake is native to Databricks and the strongest choice for Databricks-primary stacks. OPTIMIZE + Z-ORDER compact small files and cluster data for fast scans. VACUUM removes old Parquet files — never set retention below 7 days.
- Apache Hudi is optimised for high-frequency upserts and incremental queries. CoW is faster for reads; MoR is faster for writes. Incremental queries let downstream jobs read only changed records — very efficient for CDC pipelines.
- Lakehouses are eating traditional warehouses for ML workloads because training code can read the same Parquet files directly from S3, eliminating the need to copy data out of the warehouse into a separate storage system for model training.
Build on open lakehouse standards
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →