📋 Why bad data is expensive
Bad data costs money in two ways. Loud errors — a pipeline crashes because a column is NULL that cannot be NULL — are annoying but cheap. The pipeline stops, an alert fires, an engineer fixes it. Silent errors are the real danger: bad data that passes every validation and flows through to a dashboard or an ML model, producing results that are wrong but look plausible.
| Failure mode | Example | Cost |
|---|---|---|
| Loud error | Pipeline fails with NullPointerException on a missing column | Low — caught immediately, engineer fixes in hours |
| Silent data error | Currency conversion bug doubles all USD transactions — revenue reports show 2x for a week | High — trust damage, analyst rework, potential regulatory issue |
| Silent ML error | Training data includes future features (data leakage) — model looks great in evaluation but fails in production | Very high — wrong predictions served for weeks before anyone notices |
| Lineage unknown | A dashboard column is wrong — nobody knows which upstream transformation caused it | High — hours or days of manual tracing to find root cause |
📐 Data quality dimensions
Five standard dimensions characterise data quality problems. Each maps to specific test patterns in Great Expectations and dbt:
not_null, row count thresholds.unique, count distinct vs count.accepted_values, regex, range checks.✅ Great Expectations — Checkpoints
Great Expectations (GE) is a Python library for defining, running, and documenting data quality assertions against Pandas DataFrames, Spark DataFrames, or SQL sources. Key concepts: Expectation (one assertion), ExpectationSuite (a named collection), Checkpoint (wires data to a suite and triggers actions), Data Docs (auto-generated HTML validation report).
Wrap the GE checkpoint in a PythonOperator task between extract and transform. If validation fails, the task raises an exception — Airflow marks it failed, sends an alert, and the transform and load tasks never run. Bad data is quarantined at extraction, not propagated to dashboards.
🧪 dbt tests — schema.yml and dbt-expectations
dbt's four built-in tests cover most cases. The dbt-expectations package extends them with Great Expectations-style assertions directly in schema.yml — no Python required. This keeps quality rules next to the model definitions, versioned in Git.
tests/assert_revenue_matches_source.sql with a query that compares SUM(amount_kes) in stg_payments to SUM(total_revenue_kes) in mart_revenue_by_region. The test passes if the query returns zero rows (no discrepancy), fails if any rows are returned. dbt runs this as part of dbt test automatically.
🔗 OpenLineage — data lineage across systems
OpenLineage is an open specification (not a tool) for lineage events — a standard JSON format that any tool (Airflow, dbt, Spark, Flink) can emit to describe what data it reads and writes. Marquez is the reference server implementation: it stores events, provides a REST API, and shows a visual lineage graph.
| Concept | What it is |
|---|---|
| OpenLineage | Open spec — defines the JSON event format producers emit (START, COMPLETE, FAIL) |
| Marquez | Open-source lineage server — stores events, REST API, visual UI |
| Facets | Extensible metadata on events: schema facet, data quality facet, SQL facet |
| Job | A pipeline step — an Airflow task, a dbt model, a Spark job |
| Dataset | An input or output — database table, S3 path, Kafka topic |
Install apache-airflow-providers-openlineage and set OPENLINEAGE_URL=http://marquez:5000. Every Airflow task then automatically emits START/COMPLETE/FAIL lineage events with input/output dataset information extracted from operator parameters. For PostgresOperator, OpenLineage parses the SQL and extracts table names. You get full column-level lineage from your Postgres source tables all the way to your DuckDB mart tables — without writing a single line of lineage code.
📜 Data contracts — the emerging practice
A data contract is a formal agreement between the team that produces a dataset and the teams that consume it. It specifies: schema (column names and types), quality guarantees (not null columns, allowed values), SLA (freshness, availability), and ownership.
| Element | What it specifies | Enforced by |
|---|---|---|
| Schema | Column names, data types, required fields | GE schema test, dbt schema.yml, Avro/Protobuf schema registry |
| Completeness SLA | "payment_id is never null", "table has > 1,000 rows daily" | GE not_null, dbt not_null, row count threshold |
| Freshness SLA | "Data is no older than 6 hours" | dbt source freshness, GE expect_column_max_to_be_between on timestamp |
| Semantics | "amount_kes is always in Kenyan Shillings, not USD" | Documentation — enforced by code review and GE range tests |
| Ownership | Who is responsible for the dataset; who to contact on failure | Slack alert routing, PagerDuty ownership |
🧠 Quiz — 4 questions
Next up: Lesson 34 — Capstone Batch Pipeline.
🎤 Interview Q&A
not_null on primary key and mandatory columns; row count must be above a minimum threshold. Uniqueness — no duplicate primary keys. Test: unique on the primary key column; alternatively, COUNT(*) == COUNT(DISTINCT pk). Validity — values must be in the allowed domain. Test: accepted_values for categorical columns; between for numeric range; regex for format (email, phone, ID). Timeliness — data must be recent enough. Test: MAX(created_at) > NOW() - INTERVAL 6 hours; row count must not be zero for a day that should have data. Consistency — values must be consistent across systems. Test: SUM(amount) in the staging table equals SUM(total_revenue) in the mart; foreign key check (relationships in dbt) confirms every customer_id in payments exists in the customers table.
dbt test running against the transformed model before it's exposed to stakeholders. The key principle: validate at every system boundary where you transition data between ownership domains — from source to staging, from staging to mart, from mart to external consumers.
mart_revenue_by_region was produced by the daily_etl job, which read from stg_payments, which came from the borapay_raw.payments table. That is useful but not sufficient. Column-level lineage tells you that the specific column total_revenue_kes in mart_revenue_by_region was derived from SUM(amount_kes) in stg_payments, which came from the amount column in borapay_raw.payments, transformed by the cast(amount as numeric(14,2)) expression in the staging model. When a KPI is wrong, column-level lineage lets you immediately pinpoint whether the error is in the raw source column, the cast expression in staging, the aggregation formula in the mart, or the currency conversion in an intermediate model — instead of reading through every SQL file manually. OpenLineage supports column-level lineage via the columnLineage facet on output datasets. dbt generates column-level lineage from the SQL AST (abstract syntax tree) of each model and exposes it via the manifest.json. Tools like Atlan, DataHub, and Marquez visualise this as an interactive graph where clicking any column shows its full transformation ancestry.
✅ Key takeaways
- Silent data errors — bad data that passes validation and corrupts dashboards or ML models — are more expensive than loud pipeline failures. The goal of data quality tooling is to make every bad-data scenario loud and caught early.
- Great Expectations validates data at the extraction boundary before it enters your warehouse. Wrap a Checkpoint in an Airflow
PythonOperator— if validation fails, the task raises an exception and downstream tasks never run. - dbt schema tests (
not_null,unique,accepted_values,relationships) plusdbt-expectationscover 90% of data quality needs at the transformation layer. Run them withdbt buildto enforce quality before marts are exposed to stakeholders. - OpenLineage is an open specification for lineage events; Marquez is the reference server. Install the Airflow OpenLineage provider and every task emits lineage automatically — no code changes needed. Column-level lineage lets you trace any broken KPI back to its source column in minutes instead of hours.
- Data contracts formalise the agreement between data producers and consumers: schema, not-null guarantees, freshness SLA, and ownership. Start with dbt schema.yml as a lightweight contract; invest in formal contracts when multiple teams depend on your dataset.
Build pipelines stakeholders trust
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →