📋 Why bad data is expensive

Bad data costs money in two ways. Loud errors — a pipeline crashes because a column is NULL that cannot be NULL — are annoying but cheap. The pipeline stops, an alert fires, an engineer fixes it. Silent errors are the real danger: bad data that passes every validation and flows through to a dashboard or an ML model, producing results that are wrong but look plausible.

Failure modeExampleCost
Loud errorPipeline fails with NullPointerException on a missing columnLow — caught immediately, engineer fixes in hours
Silent data errorCurrency conversion bug doubles all USD transactions — revenue reports show 2x for a weekHigh — trust damage, analyst rework, potential regulatory issue
Silent ML errorTraining data includes future features (data leakage) — model looks great in evaluation but fails in productionVery high — wrong predictions served for weeks before anyone notices
Lineage unknownA dashboard column is wrong — nobody knows which upstream transformation caused itHigh — hours or days of manual tracing to find root cause

📐 Data quality dimensions

Five standard dimensions characterise data quality problems. Each maps to specific test patterns in Great Expectations and dbt:

Completeness
Required fields are present. Tests: not_null, row count thresholds.
Uniqueness
Primary keys have no duplicates. Tests: unique, count distinct vs count.
Validity
Values are in the expected domain. Tests: accepted_values, regex, range checks.
Timeliness
Data is fresh enough. Tests: max timestamp within N hours, row count anomaly detection.
Consistency
Values match across systems. Tests: cross-table reconciliation, referential integrity.

✅ Great Expectations — Checkpoints

Great Expectations (GE) is a Python library for defining, running, and documenting data quality assertions against Pandas DataFrames, Spark DataFrames, or SQL sources. Key concepts: Expectation (one assertion), ExpectationSuite (a named collection), Checkpoint (wires data to a suite and triggers actions), Data Docs (auto-generated HTML validation report).

ge_checkpoint.py Lesson 33 · Great Expectations on a Pandas DataFrame
validation output — one expectation failed
Validation success: False [PASS] expect_column_values_to_not_be_null — payment_id [PASS] expect_column_values_to_be_unique — payment_id [FAIL] expect_column_values_to_be_between — amount_kes 3 rows have amount_kes < 0.01 (negative amounts detected) [PASS] expect_column_values_to_be_in_set — status [PASS] expect_table_row_count_to_be_between — table ValueError: Data quality check failed: 1 expectation(s) failed — blocking pipeline.
GE inside Airflow

Wrap the GE checkpoint in a PythonOperator task between extract and transform. If validation fails, the task raises an exception — Airflow marks it failed, sends an alert, and the transform and load tasks never run. Bad data is quarantined at extraction, not propagated to dashboards.

🧪 dbt tests — schema.yml and dbt-expectations

dbt's four built-in tests cover most cases. The dbt-expectations package extends them with Great Expectations-style assertions directly in schema.yml — no Python required. This keeps quality rules next to the model definitions, versioned in Git.

models/staging/schema.yml Lesson 33 · dbt built-ins + dbt-expectations package
dbt test --select stg_payments
1 of 6 PASS not_null_stg_payments_payment_id .............. [PASS in 0.08s] 2 of 6 PASS unique_stg_payments_payment_id ................. [PASS in 0.07s] 3 of 6 PASS expect_column_values_to_match_regex_payment_id . [PASS in 0.10s] 4 of 6 FAIL expect_column_values_to_be_between_amount_kes .. [FAIL in 0.11s] 3 rows with amount_kes < 0.01 detected. 5 of 6 PASS accepted_values_stg_payments_status ............ [PASS in 0.09s] 6 of 6 PASS not_null_stg_payments_created_at ............... [PASS in 0.06s] Completed with 1 failure.
Custom singular test — revenue reconciliation: Create tests/assert_revenue_matches_source.sql with a query that compares SUM(amount_kes) in stg_payments to SUM(total_revenue_kes) in mart_revenue_by_region. The test passes if the query returns zero rows (no discrepancy), fails if any rows are returned. dbt runs this as part of dbt test automatically.

🔗 OpenLineage — data lineage across systems

OpenLineage is an open specification (not a tool) for lineage events — a standard JSON format that any tool (Airflow, dbt, Spark, Flink) can emit to describe what data it reads and writes. Marquez is the reference server implementation: it stores events, provides a REST API, and shows a visual lineage graph.

ConceptWhat it is
OpenLineageOpen spec — defines the JSON event format producers emit (START, COMPLETE, FAIL)
MarquezOpen-source lineage server — stores events, REST API, visual UI
FacetsExtensible metadata on events: schema facet, data quality facet, SQL facet
JobA pipeline step — an Airflow task, a dbt model, a Spark job
DatasetAn input or output — database table, S3 path, Kafka topic
openlineage_event_example.py Lesson 33 · emit a lineage event from Python
Airflow + OpenLineage — zero code changes

Install apache-airflow-providers-openlineage and set OPENLINEAGE_URL=http://marquez:5000. Every Airflow task then automatically emits START/COMPLETE/FAIL lineage events with input/output dataset information extracted from operator parameters. For PostgresOperator, OpenLineage parses the SQL and extracts table names. You get full column-level lineage from your Postgres source tables all the way to your DuckDB mart tables — without writing a single line of lineage code.

📜 Data contracts — the emerging practice

A data contract is a formal agreement between the team that produces a dataset and the teams that consume it. It specifies: schema (column names and types), quality guarantees (not null columns, allowed values), SLA (freshness, availability), and ownership.

ElementWhat it specifiesEnforced by
SchemaColumn names, data types, required fieldsGE schema test, dbt schema.yml, Avro/Protobuf schema registry
Completeness SLA"payment_id is never null", "table has > 1,000 rows daily"GE not_null, dbt not_null, row count threshold
Freshness SLA"Data is no older than 6 hours"dbt source freshness, GE expect_column_max_to_be_between on timestamp
Semantics"amount_kes is always in Kenyan Shillings, not USD"Documentation — enforced by code review and GE range tests
OwnershipWho is responsible for the dataset; who to contact on failureSlack alert routing, PagerDuty ownership
When to invest in each tool: Start with dbt schema tests — they are free and already in your dbt project. Add Great Expectations when you need to validate raw data before it enters your warehouse (at the extraction boundary). Add OpenLineage + Marquez when you have 10+ tables and debugging broken dashboards starts taking more than a day. Data contracts are worth formalising when multiple teams depend on your dataset and you need clear ownership of quality guarantees.

🧠 Quiz — 4 questions

Question 1 of 4
Why are "silent" data errors more dangerous than "loud" ones?
Question 2 of 4
In Great Expectations, what does a Checkpoint do?
Question 3 of 4
What is the difference between OpenLineage and Marquez?
Question 4 of 4
Which data quality dimension does a test like "max(created_at) must be within the last 6 hours" primarily address?
0
/ 4
Data quality mastered. Arc F complete!

Next up: Lesson 34 — Capstone Batch Pipeline.

🎤 Interview Q&A

Easy What are the five dimensions of data quality and give one test for each?
+
Completeness — required fields must be present. Test: not_null on primary key and mandatory columns; row count must be above a minimum threshold. Uniqueness — no duplicate primary keys. Test: unique on the primary key column; alternatively, COUNT(*) == COUNT(DISTINCT pk). Validity — values must be in the allowed domain. Test: accepted_values for categorical columns; between for numeric range; regex for format (email, phone, ID). Timeliness — data must be recent enough. Test: MAX(created_at) > NOW() - INTERVAL 6 hours; row count must not be zero for a day that should have data. Consistency — values must be consistent across systems. Test: SUM(amount) in the staging table equals SUM(total_revenue) in the mart; foreign key check (relationships in dbt) confirms every customer_id in payments exists in the customers table.
Medium Where in a data pipeline should you place Great Expectations validation, and why?
+
The most valuable placement is at the extraction boundary — after you pull raw data from the source system but before any transformation or loading. This is the point where you first own the data; validating here means bad data never enters your warehouse or transformation layer. If validation fails, the pipeline stops cleanly and the downstream systems remain in their last valid state. A second valuable placement is at the loading boundary — after transformation, before loading to the production mart. This catches transformation bugs: a join that accidentally duplicated rows, a window function that produced nulls, a currency conversion that resulted in negative values. In dbt projects, this second layer is naturally handled by dbt test running against the transformed model before it's exposed to stakeholders. The key principle: validate at every system boundary where you transition data between ownership domains — from source to staging, from staging to mart, from mart to external consumers.
Hard What is column-level lineage and why is it important for debugging broken KPIs?
+
Table-level lineage tells you that mart_revenue_by_region was produced by the daily_etl job, which read from stg_payments, which came from the borapay_raw.payments table. That is useful but not sufficient. Column-level lineage tells you that the specific column total_revenue_kes in mart_revenue_by_region was derived from SUM(amount_kes) in stg_payments, which came from the amount column in borapay_raw.payments, transformed by the cast(amount as numeric(14,2)) expression in the staging model. When a KPI is wrong, column-level lineage lets you immediately pinpoint whether the error is in the raw source column, the cast expression in staging, the aggregation formula in the mart, or the currency conversion in an intermediate model — instead of reading through every SQL file manually. OpenLineage supports column-level lineage via the columnLineage facet on output datasets. dbt generates column-level lineage from the SQL AST (abstract syntax tree) of each model and exposes it via the manifest.json. Tools like Atlan, DataHub, and Marquez visualise this as an interactive graph where clicking any column shows its full transformation ancestry.

✅ Key takeaways

  • Silent data errors — bad data that passes validation and corrupts dashboards or ML models — are more expensive than loud pipeline failures. The goal of data quality tooling is to make every bad-data scenario loud and caught early.
  • Great Expectations validates data at the extraction boundary before it enters your warehouse. Wrap a Checkpoint in an Airflow PythonOperator — if validation fails, the task raises an exception and downstream tasks never run.
  • dbt schema tests (not_null, unique, accepted_values, relationships) plus dbt-expectations cover 90% of data quality needs at the transformation layer. Run them with dbt build to enforce quality before marts are exposed to stakeholders.
  • OpenLineage is an open specification for lineage events; Marquez is the reference server. Install the Airflow OpenLineage provider and every task emits lineage automatically — no code changes needed. Column-level lineage lets you trace any broken KPI back to its source column in minutes instead of hours.
  • Data contracts formalise the agreement between data producers and consumers: schema, not-null guarantees, freshness SLA, and ownership. Start with dbt schema.yml as a lightweight contract; invest in formal contracts when multiple teams depend on your dataset.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Build pipelines stakeholders trust

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →