📋 What Airflow solves
A cron job runs a script. That's fine for a single script. But real data pipelines are chains of steps: extract from Postgres, validate the data, transform with PySpark, load to DuckDB, send a Slack alert. When step 3 fails at 2 AM, cron does nothing — it doesn't retry, it doesn't notify you, and it doesn't tell you which step failed or why.
Apache Airflow was built to solve exactly this. It gives you three things cron cannot:
| Problem with cron | Airflow solution |
|---|---|
| Silent failures — a job exits 1, nobody knows | Task state tracking: success, failed, up_for_retry, skipped — visible in the UI |
| No retry logic | retries=3, retry_delay=timedelta(minutes=5) per task |
| No dependency tracking between scripts | DAG: directed acyclic graph — task B only runs after task A succeeds |
| No historical record of runs | Airflow stores every DAG run and task log in a metadata DB (Postgres or MySQL) |
| No parallelism | Tasks with no dependency can run in parallel via the CeleryExecutor or KubernetesExecutor |
Airflow itself does not process your data — it triggers and monitors the tools that do (PySpark, dbt, Python scripts, SQL queries). Think of it as a very smart scheduler with a web UI, not a replacement for Spark.
🧬 DAG anatomy
A DAG (Directed Acyclic Graph) is a Python file that Airflow reads from the dags/ folder. It defines tasks and the dependencies between them. The key parameters of a DAG are:
| Parameter | What it controls | Example value |
|---|---|---|
dag_id | Unique name — shown in the UI | 'daily_etl' |
schedule_interval | When the DAG runs — cron string or preset | '@daily', '0 6 * * *' |
start_date | First logical run date | datetime(2024, 1, 1) |
catchup | Whether to backfill missed runs since start_date | False (recommended for new pipelines) |
default_args | Defaults applied to every task in the DAG | retries=2, retry_delay=timedelta(minutes=5) |
tags | Labels for filtering in the UI | ['etl', 'payments'] |
⚙️ Operators — the building blocks
An operator defines a single unit of work. Airflow ships with dozens; you pick the right one for each task:
| Operator | When to use it | Key parameter |
|---|---|---|
PythonOperator | Run any Python function | python_callable=my_func |
BashOperator | Run a shell command or script | bash_command='...' |
PostgresOperator | Execute SQL against Postgres | sql='SELECT ...' |
SparkSubmitOperator | Submit a PySpark job to a cluster | application='job.py' |
S3KeySensor | Wait until a file appears in S3 | bucket_key='s3://...' |
FileSensor | Wait until a local file exists | filepath='/tmp/ready.flag' |
HttpSensor | Poll an HTTP endpoint until 200 | endpoint='/health' |
DummyOperator | Placeholder / fan-out join point | None |
PostgresOperator example
🔄 TaskFlow API — cleaner DAGs with decorators
Airflow 2.0 introduced the TaskFlow API: Python decorators that eliminate boilerplate. Instead of creating PythonOperator instances manually, you decorate functions with @task and the framework wires everything automatically — including XCom passing between tasks.
When a @task returns a value, Airflow serialises it to the metadata database as an XCom (cross-communication) entry. The downstream task receives it as a function argument. Keep XCom payloads small — kilobytes, not megabytes. For large datasets, write to S3/GCS/disk and pass only the path via XCom.
📡 Sensors — waiting for upstream events
Sensors are operators that poke a condition on an interval until it becomes true (or they time out). They are essential when your pipeline depends on an upstream team dropping a file, an API becoming available, or a Kafka topic reaching a certain offset.
poke mode the sensor holds a worker process alive between checks — wasteful if you're waiting hours. In reschedule mode Airflow releases the worker slot between pokes and reschedules the task when it's time to check again. Always use reschedule for sensors with long wait times.
🏭 Real DAG — daily ETL: Postgres → PySpark → DuckDB
Here is a production-style DAG combining everything covered above: a daily ETL that extracts from Postgres, transforms with a PySpark script submitted via SparkSubmitOperator, and loads results to DuckDB.
🐳 Running Airflow locally with Docker Compose
The fastest way to get Airflow running locally is the official Docker Compose setup. It starts the webserver, scheduler, triggerer, and a Postgres metadata database.
After triggering a run: open Graph view to see task dependencies, click a task box → Log to see output, click Gantt to see which tasks were slow. To re-run a failed task: click the task → Clear → confirm. Airflow will re-execute only that task and any downstream dependents.
⚠️ Common mistakes
start_date=datetime(2022, 1, 1) and catchup=True, Airflow will schedule hundreds of backfill runs on first deployment and flood your executor queue. Always set catchup=False for new DAGs, and use airflow dags backfill intentionally if you need historical runs.FileSensor waiting six hours in poke mode keeps a worker process alive the entire time, blocking other tasks from running on that worker slot. Use mode='reschedule' for any sensor that may wait more than a few minutes.🧠 Quiz — 4 questions
catchup=False on an Airflow DAG do?@task-decorated functions?mode='poke' and mode='reschedule' in an Airflow Sensor?task_a >> task_b mean?Next: dbt with DuckDB — models, tests, and lineage.
🎤 Interview Q&A
a >> b >> a), Airflow raises an error and refuses to load the DAG. In practice this constraint maps naturally to real pipelines: extract always precedes transform, transform always precedes load — the flow is always forward.
ExternalTaskSensor for exactly this pattern. You configure it with the external_dag_id and external_task_id of the upstream task, and it polls Airflow's metadata database until that task reaches the success state. Alternatively, the upstream DAG can write a success flag file (_SUCCESS) to a shared location and the downstream DAG uses a FileSensor or S3KeySensor — this works across Airflow instances that don't share a metadata DB. A third option is to merge both DAGs into a single DAG if they always run together and have a fixed timing relationship; this simplifies dependency management but increases DAG complexity. In practice, cross-DAG ExternalTaskSensor is the most common pattern because it doesn't require shared filesystem access and uses Airflow's own state tracking.
✅ Key takeaways
- Airflow solves what cron cannot: task dependencies, automatic retries, historical run tracking, parallel execution, and a web UI with Gantt/log views — all defined in code.
- A DAG is a Python file with tasks connected by the
>>operator. Key parameters:schedule_interval,catchup=False,default_argsfor retries and alerts. - The TaskFlow API (
@dag,@task) eliminates boilerplate: return values automatically flow via XCom. Keep XCom payloads small — pass file paths, not data. - Sensors wait for external conditions (file exists, API responds, upstream DAG succeeds). Always use
mode='reschedule'for sensors that may wait more than a few minutes to avoid blocking worker slots. - Run Airflow locally with the official Docker Compose in five commands. In production, use CeleryExecutor (multi-node) or KubernetesExecutor (Pod-per-task isolation) depending on your infrastructure.
Build production pipelines with Airflow, Spark & Kafka
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →