📋 What Airflow solves

A cron job runs a script. That's fine for a single script. But real data pipelines are chains of steps: extract from Postgres, validate the data, transform with PySpark, load to DuckDB, send a Slack alert. When step 3 fails at 2 AM, cron does nothing — it doesn't retry, it doesn't notify you, and it doesn't tell you which step failed or why.

Apache Airflow was built to solve exactly this. It gives you three things cron cannot:

Problem with cronAirflow solution
Silent failures — a job exits 1, nobody knowsTask state tracking: success, failed, up_for_retry, skipped — visible in the UI
No retry logicretries=3, retry_delay=timedelta(minutes=5) per task
No dependency tracking between scriptsDAG: directed acyclic graph — task B only runs after task A succeeds
No historical record of runsAirflow stores every DAG run and task log in a metadata DB (Postgres or MySQL)
No parallelismTasks with no dependency can run in parallel via the CeleryExecutor or KubernetesExecutor
Airflow is an orchestrator, not a compute engine

Airflow itself does not process your data — it triggers and monitors the tools that do (PySpark, dbt, Python scripts, SQL queries). Think of it as a very smart scheduler with a web UI, not a replacement for Spark.

🧬 DAG anatomy

A DAG (Directed Acyclic Graph) is a Python file that Airflow reads from the dags/ folder. It defines tasks and the dependencies between them. The key parameters of a DAG are:

ParameterWhat it controlsExample value
dag_idUnique name — shown in the UI'daily_etl'
schedule_intervalWhen the DAG runs — cron string or preset'@daily', '0 6 * * *'
start_dateFirst logical run datedatetime(2024, 1, 1)
catchupWhether to backfill missed runs since start_dateFalse (recommended for new pipelines)
default_argsDefaults applied to every task in the DAGretries=2, retry_delay=timedelta(minutes=5)
tagsLabels for filtering in the UI['etl', 'payments']
basic_dag.py Lesson 29 · DAG anatomy
airflow dags trigger daily_payment_etl --run-id manual_test
[2024-01-15 06:00:01] check_postgres SUCCESS 0.8s [2024-01-15 06:00:03] extract_payments SUCCESS 4.2s returned: /tmp/payments_2024-01-15.parquet [2024-01-15 06:00:09] validate_extract SUCCESS 1.1s

⚙️ Operators — the building blocks

An operator defines a single unit of work. Airflow ships with dozens; you pick the right one for each task:

OperatorWhen to use itKey parameter
PythonOperatorRun any Python functionpython_callable=my_func
BashOperatorRun a shell command or scriptbash_command='...'
PostgresOperatorExecute SQL against Postgressql='SELECT ...'
SparkSubmitOperatorSubmit a PySpark job to a clusterapplication='job.py'
S3KeySensorWait until a file appears in S3bucket_key='s3://...'
FileSensorWait until a local file existsfilepath='/tmp/ready.flag'
HttpSensorPoll an HTTP endpoint until 200endpoint='/health'
DummyOperatorPlaceholder / fan-out join pointNone

PostgresOperator example

postgres_operator_example.py Lesson 29 · SQL task

🔄 TaskFlow API — cleaner DAGs with decorators

Airflow 2.0 introduced the TaskFlow API: Python decorators that eliminate boilerplate. Instead of creating PythonOperator instances manually, you decorate functions with @task and the framework wires everything automatically — including XCom passing between tasks.

taskflow_dag.py Lesson 29 · @dag and @task decorators
task logs — extract task
Extracted 14372 rows for 2024-01-15 [2024-01-15 06:02:14] extract SUCCESS 3.8s [2024-01-15 06:02:19] transform SUCCESS 0.4s [2024-01-15 06:02:21] load SUCCESS 0.9s "Loaded summary: {"total_txns": 14372, "settled_txns": 13891, "total_kes": 42185000.0, "avg_kes": 3038.7}"
XCom under the hood

When a @task returns a value, Airflow serialises it to the metadata database as an XCom (cross-communication) entry. The downstream task receives it as a function argument. Keep XCom payloads small — kilobytes, not megabytes. For large datasets, write to S3/GCS/disk and pass only the path via XCom.

📡 Sensors — waiting for upstream events

Sensors are operators that poke a condition on an interval until it becomes true (or they time out). They are essential when your pipeline depends on an upstream team dropping a file, an API becoming available, or a Kafka topic reaching a certain offset.

filesensor_dag.py Lesson 29 · FileSensor for upstream dependency
mode='reschedule' vs mode='poke': In poke mode the sensor holds a worker process alive between checks — wasteful if you're waiting hours. In reschedule mode Airflow releases the worker slot between pokes and reschedules the task when it's time to check again. Always use reschedule for sensors with long wait times.

🏭 Real DAG — daily ETL: Postgres → PySpark → DuckDB

Here is a production-style DAG combining everything covered above: a daily ETL that extracts from Postgres, transforms with a PySpark script submitted via SparkSubmitOperator, and loads results to DuckDB.

daily_etl_dag.py Lesson 29 · Production ETL pipeline
airflow dags list-runs -d daily_etl_postgres_spark_duckdb --limit 3
dag_id run_id state execution_date daily_etl_postgres_spark_duckdb scheduled__2024-01-15T05:00 success 2024-01-15 05:00:00+00:00 daily_etl_postgres_spark_duckdb scheduled__2024-01-14T05:00 success 2024-01-14 05:00:00+00:00 daily_etl_postgres_spark_duckdb scheduled__2024-01-13T05:00 failed 2024-01-13 05:00:00+00:00

🐳 Running Airflow locally with Docker Compose

The fastest way to get Airflow running locally is the official Docker Compose setup. It starts the webserver, scheduler, triggerer, and a Postgres metadata database.

shell Lesson 29 · Airflow local setup
Airflow UI tips

After triggering a run: open Graph view to see task dependencies, click a task box → Log to see output, click Gantt to see which tasks were slow. To re-run a failed task: click the task → Clear → confirm. Airflow will re-execute only that task and any downstream dependents.

⚠️ Common mistakes

catchup=True on a new DAG with an old start_date
If you set start_date=datetime(2022, 1, 1) and catchup=True, Airflow will schedule hundreds of backfill runs on first deployment and flood your executor queue. Always set catchup=False for new DAGs, and use airflow dags backfill intentionally if you need historical runs.
Returning large objects from @task via XCom
XCom stores values in the Airflow metadata database. Returning a Pandas DataFrame or a list of 100,000 rows will bloat the DB and cause slow UI queries. Return a file path string instead — write the data to disk or cloud storage, then pass the path.
Using mode='poke' for long-running sensors
A FileSensor waiting six hours in poke mode keeps a worker process alive the entire time, blocking other tasks from running on that worker slot. Use mode='reschedule' for any sensor that may wait more than a few minutes.

🧠 Quiz — 4 questions

Question 1 of 4
What does setting catchup=False on an Airflow DAG do?
Question 2 of 4
In the TaskFlow API, how is data passed between @task-decorated functions?
Question 3 of 4
What is the difference between mode='poke' and mode='reschedule' in an Airflow Sensor?
Question 4 of 4
In a DAG definition, what does task_a >> task_b mean?
0
/ 4
Airflow fundamentals locked in.

Next: dbt with DuckDB — models, tests, and lineage.

🎤 Interview Q&A

Easy What is a DAG in Airflow and why must it be acyclic?
+
A DAG (Directed Acyclic Graph) is a collection of tasks with defined dependencies between them, represented as a Python file. "Directed" means dependencies flow in one direction — task A triggers task B. "Acyclic" means there are no circular dependencies — if task B depended back on task A, the scheduler would have no valid execution order and would deadlock. Airflow enforces acyclicity at DAG parse time: if your code creates a cycle (e.g., a >> b >> a), Airflow raises an error and refuses to load the DAG. In practice this constraint maps naturally to real pipelines: extract always precedes transform, transform always precedes load — the flow is always forward.
Medium How would you handle a situation where a downstream DAG needs to wait for an upstream DAG to finish?
+
Airflow provides the ExternalTaskSensor for exactly this pattern. You configure it with the external_dag_id and external_task_id of the upstream task, and it polls Airflow's metadata database until that task reaches the success state. Alternatively, the upstream DAG can write a success flag file (_SUCCESS) to a shared location and the downstream DAG uses a FileSensor or S3KeySensor — this works across Airflow instances that don't share a metadata DB. A third option is to merge both DAGs into a single DAG if they always run together and have a fixed timing relationship; this simplifies dependency management but increases DAG complexity. In practice, cross-DAG ExternalTaskSensor is the most common pattern because it doesn't require shared filesystem access and uses Airflow's own state tracking.
Hard What are the trade-offs between LocalExecutor, CeleryExecutor, and KubernetesExecutor?
+
LocalExecutor runs tasks as subprocesses on the same machine as the scheduler. It supports parallelism (multiple subprocesses) but is limited to a single node — it's the right choice for small teams with modest workloads and the default for Docker Compose setups. CeleryExecutor distributes tasks to a pool of worker nodes via a message broker (Redis or RabbitMQ). It scales horizontally by adding worker nodes and is the most widely deployed executor in production. The trade-off is operational overhead: you must manage the broker, worker fleet, and Flower monitoring UI. KubernetesExecutor spins up a fresh Kubernetes Pod for each task and tears it down on completion. This gives perfect isolation (each task gets its own environment and resources), eliminates the persistent worker fleet, and scales to zero when idle. The trade-off is Pod startup latency (10–30 seconds per task) and the requirement to run on Kubernetes. For data teams already on Kubernetes, KubernetesExecutor is increasingly the default choice because it eliminates worker sizing decisions and eliminates dependency conflicts between tasks.

✅ Key takeaways

  • Airflow solves what cron cannot: task dependencies, automatic retries, historical run tracking, parallel execution, and a web UI with Gantt/log views — all defined in code.
  • A DAG is a Python file with tasks connected by the >> operator. Key parameters: schedule_interval, catchup=False, default_args for retries and alerts.
  • The TaskFlow API (@dag, @task) eliminates boilerplate: return values automatically flow via XCom. Keep XCom payloads small — pass file paths, not data.
  • Sensors wait for external conditions (file exists, API responds, upstream DAG succeeds). Always use mode='reschedule' for sensors that may wait more than a few minutes to avoid blocking worker slots.
  • Run Airflow locally with the official Docker Compose in five commands. In production, use CeleryExecutor (multi-node) or KubernetesExecutor (Pod-per-task isolation) depending on your infrastructure.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Build production pipelines with Airflow, Spark & Kafka

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →