Apache Airflow
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring data pipelines expressed as directed acyclic graphs (DAGs) of tasks. It originated at Airbnb and is now a top-level Apache Software Foundation project used to orchestrate ETL, ML, and infrastructure automation workflows.
What gets deployed with it
- PostgreSQL HA Cluster — required Airflow metadata database dependency. Stores DAG state, task instances, logs, and variables.
Baseline topologies
| Topology | Minimum nodes | Roles |
|---|---|---|
| scheduler-webserver | 1 | airflow-scheduler, airflow-webserver |
| scheduler-webserver-workers | 3 | airflow-scheduler, airflow-webserver, airflow-worker |
What drives the cost
- Scheduled DAGs
- DAGs that are actively scheduled and contribute to scheduler and metadata DB load.
- Peak running tasks
- Maximum task instances expected to run at the same time. Primary driver for worker capacity.
Where it runs
bare_metal, dedicated, cloud_vm
Size Apache Airflow and get a cost estimate