Prefect
Prefect is a Python-native workflow orchestration framework that turns ordinary functions into monitored, retryable flows and tasks, handling scheduling, state tracking, and failure recovery for data and ML pipelines.
What gets deployed with it
- PostgreSQL HA Cluster — required Prefect Server requires PostgreSQL for production deployments. SQLite is supported only for local development. The database stores flow run state, task runs, deployment definitions, and worker heartbeats.
Baseline topologies
| Topology | Minimum nodes | Roles |
|---|---|---|
| single-node | 1 | prefect-server |
| server-plus-workers | 2 | prefect-server, prefect-worker |
What drives the cost
- Peak concurrent flow runs
- Peak number of flow runs executing simultaneously. Primary driver for worker node count and CPU/RAM headroom on execution infrastructure.
- Active scheduled flows
- Total number of flows with active schedules. Drives Prefect Server scheduler heartbeat load and database poll frequency.
Where it runs
bare_metal, dedicated, cloud_vm
Size Prefect and get a cost estimate