Airflow
by Community
Platform created by the community to programmatically author, schedule and monitor workflows.
OSS
Airflow
Added 1 June 2026
Overview
Airflow is a community platform for programmatically authoring, scheduling, and monitoring workflows. It uses directed acyclic graphs (DAGs) defined in Python to orchestrate complex data pipelines and tasks.
Best for
Best for
Teams that need a robust, code-driven scheduler for batch-oriented data pipelines
Use cases
- Scheduling and running ETL pipelines on a recurring basis
- Orchestrating multi-step data processing workflows with dependencies
- Monitoring pipeline execution and alerting on failures
Notes
Airflow is a community platform for programmatically authoring, scheduling, and monitoring workflows. It uses directed acyclic graphs (DAGs) defined in Python to orchestrate complex data pipelines and tasks.
Use cases
- Scheduling and running ETL pipelines on a recurring basis
- Orchestrating multi-step data processing workflows with dependencies
- Monitoring pipeline execution and alerting on failures
Pros
- Open source with a large community and extensive integrations
- Python-native DAGs make pipeline logic testable and version-controllable
- Rich UI for visualizing task status, logs, and execution history
Cons
- Steep learning curve for configuring and managing the production environment
- Not designed for real-time streaming or low-latency task execution
- Scaling the scheduler and workers requires careful infrastructure planning
Indexed from awesome-llmops and enriched against its public facts.
Pros
- Open source with a large community and extensive integrations
- Python-native DAGs make pipeline logic testable and version-controllable
- Rich UI for visualizing task status, logs, and execution history
Cons
- Steep learning curve for configuring and managing the production environment
- Not designed for real-time streaming or low-latency task execution
- Scaling the scheduler and workers requires careful infrastructure planning
Pairs with
Other entries in the index that connect to this one. Click through to see the chain.
Prefect
Community
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Argo Workflows
Community
Workflow Engine for Kubernetes
Kubeflow
Community
Machine Learning Toolkit for Kubernetes
Get the free Developer’s Field Guide
A 27-page field guide to the AI coding workflow with Claude. Claude Code, MCP servers, the prompt patterns that work, and what to delegate. Free.
Enter your work email. We send it straight over, plus a few short notes worth knowing. Unsubscribe any time.