Workflow orchestration

An orchestrator answers questions a scheduler cannot: run task B only if task A succeeded, retry with backoff, alert when the whole thing exceeds its SLA, and re-run last Tuesday without hand-editing anything. Apache Airflow models a pipeline as a DAG - a directed acyclic graph of tasks written in Python, where dependencies are explicit (extract >> transform >> load) and the scheduler turns the graph into runs.

The scheduling model that trips everyone

An Airflow schedule describes a data interval, not a wall-clock time. A DAG on a daily schedule has a run for each day - and the run for “yesterday” starts after yesterday ends. Concretely: a @daily DAG’s run whose data interval covers Monday fires at 00:00 Tuesday. The first run of a new DAG starts one interval after start_date, not at it.

This is correct behaviour, not a bug: a pipeline processes a period of data, and the period must be complete before it runs. But it means “my 6am job” in Airflow runs at 6am covering the previous day’s window - which is usually what you wanted, provided you know which window the run is for. The data interval is available to tasks as data_interval_start and data_interval_end; using datetime.now() inside a task instead is how pipelines end up silently processing the wrong slice. Airflow 3 renamed execution_date to the clearer logical_date - same concept, less confusion about when versus for-when.

Task semantics that matter

• Idempotent tasks. A retried task must be safe to re-run: write to a dated partition and overwrite it, upsert on a key, or make the effect checkable. A task that appends unconditionally duplicates on every retry.
• retries and retry_delay. Defaults come from default_args; exponential backoff via retry_exponential_backoff. Retries handle transient failures - they cannot rescue a task whose input is genuinely missing; that is what depends_on_past and data checks are for.
• SLAs and catchup. An SLA missed fires a callback/alert; catchup=False on the DAG stops the scheduler from immediately queuing every missed interval since start_date - the standard surprise when a paused DAG is unpaused after a month.
• Backfills are re-runs. airflow dags backfill creates runs for past intervals; it exists because idempotent, interval-parameterised tasks can be safely re-executed for any window. If your tasks cannot be backfilled, they were not written to the scheduling model.

Airflow versus Dagster and Prefect

• Airflow is the incumbent: DAGs as code, a mature operator ecosystem, executor choices (Local, Celery, Kubernetes) for scaling workers. It can run multiple schedulers concurrently for resiliency - the schedulers coordinate through row locks on the shared metadata database (PostgreSQL 12+ or MySQL 8.0+; MariaDB is not supported for this), so the metadata DB, not the scheduler, is the single point of failure to engineer around. Its real weaknesses are that local dev is heavyweight and the DAG file is code that must be parsed on every scheduler loop, so authoring discipline matters.
• Dagster re-frames pipelines around assets (tables, files, models) rather than tasks: you declare what should exist and Dagster works out what to run. Better for teams that think in data products and want lineage as a first-class object.
• Prefect keeps the same task-graph model with lighter infrastructure and dynamic, parameterised workflows as a design goal.

None of them removes the underlying requirements - idempotent work, explicit dependencies, observable runs. They change how much of the plumbing you write versus how much the framework infers.

Failure modes and diagnostics

• Scheduler or worker death. A dead scheduler stops new runs; queued work waits. Workers dying mid-task leaves tasks in running state until the timeout or heartbeat check marks them failed - the same reason tasks must tolerate being killed.
• depends_on_past deadlocks. A task that waits for its own previous run stalls permanently if the previous interval failed and was cleared - useful for ordered backfills, dangerous as a default.
• Upstream-stuck cascades. One hung sensor or a source that never materialises holds every downstream task. execution_timeout on sensors and explicit timeouts on tasks bound how long a run can wait.
• Resource contention at scale. Thousands of DAGs parsed on every scheduler loop cost CPU and file I/O; the DAG processor and parse-time settings exist because DAG files are re-read constantly, not once at deploy.

What this is not for

An orchestrator schedules work; it is not a compute engine and should not hold your data - data moves through the tasks, not through the DAG. It is not a replacement for a queue when the work is high-frequency, latency-sensitive event processing (that is streaming territory). And for a single cron-shaped job with no dependencies, it is more machinery than the problem needs - cron plus alerting is sometimes the honest answer.