Airflow vs Cron: When a Crontab Line Isn't Enough

Prefer to watch? ⚡ The 2-minute version ✈ Telegram

"Just use cron" and "we need Airflow" are both wrong about half the time.

They look like the same tool — both run your jobs on a schedule — so teams pick one out of habit and pay for it later. But scheduling isn't the real difference. The real difference is whether your work is one command on a clock or a graph of tasks that depend on each other.

The one-line mental model:

cron asks "is it time yet?"
Airflow asks "what's the state of my whole pipeline?"

cron: one line, one clock

cron is a time-based scheduler that has shipped on essentially every Unix box for about forty years. You give it a time expression and a command. That's the entire interface.

# min hour dom mon dow   command
0    2    *   *   *      /opt/etl/run.sh

At 02:00 every day, cron runs the script. There is no daemon to operate, no database to back up, no dashboard to log into. It is one of the most battle-tested pieces of software you will ever depend on, and for a single timed job nothing is simpler or more reliable.

The important thing to notice is what cron knows: the clock, and nothing else. It does not know what your command does, whether it succeeded, or whether anything else depends on it.

Where the crontab starts to hurt

The failure mode shows up the moment you have a second step. Say load must run after extract. cron has no way to express "after", so you do the thing everyone does — you guess a gap:

0    2    *   *   *      /opt/etl/extract.sh
10   2    *   *   *      /opt/etl/load.sh    # extract "should" be done by now

That works right up until the night extract runs long, or crashes. cron does not care. At 02:10 it fires load exactly on schedule — against stale data, partial data, or no data at all. And you will not find out from cron, because:

No dependency awareness. 02:10 is a wish, not a guarantee.
No retries. A transient network blip means the run is simply lost until tomorrow.
No backfill. The box was down for two days? Those two days never happen.
No observability. "Did it run?" is answered by SSH-ing in and grepping logs.

None of these are bugs. cron does exactly what it promised. You just asked it a question it was never designed to answer.

Airflow: the pipeline is the unit

Airflow attacks the other question. Instead of scheduling commands, you describe your work as a DAG — a directed acyclic graph of tasks, where the edges are dependencies:

from airflow.decorators import dag, task
from datetime import datetime

@dag(schedule="0 2 * * *", start_date=datetime(2026, 1, 1), catchup=True)
def daily_etl():

    @task(retries=3)
    def extract():
        return fetch_rows()

    @task()
    def transform(rows):
        return clean(rows)

    @task()
    def load(rows):
        write_warehouse(rows)

    load(transform(extract()))

daily_etl()

The last line is the whole point. load does not run at a time — it runs when transform has succeeded, which runs when extract has succeeded. The schedule only starts the graph; the graph decides the rest.

That single change buys you the entire list cron was missing:

Dependencies are declared, not approximated with a ten-minute gap.
Retries are a keyword argument (retries=3), applied per task.
Backfills are first-class: catchup=True and Airflow will run the windows it missed, because each run is tied to a data interval rather than to "now".
Observability is a web UI showing every run, every task, every log, colour-coded green and red.

And the bill for that

Airflow is not a binary you drop in /usr/local/bin. It is a system you now operate:

a scheduler process that must stay up,
a metadata database (Postgres) holding every task state,
a queue (Redis/Celery) or a Kubernetes executor,
a pool of workers,
and the upgrades, migrations and on-call that come with all of it.

To fire one nightly python etl.py, that is pure overhead — you have replaced a line of text with a distributed system, and now the scheduler itself can page you at 3 a.m. The joke that writes itself: you cannot run Airflow to make sure Airflow is running.

Side by side

Dimension	cron	Apache Airflow
Core question	"Is it time yet?"	"What's the state of my pipeline?"
Unit of work	One command	A DAG of tasks
Dependencies	None — approximated with time gaps	Declared in the graph
Retries	Whatever you hand-roll in the script	Built in, per task
Missed runs / backfill	Lost forever	Backfilled by data interval
Observability	grep the logs over SSH	Web UI, run history, per-task logs
Operational cost	Zero — already installed	Scheduler + Postgres + queue + workers
Scale-out	One box	Worker pool / Kubernetes
Best fit	A single timed job	A multi-step pipeline with state

The same split shows up in LLM pipelines

If you are building AI systems rather than classic ETL, don't assume this is a data-engineering-only question — the orchestration lane is where most RAG systems quietly rot.

A retrieval pipeline is a DAG whether you admit it or not: fetch sources → chunk → embed → upsert into the vector store → rebuild the index → smoke-test retrieval. Every one of those steps depends on the one before it, and two properties of LLM workloads make cron a particularly bad fit:

Model APIs are flaky and rate-limited. An embedding job over 200k chunks will hit a 429 or a timeout at some point. With Airflow that's retries=5 plus exponential backoff on one task; with cron the whole nightly script dies and you find out when answers go stale.
Re-embedding is a backfill problem. Switch embedding models, change your chunk size, or fix a parser bug, and you need to re-run the pipeline over a historical range of documents. That is exactly what a backfill is — and cron has no concept of a past window at all.

The failure is also nastier than a missed report. If embed fails and upsert runs anyway on a partial batch, you don't get an error — you get a vector store that is silently missing a slice of your corpus, and a chatbot that confidently answers "I don't have information about that." Partial state in a retrieval index looks exactly like a working system.

The heuristic carries over cleanly: a nightly "re-embed the docs folder" script is fine on cron. A multi-source ingestion pipeline feeding a production RAG endpoint — with evaluation gates and index swaps — wants a real orchestrator.

Verdict

Reach for cron when the work is genuinely one command on a clock: a nightly backup, a certificate renewal, a single script that is idempotent and cheap to re-run. Adding an orchestrator there buys you nothing and hands you a scheduler, a database and a queue to babysit.

Reach for Airflow (or Dagster, or Prefect — the model is the same) the moment step B needs step A to have succeeded, or you need retries, backfills, and a straight answer to "what ran last night?". The tell is simple: the first time you write a crontab line whose correctness depends on guessing how long the previous line takes, you have outgrown cron.

One command on a clock, or a graph of tasks with state. Match the tool to the shape of the work — not to what's trendy.

Watch the 2-minute version for the whole comparison in one pass.