Why Your Airflow Tasks Fail at 3 AM (And Why Re-Running Them Is a Trap)

One outage, four suspects — interrogate in order.

Pull up a chair, grab a cup of chai, and let us talk honestly about what happens at 3 AM when an alert screams on your phone. A task inside your Airflow DAG — that is your Directed Acyclic Graph, fancy engineering talk for the step-by-step workflow that moves your data around — just turned bright, angry red.

What is your very first reaction? Be honest. You log in with sleep in your eyes, navigate to the Airflow user interface, find the dead task, and click “Clear” or “Re-run.” You sit back, hold your breath, and hope it goes green this time.

Congratulations, my friend: you just ran the tech industry’s most popular debugging move — and easily its most useless. The insidious thing is that hitting re-run sometimes actually works. And that occasional success is a trap. It teaches you absolutely nothing about why it failed, so the exact same bug will happily ring your phone tomorrow night, and the night after that, forever.

Treating the re-run button like a casino slot machine — pull the handle, wait for three matching cherries, celebrate when you win — keeps you glued to the screen without teaching you the actual odds. There is a far calmer, smarter way to handle this: classify what went wrong before your finger even touches retry. Almost every data pipeline crash falls neatly into one of four distinct buckets: source data , upstream API , code bug , or transient infra. Each bucket demands exactly one sensible reaction. Nail the bucket down first, and that 3 AM panic shrinks into a calm, ten-minute decision: retry it, route it to the right person, or patch the code.

The industry is finally catching up to this reality. Airflow 3.1 rolled out human-in-the-loop task types — including built-in data quality gates where a human data steward must approve weird numbers before downstream jobs chew on them. Meanwhile, AWS added managed Airflow 3 support to MWAA (Managed Workflows for Apache Airflow). The platform itself is openly admitting what engineers who hold the pager have known for a decade: automated retries are not a cure-all. Some problems simply need a human. Specifically, the right human armed with the right checklist. Here is the field-tested taxonomy to make sure you never guess again.

The blanket retry policy is delay with extra steps

Look at almost any production Airflow repository, and you will spot the classic copy-paste habit: retries = 3 and retry_delay = 5 minutes slapped right inside default_args at the top of the file. It came from an online tutorial five years ago, nobody questioned it, and it feels like a safety net. But let us look at what that default actually does when things go sideways in the real world.

Suppose an upstream partner renames a column from customer_id to client_id. Your Python script hits a KeyError — an error where Python throws its hands up because a dictionary key or column simply does not exist. That code is broken. Retrying it immediately will not magically make the old column appear. Yet Airflow obediently waits five minutes, tries again, fails, waits another five minutes, tries a third time, and fails once more. The alert finally wakes you up fifteen minutes late, completely empty-handed.

Or say a third-party vendor hits you with an HTTP 429 — an error code meaning “Too Many Requests,” where their server tells you to back off because you exceeded your API rate quota. Your blanket policy immediately hammers their endpoint again every five minutes while your quota bucket is bone-dry. You are literally launching an accidental denial-of-service attack against your own SaaS partner on a timer! If a CSV file arrives truncated or mangled, running it three times achieves nothing except wasting cloud compute credits. The only failure that actually benefits from mindless retrying is a temporary hiccup in the network — and a tiny, gentle retry loop was always going to fix that anyway.

Retries are a policy decision, not a default. The right number of retries for a malformed CSV is zero.

A blanket retry policy assumes every crash is the exact same flavor of bad luck. Sorting failures into clear categories turns retrying from blind wishful thinking into deliberate engineering.

Four buckets cover almost every failure

Do not overcomplicate your life with a forty-node academic taxonomy diagram. Nobody reads complex flowcharts when their alarm goes off in the dark. Four buckets are plenty, because there are really only four distinct, correct ways to respond :

• Source data — Log signature: file not found, malformed rows, schema mismatch. First action: stop the task; tell the data producer. Owner: data provider. Retry: 0.

• Upstream API — Log signature: HTTP 429, timeouts, 5xx. First action: back off with jitter; watch the quota. Owner: integration owner. Retry: gentle.

• Code bug — Log signature: KeyError, TypeError, bad SQL, assertion failures. First action: fix the code, then re-run. Owner: pipeline author. Retry: 0.

• Transient infra — Log signature: worker evicted, SIGTERM, connection refused. First action: retry; it usually self-heals. Owner: platform team. Retry: aggressive.

Look closely at that list. Notice something ironic? The two categories where retrying is 100% useless — bad source data and broken code — are the exact places where teams waste hours waiting on retries. Notice also that the “Owner” field turns this simple list into a triage switchboard. Each bucket belongs on the desk of a completely different human.

Where each bucket lives in the pipeline

Failures have natural habitats. Just like bugs thrive in warm, damp corners, specific pipeline errors breed in specific pipeline stages. Data issues cluster right where files land from external partners. API failures live at the boundary where your extract workers talk to outside SaaS platforms. Code bugs dwell in your transform steps where complex business logic lives. And transient infrastructure issues lurk everywhere cloud servers, networks, and containers touch each other.

Each failure bucket has a habitat. Know where to look before you open the log.

Knowing the habitat gives you an instant shortcut: an extract task pulling records from Salesforce or Stripe is an upstream API problem until proven otherwise. A database load task dumping cleaned rows into BigQuery (Google’s managed cloud data warehouse) that suddenly breaks after weeks of running fine with zero code updates is transient infra until proven otherwise.

Classify first, fix second: the three-question tree

When an alert fires, do not start reading lines from the middle of a two-thousand-line log file. Walk through three simple questions in exact order. The early questions knock out the most obvious culprits immediately.

Three questions, asked in order, route every failure to its bucket and its owner.

Question 1: Did the input data arrive intact? Check the cloud storage bucket or landing directory. If the vendor never dropped the CSV file, or if the file contains zero bytes, stop right there. Nothing downstream matters until that file exists.

Question 2: Do you see an HTTP 429, timeout, or 5xx in the log? Hit Ctrl+F in the log window. Searching for a 429 (rate-limited) or a 503 (service unavailable) takes five seconds flat. This immediately draws the line between “the vendor’s servers are struggling” versus “our internal code did something wrong.”

Question 3: Does the exact same input fail the exact same way twice? If you feed identical input data into the task and it throws the exact same line-number traceback both times, the universe is not messing with you. You have a code bug. Stop hoping it will fix itself.

A failure you can classify in thirty seconds reaches the right person in minutes, not hours.

Always keep an escape hatch ready. If an error does not fit these three questions cleanly, mark it as UNKNOWN and route it straight to the data on-call engineer with the raw log snippet attached. Forcing a weird edge case into the wrong bucket just to keep things neat is far worse than admitting you don’t know. “Unknown failure, requires human review” is honest engineering — and honesty is rare enough at 3 AM.

A runbook per bucket

Categorizing problems without clear next steps is just academic trivia. Every single bucket must have a dead-simple runbook — keep it to five steps max, or nobody on call will ever follow it:

1. Source data: First, verify the file’s size and contents in the landing storage. Second, ensure retries are set to zero so you don’t spin your wheels. Third, ping the data provider or vendor team with three clean facts: the exact filename, what schema you expected, and what corrupt columns you actually found. Fourth, move the corrupt file to a quarantine directory so it doesn’t block future runs. Fifth, trigger the pipeline only after the corrected replacement file lands. How to prevent it: Add an automated validation step at the front of your DAG that validates file presence, row counts, and schema before your heavy transformations ever touch the data.

2. Upstream API: First, check the vendor’s public status page and your internal API rate-limit dashboards. Second, confirm whether the failure is a temporary throttle or expired authentication credentials. Third, configure retries to use exponential backoff combined with “jitter” (adding a small random delay so a dozen parallel tasks do not all retry at the exact same microsecond). Fourth, dial down the DAG’s maximum concurrent worker slots for that specific vendor until their rate-limit window resets. How to prevent it: Cache heavy requests where practical, paginate query results in small batches, and hard-cap concurrency per provider.

3. Code bug: First, replicate the error in a local test using the exact data payload that crashed in production. Second, fix the underlying logic in your code rather than hacking an Airflow configuration parameter. Third, write a regression test (an automated unit test verifying that this specific input variation never breaks the pipeline again). Fourth, commit the fix through your standard Continuous Integration (CI) test pipeline. Fifth, clear the failed production task and let it process cleanly. How to prevent it: Run airflow dags test against realistic test fixtures in your pull request pipeline before code gets merged.

4. Transient infra: First, let Airflow retry automatically — this is the one and only category where hitting retry is the correct engineering remedy. Second, ensure backoff intervals do not stretch out into hours. Third, if the task still fails after burning through all configured retries, escalate the ticket to the platform infrastructure team with the attempt history attached. How to prevent it: Make your database load steps strictly idempotent — an essential engineering concept meaning that if you run the exact same operation five times in a row, the database looks identical to running it once, without duplicating a single row.

Encode the taxonomy in the DAG

Writing neat checklists on an internal company wiki is polite, but let us be real: in an emergency, nobody reads documentation. Writing the rules right into your DAG code makes them physically unavoidable. Stop relying on one giant blanket policy. Build explicit retry settings for each failure bucket and attach them directly to tasks based on their habitat:

from datetime import timedelta
# Per-bucket retry policies: tune the numbers to your own quotas and SLAs.
TRANSIENT_INFRA_ARGS = {
    "retries": 5,
    "retry_delay": timedelta(minutes=2),
    "retry_exponential_backoff": True,
    "max_retry_delay": timedelta(minutes=30),
}
UPSTREAM_API_ARGS = {
    "retries": 3,
    "retry_delay": timedelta(minutes=15),
    "retry_exponential_backoff": True,  # pair with jitter in your HTTP client
}
NO_RETRY_ARGS = {"retries": 0}  # source-data and code-bug tasks: a human owns these
with DAG("sales_pipeline", schedule="@daily", catchup=False) as dag:
    extract = PythonOperator(task_id="extract_vendor_api",
                             python_callable=pull_vendor_data, **UPSTREAM_API_ARGS)
    validate = PythonOperator(task_id="validate_source_files",
                              python_callable=check_files_and_schema, **NO_RETRY_ARGS)
    transform = PythonOperator(task_id="transform",
                               python_callable=run_transform, **NO_RETRY_ARGS)
    load = BigQueryInsertJobOperator(task_id="load_warehouse",
                                    configuration={...}, **TRANSIENT_INFRA_ARGS)

Take a look at what we did there. The retry policy is no longer an invisible default; it serves as clear documentation. Any junior engineer reading this DAG immediately understands what each task expects and what kind of failures it can withstand.

Now take it one step further: automate the classification logic itself. You can write a tiny helper function that parses the tail of the error log, identifies the category, and fires an alert callback directly into the specific team’s Slack or Teams channel, complete with the direct runbook link:

def classify_error(log_tail: str) -> str:
    """Illustrative: map a task log excerpt to one of the four buckets."""
    text = log_tail.lower()
    if any(s in text for s in ("file not found", "malformed", "schema mismatch")):
        return "source_data"
    if any(s in text for s in ("429", "rate limit", "timed out", "503")):
        return "upstream_api"
    if any(s in text for s in ("keyerror", "typeerror", "column not found")):
        return "code_bug"
    if any(s in text for s in ("evicted", "sigterm", "connection refused")):
        return "transient_infra"
    return "unknown"  # needs a human; route to data on-call
def route_alert(context: dict):
    ti = context["task_instance"]
    bucket = classify_error(fetch_log_tail(ti))  # last N lines of the task log
    routes = {"source_data": "#data-providers", "upstream_api": "#integrations",
              "code_bug": "#pipeline-authors", "transient_infra": "#platform-team",
              "unknown": "#data-oncall"}
    notify(channel=routes[bucket],  # illustrative channel names
           title=f"[{bucket}] {ti.dag_id}.{ti.task_id} failed",
           runbook=f"https://runbooks.example.com/{bucket}")  # illustrative
default_args = {"on_failure_callback": route_alert, **TRANSIENT_INFRA_ARGS}

With this setup, the alert lands in chat already triaged. The channel name tells you who owns it, the tag points straight to the five-step runbook, and your retry parameters have already done their job without anyone having to wake up in a cold sweat.

One retry policy for everything versus a policy per bucket. The shape of the table matters more than the exact numbers.

The pipeline doesn’t have a reliability problem. It has a routing problem — and routing is solvable with a callback and a table.

Roll it out in a week

You do not need to rewrite your entire production environment tomorrow. Pick your single noisiest, most irritating pipeline and follow this migration plan step-by-step:

• [ ] Tag each task in the DAG by its natural habitat (extract = upstream API, file check = source data, transform = code bug, load = transient infra).

• [ ] Rip out your catch-all default_args retry rule and replace it with the four explicit, per-bucket dictionaries shown above.

• [ ] Draft four one-page runbooks — keep every single one capped at five bullet points or fewer.

• [ ] Insert an upfront validation step right before the transform stage to fail fast if an incoming file is corrupt or missing.

• [ ] Configure the on_failure_callback hook so failure notifications automatically link to the corresponding runbook in the right channel.

• [ ] Verify that your database insert/load operations are idempotent — aggressive retry loops are only safe when re-running a step cannot duplicate records.

• [ ] Run this for two weeks, sit down with the team to examine every alert that fell into “unknown,” and add those newly discovered error patterns into your classifier.

That last habit pays massive compound interest: every single edge-case pattern you codify permanently sharpens the alerting intelligence for every team member who follows you.

The takeaway

Broken pipelines feel messy and chaotic only because every error shows up looking like the exact same red box in a monitoring dashboard. But underneath that alert, you are dealing with four completely distinct beasts. Each has only one logical answer: let temporary cloud hiccups retry on their own, back off gently when APIs throttle you, patch your code bugs, and kick corrupt files straight back to whoever sent them.

Start small. Take one noisy DAG, apply the four retry policies, and wire up the notification switchboard. Then walk away from the re-run slot machine for good. Your future self — sleeping soundly through the night while transient errors quietly resolve themselves — will thank you for it.

So here's my question for you: which of your DAGs gets re-run most often, and which of the four buckets do its failures really fall into?

Disclaimer

• This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.

• I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

• Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

• Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.

• Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

Real-world insights on monitoring, SRE, Terraform, and platform engineering for high-reliability systems.

Got a DAG that wakes your team up three times a week? Pick one pipeline, map its tasks to these four buckets this sprint, and share this guide with that teammate who reflexively clears failed tasks without reading the logs.

Examples are illustrative — validate in your own environment before running in production.