Dev.to WebDev 🛠 Dev Wed, 11 Nov 2026 16:33:17 +0000 👁 0 📖 5 min read

Postmortem (hypothetical): our robots.txt returned 503 for nine days, and crawling nearly stopped

This is a hypothetical postmortem. The site, timeline and details are invented, but the failure mode is real and well documented by search engines. It's written in a blameless format, the way we'd want a real one written

Summary

For nine days, requests for /robots.txt from search engine crawlers received HTTP 503 responses from our edge. Regular pages loaded normally for users and returned 200 to crawlers when they were requested. Because major crawlers treat a server error on robots.txt as a reason to pause crawling, the number of pages crawled per day fell sharply, newly published pages weren't discovered, and updated pages kept showing stale titles in search results. No pages were deindexed, but freshness suffered for about three weeks in total.

Impact

New articles published during the window took far longer than usual to appear in search results.
Price and stock changes on product pages were reflected late in search snippets.
Crawl stats in Search Console showed a steep drop in total crawl requests and a spike in "host status" problems for robots.txt fetching.
Users were not directly affected; the site itself was up.

Timeline (all times local, hypothetical)

Day 0, 16:10: A new edge rate-limiting rule is deployed to protect the search API from a scraping spike. The rule matches requests by path pattern and user agent category.
Day 0, 16:12: Rule begins returning 503 with a Retry-After header to "automated clients" exceeding a request threshold on matched paths. Due to an overly broad pattern, /robots.txt is included, and the threshold is low enough that crawlers hit it immediately.
Day 1 to Day 8: Crawl activity declines. Nobody notices; uptime monitoring checks the homepage and checkout, both healthy.
Day 8, 11:30: An editor asks why an article published four days ago isn't in search results yet.
Day 9, 09:40: An SEO engineer opens Search Console's crawl stats, sees robots.txt fetch failures, and fetches /robots.txt with a crawler user agent. It returns 503.
Day 9, 10:05: Rule pattern corrected to exclude /robots.txt and sitemap paths. Fix deployed.
Day 9 onward: Crawling recovers over the following days.

What crawlers do when robots.txt fails

This is the heart of the incident, so it's worth being precise. Google's documentation describes roughly this behaviour, and other major crawlers behave similarly:

4xx responses (like 404) for robots.txt are generally treated as "there is no robots.txt", meaning crawling is allowed.
5xx responses (like 503) and network errors are treated as a sign the server is struggling. The crawler temporarily treats the whole site as disallowed and retries later. For longer outages, it may fall back to a previously cached copy of the file, and eventually change its behaviour again if the errors continue for a long time.

In plain terms: a broken robots.txt that returns 404 is mostly harmless; a broken robots.txt that returns 503 tells crawlers "please stay away for now". It's a reasonable design. If the server says it's overloaded, polite crawlers back off. Our edge rule was, unintentionally, saying exactly that.

Check the current documentation for any crawler you care about; the details and time windows are the search engines' to change.

Root cause

The rate-limit rule matched by a path prefix list that included / for one user agent category, intended as a catch-all fallback for "unknown scrapers". Crawler traffic was classified into that category for paths not in an allowlist. /robots.txt and the sitemaps weren't in the allowlist because nobody thought of them as endpoints.

Contributing factors

robots.txt wasn't monitored. Our uptime checks covered user-facing pages. A file served to machines had no check at all.
Testing used a browser user agent. The engineer who tested the rule loaded /robots.txt in a browser and got 200, because the browser wasn't classified as automated.
No alert on crawl stats. The drop was visible in Search Console within a day or two, but nobody was looking.
503 felt safe. The team chose 503 with Retry-After because it's the "polite" status for rate limiting. For most endpoints that's true. For robots.txt it has a site-wide consequence.

What went well

The site itself stayed fast and available.
Once spotted, the cause was found within minutes because the crawl stats report pointed directly at robots.txt fetching.
The fix was a one-line config change with no deploy of application code.

Action items

Exempt crawler-control files from rate limiting and bot rules: /robots.txt, sitemaps and any sitemap index. Make this an explicit allowlist with a comment explaining why.
Monitor robots.txt like an endpoint. Check every few minutes that it returns 200, with expected content, for both a browser and a crawler user agent string. Alert on anything else.
Test edge rules with crawler user agents, not just browsers, before deploying.
Alert on crawl anomalies. A weekly look at crawl stats is the minimum; a scripted check against the Search Console API for sudden drops in crawl requests is better.
Document the status code rule in the runbook: for robots.txt, a 5xx is never "just a temporary error".

A quick check you can run now

for ua in "Mozilla/5.0" "Googlebot/2.1 (+http://www.google.com/bot.html)"; do
  printf '%s -> ' "$ua"
  curl -s -o /dev/null -w '%{http_code}\n' -A "$ua" https://example.com/robots.txt
done

If either line prints anything other than 200 (or a deliberate 404 if you have no file), find out why today. Note that some edge providers verify real crawlers by IP, so a spoofed user agent may be treated differently from the real thing; this check catches obvious problems, not all of them.

The lesson

Robots.txt is tiny, rarely edited and easy to forget. That's exactly why it should be monitored. Catching technical SEO regressions like this one before they cost weeks of freshness is what we built seotto.com for.