Dev.to WebDev 🛠 Dev Sat, 07 Nov 2026 14:00:51 +0000 👁 0 📖 5 min read

What your 500s, 503s and 429s tell search crawlers (and how to take a site down for maintenance without hurting SEO)

Developers think of status codes as a conversation with browsers. But a large share of the requests your server answers come from crawlers, and they read status codes much more literally than humans do. A human sees a 500 page, sighs and hits refresh. A crawler records it, adjusts how often it visits, and, if it keeps happening, starts doubting the page exists.

Here's what the server-side codes mean to a search crawler, based on what the major search engines document publicly, and how to use them deliberately.

The short version

Code	What you meant	What a crawler hears
500	"Something broke"	"This server is struggling; slow down"
502 / 504	"The upstream broke"	Same as 500
503	"Temporarily unavailable"	"Come back later", especially with `Retry-After`
429	"Too many requests"	"Slow down", treated much like a 503
200 with an error page	"Oops"	"This is the content of the page"

The last row is the dangerous one, and we'll come back to it.

5xx: the server is unhealthy

Google's documentation says that 5xx responses (and 429) cause its crawler to temporarily slow its crawl rate. That's sensible behaviour: if your server is failing, hammering it makes things worse.

What matters is duration:

A short burst of 5xx errors, during a deploy or a brief outage, is normal. Already-indexed pages generally stay indexed. The crawler comes back.
Persistent 5xx on the same URLs is different. Google documents that URLs which keep returning server errors may eventually be dropped from the index. From the crawler's point of view, a page that hasn't loaded successfully for a long time is a page that may not exist any more.

So the takeaway isn't "never return a 500". It's: don't let 5xx become the steady state for pages you care about, and watch for them in your logs and in Search Console's crawl stats.

503 + Retry-After: the polite way to say "not now"

A 503 is the code that means temporary by definition. Pair it with a Retry-After header and you're giving crawlers a direct hint about when to return:

HTTP/1.1 503 Service Unavailable
Retry-After: 3600
Content-Type: text/html

<h1>Down for maintenance, back within the hour</h1>

This is the right response for planned maintenance. The wrong responses are more common:

A 200 "maintenance" page. The crawler happily indexes "We'll be right back!" as the content of every URL it visits during the window. Your titles and snippets can briefly become the maintenance text.
A redirect to /maintenance. Every URL now says "I've moved to /maintenance". That's a strange signal to send about your whole site.
A 404. "This page doesn't exist." Not what you meant.

A maintenance switch that gets it right (nginx)

# touch /etc/nginx/maintenance.on to enable
location / {
    if (-f /etc/nginx/maintenance.on) {
        return 503;
    }
    proxy_pass http://app;
}

error_page 503 @maintenance;
location @maintenance {
    add_header Retry-After 1800 always;
    root /var/www/maintenance;
    rewrite ^ /index.html break;
}

The always flag matters: without it, nginx may not add the header to error responses.

One more caution: a 503 for a few hours is fine. A 503 for days starts to look like a persistent server error. Long migrations should be designed so the site stays up.

429: rate limiting the crawler

Some teams rate limit aggressively, and crawlers get caught in it. Google treats 429 much like a 503: a signal to slow down. That's useful if a crawler is genuinely overloading you.

But if your rate limiter returns 429 to a search crawler routinely, you're telling it to crawl less, which means new and updated pages get picked up more slowly. Check your WAF and rate-limit rules for verified crawler traffic, and make sure the limit reflects what your server can handle rather than a default someone copied.

The 200 error page: the soft error trap

The worst case isn't any 5xx. It's a framework that catches the exception and renders a friendly "Something went wrong" page with a 200 status. Common culprits:

single-page apps that render an error component client-side
a CMS error template that doesn't set the status code
edge functions that catch upstream failures and return a fallback page

To a crawler, that's not an error at all. It's a page whose content is now "Something went wrong". Search engines try to detect these "soft" errors, but you're relying on their guesswork instead of telling them the truth.

Quick test:

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/this-should-break

Then break something on purpose in staging (point the app at a wrong database URL) and confirm the status is 5xx, not 200.

Monitoring this from the outside

Status codes in your app logs show what your app returned. They don't always show what the crawler got, because CDNs, WAFs and load balancers sit in between. Two useful habits:

Grep your edge or CDN logs for crawler user agents and group by status. A rising share of 5xx or 429 is an early warning.
Check the crawl stats report in Search Console after any infrastructure change.

At seotto.com we see the same pattern often: a site's rankings wobble, and the cause turns out to be an error response that was invisible to humans but very visible to crawlers.

Summary

5xx and 429 make crawlers slow down; persistent 5xx can get pages dropped.
503 + Retry-After is the right answer for planned maintenance.
Never serve errors or maintenance pages with a 200.
Look at what the crawler actually received, at the edge, not just what your app thinks it sent.