How to fix 'Excluded by noindex tag' in Search Console when you're sure you never added noindex
Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com The page indexing report in Search Console lists a batch of URLs under Excluded by 'noindex' tag. You open the template, sea
Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com
The page indexing report in Search Console lists a batch of URLs under Excluded by 'noindex' tag. You open the template, search for noindex, and find nothing. You view source in your browser: nothing there either.
Google isn't making it up. When it reports a noindex, it saw one, either in the HTML it fetched or in the HTTP headers. The trick is that what Googlebot receives isn't always what your browser receives, and the directive doesn't have to be a <meta> tag at all.
Here are the places it hides, in the order I'd check them.
First: confirm what Google sees
Run the affected URL through URL Inspection, then click Test live URL. Under Page indexing, Google tells you whether indexing is allowed and, if not, whether it detected the noindex in a meta tag or in the X-Robots-Tag HTTP header. That one line usually halves the search.
If the live test says indexing is allowed now, the noindex was there when Google last crawled, but isn't anymore. Request indexing and the status will update after the recrawl. Something removed it in the meantime, and it's still worth finding out what added it, because it may come back.
Check 1: the HTTP headers
A noindex can be sent as a header instead of a tag:
X-Robots-Tag: noindex
You won't see it in view source. Check with curl:
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
https://www.example.com/some-page/ | grep -i x-robots
Common sources:
-
Web server or CDN config copied from a staging environment, where adding
X-Robots-Tag: noindexto every response is standard practice. - Preview or staging hosts that share config with production by mistake.
-
Rules for PDFs or other file types written too broadly, like a location match that catches more than
.pdf. - Hosting platforms that add the header automatically to non-production domains. If production is served from what the platform thinks is a preview domain, the header follows.
Check 2: the HTML Googlebot gets, not the HTML you get
Fetch the HTML with a crawler user agent and search it:
curl -sL -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
https://www.example.com/some-page/ | grep -ioE '<meta[^>]+(robots|googlebot)[^>]*>'
Look for name="robots" and name="googlebot". A googlebot-specific tag only affects Google, which is why other search engines may index the page fine.
If this shows a noindex that your browser's view source doesn't, something is serving different HTML to bots: a caching layer, an A/B testing tool, a bot-protection page, or a prerendering service with a stale or misconfigured snapshot.
Check 3: CMS and plugin settings
If you use a CMS, the tag may come from a setting rather than a template:
- WordPress: Settings → Reading → "Discourage search engines from indexing this site" adds a site-wide noindex. SEO plugins also have per-post, per-taxonomy and per-post-type noindex settings, and defaults for archives, attachments and author pages.
- Headless CMSes often have a "hide from search" boolean on each entry. Imported content sometimes arrives with it set to true.
- E-commerce platforms may noindex out-of-stock products, filtered collections or search pages by default.
If only some URLs are affected, look at what they have in common: same post type, same category, same author, same import batch.
Check 4: JavaScript
A noindex can be added by JavaScript after the page loads. Googlebot renders pages, so it sees it. Your view source doesn't. Check the rendered HTML in URL Inspection (View tested page → HTML) or in DevTools' Elements panel, rather than view source.
Note that the opposite doesn't work reliably: if the server HTML says noindex and JavaScript removes it, Google may never render the page, because it can skip rendering for pages it already knows it can't index. Never ship a noindex in server HTML expecting JavaScript to undo it.
Check 5: deploys and environment variables
Many frameworks decide whether to emit noindex based on an environment variable (NODE_ENV, NEXT_PUBLIC_ENV, APP_ENV, a "robots" config). If production was built or started with the wrong value even once, every page got the noindex. Look at your deploy history around the time the affected URLs were last crawled. The "Last crawled" date in URL Inspection tells you when to look.
A noindex that ships in a header or an env-driven template can drop a whole section from search before anyone opens Search Console. seotto.com checks for indexing blocks after deploys, in headers as well as tags, so a stray noindex gets caught the same day.
After you fix it
- Confirm with the curl checks above, from outside your network, that neither the header nor the tag is present.
- Use Test live URL to confirm Google sees the page as indexable.
- Request indexing for the most important URLs; for large batches, the Validate fix button in the page indexing report asks Google to recheck them.
- Make sure the URLs are in your sitemap, so they're recrawled sooner.
Recovery usually takes days to a few weeks, depending on how often Google crawls those pages.
Keep it from happening again
- Add a CI check that fails if a production build emits
noindexon pages that should be indexable. - Keep staging protection in a layer that can't be copied to production by accident (HTTP auth is better than a noindex header for staging anyway).
- Monitor a handful of key URLs for both the meta tag and the header after every deploy.
Related reading
- Case study (hypothetical): the staging site that got indexed, and started outranking production
- How to check if robots.txt is blocking Google (and fix it without breaking anything)
- How to fix 'Discovered – currently not indexed' in Google Search Console (a developer's list of causes)
- Google renders your page with last week's JavaScript: a 10-point checklist for assets crawlers can actually fetch
- How to find which sections Googlebot stopped crawling after a deploy (compare two weeks of access logs in Python)
Where did your mystery noindex come from? I've seen the X-Robots-Tag header from a copied staging config more than any other cause.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.