How to Debug an Old noindex Verdict in Google Search Console

You publish a site migration and open a guide at its expected address. It loads. Google Search Console still shows that a crawler found a noindex tag. Before changing the deployment again, check when Google fetched the URL and what the public server sends today.

A noindex directive can come from the HTML or the HTTP headers. A stored indexing report can reflect an older response. Comparing the two versions gives you a specific next action.

Check the first public response

Use the exact URL you want readers to find, including the scheme, host and trailing slash. Save the following script as check_noindex.py. It performs one GET and stops at a redirect so you can see the first status and its Location header.

import argparse
from html.parser import HTMLParser
from urllib.error import HTTPError, URLError
from urllib.parse import urlsplit
from urllib.request import HTTPRedirectHandler, Request, build_opener


class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None


class PageMetadata(HTMLParser):
    def __init__(self):
        super().__init__()
        self.in_head = False
        self.canonicals = []
        self.robots = []

    def handle_starttag(self, tag, attrs):
        fields = dict(attrs)
        if tag == "head":
            self.in_head = True
        if tag == "meta":
            name = (fields.get("name") or "").lower()
            if name in {"robots", "googlebot", "googlebot-news"}:
                self.robots.append((name, fields.get("content") or ""))
        rel = (fields.get("rel") or "").lower().split()
        if self.in_head and tag == "link" and "canonical" in rel:
            self.canonicals.append(fields.get("href") or "")

    def handle_endtag(self, tag):
        if tag == "head":
            self.in_head = False


def inspect(url):
    target = urlsplit(url)
    if target.scheme not in {"http", "https"} or not target.hostname:
        raise SystemExit("Enter an HTTP or HTTPS URL.")

    opener = build_opener(NoRedirect)
    try:
        request = Request(url, headers={"Accept-Encoding": "identity"})
        response = opener.open(request, timeout=12)
    except HTTPError as error:
        response = error
    except (URLError, TimeoutError) as error:
        raise SystemExit(f"GET failed: {error}") from error

    with response:
        print(f"GET {response.geturl()} -> {response.status}")
        for name in ("Location", "Content-Type", "Content-Encoding", "X-Robots-Tag", "Link"):
            for value in response.headers.get_all(name, []):
                print(f"{name}: {value}")
        if response.status != 200:
            raise SystemExit("Inspect the status or destination before reading metadata.")
        if response.headers.get_content_type() != "text/html":
            raise SystemExit("Response is not HTML.")
        encodings = response.headers.get_all("Content-Encoding", [])
        if any(value.strip().lower() != "identity" for value in encodings):
            raise SystemExit("Decode the response body before inspecting its HTML.")
        limit = 2_000_000
        body = response.read(limit + 1)
        if len(body) > limit:
            raise SystemExit("HTML exceeds the example's 2 MB inspection limit.")
        page = PageMetadata()
        encoding = response.headers.get_content_charset() or "utf-8"
        page.feed(body.decode(encoding, errors="replace"))
        print(f"Canonical links in head: {page.canonicals}")
        print(f"Robots meta tags: {page.robots}")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Show one public HTML response's indexing signals.")
    parser.add_argument("url")
    inspect(parser.parse_args().url)

Run it first on your preferred URL.

python3 check_noindex.py https://example.com/guides/import/

The probe requests an uncompressed response and prints any Content-Encoding header it receives. If a server still sends gzip or Brotli, decode the body before inspecting the HTML. Reading compressed bytes as text can make a page with a noindex tag appear to have no robots meta tags. HTTP content coding

An HTTP 200 can carry both an HTML index, follow tag and an HTTP header such as X-Robots-Tag: googlebot: noindex. That header applies to Google’s main web crawler. Inspect every header and meta tag that applies to your search surface. A none directive also includes noindex. Google's robots tag reference

On a redirect, confirm the first hop, then run the same check on the intended destination. For HTML 200 responses, check the canonical in the document head. If the page also declares a canonical in an HTTP Link header, verify that both signals point to the same preferred URL. The script reports the directives it sees for this one request. Google’s canonical URL guidance

Compare the stored crawl with a live test

In URL Inspection, record the last crawl, page fetch, and indexing allowed fields from the Google Index view. Then run Test live URL for the preferred address and check whether crawling and indexing are allowed there. The live test follows redirects. Check the first hop, then test the intended destination directly. URL Inspection documentation

Read Crawl allowed? first. When robots.txt blocks the fetch, URL Inspection can show Yes under Indexing allowed? because Google could not read the page’s directives. The public GET above can still succeed in that situation because it does not check robots.txt. Use the live test to confirm that crawling is allowed and the fetch succeeds before interpreting the indexing field. Google’s explanation of the inspection fields

A live test that allows indexing confirms the current response is eligible in the tool. Use the stored result to see whether Google has indexed that URL and which canonical it chose. Google says repeated requests for the same URL won’t make it crawl faster. Google’s recrawl guidance

Check the next stored crawl

In a synthetic UTC timeline, Google’s 09:00 crawl found noindex. The corrected route deployed at 10:00. At 10:10, the public GET returned HTTP 200 with the expected canonical, and Google’s live test allowed indexing. When a newer stored crawl appears, check Indexing allowed and Google-selected canonical. If that crawl still reports noindex, open View crawled page and compare its response headers and HTML with the public GET. Use the two capture times to trace which response supplied the directive.

For several URLs with the same issue, review the affected examples in Page indexing before using Validate fix. A submitted-sitemap filter limits validation to known affected URLs in that sitemap when you start it. Monitor the issue’s validation details and use URL Inspection for each priority URL’s stored indexing status. Google’s Page indexing report

Try a public example

On 11 November 2026, the October Sarv Blog archive returned HTTP 200, declared its own canonical, and sent no robots meta tag or X-Robots-Tag on the checked response. I work with Sarv. You can run the same command on the archive and record what the server returns today.

For your own release, keep three pieces of evidence together. Save the first HTTP response, the live test verdict, and the date of the stored Google crawl. Compare them again after Google records a newer crawl.

I used AI for research, drafting, and editing assistance.