Stop Trusting Your Sitemap: How to Reconcile It With a Crawl

I work at SEOSage, which builds desktop SEO audit software. The steps below work with any crawler.

Your sitemap says one thing. Your site says another.

A sitemap is a declaration: "these are my important pages." A crawl is the evidence: "this is what is actually reachable." Most sites I look at have never reconciled the two. When I finally do the comparison for them, the gaps are always the same three buckets — and each bucket is an action list.

Bucket 1: In the sitemap, never crawled

These URLs cannot be reached by following links from the homepage. Some are true orphans your navigation forgot. Some are pages behind crawl blocks — noindex rules, robots disallows, or login walls you did not intend.

For each one, make a deliberate call: if the page matters, link to it from relevant pages. If it was retired, remove it from the sitemap and either let it 404/410 or redirect it to the closest live equivalent. The worst choice is doing nothing — a URL you declare but nobody can reach teaches crawlers to distrust your sitemap.

Bucket 2: Crawled, not in the sitemap

Most of these are fine — ordinary pages your navigation reaches. But scan the list for what should never be indexed: staging leftovers, parameter duplicates, thin archive and tag pages, old campaign landing pages. Being crawled is the first step toward being indexed, so this bucket is where index bloat is born.

The fix is not to add everything to the sitemap. It is to clean the crawlable site: noindex or block what should not be indexed, and keep the sitemap to pages that genuinely deserve to rank.

Bucket 3: Indexed, in neither list

Google found these through external links — old URLs, moved pages, retired campaigns. You did not declare them and your crawl cannot reach them, but they are in the index anyway.

Do not just add them to the sitemap to make the lists match. That bakes the mess into your declaration. Decide deliberately: redirect to the closest live page or let them go. A sitemap should contain only URLs that return 200, are indexable, and are the canonical version.

The reconciliation habit

A sitemap should be generated from the same system that builds the site, not maintained by hand — and it should be reconciled against a crawl after every redesign, migration, or CMS change. That is when the three buckets fill up: new templates create crawlable pages that never make it into the sitemap, and retired pages linger in the sitemap long after they stopped being reachable.

I do the reconciliation in SEOSage — it can crawl a sitemap directly and flag sitemap URLs that redirect or error alongside crawl-only and orphan URLs, with GA traffic overlaid so I fix the buckets in traffic order. Any crawler that supports sitemap crawl and orphan detection gives you the same three lists; the point is the comparison, not the tool.

Once the lists agree, most of the "mystery" problems people chase — pages that never get indexed, sitemap warnings that never go away, crawl budget spent on junk — tend to disappear on their own.

Disclosure: I work at SEOSage, a desktop SEO audit tool. These steps work with any crawler you already use. When was the last time you compared your sitemap against an actual crawl?