Dev.to WebDev 🛠 Dev Wed, 11 Nov 2026 16:34:28 +0000 👁 0 📖 5 min read

How to find orphan pages with Python: diff your sitemap against your internal links

An orphan page is a page on your site that no other page links to. It might be in your sitemap, it might even be indexed, but a visitor clicking around your site can't reach it, and search engines get few signals that it

Orphans are easy to create. A campaign landing page is removed from the menu. A blog category is renamed and old posts lose their only link. A CMS publishes a page but the template that listed it was changed. Nobody notices, because the page itself still works.

In this tutorial we'll write a small Python script that finds them by comparing two lists:

Every URL your sitemap says exists.
Every URL that's actually linked from another page on your site.

Anything in the first list but not the second is an orphan candidate.

What you'll need

Python 3.10 or later
requests and beautifulsoup4 (pip install requests beautifulsoup4)
A site with an XML sitemap, and permission to crawl it. Your own site, ideally.

Step 1: read the sitemap (including sitemap indexes)

Sitemaps can be a single file or an index pointing to several files. We'll handle both.

import requests
import xml.etree.ElementTree as ET

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
session = requests.Session()
session.headers["User-Agent"] = "orphan-finder/1.0 (+contact: [email protected])"

def sitemap_urls(url):
    root = ET.fromstring(session.get(url, timeout=20).content)
    if root.tag.endswith("sitemapindex"):
        urls = set()
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            urls |= sitemap_urls(loc.text.strip())
        return urls
    return {loc.text.strip() for loc in root.findall("sm:url/sm:loc", NS)}

Gzipped sitemaps (.xml.gz) are common too; if you have them, decompress with gzip.decompress before parsing.

Step 2: normalise URLs

This step is what makes or breaks the result. The sitemap might say https://example.com/blog/post/ while a link says /blog/post or https://example.com/blog/post/?utm_source=nav. Without normalising, everything looks like an orphan.

from urllib.parse import urljoin, urlsplit, urlunsplit

def normalise(url, base):
    absolute = urljoin(base, url)
    parts = urlsplit(absolute)
    path = parts.path or "/"
    if path != "/" and path.endswith("/"):
        path = path.rstrip("/")
    return urlunsplit((parts.scheme.lower(), parts.netloc.lower(), path, "", ""))

This drops query strings and fragments and ignores trailing slashes. Adjust it to match your site's conventions: if query strings carry meaning on your site (pagination, for example), keep the relevant ones.

Step 3: crawl the site and collect internal links

We'll do a simple breadth-first crawl starting from the homepage, staying on the same host and recording every internal link target.

from bs4 import BeautifulSoup
from collections import deque
import time

def crawl(start, max_pages=2000, delay=0.5):
    host = urlsplit(start).netloc.lower()
    seen, linked = set(), set()
    queue = deque([normalise(start, start)])
    while queue and len(seen) < max_pages:
        url = queue.popleft()
        if url in seen:
            continue
        seen.add(url)
        try:
            resp = session.get(url, timeout=20)
        except requests.RequestException:
            continue
        if "text/html" not in resp.headers.get("Content-Type", ""):
            continue
        soup = BeautifulSoup(resp.text, "html.parser")
        for a in soup.find_all("a", href=True):
            href = a["href"].strip()
            if href.startswith(("mailto:", "tel:", "javascript:", "#")):
                continue
            target = normalise(href, resp.url)
            if urlsplit(target).netloc == host:
                linked.add(target)
                if target not in seen:
                    queue.append(target)
        time.sleep(delay)
    return seen, linked

The delay keeps the crawl polite. On a small site a full crawl takes a few minutes; for big sites, raise max_pages gradually and run it off-peak.

One important note: a page is only "linked" in this sense if the link is in the HTML that your server returns. If your navigation is rendered entirely by client-side JavaScript, this crawler won't see it. That's useful information in itself, but it means you'd need a headless browser for an accurate picture.

Step 4: diff the two sets

def find_orphans(site, sitemap):
    in_sitemap = {normalise(u, site) for u in sitemap_urls(sitemap)}
    crawled, linked = crawl(site)
    orphans = sorted(in_sitemap - linked - {normalise(site, site)})
    not_in_sitemap = sorted((crawled - in_sitemap))
    return orphans, not_in_sitemap

if __name__ == "__main__":
    import sys
    site, sitemap = sys.argv[1], sys.argv[2]
    orphans, missing = find_orphans(site, sitemap)
    print(f"Orphan candidates ({len(orphans)}):")
    for u in orphans:
        print("  ", u)
    print(f"\nCrawlable but not in sitemap ({len(missing)}):")
    for u in missing[:50]:
        print("  ", u)

Run it:

python orphans.py https://example.com https://example.com/sitemap.xml

You get two lists. The second one, pages reachable by links but missing from the sitemap, is a free bonus: those are often pages you forgot to add, or pages that shouldn't exist at all (tag archives, test pages, old drafts).

Step 5: decide what to do with each orphan

Not every orphan is a problem. Go through the list and sort each URL into one of these:

Important and should be linked. Add links from relevant pages: a parent category, related articles, a hub page. This is the most common fix.
Outdated. Redirect it to its closest current equivalent, or remove it and take it out of the sitemap.
Deliberately standalone. Landing pages for ads or email campaigns may be intentionally unlinked. Consider whether they should be in the sitemap at all.
A false positive. Usually a normalisation mismatch (a trailing slash, www vs non-www, http vs https). Fix the normaliser or, better, fix the inconsistency on your site.

Limitations to keep in mind

JavaScript-rendered links aren't seen, as mentioned above.
Pagination and filters can make the crawl huge. Exclude patterns you don't care about.
Links from outside your site don't count here. A page can be an orphan internally while still getting visits from search or social, and that's exactly the kind of page that deserves an internal link.
The crawl is a snapshot. Run it regularly, for example monthly or after major content changes.

Making it a habit

The script is about a hundred lines and takes minutes to run, which makes it a good candidate for a scheduled job: run it monthly, save the output, and compare it with last month's to see what's new.

If you'd rather have orphan pages, sitemap gaps and other crawl issues checked for you continuously, that's part of what seotto.com does.