Dev.to WebDev 🛠 Dev Tue, 03 Nov 2026 07:22:40 +0000 👁 0 📖 6 min read

Extract every URL from a sitemap (incl. sitemap indexes and .xml.gz) and check status codes

You need a list of every URL on a site, whether that's for an SEO audit, a migration redirect map, a crawl seed list or chunks for a RAG pipeline. The sitemap is the obvious place to start. In practice "the sitemap" is o

listed in robots.txt, or not at all,
a sitemap index pointing at 5–500 child sitemaps,
gzipped (sitemap-1.xml.gz), sometimes without a Content-Encoding header, and
full of URLs that redirect or return 404.

This post builds a small, dependency-free Python script that handles all four. It then shows a hosted version for when you need filters, many sites or a schedule.

Disclosure: I built the Sitemap URL Extractor Actor shown in part 2. Part 1 is plain Python, so use it freely.

1. Do it yourself: Python standard library only

import gzip
import urllib.request
import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor
from urllib.error import HTTPError, URLError
from urllib.parse import urljoin

UA = {"User-Agent": "Mozilla/5.0 (sitemap-check script)"}
NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"

def fetch(url):
    with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30) as r:
        body = r.read()
    if body[:2] == b"\x1f\x8b":  # gzip magic bytes (.xml.gz), regardless of headers
        body = gzip.decompress(body)
    return body

def find_sitemaps(site):
    """Sitemap: lines in robots.txt, falling back to /sitemap.xml."""
    try:
        robots = fetch(urljoin(site, "/robots.txt")).decode("utf-8", "replace")
        found = [l.split(":", 1)[1].strip() for l in robots.splitlines()
                 if l.lower().startswith("sitemap:")]
        if found:
            return found
    except (HTTPError, URLError):
        pass
    return [urljoin(site, "/sitemap.xml")]

def walk(sitemap_url, seen=None, max_sitemaps=50):
    """Yield (url, lastmod) pairs, recursing into <sitemapindex> files."""
    seen = seen if seen is not None else set()
    if sitemap_url in seen or len(seen) >= max_sitemaps:
        return
    seen.add(sitemap_url)
    root = ET.fromstring(fetch(sitemap_url))
    if root.tag == f"{NS}sitemapindex":
        for loc in root.iter(f"{NS}loc"):
            yield from walk(loc.text.strip(), seen, max_sitemaps)
    else:
        for u in root.iter(f"{NS}url"):
            lastmod = u.findtext(f"{NS}lastmod")
            yield u.findtext(f"{NS}loc").strip(), lastmod

def status(url):
    req = urllib.request.Request(url, headers=UA, method="HEAD")
    try:
        with urllib.request.urlopen(req, timeout=20) as r:
            return url, r.status
    except HTTPError as e:
        return url, e.code
    except URLError as e:
        return url, f"error: {e.reason}"

if __name__ == "__main__":
    site = "https://crawlee.dev"
    urls = []
    for sm in find_sitemaps(site):
        urls.extend(walk(sm))
    print(f"{len(urls)} URLs found")
    sample = [u for u, _ in urls[:50]]
    with ThreadPoolExecutor(max_workers=10) as pool:
        for url, code in pool.map(status, sample):
            if code != 200:
                print(code, url)
    print("checked", len(sample))

What it does:

Discovery: reads the Sitemap: lines from robots.txt and falls back to /sitemap.xml.
Gzip: checks the first two bytes (1f 8b) instead of trusting headers or file extensions. Some servers send .xml.gz files as application/octet-stream, and others gzip a file called .xml.
Sitemap indexes: if the root tag is <sitemapindex>, it recurses into every <loc>, with a seen set and a limit so a misconfigured index can't loop forever.
Status codes: sends parallel HEAD requests. Redirects are followed, so a 200 here means "200 after redirects".

Running it against https://crawlee.dev found 4,861 URLs in about a second. To be sure the index and gzip paths work, I also tested it against a local sitemap index pointing to one .xml.gz and one plain .xml child:

[('http://127.0.0.1:8777/sitemap.xml', '2026-11-01'), ('http://127.0.0.1:8777/missing', None)]
[('http://127.0.0.1:8777/sitemap.xml', 200), ('http://127.0.0.1:8777/missing', 404)]

Where the DIY version falls short

HEAD is sometimes rejected (405) or treated differently from GET. A robust checker falls back to GET.
200 after redirects hides the fact that the sitemap lists URLs that redirect, which is an SEO issue in itself. Record the final URL and a "redirected" flag.
Sites without robots.txt entries often use /sitemap_index.xml or /wp-sitemap.xml (WordPress), not just /sitemap.xml.
Large sites have millions of URLs, so you want filters (/blog/ only, lastmod in the last 30 days) before checking status codes.
<lastmod> comes in several formats (2026-11-01, 2026-11-01T10:00:00+01:00, …), so normalise it before filtering.

All of these are fixable. That list is exactly what I ended up building into the Actor below.

2. Hosted option: Sitemap URL Extractor on Apify

The Sitemap URL Extractor takes homepages or sitemap URLs. It:

discovers sitemaps via robots.txt and common paths,
handles indexes and gzip,
filters by include/exclude patterns and lastmod, and
can check every URL's status with HEAD, falling back to GET.

The output is a dataset you can download as CSV, JSON or Excel, or read from code.

Set APIFY_TOKEN (free account → Console → Settings → API & Integrations), then:

Python (`pip install apify-client`)

"""Extract all URLs from a website's sitemaps (index + gzip supported) and check their status codes.

Actor: https://apify.com/paperlinenstudio/sitemap-url-extractor
Usage: APIFY_TOKEN=... python sitemap_urls.py
"""
import os
from collections import Counter

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("paperlinenstudio/sitemap-url-extractor").call(
    run_input={
        "startUrls": [{"url": "https://crawlee.dev"}],  # homepage: sitemaps are found via robots.txt
        "checkStatus": True,  # HEAD request per URL (GET fallback)
        "maxUrls": 50,
    }
)
urls = list(client.dataset(run.default_dataset_id).iterate_items())

print(f"{len(urls)} URLs from {len({u['sitemapUrl'] for u in urls})} sitemap(s)")
print("Status codes:", dict(Counter(u.get("statusCode") for u in urls)))
for u in urls:
    if not u.get("isOk"):
        print("NOT OK:", u["url"], u.get("statusCode"), u.get("finalUrl"))

50 URLs from 1 sitemap(s)
Status codes: {200: 50}

Node.js (`npm i apify-client`)

// List every URL in a site's sitemap(s) and check the HTTP status of each.
// Usage: APIFY_TOKEN=... node sitemap_urls.mjs [https://example.com]
import { ApifyClient } from 'apify-client';

const site = process.argv[2] ?? 'https://crawlee.dev';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('paperlinenstudio/sitemap-url-extractor').call({
    startUrls: [{ url: site }],
    checkStatus: true,
    maxUrls: 50,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const sitemaps = new Set(items.map((i) => i.sitemapUrl));
const codes = {};
for (const i of items) codes[i.statusCode] = (codes[i.statusCode] ?? 0) + 1;

console.log(`${items.length} URLs from ${sitemaps.size} sitemap(s)`);
console.log('Status codes:', codes);
for (const i of items.filter((x) => !x.isOk)) console.log(`  ${i.statusCode}  ${i.url}`);

Each row looks like this (a real row from the run above, shortened):

{
  "url": "https://crawlee.dev/blog/crawlee-blog-launch",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "sitemapType": "urlset",
  "path": "/blog/crawlee-blog-launch",
  "statusCode": 200,
  "finalUrl": "https://crawlee.dev/blog/crawlee-blog-launch",
  "redirected": false,
  "contentType": "text/html",
  "isOk": true,
  "possiblyBlocked": false
}

There are also image and video counts, hreflang alternates and news titles when the sitemap has them.

Useful input combinations

Only blog posts changed in the last 30 days, no status check (fast):

{
  "startUrls": [{ "url": "https://example.com" }],
  "includePatterns": ["/blog/"],
  "lastmodWithinDays": 30
}

Migration check: every URL in the old sitemap, flagging anything that isn't a clean 200:

{
  "startUrls": [{ "url": "https://old.example.com/sitemap_index.xml" }],
  "checkStatus": true,
  "concurrency": 20
}

Wrap-up

One site, now and then: the stdlib script in part 1 is under 70 lines with no dependencies.
Filters, many sites, schedules or CSV exports for non-developers: Sitemap URL Extractor.

If you need to check the links inside pages rather than the URLs in the sitemap, see the next post on finding broken links.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.

← News 🛠 Dev 🏠 Home