Find broken links and redirects in your sitemap (HTTP status check for every URL)

Your sitemap tells search engines "these are the pages worth indexing." If some of those URLs now return 404, 500, or redirect somewhere else, you're sending crawlers to dead ends and wasting crawl budget. Search Console reports some of this, but late and only for URLs Google already tried.

A faster check: read every URL from the sitemap yourself and request each one. This post shows how to do that in one call, then keep only the rows that need fixing.

What counts as a sitemap problem

For every URL listed in a sitemap you want one of these answers:

• 200: fine, nothing to do.
• 301/302/308 redirect: the sitemap should list the final URL, not the old one.
• 404/410: the page is gone; remove it from the sitemap or restore it.
• 5xx, timeouts, DNS errors: the server is failing on a page you asked to have indexed.

The tricky parts are the sitemap side (nested sitemap indexes, .xml.gz files, sitemaps declared only in robots.txt) and doing thousands of status requests politely.

One call: extract URLs and check status

Disclosure: I packaged this as XML Sitemap URL Extractor on Apify (source on GitHub). Give it a home page and it finds the sitemaps through robots.txt and common paths, follows nested indexes, and outputs one row per URL. With checkUrlStatus: true it also requests each URL (a GET that's dropped as soon as headers arrive, so page bodies aren't downloaded) and adds three fields:

statusCheckConcurrency (default 10) controls how many checks run at once. Lower it for small servers.

import os, csv
from collections import Counter
from apify_client import ApifyClient  # pip install apify-client

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("swillz/sitemap-url-extractor").call(
    run_input={
        "startUrls": [{"url": "https://docs.apify.com"}],
        "includeUrlPatterns": ["*/academy/node-js*"],
        "maxUrls": 30,
        "checkUrlStatus": True,
        "statusCheckConcurrency": 5,
    },
    max_total_charge_usd=0.05,
)

rows = [r for r in client.dataset(run.default_dataset_id).iterate_items() if r.get("url")]
print(run.status_message)
print(Counter(r.get("httpStatus") for r in rows))

# Anything that is not a clean 200: redirects, 4xx/5xx, timeouts, DNS errors
problems = [
    r for r in rows
    if r.get("statusError") or r.get("finalUrl") or (r.get("httpStatus") or 0) >= 300
]
with open("sitemap-problems.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["url", "httpStatus", "finalUrl", "statusError", "sitemapUrl"])
    w.writeheader()
    for r in problems:
        w.writerow({k: r.get(k) for k in w.fieldnames})
print(len(problems), "problem URLs written to sitemap-problems.csv")

(This uses apify-client 3.x, where call() returns a run object with attributes like run.default_dataset_id. On 1.x/2.x use run["defaultDatasetId"] and run["statusMessage"].)

Against the Node.js section of the Apify docs, the run read 7 sitemaps and printed:

Done: 27 URLs from 1/1 sites (7 sitemaps).
Counter({200: 27})
0 problem URLs written to sitemap-problems.csv

A clean section. Each row looks like this:

{
  "url": "https://docs.apify.com/academy/node-js",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "imageCount": 0,
  "isNew": null,
  "httpStatus": 200,
  "finalUrl": null,
  "statusError": null,
  "sitemapUrl": "https://docs.apify.com/sitemap_base.xml",
  "site": "https://docs.apify.com",
  "extractedAt": "2026-11-10T17:56:13.827Z"
}

On your own site, drop includeUrlPatterns and raise maxUrls (0 means no limit) to audit everything. sitemapUrl tells you which sitemap file lists each bad URL, which matters when a CMS generates several.

Size the cost cap before a big audit

Status checks are a separate per-URL charge, so a full audit costs more per URL than a plain extract. On the free tier it's $0.40 per 1,000 URLs plus $1.00 per 1,000 status checks, about $1.40 per 1,000 URLs (less on paid plans; see the Actor's pricing tab).

Set maxUrls so it fits your max cost per run. Roughly, maxUrls × $0.0014 should stay under the cap. A $0.05 cap covers about 35 URLs with checks; a 5,000-URL site needs about $7. If the cap is much smaller than the job, the run stops early with "Stopped at your maximum cost per run" in the status message, so check that message before trusting an empty problem list.

Run it on a schedule

Broken sitemap URLs come back every time someone deletes a page without updating the CMS. In the Apify Console, save the input as a Task and add a weekly Schedule. Then attach a webhook, Slack, or email integration and send the problem rows wherever your team fixes things. To push rows into a spreadsheet, chain Google Sheets Import & Export with the run's datasetId.

Sites without a discoverable sitemap don't fail silently. With the default emitStatusRows: true you get one free row per start URL with status: "no_results" and a reason, so a scheduled audit tells you when a sitemap disappears.

Hand it to an AI agent (MCP)

An agent can run the same audit from a plain request. Add the Actor to Apify's MCP server:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=swillz/sitemap-url-extractor"
    }
  }
}

Then:

"Audit the sitemap of example.com: check the HTTP status of up to 500 URLs with a max cost of $1, then list only URLs that redirect, return 4xx/5xx, or error, with the sitemap file each one came from."

Thousands of 200 rows would waste the agent's context. Tell it to read the dataset with only the columns it needs (the dataset API's fields parameter, e.g. fields=url,httpStatus,finalUrl,statusError,sitemapUrl) and to page through large datasets instead of loading everything at once.

Wrap-up

• Your sitemap is the list of URLs you want indexed, so every entry should return a 200.
• checkUrlStatus: true adds httpStatus, finalUrl and statusError to each sitemap row.
• Keep rows with errors, redirects, or status ≥ 300, and note sitemapUrl so you know which file to fix.
• Fit maxUrls to your cost cap, and run it weekly to catch new breakage.