How to get every URL from a website sitemap (and find the 404s)
You often need a complete list of a site's pages: a content inventory, a migration, an SEO audit, or a starting list for a crawler. Crawling every internal link is slow and easy to get wrong. Most sites already publish t
You often need a complete list of a site's pages: a content inventory, a migration, an SEO audit, or a starting list for a crawler. Crawling every internal link is slow and easy to get wrong. Most sites already publish that list in a sitemap.
This walkthrough shows how to turn a domain into a clean URL list β with lastmod, hreflang, image URLs and, if you want them, live HTTP status codes β without writing a sitemap parser.
We at SteadyUtil built the Apify Actor used here, Sitemap URL Extractor Pro. Apify's free plan includes credit you can use to try it. There is no need to know the exact sitemap URL.
Why not just open /sitemap.xml?
On a small site, that works. On a real site it often does not:
- The sitemap is listed only in
robots.txt, or it lives at/sitemap_index.xml,/wp-sitemap.xml, or a.xml.gzfile. - An index points at other indexes, sometimes several levels deep.
- The site also publishes a plain-text
.txtsitemap or an RSS/Atom feed. -
robots.txtstill lists anhttp://sitemap that only works overhttps://. - Pages appear in the sitemap even though
robots.txtdisallows them β a common SEO bug.
A useful extractor has to follow all of that, then give you a table you can filter.
1. Start a run
- Open Sitemap URL Extractor Pro and sign in to Apify (a free account is enough to try this).
- In Websites or sitemap URLs, paste a domain, for example
apify.com, or a sitemap URL if you already have one. You can add several sites in one run. - Set Max URLs if you want a cost cap. At $0.50 per 1,000 URLs extracted,
10000means the extract step is at most $5. - Leave Check HTTP status off for the first run if you only need the list. Turn it on when you want 404s and redirects (that check is $1.00 per 1,000 URLs, and is optional).
- Click Start. When the run finishes, download the dataset as CSV, Excel, JSON or HTML.
You do not need to point it at a specific sitemap file. It reads Sitemap: lines in robots.txt, tries common locations, follows nested indexes, and understands gzipped XML, TXT sitemaps and RSS/Atom feeds.
2. What you get back
Each row is one URL. Typical fields:
-
url,host,lastmod,changefreq,priority -
sitemapUrlandsitemapTypeβ which file the URL came from -
blockedByRobotsTxtβtrueif the sitemap lists the URL but robots.txt disallows it -
imageUrlsandalternates(hreflang) - Google News and video titles, when the sitemap includes them
A SUMMARY record in the run's key-value store lists, per site, which sitemaps were read, which failed and why, and the URL counts. A dead sitemap or a timeout is recorded there; it does not fail the whole run.
3. Find 404s and other broken pages
Run the same input with Check HTTP status turned on. Each row then also has statusCode, finalUrl and redirected.
In the dataset (or in a spreadsheet), filter for:
-
statusCode404 or 410 β listed in the sitemap, but gone -
statusCode5xx β the page failed when checked -
redirectedtrue β the sitemap URL is not the final URL -
blockedByRobotsTxttrue β published in the sitemap, blocked from crawlers
That is usually the audit people actually want: not βevery URL,β but βURLs the sitemap is wrong about.β
If you only care about recent changes, set Modified after (for example 2026-01-01) or use include/exclude patterns such as /blog/ and \.pdf$. You are charged only for URLs that pass the filters.
4. Run it from code
Replace YOUR_API_TOKEN with your Apify API token. This starts a run, waits for it to finish, and prints the URLs.
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('steadyutil/sitemap-url-extractor-pro').call({
startUrls: [{ url: 'https://apify.com' }],
maxUrls: 1000,
checkStatus: false,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((item) => item.url));
The same Actor is also available through the Apify MCP server, so an agent can call it from the Store page.
5. A simple weekly check
For a recurring inventory or 404 watch:
- In Apify Console, save the input as a Task, with Max URLs set as a cap.
- Add a Schedule (for example every Monday).
- In Make, n8n or Zapier, trigger on a finished run and upsert each
urlinto a Google Sheet, withlastmod,statusCodeandblockedByRobotsTxtas columns. - Optionally keep only new 404s or robots.txt conflicts, and email that subset to yourself.
Pricing and limits, plainly
- $0.50 per 1,000 URLs written to the dataset.
- $1.00 per 1,000 URLs if you turn on HTTP status checks.
- Sites with no sitemap, failed sitemaps, and URLs you filter out are not charged.
- Use Max URLs (and Max URLs per site) as a hard cap.
If a site has no sitemap, you will get a summary that says so, and no URL charge. This tool does not invent URLs by crawling. Bot-protected sites are a different problem and are not bypassed here.
Built by SteadyUtil
Sitemap URL Extractor Pro is published by SteadyUtil. If something fails or you need a field we do not output yet, open an issue on the Actor's Issues tab or email [email protected].
Two related Actors, if you want to take the next step with the same URL list:
- Website Screenshot Pro β desktop and mobile screenshots, or PDFs, of the pages you just extracted ($3 per 1,000 screenshots; you are not charged for pages that did not load).
- Website Tech Stack Detector & SEO Audit β CMS, ecommerce, analytics and an on-page SEO score for each domain ($4 per 1,000 sites).
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.