I Curled All 93 URLs in My Own Sitemap. 28 Came Back 404, Including the Hub Page.

The bug wasn't in the app. The app was fine. The bug was in the file that tells Google what the app contains.

Our Chinese zodiac site publishes a sitemap with 93 <loc> entries. I fetched every one of them sequentially and twenty-eight returned 404. Number six on the list is /elements — the hub for the five-element section. It's dead. So are all five pages beneath it.

How the crawl was run

I took the <loc> values verbatim and curled each one with -L, through a local proxy, 300 ms apart, single-threaded.

Single-threaded matters here. My first pass used a thread pool of eight and got 28 errors that I initially read as server trouble. Half of them were genuine 404s and half were me hammering a small box. Running it again at 300 ms spacing separated the two: the 404s stayed 404, the timeouts didn't.

Anything non-200 got a second request. Anything that timed out got a third. Two URLs moved between passes and I recorded the majority result. Python 3.13 did the tallying.

Results by section

Survivors under /elements/: 1982, 1983, 1984, 1985. All four answer 301 and land on a root-level year page. The other nineteen are plain 404.

Three things the numbers changed in my head

The brand pages are the broken ones.

I would have bet the element pages were the most linked part of the site. They are — just not at /elements/*. The real ones live at /wood-element-personality, /fire-element-personality and so on, all 200, all with Article and FAQ markup.

Two generations of routes for one set of pages. The generator that wrote the sitemap kept the old map.

Humans never hit these 404s, which is why they lasted.

27 internal hrefs on the homepage. 15 unique paths. Zero of them begin with /elements.

So the navigation was updated when the routes moved and the sitemap wasn't. Every visitor gets the new paths. Only the crawler reads the old list — and the crawler is the audience the sitemap exists for.

The sitemap's timestamps make it worse, not better.

Newest lastmod: 4 November 2026, six days before the crawl. So this is a freshly generated file that already contains 28 dead URLs. It wasn't stale. It was generated from something other than the live route table.

That's the part worth fixing in code, not just in the file: whatever builds this sitemap is reading a manifest.

Gaps that are decisions, not failures

Same crawl, two more counts, kept separate because they aren't errors:

The compatibility set also uses two URL shapes — eleven with a -compatibility suffix, seven without — and the canonical direction flips between them. /rat-and-ox-compatibility 301s to the short form; /snake-and-pig 301s to the long one. Both work. Neither agrees with the other.

geng, xin, ren, gui day-master pages don't exist. Four of ten stems have no landing page.

What a status-code crawl can't tell you

Whether the readings are correct.

A 200 means a page exists and has text on it. It says nothing about whether the element assigned to a birthday matches what a classical chart would produce. That needs real birth dates pushed through the calculator and compared against a reference table. I didn't do that pass.

I also don't know how many of these 28 have already dropped out of the index, or when. That's Search Console data for this domain and I didn't have it open.

And I can't tell whether /elements was removed on purpose. Access logs would say. I only had the outside view.

The part that passed

All eight live year pages survive an arithmetic check. Stem index (year − 4) mod 10, branch index (year − 4) mod 12: 1980 Metal Monkey through 1991 Metal Goat, eight for eight, no mismatches.

The 1984 page even handles the annoying bit — Chinese New Year on 2 February, Li Chun on the 4th, two systems disagreeing about which date opens the year, explained instead of skipped.

So the prose is better than the plumbing. Which is the usual order of things and still annoying to write down.

Our own ledger was wrong

The site ledger updated 7 November records "sitemap 18/18 all 200" and treats it as evidence that every crawl entry point was healthy.

Eighteen sitemap files returned 200. Nobody opened the URLs inside them. For this domain the index was perfect and a third of its contents were gone.

Corrected in the ledger. Checking an index is not checking the pages.

Disclosure: this is our own site — chinesezodiacelement.com. The crawl targeted our sitemap, so the 30% failure rate is ours to publish.

Six URLs out of twenty-eight are the element hub and its children. Those go first. The other twenty-two have been dead for six days already and a week more won't change the diagnosis.