I Checked 390 Sitemap URLs on 13 of My Sites. All Returned 200. The lastmod Dates Were the Problem.

I run 13 small sites: launch boards, a directory, four press sites, a few comparison sites. This morning I wanted to know one boring thing: are the URLs in my sitemaps actually pages a crawler should index?

So I pulled every sitemap listed in each site's robots.txt and sampled 30 URLs per site. 390 URLs in total. Then I checked each one for status code, a noindex (meta tag or X-Robots-Tag header), and whether the canonical pointed back to the same URL.

Result: 390 of 390 returned 200, none were noindexed, and every one had a self-referencing canonical. Clean.

Then I looked at the <lastmod> dates, and that part was not clean at all.

What the lastmod dates said

I fetched the sitemaps at about 05:11 UTC on Nov 14. Here's what came back:

• The four press sites: 296 of their 422 URLs had a lastmod equal to the exact second I fetched the file (05:12 UTC). Mostly category, tag and tool pages. Nothing on those pages changed at 05:12. The sitemap just stamps "now" on anything that has no real date.
• Aura++ (3,243 URLs): 2,063 project pages share one lastmod down to the millisecond, 2026-11-13T08:00:06.360Z. Another 62 URLs were stamped with my fetch time. Fetching again a minute later gave the same values, so it's a cached build, but still: 2,063 launch pages did not all change in the same millisecond.
• Two comparison sites: 436 of 445 URLs on one say 2026-09-02T00:00:00, and 173 of 179 on the other say 2026-09-03T00:00:00. That's a hardcoded date. It's at least stable, but it says nothing true about when each page changed.
• The launch boards (EarlyHunt, IndieHunt, MakerHunt, SideHunt) and the directory looked more spread out, with the biggest single timestamp covering a few hundred URLs, not thousands.

None of this breaks anything you can see. Every URL works. That's why it sat there unnoticed.

Why a wrong lastmod matters

Google's sitemap documentation is short on this. It ignores <priority> and <changefreq>, and about lastmod it says Google uses the value only "if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate." You can read it in Google's guide to building a sitemap.

Bing is even more direct. In a Feb 2023 post on its webmaster blog, Bing says most sitemaps it saw had "the date of generation of the sitemap, rather than the date of content modification", and sitemaps.org now says the date "must be set to the date the linked page was last modified, not when the sitemap is generated."

So a lastmod that always says "now" doesn't make pages look fresh. More likely, the crawler learns your lastmod can't be trusted and stops using it. Then, when you actually update a page, the one signal that could have said "come back, this changed" is already ignored.

I found the same pattern a week ago in structured data: in my JSON-LD audit of 25 pages, 21 said they were modified today. Same root cause, different file.

Where "now" comes from

My guess, which I still need to confirm in the code, is the most common sitemap snippet there is. The Next.js docs example for sitemap.ts literally does this:

{
  url: 'https://acme.com',
  lastModified: new Date(),
  changeFrequency: 'yearly',
  priority: 1,
}

It's fine as a docs example. It's not fine copied into production for every route, because new Date() is the build or request time, not the page's change time.

What I'm moving to:

import type { MetadataRoute } from 'next'

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getPosts() // each has updatedAt from the DB

  return [
    // no real date? leave lastModified out instead of faking one
    { url: 'https://example.com/' },
    ...posts.map((p) => ({
      url: `https://example.com/blog/${p.slug}`,
      lastModified: p.updatedAt,
    })),
  ]
}

For category and tag pages, either use the newest updatedAt of the items on that page, or leave lastmod out. Leaving it out is honest. "Now" isn't.

And updatedAt should move only on a real edit. Google's own example: a change to the main content, structured data or links counts, a copyright-year change doesn't. I wrote a longer guide on when a refreshed post should get a new date on IndieHunt, if you're updating old posts anyway.

Check your own sitemap in two commands

First, the lastmod smell test. If the top line covers most of your URLs, or matches the time you ran it, your dates are generated, not real:

curl -s https://example.com/sitemap.xml \
  | grep -o '<lastmod>[^<]*' | sort | uniq -c | sort -rn | head -3

Run it twice a minute apart. If the timestamp moves, it's request time.

Second, the URL check I used: it samples N URLs and flags anything that isn't a 200, is noindexed, or has a canonical pointing somewhere else.

#!/usr/bin/env bash
# usage: ./check-sitemap.sh https://example.com/sitemap.xml 30
sm="$1"; n="${2:-30}"
curl -s "$sm" | grep -o '<loc>[^<]*' | sed 's/<loc>//' | shuf -n "$n" |
while read -r url; do
  tmp=$(mktemp)
  hdr=$(curl -s -m 20 -o "$tmp" -D - -w '\nCODE=%{http_code}' "$url")
  code=$(printf '%s' "$hdr" | sed -n 's/^CODE=//p')
  noidx=$( (printf '%s' "$hdr" | grep -i '^x-robots-tag:'; \
            grep -oi '<meta[^>]*name="robots"[^>]*>' "$tmp") | grep -ci noindex)
  canon=$(grep -oi '<link[^>]*rel="canonical"[^>]*>' "$tmp" | head -1 \
          | grep -oi 'href="[^"]*"' | cut -d'"' -f2)
  flag=OK
  [ "$code" != 200 ] && flag="STATUS_$code"
  [ "$code" = 200 ] && [ "$noidx" != 0 ] && flag=NOINDEX
  [ "$flag" = OK ] && [ -z "$canon" ] && flag=NO_CANONICAL
  [ "$flag" = OK ] && [ "${canon%/}" != "${url%/}" ] && flag=CANON_ELSEWHERE
  printf '%s\t%s\t%s\n' "$flag" "$code" "$url"
  rm -f "$tmp"
done

It reads the canonical anywhere in the HTML, not only in <head>, which matters on some Next.js setups. It doesn't follow sitemap indexes; point it at the child sitemap.

If you'd rather not run scripts, the free sitemap checker on SideHunt (one of my sites) finds the sitemap from robots.txt, checks size limits, https, duplicate locs and missing lastmod, and test-fetches the first 10 URLs. It flags a missing lastmod, but it won't tell you a "now" date is fake. The uniq -c line above is still the quickest way to see that.

What I'm doing next

1. Drop lastmod from pages with no real change date (categories, tags, tools) on the press sites.
2. Find out what wrote 08:00:06.360 onto 2,063 Aura++ project pages, and switch it to each project's own update time.
3. Leave the two hardcoded comparison-site dates alone until each page has a real updatedAt, then use it.
4. Re-run both checks next week and compare.

Have you ever seen lastmod actually change how fast Google or Bing recrawled a page? Or do you just leave it out?