Dev.to WebDev 🛠 Dev 👁 0 📖 18 min read

Check Whether Your Sitemap lastmod Dates Are Accurate with Python (Standard Library Only): Compare with dateModified

Most sitemap checkers ask one question about <lastmod>: is it a valid date? That is the easy half. The harder half is whether the date is true. A static site generator that stamps every URL with the build date passes val

Most sitemap checkers ask one question about <lastmod>: is it a valid date? That is the easy half. The harder half is whether the date is true. A static site generator that stamps every URL with the build date passes validation. So does a CMS that never updates <lastmod> after an edit. Both are wrong, and Google says it only uses <lastmod> when it is "consistently and verifiably" accurate.

This post builds a small auditor in Python using only the standard library (urllib, urllib.robotparser, xml.etree, html.parser, json, email.utils, datetime, csv, concurrent.futures). It reads your sitemap or sitemap index, fetches every page, and compares each <lastmod> with what the page says about itself: dateModified and datePublished in JSON-LD (including @graph), article:modified_time and similar meta tags, and the Last-Modified response header. It flags invalid and future dates, sitemap dates older or newer than the page's own date, pages whose own dates disagree, and one <lastmod> value shared by a large share of URLs, which usually means a build stamp.

Google Search Central's Build and submit a sitemap page (showing "Last updated 2026-07-08 UTC") and the sitemaps.org protocol page were opened on 20 Nov 2026, and every statement attributed to them below comes from those pages. The output in this post comes from a small fictional site ("Fernwood Ceramics") served on localhost, built for this article with planted problems. Appie free-tool descriptions are as of 20 Nov 2026, checked on each tool's page that day.

Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.

Where this fits

This is another script in a small standard-library series that reads a sitemap and checks one thing per page. An earlier post on validating an XML sitemap checks that <lastmod> is a valid W3C Datetime. Another on generating a sitemap for a static site writes one. This one asks whether the dates you publish match the pages they describe.

What Google and the protocol actually say

Google. On its sitemap page: "Google ignores <priority> and <changefreq> values. Google uses the <lastmod> value if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate." And: "The <lastmod> value should reflect the date and time of the last significant update to the page. For example, an update to the main content, the structured data, or links on the page is generally considered significant, however an update to the copyright date is not."

The protocol. sitemaps.org defines <lastmod> as "The date of last modification of the page," in W3C Datetime format, where you can "omit the time portion, if desired, and use YYYY-MM-DD." It adds: "the date must be set to the date the linked page was last modified, not when the sitemap is generated." And: "this tag is separate from the If-Modified-Since (304) header the server can return, and search engines may use the information from both sources differently."

Two practical readings follow. First, a build-time stamp on every URL is exactly what the protocol says not to do. Second, the Last-Modified header is a separate signal. On many static hosts it reflects when the file was last written, which after a full rebuild is every file, so a header match proves little on its own. The script reports header mismatches as low severity for that reason.

What neither page defines is how Google verifies "the last modification of the page." The script uses the dates a page publishes about itself as the best available proxy. That is my choice, not Google's documented method.

How the audit works

  1. Read the sitemap (or every child sitemap in an index) and collect each <loc> with its <lastmod>, if any.
  2. Check robots.txt and skip disallowed URLs.
  3. Validate each <lastmod> as W3C Datetime and flag dates more than a day in the future.
  4. Fetch the page. Parse JSON-LD blocks and walk the whole tree for dateModified and datePublished. Read article:modified_time, og:updated_time and itemprop="dateModified" meta tags. Read the Last-Modified header.
  5. Take the newest page modification date and compare it with <lastmod>, with a tolerance (default one day) so time zones and same-day edits do not trigger noise.
  6. After all pages are done, count how many URLs share each <lastmod> value.
  7. Write a CSV sorted by severity and print a summary.

What the script flags

  • invalid_lastmod: Severity: high; Meaning: Not a W3C Datetime (for example 21/07/2026)
  • future_lastmod: Severity: high; Meaning: More than a day after the time you ran the script
  • lastmod_before_page_date: Severity: high; Meaning: The page says it was updated later than the sitemap does. The sitemap is stale, so the update may go unnoticed
  • lastmod_before_published: Severity: high; Meaning: The sitemap date is earlier than the page's own publish date
  • fetch_error: Severity: high; Meaning: The URL in the sitemap did not return a page (404, timeout)
  • lastmod_after_page_date: Severity: medium; Meaning: The sitemap claims a newer change than the page shows. Often a build stamp or a trivial edit
  • page_dates_conflict: Severity: medium; Meaning: JSON-LD and meta tags on the same page give different modification dates
  • modified_before_published: Severity: medium; Meaning: The page's dateModified is earlier than its datePublished
  • shared_lastmod: Severity: medium; Meaning: One <lastmod> value appears on at least 25% of URLs (and at least 3). Usually a build stamp
  • header_mismatch: Severity: low; Meaning: <lastmod> and the Last-Modified header differ by more than the tolerance
  • missing_lastmod: Severity: low; Meaning: No <lastmod>. Optional in the protocol, but then Google has nothing to use
  • no_page_date: Severity: info; Meaning: The page publishes no modification date, so <lastmod> cannot be checked against it

The script

Save this as lastmod_audit.py. It needs Python 3.9 or newer and nothing outside the standard library.

#!/usr/bin/env python3
"""Check whether sitemap <lastmod> dates match what each page says about itself.

Standard library only. Reads a sitemap or sitemap index, fetches every page and compares
<lastmod> with the page's own dates (JSON-LD dateModified/datePublished, meta
article:modified_time) and the Last-Modified response header. Writes a CSV report.

Usage: python3 lastmod_audit.py https://example.com/sitemap.xml --out lastmod_report.csv
"""
import argparse, csv, json, re, sys, time
import urllib.request, urllib.error, urllib.robotparser
import xml.etree.ElementTree as ET
from collections import Counter
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timedelta, timezone
from email.utils import parsedate_to_datetime
from html.parser import HTMLParser
from urllib.parse import urlsplit

UA = "lastmod-audit/1.0 (+standard-library script)"
NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"

# W3C Datetime: YYYY, YYYY-MM, YYYY-MM-DD, or a full date with time and a zone.
W3C = re.compile(
    r"^\d{4}(-\d{2}(-\d{2}(T\d{2}:\d{2}(:\d{2}(\.\d+)?)?(Z|[+-]\d{2}:\d{2}))?)?)?$")

SEVERITY = {
    "invalid_lastmod": "high", "future_lastmod": "high", "lastmod_before_page_date": "high",
    "lastmod_before_published": "high", "fetch_error": "high",
    "lastmod_after_page_date": "medium", "page_dates_conflict": "medium",
    "modified_before_published": "medium", "shared_lastmod": "medium",
    "header_mismatch": "low", "missing_lastmod": "low", "no_page_date": "info",
}


def fetch(url, timeout=15):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return r.status, dict(r.headers), r.read()


def parse_w3c(value):
    """Return an aware UTC datetime for a W3C Datetime string, or None if invalid."""
    v = (value or "").strip()
    if not W3C.match(v):
        return None
    if len(v) == 4:
        v += "-01-01"
    elif len(v) == 7:
        v += "-01"
    v = v.replace("Z", "+00:00")
    try:
        dt = datetime.fromisoformat(v)
    except ValueError:
        return None
    if dt.tzinfo is None:  # date only: treat as midnight UTC
        dt = dt.replace(tzinfo=timezone.utc)
    return dt.astimezone(timezone.utc)


def parse_page_date(value):
    """Page dates are looser than W3C (ISO 8601 with or without zone). Try both."""
    dt = parse_w3c(value)
    if dt:
        return dt
    try:
        dt = datetime.fromisoformat((value or "").strip().replace("Z", "+00:00"))
    except ValueError:
        return None
    if dt.tzinfo is None:
        dt = dt.replace(tzinfo=timezone.utc)
    return dt.astimezone(timezone.utc)


def read_sitemap(url, seen=None):
    """Yield (loc, lastmod_text) from a sitemap or sitemap index."""
    seen = seen or set()
    if url in seen:
        return
    seen.add(url)
    _, _, body = fetch(url)
    root = ET.fromstring(body)
    if root.tag == NS + "sitemapindex":
        for sm in root.findall(NS + "sitemap"):
            loc = sm.findtext(NS + "loc", "").strip()
            if loc:
                yield from read_sitemap(loc, seen)
    else:
        for u in root.findall(NS + "url"):
            loc = u.findtext(NS + "loc", "").strip()
            lm = u.findtext(NS + "lastmod")
            yield loc, (lm.strip() if lm is not None else None)


class DateParser(HTMLParser):
    """Collect JSON-LD blocks and meta tags that carry modification dates."""

    def __init__(self):
        super().__init__()
        self.in_ld = False
        self.ld_chunks = []
        self.meta = {}

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "script" and (a.get("type") or "").lower() == "application/ld+json":
            self.in_ld = True
            self.ld_chunks.append("")
        elif tag == "meta":
            key = (a.get("property") or a.get("itemprop") or a.get("name") or "").lower()
            if key in ("article:modified_time", "og:updated_time", "datemodified",
                       "article:published_time", "datepublished"):
                self.meta[key] = a.get("content", "")

    def handle_endtag(self, tag):
        if tag == "script":
            self.in_ld = False

    def handle_data(self, data):
        if self.in_ld:
            self.ld_chunks[-1] += data


def walk_ld(node, found):
    """Find dateModified and datePublished anywhere in a JSON-LD tree (including @graph)."""
    if isinstance(node, dict):
        for k in ("dateModified", "datePublished"):
            if isinstance(node.get(k), str):
                found.setdefault(k, []).append(node[k])
        for v in node.values():
            walk_ld(v, found)
    elif isinstance(node, list):
        for v in node:
            walk_ld(v, found)


def page_dates(html):
    p = DateParser()
    p.feed(html)
    found = {}
    for chunk in p.ld_chunks:
        try:
            walk_ld(json.loads(chunk), found)
        except json.JSONDecodeError:
            continue
    modified = {}
    if found.get("dateModified"):
        modified["jsonld"] = max(filter(None, map(parse_page_date, found["dateModified"])), default=None)
    for key in ("article:modified_time", "og:updated_time", "datemodified"):
        if p.meta.get(key):
            modified["meta:" + key] = parse_page_date(p.meta[key])
    published = None
    pubs = found.get("datePublished", []) + [p.meta[k] for k in ("article:published_time", "datepublished") if p.meta.get(k)]
    if pubs:
        published = min(filter(None, map(parse_page_date, pubs)), default=None)
    return {k: v for k, v in modified.items() if v}, published


def days(a, b):
    return abs((a - b).total_seconds()) / 86400


def audit_url(loc, lm_text, tolerance, now, robots):
    row = {"url": loc, "lastmod": lm_text or "", "page_modified": "", "page_published": "",
           "last_modified_header": "", "flags": [], "notes": []}
    if robots and not robots.can_fetch(UA, loc):
        row["notes"].append("skipped: disallowed by robots.txt")
        return row
    lm = None
    if lm_text is None:
        row["flags"].append("missing_lastmod")
    else:
        lm = parse_w3c(lm_text)
        if lm is None:
            row["flags"].append("invalid_lastmod")
            row["notes"].append(f"not a W3C Datetime: {lm_text}")
        elif lm > now + timedelta(days=1):
            row["flags"].append("future_lastmod")
    try:
        status, headers, body = fetch(loc)
    except urllib.error.HTTPError as e:
        row["flags"].append("fetch_error")
        row["notes"].append(f"HTTP {e.code}")
        return row
    except (urllib.error.URLError, TimeoutError) as e:
        row["flags"].append("fetch_error")
        row["notes"].append(str(e))
        return row
    hdr = headers.get("Last-Modified")
    hdr_dt = None
    if hdr:
        try:
            hdr_dt = parsedate_to_datetime(hdr).astimezone(timezone.utc)
            row["last_modified_header"] = hdr_dt.strftime("%Y-%m-%d")
        except (TypeError, ValueError):
            pass
    modified, published = page_dates(body.decode("utf-8", "replace"))
    if published:
        row["page_published"] = published.strftime("%Y-%m-%d")
    if modified:
        vals = sorted(modified.values())
        newest = vals[-1]
        row["page_modified"] = newest.strftime("%Y-%m-%d")
        if days(vals[0], vals[-1]) > tolerance:
            row["flags"].append("page_dates_conflict")
            row["notes"].append("; ".join(f"{k}={v:%Y-%m-%d}" for k, v in modified.items()))
        if published and newest < published - timedelta(days=tolerance):
            row["flags"].append("modified_before_published")
        if lm:
            if lm < newest - timedelta(days=tolerance):
                row["flags"].append("lastmod_before_page_date")
            elif lm > newest + timedelta(days=tolerance):
                row["flags"].append("lastmod_after_page_date")
    else:
        row["flags"].append("no_page_date")
    if lm and published and lm < published - timedelta(days=tolerance):
        row["flags"].append("lastmod_before_published")
    if lm and hdr_dt and days(lm, hdr_dt) > tolerance:
        row["flags"].append("header_mismatch")
    return row


def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("sitemap")
    ap.add_argument("--out", default="lastmod_report.csv")
    ap.add_argument("--tolerance", type=float, default=1.0, help="days of slack before a mismatch counts")
    ap.add_argument("--shared-share", type=float, default=0.25,
                    help="flag a lastmod value shared by at least this share of URLs")
    ap.add_argument("--workers", type=int, default=4)
    ap.add_argument("--delay", type=float, default=0.2, help="seconds between requests per worker")
    args = ap.parse_args()

    now = datetime.now(timezone.utc)
    parts = urlsplit(args.sitemap)
    robots = urllib.robotparser.RobotFileParser(f"{parts.scheme}://{parts.netloc}/robots.txt")
    try:
        robots.read()
    except Exception:
        robots = None

    entries = list(read_sitemap(args.sitemap))
    print(f"Sitemap entries: {len(entries)}")

    def job(e):
        time.sleep(args.delay)
        return audit_url(e[0], e[1], args.tolerance, now, robots)

    with ThreadPoolExecutor(max_workers=args.workers) as ex:
        rows = list(ex.map(job, entries))

    # Site-level check: one lastmod value on a large share of URLs usually means a build stamp.
    counts = Counter(r["lastmod"] for r in rows if r["lastmod"])
    with_lm = sum(counts.values())
    for value, n in counts.items():
        if with_lm >= 5 and n >= 3 and n / with_lm >= args.shared_share:
            print(f"Shared lastmod: {value} on {n} of {with_lm} URLs with a lastmod")
            for r in rows:
                if r["lastmod"] == value:
                    r["flags"].append("shared_lastmod")

    order = {"high": 0, "medium": 1, "low": 2, "info": 3}
    with open(args.out, "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["url", "severity", "flags", "lastmod", "page_modified", "page_published",
                    "last_modified_header", "notes"])
        for r in sorted(rows, key=lambda r: min([order[SEVERITY[x]] for x in r["flags"]] or [9])):
            sev = min((SEVERITY[x] for x in r["flags"]), key=order.get, default="ok")
            w.writerow([r["url"], sev, " ".join(r["flags"]), r["lastmod"], r["page_modified"],
                        r["page_published"], r["last_modified_header"], " | ".join(r["notes"])])

    flagged = Counter(x for r in rows for x in r["flags"])
    clean = sum(1 for r in rows if not r["flags"])
    print(f"Pages checked: {len(rows)}  clean: {clean}")
    for flag, n in sorted(flagged.items(), key=lambda kv: (order[SEVERITY[kv[0]]], kv[0])):
        print(f"  {SEVERITY[flag]:<6} {flag:<26} {n}")
    print(f"Report: {args.out}")


if __name__ == "__main__":
    sys.exit(main())

A few design notes:

  • Date-only values are treated as midnight UTC. That is why the tolerance exists. A page that says 2026-09-30T08:15:00+02:00 and a sitemap that says 2026-09-30 are the same day and should not be flagged.
  • The newest page date wins. If JSON-LD and a meta tag disagree, the script compares <lastmod> with the newer one and flags the disagreement separately.
  • shared_lastmod is a heuristic. A small site that really did update a third of its pages on one day will trip it. Read the URLs before you act.
  • It reads the HTML the server sends. Dates injected by JavaScript after load are not seen.

Running it against a test site

The fixture is a fictional pottery studio with twelve pages and one URL that no longer exists. Each page has a different planted problem, and the files' timestamps are set so Python's built-in server sends a Last-Modified header we control. To reproduce something similar locally:

python3 build_fixture.py              # writes ./site with pages, sitemap.xml, robots.txt
python3 -m http.server 8765 --bind 127.0.0.1 --directory site
python3 lastmod_audit.py http://localhost:8765/sitemap.xml --out lastmod_report_before.csv

Console output from the run:

Sitemap entries: 13
Shared lastmod: 2026-11-14 on 4 of 12 URLs with a lastmod
Pages checked: 13  clean: 2
  high   fetch_error                1
  high   future_lastmod             1
  high   invalid_lastmod            1
  high   lastmod_before_page_date   1
  medium lastmod_after_page_date    3
  medium modified_before_published  1
  medium page_dates_conflict        1
  medium shared_lastmod             4
  low    header_mismatch            2
  low    missing_lastmod            1
  info   no_page_date               3
Report: lastmod_report_before.csv

And the CSV:

url,severity,flags,lastmod,page_modified,page_published,last_modified_header,notes
http://localhost:8765/blog/kiln-care.html,high,lastmod_before_page_date header_mismatch,2026-03-01,2026-10-28,2025-08-11,2026-10-28,
http://localhost:8765/blog/clay-bodies.html,high,future_lastmod lastmod_after_page_date header_mismatch,2026-12-25,2026-09-09,2026-02-02,2026-09-09,
http://localhost:8765/blog/firing-schedule.html,high,invalid_lastmod,21/07/2026,2026-07-21,2026-01-15,2026-07-21,not a W3C Datetime: 21/07/2026
http://localhost:8765/blog/old-post.html,high,fetch_error,2026-01-10,,,,HTTP 404
http://localhost:8765/blog/glaze-guide.html,medium,lastmod_after_page_date shared_lastmod,2026-11-14,2026-06-03,2025-11-20,2026-11-14,
http://localhost:8765/blog/studio-news.html,medium,modified_before_published lastmod_after_page_date,2026-10-05,2026-09-12,2026-10-05,2026-10-05,
http://localhost:8765/workshops.html,medium,page_dates_conflict,2026-10-19,2026-10-19,,2026-10-19,jsonld=2026-08-01; meta:article:modified_time=2026-10-19
http://localhost:8765/shop/mugs.html,medium,no_page_date shared_lastmod,2026-11-14,,,2026-11-14,
http://localhost:8765/shop/bowls.html,medium,no_page_date shared_lastmod,2026-11-14,,,2026-11-14,
http://localhost:8765/shop/planters.html,medium,no_page_date shared_lastmod,2026-11-14,,,2026-11-14,
http://localhost:8765/about.html,low,missing_lastmod,,2026-04-18,,2026-04-18,
http://localhost:8765/,ok,,2026-11-02,2026-11-02,,2026-11-02,
http://localhost:8765/contact.html,ok,,2026-09-30T08:15:00+02:00,2026-09-30,,2026-09-30,

How to read it, worst first:

  • kiln-care.html is the one that hurts. The post was updated on 28 Oct (JSON-LD and article:modified_time agree, and so does the header), but the sitemap still says 1 March. A crawler that trusts the sitemap has no reason to come back.
  • clay-bodies.html has 2026-12-25 in the sitemap: a future date, usually a typo or a scheduled-post bug.
  • firing-schedule.html uses 21/07/2026, which is not W3C Datetime. The page itself says 21 July, so the fix is just the format.
  • old-post.html is in the sitemap but returns 404. Remove it.
  • glaze-guide.html and the three shop pages share 2026-11-14, the day of the last full rebuild. The glaze guide's own dates say 3 June. Note that its Last-Modified header also says 14 Nov, because the build rewrote the file. That is why a header match is weak evidence.
  • studio-news.html says it was modified (12 Sep) before it was published (5 Oct), which is a template or migration error in the page itself.
  • workshops.html has two different modification dates on one page: JSON-LD says 1 Aug, the meta tag says 19 Oct. Pick one source of truth.
  • about.html has no <lastmod> at all, although the page knows its date.
  • The shop pages publish no date of their own, so their <lastmod> cannot be verified from the page. Adding dateModified to their structured data would fix that.

Fixing it and running again

The fixes, in the fixture:

  1. Sitemap <lastmod> now comes from each page's own modification date, not the build time.
  2. Fixed the date format, removed the future date and dropped the deleted URL.
  3. Corrected the studio news JSON-LD and made the workshops page use one modification date.
  4. Added dateModified to the shop pages and gave about.html a <lastmod>.
  5. Set file timestamps to the content dates, so the header agrees too.

Second run:

Sitemap entries: 12
Pages checked: 12  clean: 12
Report: lastmod_report_after.csv

Every row in the new CSV reads ok.

Running it on your own site

python3 lastmod_audit.py https://www.example.com/sitemap.xml --out lastmod_report.csv --workers 3 --delay 0.5
  • Keep --workers low and --delay polite on production sites, and run it against sites you own or have permission to audit.
  • Use --tolerance 2 if your CMS writes <lastmod> in UTC and pages in local time and you see one-day noise.
  • Use --shared-share 0.5 on small sites where 25% is too sensitive.
  • If every row says missing_lastmod and no_page_date, the script has nothing to compare. That is a finding in itself: decide where your real modification dates live and publish them.

What to fix first

  1. lastmod_before_page_date. Stale sitemap dates on pages you actually improved.
  2. invalid_lastmod, future_lastmod and fetch_error. Quick, mechanical fixes.
  3. shared_lastmod and lastmod_after_page_date. Usually one change in the generator: write the content's modified date, not the build time.
  4. page_dates_conflict and modified_before_published. Template fixes, so they repair many pages at once.
  5. missing_lastmod and no_page_date. Add dates where you can keep them honest.

Google's own example of what does not count as significant is a copyright date change. If your generator bumps <lastmod> when the footer year changes, fix that before anything else in this list.

Related free tools

Two of our free tools help with the sitemap side. The XML Sitemap Validator checks the root element, absolute URLs, lastmod, changefreq, priority, duplicates and the 50,000 URL limit, and gives a score that drops for errors and warnings. The Sitemap URL Extractor gives you one URL per line from a sitemap (for a sitemap index it lists the child sitemaps, which you then run one at a time), a quick way to see what you are about to audit. Neither compares <lastmod> with the pages themselves; that is what the script is for.

FAQ

Does Google use lastmod?
Google's sitemap documentation says it uses <lastmod> "if it's consistently and verifiably ... accurate," and ignores <priority> and <changefreq>.

Should lastmod change when I fix a typo or update the footer?
Google says <lastmod> should reflect the last significant update, such as main content, structured data or links, and that a copyright date change is not significant. A typo fix is a judgment call; a footer year is not.

Is the Last-Modified header the same as lastmod?
No. sitemaps.org says <lastmod> is separate from the If-Modified-Since (304) header and that search engines may use the two differently. On static hosts the header often reflects the last build, not the last content change.

My generator stamps every URL with the build date. What should I do?
Write the content's own modified date instead, for example from front matter or from your version control history for that file. If you cannot get a real date, leaving <lastmod> out is better than a false one.

Why does the script say no_page_date for my pages?
They publish no dateModified in JSON-LD and no modification meta tag, so there is nothing on the page to compare with. Many product and landing pages are like that.

If you want a quick structural check before you run audits like this one, our free XML Sitemap Validator is a fast first step.

RG, Founder, Appie.run

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.