Find Duplicate and Near-Duplicate Pages Across Your Sitemap with Python (Standard Library Only): Content Hashing, difflib Similarity, CSV Report

Most duplicate pages are not copied on purpose. A CMS keeps serving the old URL after a page is renamed. A newsletter link with a tracking parameter ends up in the sitemap. A print view, a demo copy of the home page, or a set of city pages built from one template quietly becomes a second (or fifth) version of the same content. Each page loads fine on its own. You only see the problem when you compare pages with each other.

This post builds a small duplicate finder in Python using only the standard library (urllib, html.parser, xml.etree, hashlib, difflib, csv, concurrent.futures). It reads a sitemap or sitemap index, fetches every page, strips the shared template, and compares pages three ways: identical HTML, identical main content, and near-identical main content scored with difflib. Matching pages are grouped into clusters, each cluster gets a canonical check and a suggested keeper URL, and the result is a CSV, a console summary and an exit code that works in CI.

Google Search Central's What is canonicalization (showing "Last updated 2026-08-20 UTC"), How to specify a canonical URL with rel="canonical" and other methods (showing "Last updated 2026-07-10 UTC") and Fix canonicalization issues (showing "Last updated 2026-08-21 UTC") were opened on 14 Nov 2026, and every Google statement below comes from those three pages. Appie free-tool descriptions are as of 14 Nov 2026, checked on the tool pages that day. The script was tested on Python 3.13.5 against a local fixture site with planted duplicates; the output below is real, from a run on 14 Nov 2026.

Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.

Where this fits

This is another script in a small standard-library series. An earlier post audited canonical tags and noindex directives across a sitemap, which checks what each page says its canonical is. Another found missing and duplicate H1 tags. This one asks the question underneath both: which pages actually are the same, judged by their content, and do their canonical signals agree about which one should win?

What Google actually says (and what it does not)

The canonicalization overview defines the result of the process: "a canonical URL is the URL of a page that Google chose as the most representative from a set of duplicate pages." It then lists common reasons a site ends up with duplicates, which double as a checklist for what to plant in a test fixture:

• "Region variants: for example, separate USA and UK URLs with hreflang, but essentially the same content in the same language"
• "Device variants: for example, a page with both a mobile and a desktop version"
• "Protocol variants: for example, the HTTP and HTTPS versions of a site"
• "Site functions: for example, the results of sorting and filtering functions of a category page"
• "Accidental variants: for example, the demo version of the site is accidentally left accessible to crawlers"

The same page is clear that duplicates are not a penalty in themselves: "Some duplicate content on a site is normal and it's not a violation of Google's spam policies." The reasons it gives for caring are practical. Having the same content at many URLs "can be a bad user experience" and "may make it harder for you to track how your content performs in search results."

It also describes how Google handles them, and this is the part the script imitates. Google determines "the primary content (or centerpiece) of each page," and "If Google finds multiple pages that seem to be the same or the primary content very similar, it clusters them together." It then picks one page as canonical, and "duplicates are crawled less frequently." Your own signals help but do not decide: "indicating a canonical preference is a hint, not a rule."

The page on specifying a canonical URL ranks the signals "in order of how strongly they can influence canonicalization":

1. Redirects: "A strong signal that the target of the redirect should become canonical."
2. rel="canonical" link annotations: "A strong signal that the specified URL should become canonical."
3. Sitemap inclusion: "A weak signal that helps the URLs that are included in a sitemap become canonical."

Two of its best practices map directly onto checks in the script: "Don't specify different URLs as canonical for the same page using different canonicalization techniques (for example, don't specify one URL in a sitemap, but a different URL for that same page using rel="canonical")," and "When linking within your site, link to the canonical URL rather than a duplicate URL." It also says "We don't recommend using noindex to prevent selection of a canonical page within a single site," and that none of these methods are required: "your site will likely do just fine without specifying a canonical preference."

The troubleshooting page adds the fix for near-duplicates you want kept separate: it "boils down to ensuring that the pages that are clustered together are sufficiently different," and notes that "Google might hold pages in a duplicate cluster for up to two weeks" after you change them.

What none of the three pages give is a similarity percentage. Google does not say how similar two pages must be to be clustered, so the script's --threshold (default 0.90) is a triage line for finding pairs worth a look, not Google's number. URL Inspection in Search Console is where you see which canonical Google actually chose.

How the comparison works

For each page the script keeps only the main content: text inside <main>, <article> or role="main" if the page has one, and everything outside <nav>, <aside>, <form> and top-level <header>/<footer> if it does not. Then, after all pages are fetched, any text block that appears on at least half of them (and on at least three) is dropped as template. That second step removes a "Start your free trial" box that sits inside <main> on every page and would otherwise make unrelated short pages look alike.

Pages are then compared in three passes, cheapest first:

1. exact_html: SHA-256 of the raw response bytes. Identical hashes mean the server returned the same file for two URLs. This is the old URL still serving, or a query-string variant the CMS ignores.
2. same_content: SHA-256 of the normalized main-content words (lowercased, whitespace collapsed). Different templates, same body. Print views and AMP-style copies land here.
3. near_duplicate: difflib.SequenceMatcher(...).ratio() on the two word lists, at or above --threshold. Template pages with a city name swapped land here.

difflib is accurate but slow, and comparing every pair is O(n²). So the near-duplicate pass first builds an inverted index of 5-word shingles and only sends a pair to difflib if their shingle Jaccard similarity is at least --prefilter (default 0.30). Shingles that appear on more than 50 pages are ignored, because a phrase on 50 pages says nothing about any two of them. Pages under --min-words (default 50) skip this pass, since two short pages can score high on very little text.

The prefilter is deliberately loose. Changing one number in a sentence breaks every shingle that touches it, so Jaccard drops much faster than difflib's ratio. On the fixture below, two pricing pages with a difflib ratio of 0.89 had a shingle Jaccard of 0.35, so a prefilter of 0.5 would have hidden them.

Every match is an edge, and a small union-find groups connected pages into clusters. If A matches B and B matches C, all three are one cluster even if A and C fall just under the threshold.

What the script flags

Within each cluster it reads every page's canonical (<link rel="canonical"> first, then a Link: <...>; rel="canonical" HTTP header) and checks whether the signals agree:

• competing_self_canonicals: Meaning: Two or more pages in the cluster each declare themselves canonical; What to check: Either they are meant to be different (make them more different) or one should point to the other
• canonicals_disagree: Meaning: Pages in the cluster point to more than one canonical target; What to check: Pick one target for the whole cluster
• no_canonical: Meaning: A clustered page has no canonical at all; What to check: Add one pointing to the keeper, or redirect
• sitemap_lists_non_canonical: Meaning: The sitemap lists a URL whose own canonical points elsewhere; What to check: Google's "don't specify one URL in a sitemap, but a different URL" case; list the canonical instead
• canonical_not_in_sitemap: Meaning: The canonical target is not in the sitemap; What to check: Often the target is missing, renamed or a 404
• url_variant: Meaning: The URL differs from another member only by query string, trailing slash, index.html, www or case; What to check: Usually a tracking or filter parameter; fix the link or the sitemap generator
• noindex: Meaning: A clustered page has noindex (informational); What to check: Google recommends rel="canonical" over noindex for duplicates on one site
• http_error / fetch_failed / not_html / redirected: Meaning: The sitemap URL did not load as an HTML page; What to check: Fix it or remove it from the sitemap

Each clustered row also gets a suggested keeper: the URL most members already name as canonical, preferring URLs in the sitemap, then URLs without a query string, then the shortest. It is a starting point for a decision, not the decision.

The script

Save as dup_pages.py. No pip install needed.

#!/usr/bin/env python3
"""Find duplicate and near-duplicate pages across a sitemap.

Exact HTML hashes, main-content hashes, shingle prefilter + difflib
similarity, duplicate clusters, canonical checks per cluster, CSV report.
Standard library only.
"""
import argparse
import concurrent.futures as cf
import csv
import difflib
import gzip
import hashlib
import re
import sys
import urllib.error
import urllib.parse
import urllib.request
import xml.etree.ElementTree as ET
from collections import Counter, defaultdict
from html.parser import HTMLParser

UA = "dup-pages-audit/1.0 (+https://example.com/bot)"
SM_NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
SKIP = {"script", "style", "noscript", "template", "svg", "iframe", "head"}
CHROME = {"nav", "aside", "form", "header", "footer"}
MAIN = {"main", "article"}
BLOCK = {"p", "li", "h1", "h2", "h3", "h4", "h5", "h6", "td", "th",
         "blockquote", "pre", "dd", "dt", "figcaption", "div", "section"}
VOID = {"area", "base", "br", "col", "embed", "hr", "img", "input",
        "link", "meta", "source", "track", "wbr"}
WORD = re.compile(r"[^\W_]+(?:['’-][^\W_]+)*", re.UNICODE)

def fetch(url, timeout=20):
    req = urllib.request.Request(url, headers={"User-Agent": UA,
                                               "Accept-Encoding": "gzip"})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = resp.read()
            if resp.headers.get("Content-Encoding") == "gzip":
                body = gzip.decompress(body)
            return (resp.status, resp.geturl(),
                    resp.headers.get("Content-Type", ""),
                    resp.headers.get("Link", ""), body)
    except urllib.error.HTTPError as e:
        return e.code, url, "", "", b""
    except Exception as e:  # DNS, timeout, TLS
        return None, url, str(e), "", b""

def read_sitemap(url, seen=None):
    seen = seen if seen is not None else set()
    if url in seen:
        return []
    seen.add(url)
    status, _, _, _, body = fetch(url)
    if status != 200:
        raise RuntimeError(f"{url} returned {status}")
    if body[:2] == b"\x1f\x8b":
        body = gzip.decompress(body)
    root = ET.fromstring(body)
    if root.tag == SM_NS + "sitemapindex":
        urls = []
        for loc in root.iter(SM_NS + "loc"):
            urls += read_sitemap(loc.text.strip(), seen)
        return urls
    return [loc.text.strip() for loc in root.iter(SM_NS + "loc") if loc.text]

class PageParser(HTMLParser):
    """Collects title, canonical, robots meta and text blocks by context."""

    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.stack = []
        self.title, self.canonical, self.robots = "", "", ""
        self.blocks = []          # (text, in_main, in_chrome)
        self.has_main = False
        self.buf = []

    def flush(self):
        text = " ".join("".join(self.buf).split())
        self.buf = []
        if text:
            in_main = any(m for _, _, m in self.stack)
            in_chrome = any(c for _, c, _ in self.stack)
            self.blocks.append((text, in_main, in_chrome))

    def handle_starttag(self, tag, attrs):
        a = {k: (v or "") for k, v in attrs}
        if tag == "link" and "canonical" in a.get("rel", "").lower().split():
            self.canonical = self.canonical or a.get("href", "").strip()
        if tag == "meta" and a.get("name", "").lower() in ("robots", "googlebot"):
            self.robots += " " + a.get("content", "").lower()
        if tag in VOID:
            return
        if tag in BLOCK or tag in MAIN or tag in CHROME:
            self.flush()
        role = a.get("role", "").lower()
        is_main = tag in MAIN or role == "main"
        self.has_main = self.has_main or is_main
        inside_main = any(m for _, _, m in self.stack)
        chrome = (tag in {"nav", "aside", "form"}
                  or (tag in {"header", "footer"} and not inside_main)
                  or role in {"navigation", "banner", "contentinfo"})
        self.stack.append((tag, chrome, is_main))

    def handle_endtag(self, tag):
        if tag in VOID:
            return
        if tag in BLOCK or tag in MAIN or tag in CHROME:
            self.flush()
        for i in range(len(self.stack) - 1, -1, -1):
            if self.stack[i][0] == tag:
                del self.stack[i:]
                break

    def handle_data(self, data):
        tags = [t for t, _, _ in self.stack]
        if "title" in tags:
            self.title += data
            return
        if any(t in SKIP for t in tags):
            return
        self.buf.append(data)

def norm_block(text):
    return " ".join(text.lower().split())

def analyse(url):
    status, final, ctype, link_hdr, body = fetch(url)
    row = {"url": url, "status": status if status else "fetch_failed",
           "final_url": final, "issues": []}
    if status != 200:
        row["issues"].append("http_error" if status else "fetch_failed")
        return row
    if final != url:
        row["issues"].append("redirected")
    if "html" not in ctype.lower():
        row["issues"].append("not_html")
        return row
    p = PageParser()
    p.feed(body.decode("utf-8", "replace"))
    p.flush()
    canon = p.canonical
    m = re.search(r'<([^>]+)>\s*;\s*rel="?canonical"?', link_hdr or "")
    if m and not canon:
        canon = m.group(1)
    row.update({
        "html_hash": hashlib.sha256(body).hexdigest()[:12],
        "title": " ".join(p.title.split()),
        "canonical": urllib.parse.urljoin(final, canon) if canon else "",
        "noindex": "noindex" in p.robots or "none" in p.robots.split(","),
        "blocks": [b for b, in_main, chrome in p.blocks
                   if not chrome and (in_main or not p.has_main)],
    })
    return row

def variant_key(u):
    """Same key for URLs that differ only by protocol, www, case,
    trailing slash, index.html or query string."""
    s = urllib.parse.urlsplit(u)
    host = s.netloc.lower().removeprefix("www.")
    path = re.sub(r"/(index\.html?)?$", "", s.path.lower()) or "/"
    return host + path

class DSU:
    def __init__(self):
        self.parent = {}

    def find(self, x):
        self.parent.setdefault(x, x)
        while self.parent[x] != x:
            self.parent[x] = self.parent[self.parent[x]]
            x = self.parent[x]
        return x

    def union(self, a, b):
        self.parent[self.find(a)] = self.find(b)

def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("sitemap")
    ap.add_argument("--out", default="duplicate_pages.csv")
    ap.add_argument("--threshold", type=float, default=0.90,
                    help="difflib ratio that counts as near_duplicate")
    ap.add_argument("--prefilter", type=float, default=0.30,
                    help="5-word shingle Jaccard needed before running difflib")
    ap.add_argument("--min-words", type=int, default=50,
                    help="pages with fewer main words skip the near check")
    ap.add_argument("--workers", type=int, default=4)
    ap.add_argument("--limit", type=int, default=0)
    args = ap.parse_args()

    try:
        urls = list(dict.fromkeys(read_sitemap(args.sitemap)))
    except Exception as e:
        print(f"Could not read sitemap: {e}")
        return 2
    if args.limit:
        urls = urls[:args.limit]
    in_sitemap = set(urls)
    with cf.ThreadPoolExecutor(args.workers) as pool:
        rows = list(pool.map(analyse, urls))
    pages = [r for r in rows if "blocks" in r]

    # Template by repetition: block text on >= half the pages (min 3)
    df = Counter(b for r in pages for b in {norm_block(x) for x in r["blocks"]})
    cutoff = max(3, len(pages) / 2)
    for r in pages:
        text = " ".join(b for b in r["blocks"] if df[norm_block(b)] < cutoff)
        r["words"] = [w.lower() for w in WORD.findall(text)]
        r["content_hash"] = hashlib.sha256(
            " ".join(r["words"]).encode()).hexdigest()[:12]

    dsu, edges = DSU(), {}

    def link(a, b, kind, score):
        key = tuple(sorted((a, b)))
        if key not in edges or edges[key][1] < score:
            edges[key] = (kind, score)
        dsu.union(a, b)

    # 1 and 2: identical bytes, then identical main content
    for field, kind in (("html_hash", "exact_html"), ("content_hash", "same_content")):
        groups = defaultdict(list)
        for r in pages:
            if r["words"]:
                groups[r[field]].append(r["url"])
        for members in groups.values():
            for other in members[1:]:
                link(members[0], other, kind, 1.0)

    # 3: near duplicates. Inverted shingle index finds candidate pairs,
    # difflib confirms them on the word sequence.
    by_url = {r["url"]: r for r in pages}
    sh = {}
    for r in pages:
        w = r["words"]
        if len(w) >= args.min_words:
            sh[r["url"]] = {hash(" ".join(w[i:i + 5])) for i in range(len(w) - 4)}
    index = defaultdict(list)
    for u, s in sh.items():
        for h in s:
            index[h].append(u)
    shared = Counter()
    for posting in index.values():
        if len(posting) > 50:      # a phrase on 50+ pages says nothing
            continue
        for i, a in enumerate(posting):
            for b in posting[i + 1:]:
                shared[tuple(sorted((a, b)))] += 1
    near_misses = []
    for (a, b), n in shared.items():
        jac = n / len(sh[a] | sh[b])
        if jac < args.prefilter or (a, b) in edges:
            continue
        ratio = difflib.SequenceMatcher(None, by_url[a]["words"],
                                        by_url[b]["words"], autojunk=False).ratio()
        if ratio >= args.threshold:
            link(a, b, "near_duplicate", ratio)
        else:
            near_misses.append((ratio, a, b))

    # Clusters
    clusters = defaultdict(list)
    for u in {u for pair in edges for u in pair}:
        clusters[dsu.find(u)].append(u)
    ordered = sorted(clusters.values(), key=lambda m: (-len(m), min(m)))
    out_rows = []
    for cid, members in enumerate(ordered, 1):
        members.sort()
        canon_of = {u: by_url[u]["canonical"] for u in members}
        targets = Counter(c for c in canon_of.values() if c)
        self_canon = [u for u in members if canon_of[u] == u]
        # Keeper: most-used canonical target, preferring sitemap URLs,
        # then clean, short URLs. A starting point, not a decision.
        keeper = min(set(members) | set(targets), key=lambda u: (
            -targets[u], u not in in_sitemap, "?" in u, len(u), u))
        vkeys = Counter(variant_key(u) for u in members)
        for u in members:
            r = by_url[u]
            kinds = [edges[k] for k in edges if u in k]
            kind, score = max(kinds, key=lambda e: (e[1], e[0]))
            issues = list(r["issues"])
            if not canon_of[u]:
                issues.append("no_canonical")
            elif canon_of[u] not in in_sitemap:
                issues.append("canonical_not_in_sitemap")
            if len(self_canon) > 1 and u in self_canon:
                issues.append("competing_self_canonicals")
            if len(targets) > 1:
                issues.append("canonicals_disagree")
            if canon_of[u] and canon_of[u] != u:
                issues.append("sitemap_lists_non_canonical")
            if vkeys[variant_key(u)] > 1 and u != keeper:
                issues.append("url_variant")
            if r["noindex"]:
                issues.append("noindex")
            out_rows.append({
                "cluster": cid, "size": len(members), "url": u,
                "match": kind, "similarity": f"{score:.2f}",
                "words": len(r["words"]), "canonical": canon_of[u],
                "suggested_keeper": keeper, "issues": ";".join(issues),
                "title": r["title"]})
    errors = [r for r in rows if "blocks" not in r]
    for r in errors:
        out_rows.append({"cluster": "", "size": "", "url": r["url"],
                         "match": "", "similarity": "", "words": "",
                         "canonical": "", "suggested_keeper": "",
                         "issues": ";".join(r["issues"]), "title": ""})

    with open(args.out, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=list(out_rows[0]) if out_rows else ["url"])
        w.writeheader()
        w.writerows(out_rows)

    dup_urls = sum(len(m) for m in ordered)
    print(f"{len(rows)} sitemap URLs; {len(pages)} HTML pages compared; "
          f"template blocks = text on >= {cutoff:g} pages")
    print(f"{len(ordered)} duplicate clusters covering {dup_urls} URLs; "
          f"{len(errors)} URLs not fetched as HTML")
    for cid, members in enumerate(ordered, 1):
        kinds = Counter(edges[k][0] for k in edges if k[0] in members)
        print(f"  cluster {cid}: {len(members)} pages ({', '.join(sorted(kinds))})")
        for u in members:
            print(f"      {u}")
    if near_misses:
        print(f"Closest pairs below --threshold {args.threshold}:")
        for ratio, a, b in sorted(near_misses, reverse=True)[:5]:
            print(f"  {ratio:.3f}  {a}  ~  {b}")
    counts = Counter(i for r in out_rows for i in r["issues"].split(";") if i)
    for name, n in counts.most_common():
        print(f"  {name:<28} {n}")
    print(f"Report: {args.out}")
    return 1 if ordered or errors else 0

if __name__ == "__main__":
    sys.exit(main())

A few design choices worth knowing:

• autojunk=False matters. By default SequenceMatcher treats very common items as junk once a sequence is longer than 200 elements, and in a word list that means words like "the" and "and". Turning it off makes ratios on long pages stable at some cost in speed, which the shingle prefilter pays back.
• Similarity is on words, not characters. Markup, whitespace and punctuation changes do not count; a changed price or city name does.
• Only pages that load as HTML with status 200 are compared. Redirected URLs are compared using the page they land on and flagged redirected.
• Exit codes: 0 when there are no clusters and no fetch errors, 1 when there is at least one cluster or failed URL, 2 when the sitemap cannot be read.
• Memory is modest. Each page keeps its word list and shingle hashes. For sites over a few tens of thousands of URLs, run one child sitemap at a time with --limit for a first pass.

Testing it against a local fixture

To get output you can check line by line, I built a small fictional site ("Ledgerly", a bookkeeping app) with 16 routes and an 18-URL sitemap, and planted one example of each duplicate type Google lists, plus controls:

• Old URL still serving the same file: URLs: /features/reporting, /features/reports (both canonical to /features/reporting); Expected: exact_html, sitemap lists the non-canonical
• Tracking parameter in the sitemap (a "site functions" variant): URLs: /blog/close-the-books-faster and ?utm_source=newsletter; Expected: exact_html, url_variant
• Print view with a different template, no canonical: URLs: /guides/month-end-checklist and /print; Expected: same_content, no_canonical
• Demo copy of the home page (Google's "accidental variant") with one extra line: URLs: / and /demo/; Expected: near_duplicate, no_canonical
• City pages from one template: URLs: /locations/berlin, /munich (self-canonical), /hamburg (canonical to an unlisted /locations/germany); Expected: one cluster, canonicals disagree
• Old pricing page with different prices and limits: URLs: /pricing, /pricing-2025; Expected: just below the threshold
• Unrelated pages: URLs: /about, /blog/, /blog/reconcile-bank-feeds, /privacy; Expected: not clustered
• Dead URL in the sitemap: URLs: /blog/old-webinar-recap; Expected: http_error

Every page shares a header, footer and a "Start your free trial" box inside <main>. Like many CMSs, the fixture server ignores the query string when routing, which is how the utm_source URL ends up serving the same bytes. The handler is short:

import http.server, json, pathlib, urllib.parse
HERE = pathlib.Path(__file__).parent / "fixture"
ROUTES = json.loads((HERE / "routes.json").read_text())  # path -> HTML

class H(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        path = urllib.parse.urlsplit(self.path).path  # query string ignored
        if path == "/sitemap.xml":
            body, ctype = (HERE / "sitemap.xml").read_bytes(), "application/xml"
        elif path in ROUTES:
            body, ctype = ROUTES[path].encode(), "text/html; charset=utf-8"
        else:
            self.send_response(404); self.end_headers(); return
        self.send_response(200)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

http.server.ThreadingHTTPServer(("127.0.0.1", 8773), H).serve_forever()

Start it, then run the script:

python3 serve_fixture.py &
python3 dup_pages.py http://127.0.0.1:8773/sitemap.xml --out fixture_report.csv

Console output:

18 sitemap URLs; 17 HTML pages compared; template blocks = text on >= 8.5 pages
5 duplicate clusters covering 11 URLs; 1 URLs not fetched as HTML
  cluster 1: 3 pages (near_duplicate)
      http://127.0.0.1:8773/locations/berlin
      http://127.0.0.1:8773/locations/hamburg
      http://127.0.0.1:8773/locations/munich
  cluster 2: 2 pages (near_duplicate)
      http://127.0.0.1:8773/
      http://127.0.0.1:8773/demo/
  cluster 3: 2 pages (exact_html)
      http://127.0.0.1:8773/blog/close-the-books-faster
      http://127.0.0.1:8773/blog/close-the-books-faster?utm_source=newsletter
  cluster 4: 2 pages (exact_html)
      http://127.0.0.1:8773/features/reporting
      http://127.0.0.1:8773/features/reports
  cluster 5: 2 pages (same_content)
      http://127.0.0.1:8773/guides/month-end-checklist
      http://127.0.0.1:8773/guides/month-end-checklist/print
Closest pairs below --threshold 0.9:
  0.890  http://127.0.0.1:8773/pricing  ~  http://127.0.0.1:8773/pricing-2025
  canonicals_disagree          3
  sitemap_lists_non_canonical  3
  competing_self_canonicals    2
  no_canonical                 2
  canonical_not_in_sitemap     1
  url_variant                  1
  http_error                   1
Report: fixture_report.csv
exit code: 1

The CSV (the size and title columns are left out here to keep lines readable):

cluster,url,match,similarity,words,canonical,suggested_keeper,issues
1,http://127.0.0.1:8773/locations/berlin,near_duplicate,0.95,140,http://127.0.0.1:8773/locations/berlin,http://127.0.0.1:8773/locations/berlin,competing_self_canonicals;canonicals_disagree
1,http://127.0.0.1:8773/locations/hamburg,near_duplicate,0.99,143,http://127.0.0.1:8773/locations/germany,http://127.0.0.1:8773/locations/berlin,canonical_not_in_sitemap;canonicals_disagree;sitemap_lists_non_canonical
1,http://127.0.0.1:8773/locations/munich,near_duplicate,0.99,143,http://127.0.0.1:8773/locations/munich,http://127.0.0.1:8773/locations/berlin,competing_self_canonicals;canonicals_disagree
2,http://127.0.0.1:8773/,near_duplicate,0.97,98,http://127.0.0.1:8773/,http://127.0.0.1:8773/,
2,http://127.0.0.1:8773/demo/,near_duplicate,0.97,104,,http://127.0.0.1:8773/,no_canonical
3,http://127.0.0.1:8773/blog/close-the-books-faster,exact_html,1.00,130,http://127.0.0.1:8773/blog/close-the-books-faster,http://127.0.0.1:8773/blog/close-the-books-faster,
3,http://127.0.0.1:8773/blog/close-the-books-faster?utm_source=newsletter,exact_html,1.00,130,http://127.0.0.1:8773/blog/close-the-books-faster,http://127.0.0.1:8773/blog/close-the-books-faster,sitemap_lists_non_canonical;url_variant
4,http://127.0.0.1:8773/features/reporting,exact_html,1.00,74,http://127.0.0.1:8773/features/reporting,http://127.0.0.1:8773/features/reporting,
4,http://127.0.0.1:8773/features/reports,exact_html,1.00,74,http://127.0.0.1:8773/features/reporting,http://127.0.0.1:8773/features/reporting,sitemap_lists_non_canonical
5,http://127.0.0.1:8773/guides/month-end-checklist,same_content,1.00,101,http://127.0.0.1:8773/guides/month-end-checklist,http://127.0.0.1:8773/guides/month-end-checklist,
5,http://127.0.0.1:8773/guides/month-end-checklist/print,same_content,1.00,101,,http://127.0.0.1:8773/guides/month-end-checklist,no_canonical
,http://127.0.0.1:8773/blog/old-webinar-recap,,,,,,http_error

Notes on that result:

• All five planted clusters were found, and nothing else. The four unrelated pages share the same header, footer and trial box, and none of them clustered, because that text was dropped as template (it appeared on 16 of the 17 HTML pages, above the 8.5-page cutoff).
• The utm_source URL is exact_html, url_variant and sitemap_lists_non_canonical. The page's canonical is right; the sitemap is wrong. The fix is in whatever generates the sitemap, not on the page.
• /features/reports is the same story without the parameter. Both URLs return the same file and the canonical already names /features/reporting. Google's guidance for retiring a duplicate is a permanent redirect: "Use this only when deprecating a duplicate page."
• The print view has a different title and no template, so its HTML hash differs, but its main content hashes the same. A byte-level comparison alone would have missed it.
• The demo copy scored 0.97. One extra sentence ("Demo mode. Data resets every night.") is enough to change the content hash, which is why the difflib pass exists.
• The city cluster shows three different canonical targets for three near-identical pages. Berlin and Munich each claim to be canonical, Hamburg points to a page that is not in the sitemap (and in this fixture does not exist). The suggested keeper is Berlin only because it ties with Munich and sorts first. Whether these pages should exist separately is a content decision: if they should, the troubleshooting page's advice is to make them "sufficiently different."
• The old pricing page scored 0.890, just under the 0.90 threshold, and appears in the "closest pairs" list rather than as a cluster. It has different prices and limits, so it is not a duplicate in the sense of identical content, but it is very likely a page nobody meant to keep live.

That last pair is why the threshold is a judgment call, not a rule. Rerunning with --threshold 0.85 pulls it in as a sixth cluster:

18 sitemap URLs; 17 HTML pages compared; template blocks = text on >= 8.5 pages
6 duplicate clusters covering 13 URLs; 1 URLs not fetched as HTML
  ...
  cluster 6: 2 pages (near_duplicate)
      http://127.0.0.1:8773/pricing
      http://127.0.0.1:8773/pricing-2025

The CSV marks both pricing pages competing_self_canonicals and suggests /pricing as the keeper. Lower thresholds find more real problems and more false alarms; on a site with many pages built from one template, start at the default and read the "closest pairs" list before lowering it.

When the sitemap URL is wrong, the script exits early:

Could not read sitemap: http://127.0.0.1:8773/missing.xml returned 404
exit code: 2

Running it on your own sitemap

Point the script at your sitemap, start with a low worker count, and use --limit for a first pass on a large site:

python3 dup_pages.py https://example.com/sitemap.xml --out duplicate_pages.csv --workers 2 --limit 200

Sort the CSV by cluster, then read each cluster as a group. The match column tells you how they matched, and the canonical and suggested_keeper columns tell you whether the site already agrees with itself.

Expect some clusters you meant to have. Product, tool or integration pages built from one template often share most of their text, and the script will cluster them. That is not an error in the script; it is the same thing Google's overview describes when it says it clusters pages whose "primary content" is "very similar." If those pages are meant to rank separately, the fix is more page-specific content, not a lower threshold.

If you want to check one suspect pair by eye first, two free tools on our site can help. The Sitemap URL Extractor takes a sitemap URL or pasted XML and returns one URL per line (for a sitemap index it lists the child sitemaps, so you run it again on each child), which is a quick way to see whether parameter URLs or old paths are in the file at all. Convert a Webpage to Markdown fetches a public page's main content as Markdown, leaving out navigation and scripts, so you can paste two pages into any diff viewer and see what actually differs. Both ask for a captcha before each run and work one URL at a time; the script is what you want for a whole site.

Fixing what it finds

1. Retire true duplicates with a permanent redirect. For exact_html and same_content clusters where one URL is simply old (/features/reports), redirect it to the keeper. Google lists redirects as the strongest signal and says to use them "when you want to get rid of existing duplicate pages."
2. Point every member at one canonical. For duplicates that must stay reachable (print views, parameter URLs, demo copies), add <link rel="canonical" href="https://example.com/keeper"> with an absolute URL, and a self-referencing canonical on the keeper. Google says "Use absolute paths rather than relative paths with the rel="canonical" link element."
3. List only canonical URLs in the sitemap. Every sitemap_lists_non_canonical row is a mixed signal Google's page asks you to avoid. Fix the generator, not the individual entries.
4. Link to the keeper. Update internal links and campaign templates so they use the canonical URL. Tracking parameters in a newsletter are fine; tracking parameters in the sitemap or the navigation are the problem.
5. Make near-duplicates different, or merge them. For template clusters you want to keep, add content that is specific to each page. If they do not each serve a purpose, merge them into one page and redirect the rest. Expect Google to take time to re-evaluate: the troubleshooting page mentions "up to two weeks."
6. Do not use robots.txt or noindex for this. Google's page says "Don't use the robots.txt file for canonicalization purposes" and does not recommend noindex for choosing a canonical within one site.
7. Check what Google chose. Run the keeper and one duplicate through URL Inspection in Search Console. Your canonical is a hint; Google's choice is what shows in results.
8. Rerun after the fix. Keep it in CI so a template change that duplicates a section fails the build.

Extending it

• Cross-host checks. Pass several sitemaps (for example www and a docs subdomain) by concatenating them into one list before fetching, and the variant_key function already treats www and non-www as the same host.
• Hreflang awareness. Google's overview says regional variants in the same language should "use both canonicalization and hreflang." If your site has those, read <link rel="alternate" hreflang> in PageParser and skip pairs that are declared alternates of each other.
• Faster near-duplicate search. For very large sites, replace the inverted index with MinHash over the same shingles, still standard library (hashlib with a few salts), and only send the top candidates to difflib.
• Diff runs. Save the CSV per deploy and compare cluster membership. A cluster that grows from 2 to 40 pages after a release is a template change worth catching early.

FAQ

Does Google penalize duplicate content?
Google's canonicalization overview says "Some duplicate content on a site is normal and it's not a violation of Google's spam policies." Google groups duplicates and picks one URL to show. The reasons to fix duplicates are the ones Google gives: user confusion, split tracking, and crawl time spent on copies.

What similarity score does Google use?
None of the three Search Central pages give one. The 0.90 default is a starting point for review. Pages with different prices, dates or names can still score high, so read the cluster before acting.

Why not just compare titles or H1s?
Different URLs often share a title and have different content (paginated archives), and true duplicates sometimes have different titles (the print view in the fixture). Comparing main content catches both cases; the H1 post linked above covers headings on their own.

Why does the script ignore my header and footer?
Because they are on every page. Text inside <nav>, <aside>, <form> and top-level <header>/<footer> is dropped by structure, and any block that appears on half or more of the pages is dropped by repetition. Without that, every short page on a site would look like every other one.

Can I run it in CI?
Yes. It exits 1 when it finds a cluster or a URL that failed to load, 0 when clean and 2 when the sitemap cannot be read. If you have template clusters you have decided to keep, filter them out of the CSV before failing the build.

Wrap-up

Duplicate pages are usually a by-product of how a site is built: an old URL that still answers, a parameter that leaked into the sitemap, a print view, a demo copy, a template filled in once per city. Each one looks fine alone. Comparing main content across the whole sitemap finds them, grouping them into clusters shows which URLs belong together, and checking canonicals per cluster shows whether your redirects, tags and sitemap all name the same winner. Run it after a migration or a template change, start with the exact_html and same_content clusters, and decide for each near-duplicate cluster whether to merge the pages or make them different.

RG, Founder, Appie.run