How to Verify SEO Fixes After Every Deploy (Crawl Diffing Explained)

Your SEO ticket says "fixed." The deploy went out. How do you know it actually worked, and that it didn't break something else?

Most SEO audit tools answer with a net count: "Broken internal links: 16 → 17." That number is almost useless for a developer. It could mean one new broken link. It could also mean ten new ones hidden behind nine fixes.

What you want is a diff: for each issue type, which URLs were fixed and which are new. This post shows the logic, a small script you can run on any two crawl exports, and the gotchas we hit doing this on our own site.

The model: issues are sets of URLs

A crawl audit is just a mapping from issue type to the set of URLs that have it:

before["Client error (4xx)"] = {/blog/a, /blog/b, /pricing-old, ...}
after["Client error (4xx)"]  = {/pricing-old, ...}

Two set differences give you everything:

• fixed = before minus after
• new = after minus before

Net change (len(after) - len(before)) throws that information away.

A real example

We ran this on siteauditlint.com between two audits on the same day:

Net change says "one better, one worse." The diff says the fixes worked, and a post published that afternoon introduced exactly one new broken link. That is a ticket a developer can close in five minutes.

The script

Export issue rows from your crawler as CSV with at least an issue column and a url column (most SEO crawlers can do this). Export the full list of crawled URLs too, so you can tell "fixed" apart from "disappeared."

import csv
import sys
from collections import defaultdict

def load_issues(path):
    issues = defaultdict(set)
    with open(path, newline="", encoding="utf-8") as f:
        for row in csv.DictReader(f):
            issues[row["issue"]].add(row["url"])
    return issues

def load_urls(path):
    with open(path, newline="", encoding="utf-8") as f:
        return {row["url"] for row in csv.DictReader(f)}

before, after = load_issues(sys.argv[1]), load_issues(sys.argv[2])
urls_before, urls_after = load_urls(sys.argv[3]), load_urls(sys.argv[4])

# Scope check: comparisons only mean something if both crawls cover the same site
ratio = len(urls_after) / max(len(urls_before), 1)
if not 0.9 <= ratio <= 1.1:
    print(f"WARNING: crawl scope changed ({len(urls_before)} vs {len(urls_after)} URLs)")

for issue in sorted(set(before) | set(after)):
    b, a = before.get(issue, set()), after.get(issue, set())
    fixed, new = b - a, a - b
    gone = {u for u in fixed if u not in urls_after}  # left the crawl, not truly fixed
    if fixed or new:
        print(f"{issue}: {len(b)} -> {len(a)} | fixed {len(fixed) - len(gone)} "
              f"| dropped {len(gone)} | new {len(new)}")
        for u in sorted(new):
            print(f"    NEW  {u}")

Run it:

python crawl_diff.py issues_before.csv issues_after.csv urls_before.csv urls_after.csv

Three gotchas we hit

1. "Fixed" can mean "not crawled." In our example, one of the two fixed 4xx URLs didn't start returning 200. It dropped out of the crawl because the links pointing to it were removed. Total URLs went from 711 to 710. That still counts as resolved for most purposes, but your report should say which is which. That's what the dropped count is for.

2. Scope changes fake regressions. Compare a 100-URL crawl with a 710-URL crawl and critical issues will "rise" even if nothing broke. We had exactly this in our own history. The 10% scope check above catches it.

3. Settings must match. If the baseline ran with JavaScript rendering on and the new crawl ran with it off, rendered-only links and canonicals will appear as new issues. Same user agent, same robots.txt handling, same limits.

Putting it in the pipeline

The useful version of this runs automatically:

1. Save a baseline crawl of production (or staging) as your reference.
2. After each deploy, run a headless crawl with the same settings.
3. Diff against the baseline and fail or flag the job if any critical issue has new URLs.
4. Post the new URLs to your team channel or issue tracker, with the source page and broken target.

Treat it like a regression test: you don't need zero issues, you need zero new critical ones.

Tools that do this for you

If you'd rather not maintain a script:

• Screaming Frog (paid) has a compare mode showing URLs added to and removed from each filter.
• Semrush and Ahrefs show new and fixed or removed issues between scheduled cloud crawls.
• Sitebulb Pro has audit comparison and history.
• SiteAuditLint, which we build, shows fixed and new URLs per issue, labels every issue row New or Existing, supports headless scheduled audits on Windows, and the Agency edition can send issues to Linear and Slack.

We wrote a longer comparison of eight site audit tools, including pricing and JavaScript rendering support: the best SEO audit tools in 2027, compared.

A closed ticket isn't proof. A passing recrawl is.