I rebuilt my backlink ledger as a Python script. Here is the whole thing
I had a spreadsheet of 147 backlink records. It said 20 of them were live. When I fetched those 20 pages myself, eight carried a link that search engines would follow. That gap is not a data-entry problem. It is a model
I had a spreadsheet of 147 backlink records. It said 20 of them were live. When I fetched those 20 pages myself, eight carried a link that search engines would follow.
That gap is not a data-entry problem. It is a modelling problem: a spreadsheet can hold the claim that a link exists, but it cannot hold the evidence, and it never re-checks. So I rebuilt the ledger around one rule — a link counts when the page says so, today — and the whole thing is four small functions and three tables.
This post is the script, in the order it runs, with the three tables it writes to.
The three verdicts
Every check returns one of three values, and the middle one is the important one:
-
yes — the page is indexable, canonical to itself, and carries an anchor to my domain with no
nofollow. - no — the page is there, and one of those is false.
-
unknown — I could not read the page.
401,403,429,503, a captcha interstitial, a timeout.
The reason unknown exists is a mistake I made early: one site answers 403 to a plain request and serves the page fine to a browser. I read that as "no link", wrote it down, and spent a week believing a listing had failed when I had simply never seen it. Collapsing "no" and "unknown" makes your ledger lie in the direction that feels safe, which is the worst kind.
def verdict(status, robots, canonical, anchors, url):
if status != 200:
return "unknown" # 403/429/503 and friends are not a 'no'
if any("noindex" in r.lower() for r in robots):
return "no"
if canonical and canonical.rstrip("/") != url.rstrip("/"):
return "no"
followed = [a for a in anchors if "nofollow" not in a["rel"].lower().split()]
return "yes" if followed else "no"
Reading the page
Three facts, three regexes. This is deliberately not a headless browser: the fields I need are in the HTML, and a browser adds minutes per page and a whole class of "it rendered differently" problems.
import re, urllib.request
UA = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
def fetch(url, timeout=30):
req = urllib.request.Request(url, headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=timeout) as r:
return r.status, r.geturl(), r.read().decode("utf-8", "replace")
def robots_meta(html):
return [m.group(1) for m in re.finditer(
r'<meta[^>]+name=["\']robots["\'][^>]+content=["\']([^"\']+)', html, re.I)]
def canonical(html):
m = re.search(r'<link[^>]+rel=["\']canonical["\'][^>]+href=["\']([^"\']+)', html, re.I)
return m.group(1) if m else ""
def anchors_to(html, domain):
out = []
for m in re.finditer(r"<a\b([^>]*)>", html, re.I):
href = re.search(r'href=["\']([^"\']+)', m.group(1), re.I)
if not href or domain not in href.group(1):
continue
rel = re.search(r'rel=["\']([^"\']*)["\']', m.group(1), re.I)
out.append({"href": href.group(1), "rel": rel.group(1) if rel else ""})
return out
Two traps that cost me real time:
The domain appearing on a page is not a link. One "live" listing mentions my domain twelve times — meta description, structured data, screen-reader text — and has zero href pointing at it. The play button went through /go/your-product, a redirect wrapper carrying rel="nofollow". Grep for <a href, not for the word.
A stale file makes the greps lie. In a batch run, one URL returned 301 and curl did not write the body. The next three greps happily reported a clean followed anchor — from the previous site, still in the file. Write each URL to its own path and print the status code and byte count first.
The schema that made the numbers agree
Three tables, simplified to the columns that matter, and the split is what fixed my accounting:
CREATE TABLE link_surfaces ( -- the place (a directory, a blog, a profile)
id INTEGER PRIMARY KEY,
canonical_url TEXT NOT NULL,
domain TEXT NOT NULL,
surface_type TEXT DEFAULT 'unknown'
);
CREATE TABLE submissions ( -- one attempt: what we sent, where it landed
id INTEGER PRIMARY KEY,
surface_id INTEGER NOT NULL REFERENCES link_surfaces(id),
target_site TEXT NOT NULL, -- the asset a link should reach (mine)
public_url TEXT, -- the page that carries it (theirs)
status TEXT NOT NULL, -- planned .. submitted .. published .. verified
submitted_at TEXT, published_at TEXT
);
CREATE TABLE observations ( -- what the page actually said, with a timestamp
id INTEGER PRIMARY KEY,
source_url TEXT NOT NULL,
target_url TEXT NOT NULL,
rel TEXT,
observed_at TEXT NOT NULL,
evidence_id INTEGER
);
The rule that came out of this: target_site is my asset, public_url is their page. I had it backwards for my first three articles — I put the article URL in the target field, and the verifier could never match the anchor to a target, so those rows sat in submitted forever while the pages were live. Three live links, invisible in my own report, because two columns were swapped.
Then the checks write themselves:
db.execute("""
INSERT INTO observations(source_url, target_url, rel, observed_at, evidence_id)
VALUES (?, ?, ?, ?, ?)
""", (public_url, anchor["href"], anchor["rel"], now_iso(), evidence_id))
and the summary is a join, not a formula in a spreadsheet:
SELECT s.id, s.public_url, o.rel, o.observed_at
FROM submissions s
JOIN observations o ON o.source_url = s.public_url
WHERE s.status = 'verified'
ORDER BY o.observed_at DESC;
rel is stored as the page literally wrote it (noopener noreferrer, ugc nofollow, dofollow), so the filter for "followable" is rel NOT LIKE '%nofollow%' — and because the value is verbatim, I can also see which kind of nofollow a site uses, which turned out to be a useful signal about the site rather than about my listing.
What the script found in 147 records
- 20 records were marked live. Fetched one by one: 8 followed, 7
nofollow, 4 unreadable, 1noindex. - 13 of the "submitted" records had a public URL at all. Fetched: 0 followed. Five were
nofollow, four had no anchor to my domain, one wasnoindex, two were 404, one blocked. - The other 102 "submitted" records had no URL stored. There is nothing to check and nothing to claim.
That last line is the one that changed how I work. A record without a URL is not a backlink; it is a note that I once filled in a form. My spreadsheet had 115 of those, and it counted them as work.
Running it daily
One command, one row per surface, an exit 1 if anything that was yes yesterday is no today. The point is not the dashboard — it is that the number on the dashboard is a query result that can be reproduced by anyone with the script and the database, and cannot be typed in by hand.
Two things I would tell anyone starting this: store the timestamp and a hash of the page next to every verdict, so "it changed" and "I remembered wrong" stay distinguishable; and re-check on a schedule, because one of my listings was dofollow on 2026-11-11 and reads rel="noopener noreferrer ugc nofollow" today.
I build Copero, a free browser football career game — no download, no account:
https://copero.games/?utm_source=devto&utm_medium=blog&utm_campaign=backlink-2026-11
The script above is the shape of the one that produced these numbers; the 147-record audit ran on 2026-11-18 and every verdict came from the page HTML, not from a directory's status badge.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.