My templated pages were near-duplicates. A small Python check proved it, and now it runs in CI
I run a small static site of browser-based tools: image compressors, PDF converters, translators. The pages are generated by a Python script. That's convenient, until you notice Google has crawled most of them and decide
I run a small static site of browser-based tools: image compressors, PDF converters, translators. The pages are generated by a Python script. That's convenient, until you notice Google has crawled most of them and decided not to index them.
Search Console's verdict for 38 pages was "Crawled β currently not indexed". The site is new, so part of that is just age. But when I looked at my own generator, part of it was clearly my fault: templates.
How templated pages quietly become duplicates
Each "compress image to N KB" page came from one function. So did each "X to Y converter" and each "language A to language B translator". The function filled in the numbers and names, but everything around them was identical:
- the same "How toβ¦" steps with one word swapped,
- the same FAQ answers ("Is it free? Yes, completely freeβ¦"),
- the same privacy sentence on every single page.
To a person, the 20 KB page and the 50 KB page look different because the number in the H1 differs. To a crawler comparing text, they're mostly the same document.
Measuring it instead of guessing
I wrote a small checker that does two things over the built HTML.
1. Count repeated sentences. Pull the main content out, split it into sentences, normalise them (lowercase, letters and digits only) and count how many pages each sentence of 8+ words appears on.
def sentences(text):
out = set()
for s in re.split(r'(?<=[.!?])\s+', text):
w = re.findall(r"[a-z0-9']+", s.lower())
if len(w) >= 8:
out.add(" ".join(w))
return out
2. Measure overlap between pages. Break each page into 5-word shingles and compute the Jaccard similarity for every pair.
def shingles(text, n=5):
w = re.findall(r"[a-z0-9']+", text.lower())
return {" ".join(w[i:i+n]) for i in range(len(w) - n + 1)}
def jaccard(a, b):
return len(a & b) / max(1, len(a | b))
The first run was humbling. My privacy line appeared on 21 pages. The four compress-to-KB pages scored about 0.55 against each other, meaning more than half of their phrasing was shared. The converters sat around 0.40β0.45 and the translators around 0.36.
What I changed
- Deleted the boilerplate. The privacy promise already sits in the upload box ("files stay on your device"), so the paragraph repeated on every page added nothing.
-
Moved page-specific content into data. A separate
page_extras.pyholds an HTML block and FAQs for each slug. The template appends them if they exist. The 20 KB page now talks about signatures and tight crops. The 200 KB page covers product photos and visa uploads. The HEIC page explains why the JPG comes out bigger. - Cut generic FAQs. The ones that remain answer a question someone would ask about that specific page.
After the rewrite, no sentence appears on 3 or more pages, and the most similar pair dropped from 0.55 to 0.09.
Making it stick
The real risk is the future. New blog posts get written over time, by me or with AI help, and both reach for the same stock phrases. So the checker now runs in CI on every pull request:
- name: Check for repeated text
run: python3 tools/check_repeats.py --posts
In --posts mode it fails the build if a post shares any 8+ word sentence with another post or page, or if two posts share more than 25% of their shingles. The output names the sentence and the other file, so fixing it takes a minute.
Did it fix indexing?
Too early to say. I've only just asked Search Console to re-validate. Duplicate text isn't the only reason a new site gets ignored: it also needs time and links. What I can say is that I was handing Google a clear reason to skip those pages, and that reason is gone. I'll report back if the numbers move.
If you generate pages from templates, run something like this over your build output once. It costs nothing and the results might surprise you. The site, if you're curious, is 14FreeTools. Compare, for example, compress to 20 KB with compress to 200 KB.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.