36 pages were live and in the sitemap. The directory page linked to none of them
Thirty-six market pages went live on a Tuesday. All of them returned 200, every canonical pointed at itself, the sitemap went from 568 to 574 URLs, and the structured data validated. Then I opened the directory page that
Thirty-six market pages went live on a Tuesday. All of them returned 200, every canonical pointed at itself, the sitemap went from 568 to 574 URLs, and the structured data validated. Then I opened the directory page that is supposed to be their entry point and counted the links in it: 112 cards, and not one of them was mine.
That is the failure mode this post is about. Not a broken page, not a 404, not a missing tag — a page that is completely fine and effectively invisible. Nothing throws, nothing fails validation, and every monitoring check I had was green.
What the count said, before and after
Same URL, same function, two snapshots of the HTML:
| measurement | before | after |
|---|---|---|
| bytes of HTML | 480,224 | 604,206 |
anchors matching /markets/<cat>/<slug>/
|
239 | 311 |
| unique target URLs | 119 | 155 |
ItemList entries in JSON-LD |
112 | 148 |
| pages from this batch present in any of them | 0 | 36 |
The +72 anchors for +36 pages is not a coincidence: every card links twice, once through the image and once through the title. If your write is right, the deltas are exact integers, and that is the whole value of measuring — "+36 links" is a testable claim, "looks right in a screenshot" is not.
import json, re, urllib.request
UA = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/141.0 Safari/537.36"}
CARD = re.compile(r'href="(/markets/[a-z]+/[a-z0-9-]+/)"')
def crawl_stats(url):
req = urllib.request.Request(url, headers=UA)
html = urllib.request.urlopen(req, timeout=60).read().decode("utf-8", "replace")
anchors = CARD.findall(html)
item_list = 0
for block in re.findall(r'<script[type="application/ld\+json"][^>]*>(.*?)</script>', html, re.S):
nodes = json.loads(block)
for node in (nodes if isinstance(nodes, list) else [nodes]):
if node.get("@type") == "ItemList":
item_list = len(node.get("itemListElement") or [])
return {
"bytes": len(html.encode("utf-8")),
"anchors": len(anchors),
"unique": len(set(anchors)),
"item_list": item_list,
}
print(crawl_stats("https://eventmarketguide.com/markets/"))
Run it twice — once against the deployed page, once against the copy you saved before uploading — and you have a regression test instead of an impression.
Why the HTML edit disappeared
The obvious move is to append 36 <article> cards to the static grid. I did that first. They were in the served HTML, they were gone in the browser.
The card list is not HTML. It is a client component boundary plus a data array:
["$","$L30",null,{"initialMarkets":[{"category":"politics","slug":"will-the-national-turnout-rate-exceed-65","question":"...","yes":18,"volume":12000,"closes":"Sep 30, 2026"}]}]
$L30 is a reference to a lazily loaded client component. The server ships the data; the browser builds the cards from it. When the component mounts it owns that subtree, and any node it did not produce is discarded. This is the same rule that eats injected markup in any hydrating app — the client render is authoritative.
So one card has to exist in three places:
-
The flight payload (
initialMarkets) — what actually renders. If the array is short, the grid is short, permanently. - The static HTML grid — what a crawler that does not execute JS sees, and what the page looks like in the 200ms before hydration.
-
ItemListJSON-LD — two copies in these pages, a plain script block and an escaped copy inside the payload.
Three writes, one source of truth. I generate all three from the same pages list, and assert that all three end up with the same length.
The part that bites: length prefixes
Flight text chunks are length-prefixed:
3:TL5b,{"title":"Bitcoin forecast"}
L5b is the byte count of what follows, in hex. It is not decoration. Change text inside a chunk and the declared length is now wrong, and both directions are fatal: too large and the client consumes the next row's header as content; too small and it stops mid-JSON, and the leftover fragment gets parsed as a new row. Neither produces a readable error. You get a blank page and React #412 in the console.
An earlier batch gave me the cleanest demonstration I could ask for: 33 of 36 pages rendered blank, and the 3 that worked were exactly the 3 whose byte length did not change. 36 out of 36 matched the diff — the prefix was the whole bug.
import json, re
HEADER = re.compile(r"(?<![0-9a-fA-F])([0-9a-f]{1,6}):([TB])([0-9a-f]{1,8}),")
NEXT_ROW = re.compile(r"\n([0-9a-f]{1,6}):")
DEC = json.JSONDecoder()
def chunk_end(stream, start):
"""A T chunk ends at a JSON value boundary or at the next row header."""
if stream[start:start + 1] in ("{", "["):
try:
_, end = DEC.raw_decode(stream, start)
return end
except ValueError:
pass
m = NEXT_ROW.search(stream, start)
return m.start() if m else len(stream)
def resync_prefixes(stream):
fixes = []
for m in HEADER.finditer(stream):
start = m.end()
end = chunk_end(stream, start)
real = len(stream[start:end].encode("utf-8"))
if real != int(m.group(3), 16):
fixes.append((m.start(), m.end(), "%s:%s%x," % (m.group(1), m.group(2), real)))
for a, b, token in reversed(fixes): # right to left: offsets stay valid
stream = stream[:a] + token + stream[b:]
return stream, len(fixes)
Two details that cost me real time:
-
The newline is not the boundary. T chunks can contain newlines, and in these pages the chunk is often glued directly to the next row with no
\nat all. Splitting on\nsilently truncates the chunk you just measured, which produces a correct-looking fix and a broken page. -
The prefix counts UTF-8 bytes.
len(str)is wrong the moment the text is not ASCII. One Chinese title is enough to make a page blank. -
If the chunk is inside
self.__next_f.push([1,"…"]), fix the decoded stream, not the HTML. The JS string literal has its own escaping (\",\\,\uXXXX); a prefix computed over the escaped form is a prefix for the wrong string. Decode, fix, re-encode.
Verify in a browser, not with grep
The raw HTML check is necessary and insufficient — it cannot tell you whether the client threw away your work. One page load, four numbers, all asserted:
const { chromium } = require("playwright-core");
const browser = await chromium.launch();
const page = await browser.newPage({ viewport: { width: 1366, height: 900 } });
await page.goto("https://eventmarketguide.com/markets/", { waitUntil: "networkidle" });
const stats = await page.evaluate(() => {
const hrefs = [...document.querySelectorAll('a[href^="/markets/"]')]
.map(a => a.getAttribute("href"));
let itemList = 0;
for (const s of document.querySelectorAll('script[type="application/ld+json"]')) {
try {
for (const n of [].concat(JSON.parse(s.textContent)))
if (n["@type"] === "ItemList") itemList = (n.itemListElement || []).length;
} catch {}
}
return {
anchors: hrefs.length,
unique: new Set(hrefs).size,
cards: document.querySelectorAll("article").length,
itemList,
};
});
if (stats.unique !== 155) throw new Error(`unique targets ${stats.unique}, expected 155`);
if (stats.itemList !== 148) throw new Error(`itemList ${stats.itemList}, expected 148`);
What it returned after the fix: 314 anchors, 156 unique targets, 152 <article> nodes, ItemList 148. Slightly higher than the raw HTML because two of those links only exist after the header hydrates — which is exactly why the rendered number and the static number are two separate assertions, not one.
A second bug, same root cause: string surgery on a URL
The page generator clones a template and replaces text. The category swap ran before the related links were built:
html = template.replace("sports", category) # runs first
# ...
card = card.replace("{{RELATED_URL}}", old_url) # old_url still starts with /markets/sports/
Thirty non-sports pages ended up with related cards pointing at URLs whose category segment did not match the slug. 144 card links, every one of them a 404. The pages themselves were fine; the links inside them were not, and no build step noticed because the build only checked that generation succeeded.
Pick a slug, swap the segment, ask for it:
| URL | status |
|---|---|
/markets/sports/will-a-marathon-world-record-be-set-this-year/ |
200 |
/markets/economy/will-a-marathon-world-record-be-set-this-year/ |
404 |
I tested three slugs; three of three mutated URLs returned 404. The paths are structured — /markets/<category>/<slug>/ — and a structured path is data, so build it from data:
def market_url(category, slug):
return "/markets/%s/%s/" % (category, slug)
If you genuinely need document-wide substitution, use a token that cannot appear in real content ({{CATEGORY}}, not sports) and then assert the result: every internal href resolves. The same structural mistake shows up in every generated view of the same data — a topic hub that lists every crypto market in one view is just a filter over the same array, so if the array is short the hub is short, and if a slug is wrong there it is wrong in both places.
What I gate on now
The card fields are also prose, and prose is where "I'll check it visually" dies. A resolution string that got truncated by a template, or a closes date rendered from a different field than the array carries, is invisible on the card and wrong on the market detail page that renders the same fields. So the gate is:
-
Delta, not presence. N new pages → unique internal targets +N, anchors +2N,
ItemList+N. Exact integers. -
All three writes agree. Payload length == grid length ==
ItemListlength. - Every T chunk is honest. Declared byte length == real UTF-8 byte length, checked on every generated page, hard fail at build time.
-
Rendered counts. Chrome, after
networkidle, asserted — a raw grep cannot see hydration deleting your markup. - Every internal link resolves. HEAD a sample of pages; on the 8-page sample I ran, 32 unique related URLs, 32 of 32 returned 200.
The page was never broken. It just never mentioned 36 pages that existed, and "it looks fine" is not a measurement of that. A link count is.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.