Our rank checker prints the URL, which is how we found out we rank with the wrong page

Search Console will not tell you where you rank. It tells you your average position, averaged across every query, country and device where you appeared at all. A page can show an average position of 7.4 and be nowhere in the top 30 of the query you actually care about, because the 7.4 is made of long tail phrases you have never thought about.

So CogniPrep has a 44 line script that asks Google directly, for 24 queries, one per assessment provider. It is the least clever piece of tooling in the repo and it has produced more actionable information than anything else in the SEO stack.

Why it needs a real browser

#!/usr/bin/env bash
# Where does cogniprep.app rank on Google for "<provider> assessment practice"?
# Needs the ego-browser CLI (a real Chromium session; plain HTTP gets a consent wall).
# Google rate-limits after roughly 50 queries; if a row says BLOCKED, wait and rerun.
set -euo pipefail

curl against a Google search URL gets a consent interstitial, not results. Every scraping library that promises otherwise is in a losing race. Driving an actual browser session sidesteps the whole category of problem, and the query string carries the three parameters that make the measurement mean something:

https://www.google.com/search?q=<query>&hl=en&gl=us&pws=0&start=<0|10|20>

hl=en fixes the interface language. gl=us fixes the market being measured, which matters when your audience is not where you are. pws=0 disables personalisation, without which you are measuring your own browsing history. start walks three pages, so "not in the top 30" is a real answer rather than "not on page one".

Two details that decide whether the numbers are real

Scroll before you read. Results render lazily, so extracting immediately gives you a partial list and silently inflates the position of whatever is at the bottom:

for (let i = 0; i < 4; i++) {
  await page.evaluate(() => window.scrollBy(0, 3000));
  await page.waitForTimeout(300);
}

Identify organic results structurally, not by class name. Google's class names are generated and change constantly. What does not change is that an organic result is a link wrapping a heading:

const links = await page.evaluate(() => {
  const out = []; const seen = new Set();
  for (const a of document.querySelectorAll("a[href^='http']")) {
    if (!a.querySelector("h3")) continue;
    const href = a.href.split("#")[0];
    if (seen.has(href) || href.includes("google.com")) continue;
    seen.add(href); out.push(href);
  }
  return out;
});

Dedupe by href, because sitelinks and expanded results repeat the same URL and would each consume a position. Drop google.com to skip the "People also ask" scaffolding.

Being blocked is data, not an error

if ((await page.url()).includes("/sorry/")) { found = "BLOCKED"; break; }

Two things here, and the second is the one I would argue for in a review.

Randomised backoff between queries, 2.5 to 4.5 seconds, because a fixed interval is both more detectable and more annoying to the service you are asking for a favour.

And on a block, the loop stops entirely. It does not retry, and it does not carry on hammering through the remaining 20 queries so that they can all fail too. The row prints BLOCKED, which is different from not in top 30, and that distinction ends up in the recorded table.

The first run of this hit exactly that case: one provider came back blocked, and the baseline table in the repo says blocked in that cell. Writing >30 there would have invented a data point, and three weeks later nobody would remember that the cell was a guess. A measurement tool that cannot express "I do not know" is a tool that produces confident fiction.

The column that turned a vanity metric into a finding

The script prints the position and the URL:

console.log(slug.padEnd(18), found || "not in top 30");
// found = "#" + (start + i + 1) + " " + links[i]

That is a two word change from a boolean, and here is what it bought. Today's run, three days after a round of on page changes:

arctic-shores      #4  https://cogniprep.app/games/arctic-shores
saville            #7  https://cogniprep.app/games/saville
pymetrics          #11 https://cogniprep.app/games/pymetrics
cappfinity         #13 https://cogniprep.app/games/cappfinity
testgroup          #15 https://cogniprep.app/employers/young-group
thomas             #16 https://cogniprep.app/games/thomas
test-partnership   #27 https://cogniprep.app/blogs/test-partnership-test-tips
cubiks             #28 https://cogniprep.app/games/cubiks
criteria           #29 https://cogniprep.app/games/criteria

Look at rows five and seven. For those two providers the page that ranks is not the page we spent the afternoon optimising. One is a blog post about that provider's tests. The other is an employer page that happens to mention them, on a query that is the provider's own name.

One of the two has a defensible explanation: that provider's hub is written in Dutch, for the Dutch phrasing of the query, so an English language search reasonably surfaces a different page of ours instead. The other does not. Both of those conclusions require the URL. A checker that answered "are we in the top 30, yes or no" would have reported a plain success for both. Instead the honest reading is that the hub page is being outranked by our own content, which is a completely different problem with completely different fixes: internal linking, canonical intent, and whether the hub deserves the query at all.

Write the baseline down before you change anything

The output lives in a dated markdown file alongside the changes it measures, with the live position next to the Search Console average for the same page over the same period, and a last line that says:

Re-run: scripts/seo/serp-rank-check.sh. Compare against this table in 4 to 6 weeks.

Two reasons that file exists, and neither is documentation for its own sake.

You cannot take a "before" measurement afterwards. The changes shipped the same day the table was written, and the only version of the world where the comparison is possible is the one where somebody spent five minutes recording the baseline first.

And three days is far too soon to attribute anything. Today's run shows one provider newly appearing at 29, one drifting from 26 to 28, one dropping out of the top 30 entirely. None of that is evidence about anything. It is noise, and knowing it is noise is only possible because the baseline exists and the interval was written down before the temptation to read the tea leaves arrived.

See it

Everything above is reproducible without an account, and against a competitor if you prefer.

• Search Arctic Shores assessment practice on Google with &gl=us&pws=0 appended. cogniprep.app/games/arctic-shores is currently in the top handful. Then try Hogan assessment practice, where the hub is not in the top 30 at all, and compare the two pages to form your own theory about why.
• The two pages that rank for a query their hub was built for: cogniprep.app/blogs/test-partnership-test-tips and cogniprep.app/employers/young-group. Both outrank their hubs today.
• For your own site, the whole tool is: a browser session, three start values, a scroll loop, an h3 heuristic, and printing the URL you found rather than the fact that you found one.