We pin the exact sentence each fact rests on, and a script tells us when it changes

We run BaseAcross, a site for founders outside the US who are forming US companies. Most of what it says comes from someone else's page: IRS form instructions, a state's fee schedule, a payment provider's list of supported countries. Those pages change without notice. A form gets a new revision, a fee schedule PDF moves to a new folder, a provider drops a country from a list. Our page keeps saying the old thing, and so does any AI answer that was built from our page.

This post is about the plumbing we use to notice. It lives in a Next.js and TypeScript repo and has three parts: a registry of every outside page we rely on, a script that re-reads those pages and compares the words we quote, and a Vitest test that fails CI when someone cites a page without registering it.

Every source is registered with the words we rely on

The registry is a typed array in src/lib/site/sources.ts. Each entry is a URL plus the phrases on that page our copy depends on (comments trimmed):

export type Passage = {
  /** Words that must still be on the page. */
  find: string;
  /** What our copy says because of them. */
  why: string;
  /** For a list: watch everything from the phrase up to these words. */
  until?: string;
  /** Watch the phrase alone. Sales pages change their copy all the time. */
  alone?: boolean;
};

export type Source = {
  id: string;
  label: string;
  url: string;
  kind: "official" | "provider" | "competitor" | "review";
  /** Where the copy that depends on it lives. */
  usedIn: string[];
  passages: Passage[];
  /** Why the script can't read it. A person opens it instead. */
  manual?: string;
};

A real entry, shortened:

{
  id: "irs-ss4",
  label: "IRS instructions for Form SS-4",
  url: "https://www.irs.gov/instructions/iss4",
  kind: "official",
  usedIn: [`${DATA}/guides/ein-and-tax.ts`, `${DATA}/glossary.ts` /* , … */],
  passages: [
    { find: "Only international applicants can receive an EIN by telephone",
      why: "EIN guide: the phone route is for applicants abroad." },
    { find: 'Enter "foreign" or N/A on line 7b',
      why: "Line 7b for an owner with no SSN or ITIN." },
  ],
},

why and usedIn are for the person reading the report. When a phrase disappears, they say which of our sentences is now at risk and which files hold it.

The file's header gives one rule for writing find: pick the words that carry the fact, so the phrase disappears when the fact changes. On a competitor's pricing page that means the price string, because a plan name can survive a price change untouched.

We compare one sentence, never the whole page

Diffing whole pages is useless here. A menu change, a cookie banner or a date in a footer would flag every source on every run. So source-watch.ts finds the phrase and keeps the sentence around it. A change to that sentence is a finding; a change anywhere else on the page is ignored.

The sentence is what gets stored, in sources.lock.json:

"irs-ss4": {
  "url": "https://www.irs.gov/instructions/iss4",
  "checked": "2026-10-30",
  "passages": {
    "Enter \"foreign\" or N/A on line 7b": "Enter \"foreign\" or N/A on line 7b if the responsible party doesn't have and is ineligible to obtain an SSN or ITIN."
  }
}

We store the text itself rather than a hash of it, so the report can print what the sentence said before and what it says now, side by side. A hash would tell you something moved and leave you to go and find it.

Finding "the sentence" on a government page took more care than it sounds. The splitter has to read through "U.S. disregarded entity" and "11:00 p.m. (Eastern time)" without stopping at the full stops, and those two cases are unit tests now. Before matching, flatten() turns curly quotes into straight ones, every kind of dash into a hyphen, odd spaces into plain ones, and joins a dollar sign to a figure when a page splits them across two elements. That way a phrase typed on a keyboard matches what the browser shows.

What the check reports, and who acts on it

npm run sources:check runs scripts/sources-check.ts. It fetches each registered page, four at a time, with browser-like headers because several government sites turn away anything else. A response with under 200 characters of text is treated as unreadable, since a page that renders in the browser comes back from fetch as an empty shell.

The core of the comparison (bookkeeping trimmed):

for (const p of source.passages) {
  const now = excerpt(page.text, p);
  const before = was[p.find] ?? null;
  if (now === null) {
    found.push({ ...base, kind: "missing", find: p.find, why: p.why, was: before });
  } else if (before === null) {
    found.push({ ...base, kind: "new", find: p.find, now });
  } else if (before !== now) {
    found.push({ ...base, kind: "changed", find: p.find, why: p.why, was: before, now });
  }
}

There are six kinds of finding: missing, changed, new, moved (the URL now redirects somewhere else), unreachable and manual. A missing, changed, moved or unreachable finding sets the exit code to 1, unless the run is recording with --update. A --json flag prints the same report for an agent to read.

The lock only moves when a person says so, with --update, and the script's header asks you to run that only after the copy that depends on a change is fixed or the change is judged harmless. A missing phrase never changes what the lock holds: the last sentence a person confirmed stays there. If the words we quote are gone, the registry has to change first, because we no longer know where the source says what we say it says.

CI doesn't run this script. It makes network requests to sites we don't control, and we'd rather a merge didn't depend on whether a state server is up. The header of source-watch.ts calls it "the weekly source check"; the scheduling sits outside this repo, so we'll leave it there.

A test refuses citations nobody registered

A registry like this only helps if it's complete, and the easy failure is someone adding a fact with a "checked against" note and never registering the page. So sources.test.ts, which does run in CI, scans the comments in our data files for anything that looks like a domain and fails if no registered URL covers it:

/* Comments only, so the state offices' homepages and other links we show
   readers aren't mistaken for citations. */
function comments(source: string) {
  return source.match(/\/\*[\s\S]*?\*\/|(?<![:"'\w])\/\/[^\n]*/g) ?? [];
}

it("lists every source a data file's notes cite", () => {
  const missing = citations().filter((c) => !SOURCES.some((s) => covers(s.url, c.host, c.path)));
  expect(missing.map((c) => `${c.host}${c.path} (${c.file})`)).toEqual([]);
});

The lookbehind on // is there so the middle of https:// inside a string isn't read as a comment. covers() matches on host plus a path prefix, so a note that cites irs.gov/instructions/iss4 is satisfied by the registered page.

The same file holds a few neighbouring checks. Every page a guide, glossary term or use-case page lists as a source must be in the registry. Every competitor price source must be in it too. And the lock can't keep a phrase the registry no longer watches, so a stale recording can't hide.

The date on the page is still a constant

A small byline component prints a "Checked [date]" line under a page's heading, with a link to how we check. Its default date is a constant in seo.ts, next to a month-only version of the same date:

export const CONTENT_CHECKED = "September 2026";
export const CONTENT_CHECKED_ISO = "2026-09-17";

The ISO form feeds the byline and the pages' structured data. The month form appears in our llms.txt, under a heading addressed to AI systems, next to a note that one state's annual LLC tax changed on 1 January 2026 and that many sources still quote the old figure.

This is the weakest link in the setup. The lock records a checked date per source, and those dates are more recent than the site-wide constant. Nothing connects the two, so the date a reader sees is set by hand and can lag what we've verified. A page can pass its own date to the byline component, but the default is a person remembering to bump a string.

What it has had to absorb, and what it costs

The registry's own comments record a few of the changes it has had to absorb. Delaware moved its fee schedule PDF into a new folder, and the old address now returns a 404. Wyoming's statutes moved to a new host; the old address still redirects, so the check reports that source as moved until we change the URL. A note there says to change it together with the citation in the guide's comments, because the test above matches by host and would otherwise fail. A payment provider's help centre answered scripts with a 401 for a while, so it became a manual entry; when it later answered normally three times running, it went back to being watched.

It also has costs.

• Manual sources only produce a reminder. PDFs, sites that refuse scripts and client-rendered pages get a manual note saying why, and the script prints them for a person to open. Our own internal docs say that wasn't good enough for the state-by-state guides, which now have a separate monitor.
• Phrases are brittle by design. If a source rewords a sentence and keeps the same fact, we get a missing finding and someone has to re-pick the phrase. We accept that false alarm because the alternative failure is silence.
• It can't tell whether our copy is right. It knows that the words we relied on are still on the page, word for word. Deciding what a change means for a founder's filing is still a person reading the source.

Drafted with AI assistance and checked against our code and the sources it cites.

Cover photo: Ximin Lin on Unsplash