One Month After the Hackathon: Why My 4-Service App Stopped Working (and What @mansio Caught in the Verifiers)

What breaks a month after a hackathon, and what a fellow DEV writer caught in the verifiers.

TL;DR: A month after shipping README Clew 2, I opened it up and every scan failed. Nothing was wrong with the deterministic verifiers or with Jev on Glasser. Instead, three small operational traps lined up at once: an expired 30-day GitHub token on a tool that only reads public repos, a rotated Claude Haiku key scoped only to Development in Vercel, and a substring bug that fellow DEV writer Mikhail (@mansio) found and fixed in PR #2.

In October I built README Clew 2 for the AdaL "What Would Jev Do?" hackathon. It checks whether a public JavaScript or TypeScript repo's README matches its code, sorting every checkable line into four buckets: Contradicted, Missing, Unverifiable, and Verified.

The core rule of the build is that models never decide what is true:

When you wire that many moving parts together (AdaL, Jev, Glasser, Haiku, GitHub, and Vercel) and come back a month later to a red SCAN FAILED banner, it is hard to tell at first glance which piece broke.

Here is what was actually wrong, and the two fixes that came out of it.

1. When "javascript" hides "ava": @mansio's PR #2

Before I even got to the deployment keys, Mikhail (@mansio) had opened and landed PR #2, the project's first external code contribution.

Mikhail was testing README Clew 2 against real repositories and noticed something strange on sindresorhus/yocto-queue. The repo uses ava as a test runner, and the README never mentions ava. The coverage verifier should have flagged ava in the Missing bucket. Instead, it stayed silent.

Why? Because the README contained a link to Mozilla's documentation:

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols

And inside the word javascript, in lowercase, sits the three-letter substring ava.

Our coverage check was using a plain substring match:

// Before: any substring match counted as "mentioned in the README"
if (readme.includes(name.toLowerCase())) continue;

Even worse, the same substring check lived in the dependency verifier's prose guard (inCodeContext), which checks whether a candidate package name appears on an install line. A short name like can inside npm i canvas-parser or lodash inside npm i lodash-es could bypass the prose guard and reach Contradicted, the exact false positive the guard was built to prevent.

The fix

Mikhail replaced both substring checks with a single whole-token rule, mentionsWord(), bounded by npm's own package name characters (A-Za-z0-9_-) plus @ and /:

export function mentionsWord(text, name) {
  if (!name) return false;
  const esc = name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
  return new RegExp(`(?<![A-Za-z0-9_@/-])${esc}(?![A-Za-z0-9_-])`, 'i').test(text);
}

With that in place:

• We use ava. and Tests run with \ava. still count as real mentions.
• mozilla.org/.../javascript/... no longer hides ava.
• npm i lodash-es no longer treats lodash as an installed package claim.

Mikhail shipped the fix with a 27-case test matrix and verifier controls in tests/mentions-word.test.js, matching the repo's file-size cap and offline test rules on the first try. If you have not read his engineering field notes on DEV (like The Dataset Was Lying: 4 of 6 "False" Facts Were True), go read him.

2. The 30-day time bomb: sending an expired token to a public API

Once Mikhail's PR was merged, I still had a live site that refused to scan.

Every click on SCAN returned HTTP 502:

{"error":{"code":"github_forbidden","message":"GitHub returned HTTP 401"}}

The ironic part: README Clew 2 only scans public repositories. It makes 4 metadata calls to api.github.com per scan and fetches the files from raw.githubusercontent.com. You do not even need a GITHUB_TOKEN for low-volume public reads (GitHub allows 60 unauthenticated REST calls per hour per IP, or 15 full scans per hour).

During the October hackathon, I had added a 30-day GitHub Personal Access Token so repeated test scans wouldn't hit the unauthenticated rate limit. Thirty days passed, the token expired, and lib/github.js kept sending Authorization: Bearer <expired_token> on every request.

GitHub allows requests with no Authorization header, but immediately rejects requests carrying an expired Authorization header with 401 Bad credentials. Because step 1 threw a GithubError, the pipeline stopped before Jev or Haiku ever ran.

The 4-line fix

First, I removed the expired GITHUB_TOKEN from Vercel. Second, we hardened lib/github.js so that if an expired token is ever present in .env or server environment variables and GitHub answers 401, the client drops the authorization header from the shared headers object and retries unauthenticated:

if (res.status === 401 && headers?.authorization) {
  delete headers.authorization; // expired or bad token: public repos work without it
  return request(url, { fetchImpl, headers, as });
}

Because headers is shared across the 4 metadata calls in fetchRepo(), dropping headers.authorization on the first 401 means the remaining 3 calls go straight out without the bad token.

3. Silent fail-open + Vercel environment scopes

The last piece was Claude Haiku. By design, Haiku is fail-open in README Clew 2: if the summary call fails or times out, the report still renders in full and the clew mascot falls back to a fixed one-line summary per mood.

My October Anthropic key had also been rotated. When I added the new ANTHROPIC_API_KEY in Vercel, two classic traps bit me at the same time:

1. The new key was scoped to Development in Vercel, while Production and Preview still held the 34-day-old key.
2. Vercel environment variable changes only take effect on the next deployment.

Running vercel env ls made the split visible right away:

Overriding ANTHROPIC_API_KEY across all three environments (Production, Preview, Development) and running vercel --prod brought the Haiku summary and bucket notes back on the first scan. Meanwhile, Glasser and Jev (jev-1.13.0) never missed a beat: all 4 batched calls completed in 1.37 seconds for about half a cent ($0.0055), with $10.80 of the original hackathon balance still untouched.

Related reading on DEV

Two posts from the community hit the exact operational edges from today's debugging session:

• ["Invalid API key" or undefined env vars on Vercel: how to fix it] by @unbreakkit walks through how environment variable scopes (Production vs Preview vs Development) and missing redeploys cause 401 Invalid API key errors on Vercel even after you paste a fresh key.
• [Your AI Agent Runs for Hours. Its GitHub Token Might Not.] by @__be2942592 looks at what happens when code assumes a GitHub token in the environment is valid forever instead of handling 401 expiration explicitly.

Try it

• Live page: readme-clew-2.vercel.app (click USE DEMO REPO, then SCAN)
• Source and 478 offline tests: github.com/earlgreyhot1701D/readme-clew-2

Thank you again to Mikhail (@mansio) for finding the substring bug on a real repo and shipping a clean, well-tested fix.

Quick context if you are new here. I work in the California courts, running court operations for the county. I started building with AI in July 2025 and I have been learning in public ever since. I do not write the code. I direct, the agents generate, I validate and decide. I build the Clew Suite, a set of civic tech tools for making complex systems easier to inspect. That is the lens I am writing from.

AI Assisted. Human Approved. Powered by NLP.