Our JavaScript chunks were indexed because robots.txt blocked them, and the fix was to let crawlers in

Our app/robots.ts is 131 lines of TypeScript that generates 968 lines and 28,047 bytes of robots.txt: 906 Disallow lines and 56 Allow lines, written twice because there are two user-agent groups and a crawler only obeys one of them. 856 of those Disallows are the 428 game routes, doubled.

Until yesterday, two of the lines were these:

  // Next.js internals
  '/_next/',

Reasonable looking. Build output is not content, nobody wants a webpack chunk in a search result, so tell crawlers not to fetch it. Both halves of that sentence are wrong in an interesting way.

Blocked is not the same as not indexed

robots.txt governs fetching. The index is governed by noindex, which is a thing you put in a response: a meta tag in a document, or an X-Robots-Tag header on anything. The two instructions travel by different routes, and a URL you have disallowed can never deliver the second one, because nobody fetches it.

So a blocked URL that gets linked often enough can be indexed on the strength of the links alone, and Search Console has a status for exactly that: "Indexed, though blocked by robots.txt". That is what our chunk URLs were doing, and the comment that replaced the rule says so:

  // `/_next/` is deliberately NOT blocked. Googlebot needs the JS and CSS
  // chunks to render pages; blocking them also left the chunk URLs "Indexed,
  // though blocked by robots.txt". The chunks carry an X-Robots-Tag: noindex
  // header instead (next.config.mjs), which crawlers can only see if allowed in.

The second cost was worse than the cosmetic one. Our pages render client components, and a crawler that cannot fetch the JavaScript is rendering a different page from the one a visitor sees. Blocking build assets is a decision about rendering dressed up as a decision about indexing.

The replacement is four lines of config

// Keep JS/CSS build chunks out of the search index. They stay crawlable
// (see app/robots.ts) so Googlebot can render pages, and noindex does not
// stop a resource being used for rendering. Scoped to /_next/static so
// optimised images under /_next/image remain eligible for image search.
{
  source: '/_next/static/:path*',
  headers: [
    {
      key: 'X-Robots-Tag',
      value: 'noindex',
    },
  ],
},

The clause about rendering is the one that makes the whole arrangement work. A noindex on a resource says "do not list this URL in results". It does not say "do not use this file". The crawler fetches the chunk, renders the page with it, indexes the page, and leaves the chunk out of the index. That is the outcome we wanted from the Disallow and never got.

The scope is a path prefix rather than a file type, and that has two consequences worth knowing before you copy it.

The one we wanted: /_next/image is outside the scope, so optimised images keep their shot at image search. The one we did not plan: /_next/static/media is inside it, so every font file is now served with a noindex too. That costs nothing, because nobody is searching for a woff2, but it is a reminder that :path* under a directory catches whatever else the framework decides to put there.

What the file looks like afterwards

Nothing. There is no Allow: /_next/ line, because robots.txt has no need for one: line 2 is already Allow: /, and the change was a deletion. The only way to see this change in the file is to know it used to be there.

The place it is visible is the response:

$ curl -sI https://cogniprep.app/_next/static/chunks/<name>.js
HTTP/2 200
cache-control: public,max-age=31536000,immutable
content-type: application/javascript; charset=utf-8
x-robots-tag: noindex

An immutable year-long cache and a noindex on the same response is the honest description of a build asset: keep it forever, never list it.

See it

Open cogniprep.app and view source. There are 239 references to /_next/static/chunks in the 533 KB of HTML, resolving to 12 distinct JavaScript files. Copy any one of them and request it with curl -I. You get the x-robots-tag: noindex above.

Then try the three neighbours, which is where the scope becomes real:

• A font under /_next/static/media/...woff2 has the header, because it is under /_next/static.
• An image under /_next/image?url=... does not, by design.
• A page such as /pricing does not either, and should not: pages are what we want indexed.

Finally, fetch robots.txt and search it for _next. Zero matches, on a site whose robots.txt is 968 lines long. The rule that was keeping our build output indexed is the one that is no longer there.