Our IndexNow key is committed to the repository, and the auth gate was the only thing that nearly broke it

Try this:

curl -s https://pub-trivia.app/2382c7da66fd467eb45e9c97aadf605a.txt
# 2382c7da66fd467eb45e9c97aadf605a

That file is how we prove to Bing that we control the host. The filename is the key, the body is the key, and anybody can read it. That is not a leak, it is the entire mechanism, and getting comfortable with that is the first thing IndexNow asks of you.

We run a quiz app for pubs with a 72 page marketing site. The sitemap is a standing statement of what exists. IndexNow is an event that says what just changed, and the engines that take part act on it in minutes rather than whenever they next feel like crawling. Here is the whole implementation, which is one constant, one route, one submitter and a test file that is mostly about one failure mode.

The key is not a secret, so an env var would protect nothing

/** 8 to 128 hex characters, per the spec. Ours is a UUID with the dashes removed. */
export const INDEXNOW_KEY = '2382c7da66fd467eb45e9c97aadf605a'

The instinct when a thing is called a key is to put it in an env var. Resist it here. The protocol verifies ownership by having you publish the key at a URL on your host and then quote that URL back inside the submission body. Its readability is load bearing.

What an env var would buy is nothing. What it would cost is separating the key from the one thing it has to agree with, which is the route that serves it, and moving a value that must be identical in two places into a system where only one of them can see it.

The directory name is the key, so it is written twice and tested once

app/2382c7da66fd467eb45e9c97aadf605a.txt/route.ts

This is the one place the key cannot be derived from the constant, because the directory name is what decides the URL. So the route reads the body from the module rather than typing the key a second time:

import { INDEXNOW_KEY } from '@/lib/indexnow'

export const dynamic = 'force-static'

export function GET() {
    return new Response(INDEXNOW_KEY, {
        headers: {
            'Content-Type': 'text/plain; charset=utf-8',
            'Cache-Control': 'public, max-age=86400',
        },
    })
}

And the test closes the loop by checking the filesystem, which is not a kind of assertion I write often:

it('has a route behind it, named to match', () => {
    const routeDir = path.join(process.cwd(), 'app', `${INDEXNOW_KEY}.txt`)
    expect(existsSync(path.join(routeDir, 'route.ts')), `${routeDir} is missing`).toBe(true)
})

It earns its place because of what the failure looks like. If the constant and the directory disagree, every submission comes back as a bare 403 with no body, and nothing in that response tells you which half is wrong. A test that reads a directory name is ugly. Spending an afternoon on a 403 that means "your file is at a slightly different URL than you said" is uglier.

force-static because the body is a constant and nothing about it varies per request. There is no reason to wake a serverless function for 32 bytes of hex.

public/ was the wrong place, because the auth gate only exempts images

This was the real bug, and it is specific to how the app is routed.

Our auth boundary is a proxy (which is what Next 16 calls middleware) with a matcher that excludes static assets by extension:

'/((?!_next/static|_next/image|favicon\\.ico|manifest\\.webmanifest|robots\\.txt|sitemap\\.xml|.*\\.(?:svg|png|jpg|jpeg|gif|webp)$|webhook|api|monitoring).*)'

Read the extension group: svg, png, jpg, jpeg, gif, webp. No txt. So a .txt dropped into public/ is matched by the proxy, reaches the auth gate, and is answered with a 307 to /login. IndexNow fetches the key file, gets a redirect to a sign-in page, and rejects the submission. Nothing is in the logs, because nothing failed.

Hence a route segment under app/, plus an entry in the allowlist that the gate consults:

export const PUBLIC_ROUTES = [
    // ...
    // The IndexNow key file, app/<key>.txt/route.ts. Unlike robots.txt and
    // sitemap.xml it is NOT excluded from the proxy matcher: those two are
    // fetched by every crawler on every pass, while this one is read a handful
    // of times a year, so the Redis and Supabase round trip costs nothing worth
    // hard-coding the key into a regex to avoid. Being listed here is what stops
    // the auth gate answering IndexNow's verification fetch with a 307 to /login.
    INDEXNOW_KEY_PATH,
] as const

That comment is the decision, not a description of it. robots.txt and sitemap.xml are excluded from the matcher entirely, because every crawler asks for them on every pass and a Redis plus Supabase round trip on each of those is real. The key file is read a handful of times a year, so it takes the cheap route: let it reach the gate, and let the allowlist wave it through. Adding a 32 character hex literal to a middleware regex to save a few requests a year is a trade I would lose.

Verify your own key file first, with redirects left unfollowed

Because the 403 is uninformative, the submitter checks the thing most likely to be wrong before it asks:

export async function verifyKeyFile(): Promise<KeyFileCheck> {
    const url = absoluteUrl(INDEXNOW_KEY_PATH)
    const response = await fetch(url, { redirect: 'manual' })

    if (response.status >= 300 && response.status < 400) {
        const target = response.headers.get('location') ?? 'elsewhere'
        return {
            ok: false,
            url,
            detail: `redirects (${response.status}) to ${target}, the key file must return 200 with the key as its body`,
        }
    }
    if (response.status !== 200) {
        return { ok: false, url, detail: `returned ${response.status}` }
    }

    const body = (await response.text()).trim()
    if (body !== INDEXNOW_KEY) {
        return { ok: false, url, detail: `served ${JSON.stringify(body.slice(0, 60))}, expected the key` }
    }

    return { ok: true, url, detail: 'serves the key' }
}

redirect: 'manual' is the whole point of that function. With the default follow, a key file being redirected to /login comes back as a 200 whose body is a sign-in page, and the check that would catch it is the body comparison rather than the status, which reports "served a page of HTML" instead of "your auth gate ate it". Leaving redirects unfollowed turns the most likely failure on this site into the message that names it.

Note also that the body is compared after trim(). A trailing newline in a text file is the most ordinary thing in the world and it is not the key.

One stray origin fails all 72 URLs

Every URL in a submission must be on the declared host or the whole batch is rejected with a 422. So the payload builder refuses rather than letting the API discover it:

const offOrigin = paths.filter((path) => !path.startsWith('/'))
if (offOrigin.length > 0) {
    throw new Error(`IndexNow: paths must be root-relative, got ${offOrigin.join(', ')}.`)
}

Taking root-relative paths only, then making them absolute against one SITE_URL, means an off-origin URL cannot be expressed. The function accepts a shape that has no room for the mistake, which is better than validating a shape that does.

The CLI that drives it has the same attitude about which paths exist at all:

pnpm indexnow                          every URL in the sitemap
pnpm indexnow /guides/pub-quiz-format  only the paths given
pnpm indexnow --dry-run                print the payload, send nothing

Naming a path that is not in INDEXABLE_ROUTES is an error, not a submission, because at that point it is either a typo or a page that should have been added to the sitemap first. And the URL list comes from the same INDEXABLE_ROUTES the sitemap is generated from, so the two cannot drift.

There is also a guard that cost me twenty minutes before it existed. SITE_URL falls back to localhost outside production, and a submission full of http://localhost:3000 URLs is rejected with a 422 that reads exactly like a key mismatch. Now it refuses to run:

if (!IS_PRODUCTION_SITE) {
    fail(`SITE_URL is ${SITE_URL}, not the production site.`)
}

The default is the whole sitemap, once

The default is the whole sitemap, which is what a first submission wants. For
everything after that, name the paths that actually changed: IndexNow is a
"this URL is new or different" signal, and re-submitting seventy unchanged
pages every deploy trains the receiving end to discount it.

That is the comment at the top of the script, and it is the part people get wrong. It is tempting to wire the submitter into the deploy and fire all 72 URLs on every push. The protocol is a change notification. A change notification that fires for things that did not change is not a notification.

200 and 202 are both success, and 403 is one sentence

export const INDEXNOW_STATUS_MEANING: Record<number, string> = {
    200: 'accepted',
    202: 'accepted, key validation pending',
    400: 'bad request, the payload is malformed',
    403: 'forbidden, the key file could not be fetched or did not match',
    422: 'unprocessable, a URL is not on this host, or the key does not match',
    429: 'rate limited, too many submissions',
}

Six documented statuses, and the mapping exists so the failure message says what happened instead of printing a number. A 202 in particular looks like a problem and is not: it means accepted with key validation still pending, which is the normal answer the first time you submit from a new host.

Google does not participate

Bing, Yandex, Seznam and Naver do. Google has said publicly it does not, so all of this is additive to a sitemap in Search Console rather than a replacement for it. Submitting to api.indexnow.org rather than bing.com/indexnow notifies every participating engine in one request, and there is no reason to prefer the single-engine endpoint.

Worth being clear-eyed: for most sites this is a small effect, and it is a few hours of work. We did it because the content plan adds pages in batches, and a batch of new pages is exactly the case where "crawl this now" is worth more than "it is in the sitemap, see you next month".

Check ours

The key file is the whole verification surface, so you can check our side of it completely from a terminal:

# 200, text/plain, body is the key and nothing else
curl -si https://pub-trivia.app/2382c7da66fd467eb45e9c97aadf605a.txt | head -5

# and the sitemap that feeds the submission, 72 URLs
curl -s https://pub-trivia.app/sitemap.xml | grep -c "<loc>"

If you want to see what gets submitted, pub-trivia.app/sitemap.xml is the list, and the newest cluster in it is the free tools, which was the batch that made this worth building. The free tier needs no card if you want to see what the pages are selling.