What robots.txt can and can't do for paid content — and the setup that actually holds

You cannot make paid content unscrapable. You can make sure the bytes never leave your server unless someone paid, and you can make scraping expensive and pointless. This post is about where that line actually sits, and what a workable setup looks like.

What robots.txt actually does

robots.txt is a file of requests at the root of a domain. A crawler that chooses to follow it reads it before fetching anything else and skips the paths you list.

Three properties matter:

• It is advisory. Nothing enforces it. A crawler that ignores it gets whatever your server hands out.
• It is per-user-agent and prefix-matched. Disallow: /premium/ covers /premium/a and /premium/b/c, but not /articles/premium-a.
• It is public. Anyone can read it, including the person deciding which URL to hammer.

What it genuinely buys you: a machine-readable statement of intent to crawlers that do comply, and — for the major search engines — a reason not to crawl and, eventually, not to index those URLs.

One trap worth naming: Disallow and noindex fight each other. If a search engine is blocked from crawling a URL it never sees the noindex header, so it may keep the URL indexed on the strength of external links. If removal is the goal, allow the crawl and serve noindex. If gating is the goal and the page should not exist for anonymous users at all, Disallow is fine — there is nothing to index anyway.

Also: not every token in a robots.txt file is a crawler. Some are training opt-out tokens that control whether content is used to build models and have no effect on crawling. Google-Extended is one of these — it is not a user agent that fetches pages. Mixing the two kinds of tokens is a common mistake in AI-related robots.txt files.

What robots.txt cannot do

• Stop a non-compliant scraper. Plenty of them never read the file.
• Stop a headless browser driven by a paid account. It has a session cookie and a residential IP; it looks like a customer.
• Stop a human copy-paste, a screenshot, or a PDF passed around a team.
• Protect content that was already served. If your gated page ships the full body in HTML to anyone who requests it, robots.txt is decoration.
• Cover your other surfaces: sitemaps, RSS, oEmbed, JSON endpoints, image CDN paths, AMP or print variants, staging domains, the search-results page.

The last two points are where most "we blocked the AI crawlers" setups actually leak.

The real boundary is authentication, enforced before rendering

Rule: an anonymous request must never receive the paid payload. Not hidden with CSS, not fetched after a JS check, not truncated in the template — never sent.

Enforce it at the reverse proxy or the application's entry point, not in the theme. A gated path rendered by the same code that renders public pages will eventually leak through a preview, an API, or an excerpt filter.

Sketch with nginx auth_request (requires http_auth_request_module):

location /premium/ {
    auth_request /_auth;
    error_page 401 =200 /teaser.html;
}

location = /_auth {
    internal;
    proxy_pass http://127.0.0.1:9000/verify;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header X-Original-URI $request_uri;
}

location = /teaser.html {
    internal;
    add_header Cache-Control "no-store" always;
    add_header X-Robots-Tag "noindex, nofollow" always;
}

/verify returns 200 for a valid session and 401 otherwise. The 401 becomes a teaser page with noindex, so the URL stays shareable and link-previewable without ever exposing the body. Test the error_page rewrite on your own server before trusting it — behaviour differs between versions and when the backend is a proxy rather than a static file.

For files (PDFs, video, downloadable assets) do not serve from a public path. Issue short-lived signed URLs bound to the session, and log issuance. A signed URL that lives five minutes and is tied to one account does more than any robots rule.

Close the side doors

Checklist, roughly in the order leaks happen:

1. Sitemaps — gated URLs excluded. If the generator pulls from the post table, filter by the same condition that gates the page.
2. Feeds — excerpt only, or a separate feed that omits gated items. A full-content feed is a public unauthenticated copy of your paid body.
3. oEmbed / JSON / GraphQL — every read path needs the same check as the HTML path. This is the one people forget.
4. Search results page — if it indexes gated content, the excerpt becomes the leak.
5. Preview and staging — token-gated, noindex, taken down when the work is done.
6. Image derivatives — thumbnails of paid assets on a public CDN are a leak, and unguessable only until someone reads the page source.
7. Print and AMP variants — separate templates, same rule.
8. Directory listing — off everywhere.

A robots.txt that matches your intent

# Compliant crawlers: stay out of gated and downloadable paths.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /premium/
Disallow: /downloads/

# Training opt-out token, not a crawler. Disallow here means
# "don't use this content to improve the models".
User-agent: Google-Extended
Disallow: /premium/

# Search engines: index the teaser, not the payload.
User-agent: Googlebot
Disallow: /premium/
Disallow: /downloads/

User-agent: *
Disallow: /premium/
Disallow: /downloads/

Two notes. Prefix matching means you should test your paths with a checker rather than trusting your eye. And a User-agent: * group does not cover crawlers that have their own group — that is the whole point of separate groups, so list the ones you care about explicitly.

Tell search engines the truth about the teaser

You want the teaser indexed; that is how paid content gets discovered. What you don't want is a search engine treating it as free content. The signal Google documents for this is paywall structured data on the teaser page: isAccessibleForFree: false plus a hasPart pointing at the gated section.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "…",
  "isAccessibleForFree": false,
  "hasPart": {
    "@type": "WebPageElement",
    "isAccessibleForFree": false,
    "cssSelector": ".paid-body"
  }
}
</script>

This does not stop scraping. It stops the wrong thing from being indexed as free, which is the failure mode that costs the most in search.

Detect, then respond

Blocking is a one-time decision; scraping is ongoing. Read your access log.

With the combined log format, the user agent is the sixth quoted field:

awk -F'"' '$2 ~ /^GET \/premium\// {print $6}' access.log \
  | sort | uniq -c | sort -rn | head -20

Then look at the shape, not just the count: many distinct accounts from one ASN, sequential ID enumeration, or a single session pulling pages faster than a human can read. Those are account-level problems — rate-limit the session, not just the IP, and require re-authentication on anomalies.

Per-user watermarking (a visible user ID or hashed token in the corner of a PDF) does not stop the leak, but it makes the source attributable, which changes behaviour more than another Disallow line.

Priority order

If you only do three things:

1. Move the gate to the proxy. No paid bytes for anonymous requests.
2. Kill full-content feeds and ungated JSON/API reads of paid items.
3. Add paywall structured data to the teaser and noindex to the gated payload.

Then, and only then, spend time on robots.txt — a statement of intent, not a lock.

If you run WordPress and want to see which crawlers are actually hitting which paths before writing the rules, we maintain a free plugin for that: AumCrawl logs AI crawler hits and lets you allow or block per crawler. There is also a per-crawler breakdown at aumcreate.com/crawlers.