Cloudflare split "Block AI Bots" into Search, Training and Agent. I checked all 26 of my sites through the API

On September 16, 2026, I got an email from Cloudflare titled "Updates to managing AI crawlers on your account."
It said the setting for blocking AI crawlers was changing from a single switch into three categories: Search, Training and Agent.

I manage 26 domains on Cloudflare.
Instead of opening the dashboard 26 times, I pulled the settings for every domain at once through the API.
This post covers what changed, how I checked, and how I think a site that wants to be cited in AI answers should be set up.

What changed

The email listed three changes.

• The recommended setting for training crawlers changed from "Block" to "Disallow AI Training"
• Sites that had turned on "Block AI Bots" or "Managed Robots.txt" are moved to the new three-category setting automatically
• Sites that never touched the setting stay on "Allow," and nothing changes for them

According to Cloudflare, "Disallow AI Training" lets search crawlers such as Applebot, Googlebot and Bingbot keep crawling for search results, while telling them not to use the content for training.

Sites that are moved automatically end up with these values.

• Search: Allow
• Training: Disallow AI Training
• Agent: Block only on pages with ads

According to the email, the "Block AI Bots" switch disappears from the dashboard once the migration is done.

Reading the settings through the API

Each zone's settings can be read with GET /zones/{zone_id}/bot_management.
I listed all zones, read the settings one by one, and printed only the AI-related fields.

#!/bin/bash
# CF_API_TOKEN needs the "Bot Management Read" permission
AUTH="Authorization: Bearer $CF_API_TOKEN"

curl -s -H "$AUTH" "https://api.cloudflare.com/client/v4/zones?per_page=100" \
  | jq -r '.result[] | "\(.id) \(.name)"' \
  | while read -r id name; do
      curl -s -H "$AUTH" "https://api.cloudflare.com/client/v4/zones/$id/bot_management" \
        | jq -c --arg name "$name" '{name: $name} + (.result | {
            ai_bots_protection, ai_search, ai_training, ai_user,
            is_robots_txt_managed, cf_robots_variant, crawler_protection
          })'
    done

My first try used the API token I use for DNS.
It came back with Authentication error (code 10000).
The API reference says reading this setting needs either "Bot Management Read" or "Bot Management Write."
If you reuse a DNS token, you will hit the same wall.

What the fields mean

Summarizing the API reference:

In the reference, the three new categories (ai_search, ai_training, ai_user) are described as "robots.txt policies."
Only ai_training has a disallow value.

The point to keep in mind is that robots.txt is only a request.
Cloudflare's own documentation says whether to follow robots.txt is up to each crawler, and that it is not a technical block.
If you really want to stop a crawler, you use a rule such as AI Crawl Control.

Results for 26 sites

All 26 sites came back the same.

• ai_bots_protection: disabled
• ai_search, ai_training, ai_user: all disabled
• is_robots_txt_managed: false
• crawler_protection: disabled

In other words, none of my sites were blocking AI crawlers.
That matches "sites that never touched the setting stay the same" in the email.
They are not part of the automatic migration, so this change has no effect on them.

One field did differ between sites.
cf_robots_variant was policy_only on 10 of the 26 sites and off on the other 16.
The reference only calls it "the type of Robots Access Control License," and I do not remember ever setting it.

I fetched robots.txt from one of the policy_only sites, and Cloudflare had not added any lines.
Since is_robots_txt_managed is false, my reading is that Cloudflare is not touching robots.txt.
I have not yet confirmed what this value actually means.

I allow AI crawlers explicitly in my own robots.txt

I do not leave robots.txt to Cloudflare. I write it myself.
AI crawlers are allowed by name (excerpt):

User-Agent: *
Allow: /

# AI crawlers: fully allowed
User-Agent: GPTBot
User-Agent: ChatGPT-User
User-Agent: OAI-SearchBot
User-Agent: ClaudeBot
User-Agent: Claude-SearchBot
User-Agent: Claude-User
User-Agent: Google-Extended
User-Agent: PerplexityBot
User-Agent: Applebot-Extended
Allow: /

The reason is simple: the smaller the site, the more it gains from being named in an AI answer.
To be a candidate when someone asks "what do you recommend?", the site first has to be readable.

Settings for a site that wants to be cited in AI answers

Putting this together, here is how I think a site that wants to be cited in AI answers (ChatGPT search, Perplexity and so on) should approach it.

Do not block Search or Agent

Blocking AI search bots means losing the chance to appear in AI search results at all.
Agents that read a page because a user asked them to (ChatGPT-User and similar) cannot use your content in their answer if you block them.

Decide Training as a policy

Being used for training and being cited in a search answer are two different things.
If you do not want your content used for training, you can set only Training to Disallow AI Training and keep Search and Agent allowed.
This change is exactly what makes that split selectable in the dashboard.

If your site was migrated, check the Agent setting

Sites that had Block AI Bots turned on are moved to "Agent: Block only on pages with ads."
If you want AI agents to read pages that show ads, check the setting once after the migration.

Check that nothing else is blocking them

Even if every Cloudflare setting is disabled, a WAF custom rule or a rate limit may still be turning AI crawlers away.
Allowing them in robots.txt does nothing if they are stopped before they get there.

Summary

• Cloudflare's AI crawler setting is now split into Search, Training and Agent
• Only sites that were blocking before are migrated automatically. Sites that never touched it see no change
• The settings can be read through the bot_management API, so you can check every zone at once. The token needs Bot Management Read
• The three new categories are robots.txt policies and are not enforced
• If you want to be cited in AI answers, leave Search and Agent open and decide Training as a policy