How to read Cloudflare's AI Crawl Control report and see which AI crawlers get in

Published

Short answer

Open AI Crawl Control in your Cloudflare dashboard: the crawler list shows every AI crawler Cloudflare has seen on your domain, its requests, its robots.txt violations and whether its action is Allow or Block. To keep AI search while refusing training, allow search and agent crawlers and block or disallow training crawlers there or under AI Crawler and Agent access policies. Then confirm from outside with curl, because other Cloudflare features can still block a crawler you allowed.

Cloudflare now has two places that decide what AI crawlers receive: AI Crawl Control, the per-crawler report and switchboard that used to be called AI Audit, and the zone-wide AI Crawler and Agent access policies under Security settings. Both are available on the Free plan, both changed during 2026, and neither one tells you about every other feature that can still answer a crawler with a 403. This guide walks through the report column by column, shows where to set allow and block, and ends with a test you can run from your own machine.

On this page
  1. Where is the AI crawler report in Cloudflare?
  2. What does each column and metric in the AI Crawl Control report mean?
  3. How do I allow AI search crawlers and block training crawlers in Cloudflare?
  4. What changed in Cloudflare's AI crawler defaults?
  5. Why is Cloudflare still blocking AI crawlers after I allowed them?
  6. How do I confirm from outside which AI crawlers Cloudflare lets in?
  7. How does AISeeWhy check what Cloudflare sends to AI crawlers?

Where is the AI crawler report in Cloudflare?

In the dashboard, select your account and domain, then open AI Crawl Control. Cloudflare says it works without configuration and is available on all plans, as long as your domain's traffic is proxied through Cloudflare. It has four areas:

Two plan limits matter when you read the numbers. On the Free plan, AI Crawl Control recognises crawlers by their user agent string only, so it sees crawlers that announce themselves and nothing else, and the Metrics tab covers the past 24 hours. Paid plans add detection through the Bot Management detection ID, and Enterprise with Bot Management gets configurable time frames.

What does each column and metric in the AI Crawl Control report mean?

Cloudflare documents the fields as follows. The definitions matter, because several numbers count more than you might expect.

Read the report from the action you chose. A crawler set to Allow should show mostly allowed requests and 2xx codes. If it shows many unsuccessful requests, open the status code distribution filtered to that crawler, and check Security Analytics to find the rule that answered.

How do I allow AI search crawlers and block training crawlers in Cloudflare?

You can do it per crawler or by behaviour. Per crawler, open AI Crawl Control and set the Action for each row. Cloudflare says that blocking there creates or updates a WAF custom rule on your zone, which you can extend in the WAF for path exceptions. On paid plans you can choose whether blocked crawlers get a 403 or a 402 and set a plain text body.

By behaviour, go to Security, then Settings, filter by Bot traffic and open AI Crawler and Agent access policies. Cloudflare sorts AI traffic into three behaviours, and each policy applies to verified bots with that behaviour and to matching unverified bots:

Each behaviour can be set to Block, Block on pages with ads, or Allow (do not block). Training also has Disallow, which only asks crawlers through robots.txt. With Bot Preference Sync turned on, Cloudflare prepends managed rules to your robots.txt that disallow Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, adds a Content Signals line, and also blocks AI training crawlers at the edge, except crawlers used for both search and training.

For most sites that want AI citations, the safe combination is Search and Agent on Allow and Training on Disallow, then individual Block actions in AI Crawl Control for training-only crawlers you want to stop at the edge.

What changed in Cloudflare's AI crawler defaults?

On July 1, 2025, Cloudflare announced that it was changing the default to block AI crawlers unless they pay creators. On July 1, 2026, it replaced the single Block AI bots switch with the separate Search, Agent and Training policies described above, for all plans including Free. The same announcement said that new domains onboarding from September 15, 2026 would have Training and Agent blocked on pages that display ads, with Search allowed.

Cloudflare's current policy documentation, last updated November 2, 2026, says new domains default to Allow (do not block) for all three behaviours. Because Cloudflare's own pages have described different defaults at different times, do not rely on any of them for your zone: open the policy page and read what is set.

Why is Cloudflare still blocking AI crawlers after I allowed them?

The Allow action in AI Crawl Control only stops AI Crawl Control from blocking. Other features still act on the request. Common causes:

How do I confirm from outside which AI crawlers Cloudflare lets in?

Request the same page with a browser user agent and with each crawler's user agent, then compare status and size. The OpenAI, Perplexity and Google strings below are the ones those companies publish.

URL=https://example.com/
check() { printf '%-16s %s\n' "$1" "$(curl -s -o /dev/null -w '%{http_code} %{size_download} bytes' -A "$2" "$URL")"; }

check browser       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
check GPTBot        "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
check OAI-SearchBot "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
check PerplexityBot "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
# Anthropic publishes the token, not a full user agent string
check ClaudeBot     "Mozilla/5.0 (compatible; ClaudeBot)"
check Googlebot     "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

A crawler line with 403, 429 or 503, or a much smaller size than the browser line, means something in front of your origin answered differently. On the Free plan, AI Crawl Control detects crawlers by user agent, so this test is a reasonable check of its blocks. Rules based on verified bot status or bot scores can treat your request differently from the real crawler, because it comes from your IP address and not from the operator's published ranges. After testing, look for your own requests in Security Analytics to see which rule matched.

How does AISeeWhy check what Cloudflare sends to AI crawlers?

The free AISeeWhy check requests your page as 16 AI and search crawlers next to a real browser, applies your robots.txt per crawler and recognises firewall challenge pages, so you can see whether the result in Cloudflare's dashboard matches what crawlers actually receive.

Fixes for this problem

Free tools

Next steps

Crawlers in this article

Frequently asked questions

Is Cloudflare AI Crawl Control free?

Yes. Cloudflare lists AI Crawl Control as available on all plans. The Free plan detects crawlers by user agent only and shows the past 24 hours in Metrics; referral data and custom block responses need a paid plan.

What happened to Cloudflare AI Audit?

It was renamed. Cloudflare's documentation describes AI Crawl Control as formerly AI Audit, with the same purpose: seeing which AI crawlers visit and setting an action for each.

Does blocking a crawler in AI Crawl Control change my robots.txt?

No. A Block action creates or updates a WAF custom rule. robots.txt only changes if you use the Disallow option for Training with Bot Preference Sync, which prepends Cloudflare's managed rules to your file.

What counts as a robots.txt violation in Cloudflare?

A request from an AI crawler to a path your robots.txt disallows. Cloudflare compares past requests with your current rules, so a new rule can flag older requests that were allowed at the time.

What is pay per crawl?

A Charge action that asks AI crawlers to pay per request, answering with HTTP 402 and a price otherwise. Cloudflare describes it as a closed beta, and WAF or Bot Management blocks take precedence over it.

Sources

More articles