How to read Cloudflare's AI Crawl Control report and see which AI crawlers get in
Open AI Crawl Control in your Cloudflare dashboard: the crawler list shows every AI crawler Cloudflare has seen on your domain, its requests, its robots.txt violations and whether its action is Allow or Block. To keep AI search while refusing training, allow search and agent crawlers and block or disallow training crawlers there or under AI Crawler and Agent access policies. Then confirm from outside with curl, because other Cloudflare features can still block a crawler you allowed.
Cloudflare now has two places that decide what AI crawlers receive: AI Crawl Control, the per-crawler report and switchboard that used to be called AI Audit, and the zone-wide AI Crawler and Agent access policies under Security settings. Both are available on the Free plan, both changed during 2026, and neither one tells you about every other feature that can still answer a crawler with a 403. This guide walks through the report column by column, shows where to set allow and block, and ends with a test you can run from your own machine.
On this page
- Where is the AI crawler report in Cloudflare?
- What does each column and metric in the AI Crawl Control report mean?
- How do I allow AI search crawlers and block training crawlers in Cloudflare?
- What changed in Cloudflare's AI crawler defaults?
- Why is Cloudflare still blocking AI crawlers after I allowed them?
- How do I confirm from outside which AI crawlers Cloudflare lets in?
- How does AISeeWhy check what Cloudflare sends to AI crawlers?
Where is the AI crawler report in Cloudflare?
In the dashboard, select your account and domain, then open AI Crawl Control. Cloudflare says it works without configuration and is available on all plans, as long as your domain's traffic is proxied through Cloudflare. It has four areas:
- Overview: an executive summary with total requests, the change in volume, the most common status code, the most popular path and high-volume crawlers, plus whether Cloudflare's managed robots.txt is on, and crawlers grouped by operator such as OpenAI, Microsoft, Google, ByteDance, Anthropic and Meta.
- The crawler list: one row per AI crawler, with an Action column where you choose Allow or Block. Cloudflare's documentation calls this tab Crawlers on one page and Security on another, so look for the table with the Action column.
- Metrics: requests over time, status code distribution, content format, most popular paths and, on paid plans, referrals. Filters cover date range, crawler, operator, hostname and path, and you can export a CSV.
- Directives: the health of your robots.txt file and the list of robots.txt violations.
Two plan limits matter when you read the numbers. On the Free plan, AI Crawl Control recognises crawlers by their user agent string only, so it sees crawlers that announce themselves and nothing else, and the Metrics tab covers the past 24 hours. Paid plans add detection through the Bot Management detection ID, and Enterprise with Bot Management gets configurable time frames.
What does each column and metric in the AI Crawl Control report mean?
Cloudflare documents the fields as follows. The definitions matter, because several numbers count more than you might expect.
- Crawler and Category: the crawler's name, its operator, and its category, such as AI crawler, AI assistant or archiver.
- Requests: allowed plus unsuccessful requests, with a trend chart. Unsuccessful requests can come from any rule or response error, not only from an AI Crawl Control block, so a high number for a crawler you allowed points to another rule or to errors on your origin.
- Allowed requests: requests that received a status between 200 and 299. Data transfer is measured from the bytes Cloudflare sent for allowed requests.
- Robots.txt violations: how many times the crawler requested a path your robots.txt disallows. Cloudflare compares past requests with your current rules, so a rule you add today can flag requests that were allowed when they were made.
- Action: Allow, Block, or, for accounts in the pay per crawl beta, Charge.
- Status code distribution: 2xx, 3xx, 4xx and 5xx. The 4xx group includes 403 from blocks and 402 from pay per crawl.
- Referrals and top referrers, paid plans only: visits that arrived from AI services, for example from chatgpt.com, and the pages they landed on.
Read the report from the action you chose. A crawler set to Allow should show mostly allowed requests and 2xx codes. If it shows many unsuccessful requests, open the status code distribution filtered to that crawler, and check Security Analytics to find the rule that answered.
How do I allow AI search crawlers and block training crawlers in Cloudflare?
You can do it per crawler or by behaviour. Per crawler, open AI Crawl Control and set the Action for each row. Cloudflare says that blocking there creates or updates a WAF custom rule on your zone, which you can extend in the WAF for path exceptions. On paid plans you can choose whether blocked crawlers get a 403 or a 402 and set a plain text body.
By behaviour, go to Security, then Settings, filter by Bot traffic and open AI Crawler and Agent access policies. Cloudflare sorts AI traffic into three behaviours, and each policy applies to verified bots with that behaviour and to matching unverified bots:
- Search: collecting or indexing content so it can answer questions later.
- Agent: acting in real time on a person's behalf, such as chat fetches and browser agents.
- Training: collecting content to train or fine-tune models, including crawlers used for both training and search.
Each behaviour can be set to Block, Block on pages with ads, or Allow (do not block). Training also has Disallow, which only asks crawlers through robots.txt. With Bot Preference Sync turned on, Cloudflare prepends managed rules to your robots.txt that disallow Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, adds a Content Signals line, and also blocks AI training crawlers at the edge, except crawlers used for both search and training.
For most sites that want AI citations, the safe combination is Search and Agent on Allow and Training on Disallow, then individual Block actions in AI Crawl Control for training-only crawlers you want to stop at the edge.
What changed in Cloudflare's AI crawler defaults?
On July 1, 2025, Cloudflare announced that it was changing the default to block AI crawlers unless they pay creators. On July 1, 2026, it replaced the single Block AI bots switch with the separate Search, Agent and Training policies described above, for all plans including Free. The same announcement said that new domains onboarding from September 15, 2026 would have Training and Agent blocked on pages that display ads, with Search allowed.
Cloudflare's current policy documentation, last updated November 2, 2026, says new domains default to Allow (do not block) for all three behaviours. Because Cloudflare's own pages have described different defaults at different times, do not rely on any of them for your zone: open the policy page and read what is set.
Why is Cloudflare still blocking AI crawlers after I allowed them?
The Allow action in AI Crawl Control only stops AI Crawl Control from blocking. Other features still act on the request. Common causes:
- Blocking Training also blocks search crawlers that do both. Cloudflare's documentation says Training actions apply to mixed-purpose crawlers, and its July 2026 announcement names Googlebot, Applebot and bingbot as crawlers that customers who block Training will block from September 15, 2026. Disallow is the narrower choice, because its edge block skips mixed-purpose crawlers.
- Bot Fight Mode is on. Cloudflare states that you cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules. If it challenges a crawler, turn it off, or on a plan that has it use Super Bot Fight Mode, which supports skip rules.
- An older WAF custom rule blocks user agents containing
bot, or a rate limiting rule answers bursts with 429. These run regardless of the Action in AI Crawl Control. - The block rule was edited by hand. If the WAF rule behind AI Crawl Control was changed to another response code, the dashboard can no longer enforce your selection.
- robots.txt still disallows the crawler. Cloudflare's Allow does not change your robots.txt, so a polite crawler stays away anyway. In our study of the 1,000 most popular domains, 98 of the 155 sites that blocked an AI training crawler in robots.txt also blocked at least one AI search or user crawler.
How do I confirm from outside which AI crawlers Cloudflare lets in?
Request the same page with a browser user agent and with each crawler's user agent, then compare status and size. The OpenAI, Perplexity and Google strings below are the ones those companies publish.
URL=https://example.com/
check() { printf '%-16s %s\n' "$1" "$(curl -s -o /dev/null -w '%{http_code} %{size_download} bytes' -A "$2" "$URL")"; }
check browser "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
check GPTBot "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
check OAI-SearchBot "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
check PerplexityBot "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
# Anthropic publishes the token, not a full user agent string
check ClaudeBot "Mozilla/5.0 (compatible; ClaudeBot)"
check Googlebot "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" A crawler line with 403, 429 or 503, or a much smaller size than the browser line, means something in front of your origin answered differently. On the Free plan, AI Crawl Control detects crawlers by user agent, so this test is a reasonable check of its blocks. Rules based on verified bot status or bot scores can treat your request differently from the real crawler, because it comes from your IP address and not from the operator's published ranges. After testing, look for your own requests in Security Analytics to see which rule matched.
How does AISeeWhy check what Cloudflare sends to AI crawlers?
The free AISeeWhy check requests your page as 16 AI and search crawlers next to a real browser, applies your robots.txt per crawler and recognises firewall challenge pages, so you can see whether the result in Cloudflare's dashboard matches what crawlers actually receive.
Fixes for this problem
Next steps
- robots.txt tester and checker for AI crawlers
- AI crawler IP check
- See an example full report
- AI crawler monitoring
- Plans and prices
Crawlers in this article
Frequently asked questions
Is Cloudflare AI Crawl Control free?
Yes. Cloudflare lists AI Crawl Control as available on all plans. The Free plan detects crawlers by user agent only and shows the past 24 hours in Metrics; referral data and custom block responses need a paid plan.
What happened to Cloudflare AI Audit?
It was renamed. Cloudflare's documentation describes AI Crawl Control as formerly AI Audit, with the same purpose: seeing which AI crawlers visit and setting an action for each.
Does blocking a crawler in AI Crawl Control change my robots.txt?
No. A Block action creates or updates a WAF custom rule. robots.txt only changes if you use the Disallow option for Training with Bot Preference Sync, which prepends Cloudflare's managed rules to your file.
What counts as a robots.txt violation in Cloudflare?
A request from an AI crawler to a path your robots.txt disallows. Cloudflare compares past requests with your current rules, so a new rule can flag older requests that were allowed at the time.
What is pay per crawl?
A Charge action that asks AI crawlers to pay per request, answering with HTTP 402 and a price otherwise. Cloudflare describes it as a closed beta, and WAF or Bot Management blocks take precedence over it.
Sources
- Cloudflare: AI Crawl Control
- Cloudflare: Get started with AI Crawl Control
- Cloudflare: Manage AI crawlers
- Cloudflare: Analyze AI traffic
- Cloudflare: Track robots.txt
- Cloudflare: What is pay per crawl
- Cloudflare: AI Crawler and Agent access policies
- Cloudflare: Managed robots.txt
- Cloudflare: Bot Fight Mode
- Cloudflare: Verified bots
- Cloudflare changelog: New options to manage AI traffic
- Cloudflare blog: Your site, your rules: new AI traffic options for all customers
- Cloudflare blog: Content Independence Day: no AI crawl without compensation
- OpenAI: Overview of OpenAI crawlers
- Perplexity: Perplexity crawlers
- AISeeWhy: AI Crawler Access Report, 2026-11