AISeeWhy
EN
Start free trial

Is Cloudflare blocking AI crawlers? How to find out and fix it

Published

Short answer

Cloudflare and other firewalls can answer AI crawlers with a 403, a 429 or a "Just a moment..." challenge page while browsers get the real page, even when robots.txt allows the crawler. Check by requesting the page with the crawler's user agent, then allow AI search crawlers in AI Crawl Control or with a WAF skip rule.

Bot protection exists to stop scrapers and abuse, and it is good at it. The trouble is that AI search crawlers are bots too. A security setting turned on years ago, a rate limit, or a broad rule against unknown user agents can quietly keep the crawlers that cite you out, while your own visits look perfectly normal.

On this page
  1. Why would a firewall block AI crawlers?
  2. How do I tell a firewall block from a robots.txt block?
  3. How do I allow AI search crawlers in Cloudflare?
  4. What about Bot Fight Mode?
  5. Which crawlers should I let through?
  6. How does AISeeWhy detect firewall blocks?

Why would a firewall block AI crawlers?

Firewalls judge each request. A crawler does not run JavaScript challenges, does not keep cookies like a browser and often requests many pages in a row, so it can look exactly like the traffic bot protection is built to stop. Common causes are:

How do I tell a firewall block from a robots.txt block?

A robots.txt block is a polite request: the crawler reads your rules and does not fetch the page. A firewall block is an answer: the crawler requests the page and gets something else. You can see the second kind directly. Request the page twice, once with a browser user agent and once with the crawler's, and compare.

for UA in "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
          "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"; do
  curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "$UA" https://example.com/page
done

Signs of a firewall block: status 403, 429 or 503 for the crawler only; a page titled Just a moment... or Attention Required! | Cloudflare; a body that contains /cdn-cgi/challenge-platform; or a 200 response that is much shorter than the browser's. Keep in mind that your test comes from your own IP, not the crawler's verified IP range, so it shows how the firewall treats the user agent, not every detail of how it treats the real crawler.

How do I allow AI search crawlers in Cloudflare?

Cloudflare's AI Crawl Control, available on all plans, lists the AI crawlers that visit your domain and lets you set an action for each. In the dashboard, open AI Crawl Control, go to the Crawlers list and choose Allow or Block in the Action column for each crawler. Cloudflare says that blocking there creates or updates a WAF custom rule on your zone.

If a WAF custom rule, Super Bot Fight Mode or rate limiting is what blocks the crawler, add a custom rule that skips those features for verified bots and for the crawler's token:

Security → WAF → Custom rules → Create rule
  Expression: (cf.client.bot) or (http.user_agent contains "OAI-SearchBot")
  Action: Skip → All remaining custom rules, Super Bot Fight Mode, Rate limiting
Also: AI Crawl Control → OAI-SearchBot → Allow

Place the rule first. Note that matching on the user agent also lets through anyone who fakes that user agent. Where you can, prefer the verified bot field (cf.client.bot) or the IP ranges the vendor publishes.

What about Bot Fight Mode?

Cloudflare's documentation is explicit that Bot Fight Mode cannot be bypassed or skipped with WAF custom rules or Page Rules. If it is the feature challenging crawlers, a skip rule will not help. The options are to turn Bot Fight Mode off, or to use Super Bot Fight Mode on a plan that has it, which runs on the Ruleset Engine and supports skip rules.

Which crawlers should I let through?

Separate crawlers by what they do. Search and user-fetch crawlers decide whether AI answers can find and cite you. Training crawlers collect data for models, and blocking them is a legitimate choice that does not remove you from AI search.

Cloudflare's own policy settings use the same split into Search, Agent and Training, so you can allow the first two and decide separately on the third. If you use another firewall or CDN, the principle is the same: allowlist the crawler's token, verified by the vendor's published IP ranges or reverse DNS where offered, so it gets the same response as a browser.

How does AISeeWhy detect firewall blocks?

AISeeWhy requests your page as 14 crawlers and a normal browser, compares status, size and redirects, recognises challenge pages and names the firewall or CDN that answered. When robots.txt allows a crawler but the server blocks it, the report shows the conflict and gives the Cloudflare rule or the allowlist step to fix it.

Frequently asked questions

Does Cloudflare block AI crawlers by default?

It depends on your account, plan and the settings in your zone, and Cloudflare has changed these options over time. Check AI Crawl Control and your bot settings, and test with the crawler's user agent rather than assuming.

Why does my site work in a browser but not for crawlers?

Firewalls judge each request. A browser passes JavaScript challenges and keeps cookies; a crawler does not, so it can receive a challenge page or a 403 that you never see.

Is it safe to allow AI crawlers through the WAF?

Allowing a named crawler is low risk if you verify it by the vendor's IP ranges or Cloudflare's verified bot signal. Rules that trust the user agent alone also admit anyone who copies it.

Does the Cloudflare email obfuscation feature affect AI crawlers?

It replaces email addresses with a link that only a script decodes, so crawlers see a placeholder instead of the address. If you want your contact email to be readable, turn off Email Address Obfuscation under Scrape Shield and show the address as plain text.

Sources

More articles