Why is my site suddenly blocking AI crawlers?

Published

Short answer

A site that starts blocking AI crawlers has almost always changed in one of five places: the robots.txt that is actually served, a CDN setting such as Cloudflare's AI crawler policies, bot protection or a rate limit, a hosting default, or a robots.txt that now answers with an error. Fetch today's robots.txt and a page as each crawler, compare them with a browser and with what you had before, and check those layers in that order.

Nobody decides to disappear from AI answers. It happens as a side effect of a theme update, a security setting, a migration or a changed default at a provider. A browser still sees the site normally, so the change can go unnoticed until someone asks ChatGPT or Perplexity about the product and gets a competitor. Each layer leaves traces you can check from outside or in your logs.

On this page
  1. Did the robots.txt that crawlers receive change?
  2. Did Cloudflare's 2026 AI crawler policies change what my site allows?
  3. Is bot protection or a rate limit catching the crawlers?
  4. Did my hosting provider change its defaults?
  5. Is robots.txt answering with an error?
  6. How do I find out when it started?
  7. How do I catch it sooner next time?

Did the robots.txt that crawlers receive change?

Fetch https://yourdomain.com/robots.txt and compare it with the file in your repository or backup. Look at the served file, not the one on disk, because several layers can change it on the way:

Google Search Console's robots.txt report shows the versions Google fetched in the last 30 days, with a new entry each time the file or the fetch result changed. It covers only what Google saw, but unless your server varies robots.txt by user agent, AI crawlers got the same file.

Did Cloudflare's 2026 AI crawler policies change what my site allows?

Cloudflare has changed its AI crawler controls several times, and some changes alter what a zone does without anyone editing it. On 1 July 2025 it began asking new domains at sign-up whether to allow AI crawlers, with blocking as the starting state. On 1 July 2026 it replaced the single Block AI bots switch with separate Search, Agent and Training policies, for all plans including Free.

The detail that catches most sites is mixed-purpose crawlers. Cloudflare's documentation says Training actions also apply to crawlers used for both search and training. Its July 2026 announcement said that from 15 September 2026, multi-purpose crawlers such as Googlebot, Applebot and BingBot would be blocked for customers who chose to block Training, whether through the new options or the legacy Block AI bots setting. A training opt-out set a year ago can therefore now reach search crawlers.

Defaults for new domains have been described differently over time. The same announcement said new domains would have Training and Agent blocked on pages that display ads from 15 September 2026, while the policy documentation updated on 2 November 2026 says new domains start with all three on Allow. Open Security, then Settings, filter by Bot traffic and read AI Crawler and Agent access policies for your own zone. Our guide to Cloudflare AI Crawl Control explains each setting.

Is bot protection or a rate limit catching the crawlers?

Bot protection judges each request, and crawlers fetch many pages quickly from data centres, which is what bots do. A crawler that got through last month can start receiving challenge pages after its operator changes IP ranges or crawl rate, or after the protection rules are updated. Cloudflare's Bot Fight Mode is a common cause, and Cloudflare says it cannot be bypassed or skipped with WAF custom rules or Page Rules, so an allow rule added later will not help while it is on. A new rate limit that answers bursts with 429 has the same effect on a busy crawler.

Test from outside: request a page as a browser and as two AI search crawlers, then request robots.txt as a crawler.

URL=https://example.com/page
for UA in "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
          "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
          "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"; do
  curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "$UA" "$URL"
done

# robots.txt itself, as a crawler
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" \
  -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  https://example.com/robots.txt

A 403, 429 or 503 for a crawler only, a page titled Just a moment..., or a body much smaller than the browser's means the crawler is treated differently. The test comes from your IP, not the vendor's, so rules based on verified bot status can still treat the real crawler differently. The firewall's event log shows which rule answered.

Did my hosting provider change its defaults?

Some hosts filter AI traffic at the server, before your own rules run. SiteGround, for example, wrote in August 2025 that it blocks AI crawlers designed to collect training data by default and allows crawlers that visit when a person asks an AI assistant about a page. Rules like these do not show up in your robots.txt, and they can change when the host updates its bot lists.

If a block appears on several sites with the same host at once, or in responses that never reach your application logs, ask the host which bot rules apply to your account and whether you can exempt the crawlers you want to keep.

Is robots.txt answering with an error?

A robots.txt that fails can block more than any Disallow line. Under RFC 9309, a 4xx response means a crawler may fetch everything, but a server error (5xx) or no answer at all means it must assume the whole site is disallowed. Google stops crawling for the first 12 hours of such errors and then uses its last good copy for up to 30 days.

This is how a load spike, a firewall rule that challenges /robots.txt, or a rate limit that includes it can shut out well-behaved crawlers without a single rule changing. The last command in the test above should return 200 and text/plain. A 5xx, a timeout or a challenge page there is the first thing to fix.

How do I find out when it started?

A date turns a mystery into a diff. In your access logs, filter requests from AI crawler user agents and look for the day their status codes moved from 200 to 403, 429 or 503, or the day they stopped requesting pages after reading robots.txt; our guide to checking server logs for AI crawlers has the commands. Requests your CDN blocked never reach the origin, so check the CDN as well. On Cloudflare, AI Crawl Control shows a status code distribution over time that you can filter by crawler.

Then line that date up with deploys, plugin and theme updates, CDN setting changes and provider announcements. Before acting, confirm the source IPs belong to the vendor, so you are looking at the real crawler and not an impostor using its name.

How do I catch it sooner next time?

Check access after every deploy, CDN change and plugin update, not only when traffic drops. The free AISeeWhy check fetches a page as 16 AI and search crawlers next to a real browser, applies your robots.txt per crawler and recognises firewall challenge pages. The free robots.txt checker shows, per crawler, whether a page may be fetched under the file that is live today.

Free tools

Next steps

Crawlers in this article

Frequently asked questions

Why did ChatGPT stop showing my site?

Often because OAI-SearchBot, the crawler behind ChatGPT search, became blocked by a robots.txt change, a CDN policy or bot protection. OpenAI says sites that block it will not appear in ChatGPT search answers. Check each layer for that crawler, and allow about 24 hours after a robots.txt fix, which is how long OpenAI says its systems take to adjust.

Did Cloudflare start blocking Googlebot on my site?

It can happen. Cloudflare said that from 15 September 2026 customers who block Training also block multi-purpose crawlers such as Googlebot, Applebot and BingBot. Its documentation says the Training Disallow option blocks training crawlers at the edge but leaves out mixed-use crawlers, so check which option your zone uses.

Can a website builder or plugin change robots.txt for me?

Yes. Site builders and SEO or security plugins can write AI crawler rules into robots.txt, and some extend their bot lists over time. After each update, fetch the live file and read the groups that apply to the search crawlers you care about.

Why is my live robots.txt different from the file on my server?

Something in front of your server is changing it. Cloudflare's managed robots.txt, for example, prepends its own rules and content signals to your file in the same response. A CMS that generates robots.txt on request, or a host-level setting, can also replace or extend it.

How long until AI crawlers return after I fix it?

RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours, OpenAI says about 24 hours and Perplexity up to 24 hours. Firewall changes apply as soon as your provider deploys them, but crawlers come back on their own schedule.

Sources

More articles