How to slow down AI crawlers without blocking them
Rate-limit instead of blocking. Answer excess crawler requests with 429 or 503 and a Retry-After header, give each crawler its own budget, keep /robots.txt out of every limit, and serve crawlers from cache. Crawl-delay in robots.txt helps only where the operator documents it: Anthropic and Bing do, Google says it does not support it, and OpenAI and Perplexity do not mention it. Keep the AI search crawlers allowed, or you also leave the answers they feed.
When a crawler sends thousands of requests an hour, the quick fix is Disallow: / or a firewall block. For a training crawler that can be a fair choice. For OAI-SearchBot, Claude-SearchBot or PerplexityBot it removes your pages from the AI answers you want to appear in. Most of the time the load can be cut without giving that up, with signals crawlers already understand: status codes, caching and, for some operators, Crawl-delay.
On this page
Which crawler is actually causing the load?
Look before you change anything. Count requests per user agent and per hour in your access logs, and check the source IPs against the ranges each vendor publishes; our guide to checking server logs for AI crawlers has the commands. Two findings come up often:
- The heavy traffic only claims to be an AI crawler. If the IPs are not in the vendor's list, rules for the real crawler will not help. Limit or block that traffic by its origin instead.
- A real crawler is stuck in a crawl trap, such as endless calendar pages, filter combinations or session IDs in URLs. Closing the trap with
Disallowrules for those paths saves more than any rate limit.
Also sort crawlers by job. Training crawlers such as GPTBot and ClaudeBot are not the ones that make you citable in AI search, so if one of them is the problem, disallowing it is a reasonable option. The rest of this guide is for crawlers you want to keep.
Does Crawl-delay work for AI crawlers?
Crawl-delay is not part of RFC 9309, so each operator decides whether to read it. This is what the documentation says as of November 2026:
- Anthropic: supported. Its help page says it supports the non-standard
Crawl-delayextension to limit crawling activity, withCrawl-delay: 1in aClaudeBotgroup as the example. - Microsoft Bing: supported. Bing's webmaster blog describes the value as a time window of 1 to 30 seconds in which
bingbotcrawls at most one page, and says a robots.txt crawl delay overrides the crawl settings in Bing Webmaster Tools. - Google: not supported. Google's robots.txt documentation names
crawl-delayas a field it does not support. - OpenAI and Perplexity: their crawler pages do not mention it, so its effect on
GPTBot,OAI-SearchBotorPerplexityBotis undocumented.
User-agent: ClaudeBot
Crawl-delay: 5
Disallow: /search/
Disallow: /cart/
User-agent: *
Disallow: /search/
Disallow: /cart/ Watch the group logic. A crawler that finds a group with its own name ignores the User-agent: * group completely, so a ClaudeBot group that holds only Crawl-delay would drop every Disallow you set for everyone else. Repeat them, as above. Bing describes Crawl-delay as an exception it reads from either group, but you should not assume that for other crawlers.
How do 429 and 503 responses slow crawlers down?
For load peaks, the status code is the signal crawlers are built to respect. 429 Too Many Requests, defined in RFC 6585, means the client sent too many requests in a given time, and may carry a Retry-After header saying how long to wait. 503 Service Unavailable says the server cannot cope right now.
Google documents this as the way to reduce Googlebot's crawl rate quickly: return 500, 503 or 429 instead of 200 to some crawl requests. It says this lowers the crawl rate for the whole hostname, and that Retry-After can hold a number of seconds or a date.
HTTP/1.1 429 Too Many Requests
Retry-After: 120
Content-Type: text/plain
Too many requests, please retry later. Keep it short. Google advises against doing this for longer than one or two days, because a URL that returns these codes for several days may be dropped from its index. For a lasting reduction, Google offers a special request through Search Console instead.
OpenAI, Anthropic and Perplexity do not document how their crawlers react to 429, 503 or Retry-After. The codes are the standard way to ask for less, but how fast each AI crawler backs off is not something you can look up.
How do I set a rate limit for AI crawlers?
Give each crawler its own budget instead of one limit for all traffic, and answer requests over it with 429. In nginx, the limit_req module does this. Requests with an empty key are not counted, so a map that returns a key only for crawler user agents leaves human visitors alone. Here each crawler gets 60 requests a minute with a burst of 20:
# In the http block
map $http_user_agent $ai_crawler {
default "";
~*GPTBot gptbot;
~*OAI-SearchBot oai-searchbot;
~*ClaudeBot claudebot;
~*Claude-SearchBot claude-searchbot;
~*PerplexityBot perplexitybot;
}
limit_req_zone $ai_crawler zone=ai_crawlers:10m rate=60r/m;
# In the server block
location = /robots.txt {
# serve robots.txt as before, without limit_req
}
location / {
limit_req zone=ai_crawlers burst=20 nodelay;
limit_req_status 429;
# ... your existing configuration
} The key here is the user agent, so anyone who copies a crawler's name shares that crawler's budget. That is acceptable for a rate limit. For an allow rule, match the vendor's published IP ranges as well.
On Cloudflare, rate limiting rules do the same job at the edge. On the Free plan you get one rule, counted per IP over 10 seconds, and the rule expression can use the path and the Verified Bot field. Counting per IP is coarse for crawlers that spread requests over many addresses, and higher plans add more counting options. The default response is 429, and you can set another 4xx code and a custom body. AI Crawl Control offers Allow, Block and, in a private beta, Charge for each crawler, but no rate setting.
Whatever you use, keep /robots.txt out of the limit. RFC 9309 says a server error for robots.txt means the crawler must assume the whole site is disallowed, and Google excludes 429 from the 4xx codes it treats as a missing file.
Can caching reduce crawler load?
Often more than a rate limit. Crawlers mostly ask for public pages that are the same for every visitor, so serving them from a CDN or page cache turns expensive renders into cheap hits. Check that your cache does not keep a separate, uncached variant for crawler user agents.
Conditional requests help too. Google says its crawlers support HTTP caching with ETag and If-None-Match, and with Last-Modified and If-Modified-Since, and use ETag when both are present. The OpenAI, Anthropic and Perplexity crawler pages do not say whether their crawlers send conditional requests, so count on that saving for Google only.
What should I avoid when slowing crawlers down?
- Blocking a vendor's IP ranges at the firewall. Anthropic warns that this may not work correctly or keep an opt-out in place, because it stops its bots from reading your robots.txt.
- Serving
5xxor429for days. Google says URLs that keep returning them may be dropped from its index. - Using bot protection as a throttle. Cloudflare's Bot Fight Mode issues computationally expensive challenges to bots and cannot be skipped with WAF custom rules, so you cannot exempt the crawlers you want to keep.
- Disallowing a search crawler to save bandwidth. A
DisallowforOAI-SearchBot,Claude-SearchBotorPerplexityBottakes you out of the answers those crawlers feed.
How can I check that crawlers still get through?
Rate limits are easy to set too tight. The free AISeeWhy check fetches a page as 16 AI and search crawlers next to a real browser, applies your robots.txt per crawler and recognises firewall challenge pages, so a rule that answers crawlers differently from browsers shows up. One request per crawler will not trip a sensible limit, which is the point: run it after changing rules to confirm normal requests still pass. The free AI crawler IP check tells you whether an IP from your logs really belongs to GPTBot or another crawler.
Next steps
- robots.txt tester and checker for AI crawlers
- AI crawler IP check
- See an example full report
- AI crawler monitoring
- Plans and prices
Crawlers in this article
Frequently asked questions
Does Google support Crawl-delay?
No. Google's robots.txt documentation lists crawl-delay among the fields it does not support. To slow Googlebot briefly, return 503, 500 or 429 to some requests; for a longer reduction, Google offers a special request through Search Console.
Does ClaudeBot respect Crawl-delay?
Anthropic says it supports the non-standard Crawl-delay extension. Put it in a ClaudeBot group and repeat any Disallow rules from your User-agent: * group there.
Do GPTBot and PerplexityBot support Crawl-delay?
OpenAI's and Perplexity's crawler pages do not mention Crawl-delay, so its effect is undocumented. Use a server-side limit that answers excess requests with 429.
Will a 429 response hurt my rankings?
Not for a short period. Google documents 429 and 503 as a way to reduce crawling temporarily, but warns that URLs returning them for several days may be dropped from its index.
Should I block GPTBot to reduce load?
You can. OpenAI says GPTBot crawls content that may be used for training and that each robots.txt setting is independent, so disallowing it does not remove you from ChatGPT search, which uses OAI-SearchBot. Check your logs first to make sure the load comes from OpenAI's addresses.
Sources
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Bing Webmaster Blog: To crawl or not to crawl, that is BingBot's question
- Google: How Google interprets the robots.txt specification
- Google: Reduce the Google crawl rate
- Google: Overview of Google crawlers and fetchers (HTTP caching)
- RFC 9309: Robots Exclusion Protocol
- RFC 6585: Additional HTTP Status Codes (429)
- nginx: Module ngx_http_limit_req_module
- Cloudflare: Rate limiting rules
- Cloudflare: Rate limiting parameters
- Cloudflare: Manage AI crawlers (AI Crawl Control)
- Cloudflare: Bot Fight Mode
- OpenAI: Overview of OpenAI crawlers
- Perplexity: Perplexity crawlers