How to slow down AI crawlers without blocking them

Published

Short answer

Rate-limit instead of blocking. Answer excess crawler requests with 429 or 503 and a Retry-After header, give each crawler its own budget, keep /robots.txt out of every limit, and serve crawlers from cache. Crawl-delay in robots.txt helps only where the operator documents it: Anthropic and Bing do, Google says it does not support it, and OpenAI and Perplexity do not mention it. Keep the AI search crawlers allowed, or you also leave the answers they feed.

When a crawler sends thousands of requests an hour, the quick fix is Disallow: / or a firewall block. For a training crawler that can be a fair choice. For OAI-SearchBot, Claude-SearchBot or PerplexityBot it removes your pages from the AI answers you want to appear in. Most of the time the load can be cut without giving that up, with signals crawlers already understand: status codes, caching and, for some operators, Crawl-delay.

On this page
  1. Which crawler is actually causing the load?
  2. Does Crawl-delay work for AI crawlers?
  3. How do 429 and 503 responses slow crawlers down?
  4. How do I set a rate limit for AI crawlers?
  5. Can caching reduce crawler load?
  6. What should I avoid when slowing crawlers down?
  7. How can I check that crawlers still get through?

Which crawler is actually causing the load?

Look before you change anything. Count requests per user agent and per hour in your access logs, and check the source IPs against the ranges each vendor publishes; our guide to checking server logs for AI crawlers has the commands. Two findings come up often:

Also sort crawlers by job. Training crawlers such as GPTBot and ClaudeBot are not the ones that make you citable in AI search, so if one of them is the problem, disallowing it is a reasonable option. The rest of this guide is for crawlers you want to keep.

Does Crawl-delay work for AI crawlers?

Crawl-delay is not part of RFC 9309, so each operator decides whether to read it. This is what the documentation says as of November 2026:

User-agent: ClaudeBot
Crawl-delay: 5
Disallow: /search/
Disallow: /cart/

User-agent: *
Disallow: /search/
Disallow: /cart/

Watch the group logic. A crawler that finds a group with its own name ignores the User-agent: * group completely, so a ClaudeBot group that holds only Crawl-delay would drop every Disallow you set for everyone else. Repeat them, as above. Bing describes Crawl-delay as an exception it reads from either group, but you should not assume that for other crawlers.

How do 429 and 503 responses slow crawlers down?

For load peaks, the status code is the signal crawlers are built to respect. 429 Too Many Requests, defined in RFC 6585, means the client sent too many requests in a given time, and may carry a Retry-After header saying how long to wait. 503 Service Unavailable says the server cannot cope right now.

Google documents this as the way to reduce Googlebot's crawl rate quickly: return 500, 503 or 429 instead of 200 to some crawl requests. It says this lowers the crawl rate for the whole hostname, and that Retry-After can hold a number of seconds or a date.

HTTP/1.1 429 Too Many Requests
Retry-After: 120
Content-Type: text/plain

Too many requests, please retry later.

Keep it short. Google advises against doing this for longer than one or two days, because a URL that returns these codes for several days may be dropped from its index. For a lasting reduction, Google offers a special request through Search Console instead.

OpenAI, Anthropic and Perplexity do not document how their crawlers react to 429, 503 or Retry-After. The codes are the standard way to ask for less, but how fast each AI crawler backs off is not something you can look up.

How do I set a rate limit for AI crawlers?

Give each crawler its own budget instead of one limit for all traffic, and answer requests over it with 429. In nginx, the limit_req module does this. Requests with an empty key are not counted, so a map that returns a key only for crawler user agents leaves human visitors alone. Here each crawler gets 60 requests a minute with a burst of 20:

# In the http block
map $http_user_agent $ai_crawler {
    default            "";
    ~*GPTBot           gptbot;
    ~*OAI-SearchBot    oai-searchbot;
    ~*ClaudeBot        claudebot;
    ~*Claude-SearchBot claude-searchbot;
    ~*PerplexityBot    perplexitybot;
}
limit_req_zone $ai_crawler zone=ai_crawlers:10m rate=60r/m;

# In the server block
location = /robots.txt {
    # serve robots.txt as before, without limit_req
}
location / {
    limit_req zone=ai_crawlers burst=20 nodelay;
    limit_req_status 429;
    # ... your existing configuration
}

The key here is the user agent, so anyone who copies a crawler's name shares that crawler's budget. That is acceptable for a rate limit. For an allow rule, match the vendor's published IP ranges as well.

On Cloudflare, rate limiting rules do the same job at the edge. On the Free plan you get one rule, counted per IP over 10 seconds, and the rule expression can use the path and the Verified Bot field. Counting per IP is coarse for crawlers that spread requests over many addresses, and higher plans add more counting options. The default response is 429, and you can set another 4xx code and a custom body. AI Crawl Control offers Allow, Block and, in a private beta, Charge for each crawler, but no rate setting.

Whatever you use, keep /robots.txt out of the limit. RFC 9309 says a server error for robots.txt means the crawler must assume the whole site is disallowed, and Google excludes 429 from the 4xx codes it treats as a missing file.

Can caching reduce crawler load?

Often more than a rate limit. Crawlers mostly ask for public pages that are the same for every visitor, so serving them from a CDN or page cache turns expensive renders into cheap hits. Check that your cache does not keep a separate, uncached variant for crawler user agents.

Conditional requests help too. Google says its crawlers support HTTP caching with ETag and If-None-Match, and with Last-Modified and If-Modified-Since, and use ETag when both are present. The OpenAI, Anthropic and Perplexity crawler pages do not say whether their crawlers send conditional requests, so count on that saving for Google only.

What should I avoid when slowing crawlers down?

How can I check that crawlers still get through?

Rate limits are easy to set too tight. The free AISeeWhy check fetches a page as 16 AI and search crawlers next to a real browser, applies your robots.txt per crawler and recognises firewall challenge pages, so a rule that answers crawlers differently from browsers shows up. One request per crawler will not trip a sensible limit, which is the point: run it after changing rules to confirm normal requests still pass. The free AI crawler IP check tells you whether an IP from your logs really belongs to GPTBot or another crawler.

Free tools

Next steps

Crawlers in this article

Frequently asked questions

Does Google support Crawl-delay?

No. Google's robots.txt documentation lists crawl-delay among the fields it does not support. To slow Googlebot briefly, return 503, 500 or 429 to some requests; for a longer reduction, Google offers a special request through Search Console.

Does ClaudeBot respect Crawl-delay?

Anthropic says it supports the non-standard Crawl-delay extension. Put it in a ClaudeBot group and repeat any Disallow rules from your User-agent: * group there.

Do GPTBot and PerplexityBot support Crawl-delay?

OpenAI's and Perplexity's crawler pages do not mention Crawl-delay, so its effect is undocumented. Use a server-side limit that answers excess requests with 429.

Will a 429 response hurt my rankings?

Not for a short period. Google documents 429 and 503 as a way to reduce crawling temporarily, but warns that URLs returning them for several days may be dropped from its index.

Should I block GPTBot to reduce load?

You can. OpenAI says GPTBot crawls content that may be used for training and that each robots.txt setting is independent, so disallowing it does not remove you from ChatGPT search, which uses OAI-SearchBot. Check your logs first to make sure the load comes from OpenAI's addresses.

Sources

More articles