AISeeWhy
EN
Start free trial

Anthropic's crawlers: ClaudeBot, Claude-SearchBot and Claude-User

Published

Short answer

Anthropic runs three agents: ClaudeBot collects content that may be used to train its models, Claude-SearchBot crawls to improve Claude's search results, and Claude-User fetches pages when a Claude user's question needs them. Each has its own robots.txt token, and Anthropic says all three honor robots.txt. Blocking ClaudeBot alone does not stop search or user fetches.

Anthropic documents its crawlers on a single help page, and the structure mirrors other AI vendors: one agent for training, one for search and one for live fetches on a user's behalf. If you want Claude to be able to find and quote your pages, the last two matter most.

On this page
  1. What does ClaudeBot do?
  2. What does Claude-SearchBot do?
  3. What does Claude-User do?
  4. How do I write robots.txt rules for Claude?
  5. How does Anthropic's setup compare with OpenAI and Perplexity?
  6. Should I block Anthropic by IP address?
  7. Why would Claude still fail to read my page?

What does ClaudeBot do?

ClaudeBot collects web content that could contribute to training Anthropic's generative AI models. According to Anthropic, disallowing it signals that your site's future content should be excluded from training datasets.

Anthropic also says it honors a machine-readable no-AI-training signal based on an IETF draft, and that rules naming ClaudeBot in robots.txt take priority over that general setting. As with other training crawlers, blocking it is a policy choice.

What does Claude-SearchBot do?

Claude-SearchBot crawls the web to improve the relevance and accuracy of search results in Claude. Anthropic says blocking it prevents your content from being indexed for search, which may reduce your visibility and accuracy in user search results.

This is the agent to allow if you want Claude's search features to know about your pages before anyone asks about them. Because it is a search crawler, it benefits from the same things any search crawler does: a sitemap listed in robots.txt, clean internal links, and pages that return their content in the first response.

What does Claude-User do?

Claude-User accesses a website when a Claude user asks a question that requires it, for example when someone shares a URL or asks Claude to look something up. Anthropic says blocking it prevents retrieval in response to user queries, which may reduce your visibility in user-directed web search.

Note a difference from some other vendors. Anthropic states that its bots honor robots.txt directives, and it describes blocking Claude-User through robots.txt. OpenAI and Perplexity, by contrast, say their user-requested fetchers may not follow robots.txt.

How do I write robots.txt rules for Claude?

Each agent has its own token, and a group that names a crawler replaces the * group for that crawler under RFC 9309. A common choice is to allow search and user fetches and opt out of training:

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Anthropic also supports Crawl-delay as a non-standard extension, for example Crawl-delay: 1 under User-agent: ClaudeBot, if you want to slow crawling instead of blocking it. Rules must sit in the robots.txt at the top-level directory of each host, so add them to every subdomain you want covered.

How does Anthropic's setup compare with OpenAI and Perplexity?

The three vendors follow the same pattern with different names, which makes a combined robots.txt easy to reason about. The main difference is how each treats robots.txt for live fetches on a user's request.

If you write one robots.txt for all three, put the search and user tokens in one group and the training tokens in another. That keeps the policy readable and makes it obvious which choice you made for each purpose.

Should I block Anthropic by IP address?

Anthropic advises against it. Its documentation says that blocking IP addresses may not work correctly or persistently guarantee an opt-out, because it also stops the crawler from reading your robots.txt. Use robots.txt to state your policy.

Anthropic does publish its crawler IP addresses at claude.com/crawling/bots.json. The useful way to use that list is the opposite of blocking: let your firewall verify that a request claiming to be Claude-SearchBot or Claude-User really comes from Anthropic, and let it through.

Why would Claude still fail to read my page?

Even with permissive robots.txt, your firewall or CDN can answer Anthropic's agents with a 403 or a challenge page. Anthropic says its bots respect anti-circumvention technologies and will not try to bypass CAPTCHAs, so a challenge page simply ends the visit. Pages whose text only appears after JavaScript runs can also arrive nearly empty.

AISeeWhy requests your page as ClaudeBot, Claude-SearchBot and Claude-User next to a browser and checks your robots.txt rules for each, so you can see whether the policy you wrote is the one your server enforces.

Frequently asked questions

Does blocking ClaudeBot remove my site from Claude's answers?

Not by itself. Search indexing and user-requested fetches use Claude-SearchBot and Claude-User, which have their own robots.txt tokens.

Does Claude-User follow robots.txt?

Anthropic says its bots honor robots.txt and describes blocking Claude-User there, with the effect that Claude cannot retrieve your pages for user questions.

Will Anthropic's crawlers solve a CAPTCHA?

No. Anthropic says its bots respect anti-circumvention technologies and will not attempt to bypass CAPTCHAs.

Can I slow ClaudeBot down instead of blocking it?

Yes. Anthropic supports Crawl-delay as a non-standard extension, for example Crawl-delay: 1 in the ClaudeBot group of your robots.txt.

How can I verify that a request really comes from Anthropic?

Compare the source IP with the list Anthropic publishes at claude.com/crawling/bots.json. A user agent alone can be copied by anyone.

Sources

More articles