ClaudeBot
ClaudeBot is Anthropic's crawler for web content that may be used to train its generative AI models. Disallowing it in robots.txt signals that your site's future content should be excluded from training. Blocking it alone does not stop Claude-SearchBot or Claude-User, which serve search and live answers.
What it does
Anthropic says ClaudeBot collects web content that could contribute to training its models, to help enhance their utility and safety. Disallowing ClaudeBot signals that your site's future materials should be excluded from training datasets.
Anthropic also supports the non-standard Crawl-delay directive, so you can slow ClaudeBot down instead of blocking it. Rules naming ClaudeBot in robots.txt take priority over Anthropic's general no-AI-training signal. Robots.txt rules apply per host, so add them to every subdomain you want covered.
In our 2026-11 study, 21.2% of popular sites with a robots.txt blocked it. →
robots.txt token
ClaudeBot
How to allow it
User-agent: ClaudeBot
Allow: /How to block it
User-agent: ClaudeBot
Disallow: /How to verify real visits
Anthropic publishes the IP addresses of its crawlers at claude.com/crawling/bots.json and says a crawler whose IP is on that list comes from Anthropic. Match the source IP against the list rather than trusting the user agent. Anthropic advises against using IP blocking to opt out, because it also stops the crawler from reading robots.txt.
Check if your site lets it in →
Anthropic's crawlers: ClaudeBot, Claude-SearchBot and Claude-User →