Claude-SearchBot
Claude-SearchBot crawls the web to improve the relevance and accuracy of search results in Claude. Anthropic says blocking it prevents your content from being indexed for search, which may reduce your visibility and accuracy in user search results. Allow it if you want Claude to find and cite your pages.
What it does
Anthropic uses Claude-SearchBot to navigate the web and improve the quality of search results for Claude users. It is separate from ClaudeBot, the training crawler, so you can allow search while opting out of training.
Anthropic says disabling Claude-SearchBot prevents indexing of your content for search optimization, which may reduce your site's visibility and accuracy in user search results. Anthropic's bots honor robots.txt and respect anti-circumvention measures, so a CAPTCHA or challenge page ends the visit.
In our 2026-11 study, 12.9% of popular sites with a robots.txt blocked it. →
robots.txt token
Claude-SearchBot
How to allow it
User-agent: Claude-SearchBot
Allow: /How to block it
User-agent: Claude-SearchBot
Disallow: /How to verify real visits
Anthropic publishes the IP addresses of its crawlers at claude.com/crawling/bots.json and says a crawler whose IP is on that list comes from Anthropic. Match the source IP against the list rather than trusting the user agent. Anthropic advises against using IP blocking to opt out, because it also stops the crawler from reading robots.txt.
Check if your site lets it in →
Anthropic's crawlers: ClaudeBot, Claude-SearchBot and Claude-User →