Claude-User
Claude-User visits a website when a Claude user asks a question that needs it, such as reading a shared link. Anthropic says its bots honor robots.txt, and that blocking Claude-User prevents retrieval of your content in response to user queries, which may reduce your visibility in user-directed web search.
What it does
Anthropic uses Claude-User to support Claude users: when someone asks Claude a question, it may access a website to answer it. These are individual fetches on a user's behalf, not a general crawl.
Unlike OpenAI and Perplexity, which say their user-requested fetchers may not follow robots.txt, Anthropic states that its bots honor robots.txt directives and describes blocking Claude-User there. Doing so prevents Claude from retrieving your pages when users ask about them.
In our 2026-11 study, 13.1% of popular sites with a robots.txt blocked it. →
robots.txt token
Claude-User
How to allow it
User-agent: Claude-User
Allow: /How to block it
User-agent: Claude-User
Disallow: /How to verify real visits
Anthropic publishes the IP addresses of its crawlers at claude.com/crawling/bots.json and says a crawler whose IP is on that list comes from Anthropic. Match the source IP against the list rather than trusting the user agent. Anthropic advises against using IP blocking to opt out, because it also stops the crawler from reading robots.txt.
Check if your site lets it in →
Anthropic's crawlers: ClaudeBot, Claude-SearchBot and Claude-User →