AI crawler directory
Which bots read your site for AI training, AI search and live answers, what each one does according to its vendor, and how to allow, block or verify it.
| Crawler | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | AI training |
| OAI-SearchBot | OpenAI | AI search |
| ChatGPT-User | OpenAI | User-requested fetch |
| ClaudeBot | Anthropic | AI training |
| Claude-SearchBot | Anthropic | AI search |
| Claude-User | Anthropic | User-requested fetch |
| PerplexityBot | Perplexity | AI search |
| Perplexity-User | Perplexity | User-requested fetch |
| Google-Extended | AI training | |
| Googlebot | Search engine | |
| Google-CloudVertexBot | User-requested fetch | |
| bingbot | Microsoft | Search engine |
| Applebot | Apple | Search engine |
| Applebot-Extended | Apple | AI training |
| CCBot | Common Crawl | AI training |
| meta-externalagent | Meta | AI training |
| meta-externalfetcher | Meta | User-requested fetch |
| Amazonbot | Amazon | AI training |
| DuckAssistBot | DuckDuckGo | AI search |
| MistralAI-User | Mistral AI | User-requested fetch |