Direktoryo ng AI crawler
Aling mga bot ang nagbabasa ng site mo para sa AI training, AI search at live na sagot, ano ang ginagawa ng bawat isa ayon sa vendor nito, at paano ito papayagan, iba-block o beripikahin.
| Crawler | Kumpanya | Layunin |
|---|---|---|
| GPTBot | OpenAI | AI training |
| OAI-SearchBot | OpenAI | AI search |
| ChatGPT-User | OpenAI | Pagkuha na hiniling ng user |
| ClaudeBot | Anthropic | AI training |
| Claude-SearchBot | Anthropic | AI search |
| Claude-User | Anthropic | Pagkuha na hiniling ng user |
| PerplexityBot | Perplexity | AI search |
| Perplexity-User | Perplexity | Pagkuha na hiniling ng user |
| Google-Extended | AI training | |
| Googlebot | Search engine | |
| Google-CloudVertexBot | Pagkuha na hiniling ng user | |
| bingbot | Microsoft | Search engine |
| Applebot | Apple | Search engine |
| Applebot-Extended | Apple | AI training |
| CCBot | Common Crawl | AI training |
| meta-externalagent | Meta | AI training |
| meta-externalfetcher | Meta | Pagkuha na hiniling ng user |
| Amazonbot | Amazon | AI training |
| DuckAssistBot | DuckDuckGo | AI search |
| MistralAI-User | Mistral AI | Pagkuha na hiniling ng user |