User agents na safu za IP za crawler za AI: GPTBot, ClaudeBot, PerplexityBot na nyinginezo
GPTBot hujitambulisha kwa user agent yenye GPTBot/1.4; +https://openai.com/gptbot, na OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple na Common Crawl kila moja huchapisha orodha ya JSON ya safu za IP ambazo crawler zao hutumia. Ombi ni halisi tu pale vyote viwili vinalingana: user agent inataja crawler na IP ya chanzo iko katika orodha ya mtoa huduma huyo, au, kwa Googlebot, bingbot, Applebot na CCBot, inapita ukaguzi wa reverse na forward DNS. User agent peke yake haithibitishi chochote, kwa sababu mtu yeyote anaweza kuituma.
Unapoona GPTBot au ClaudeBot katika kumbukumbu zako, au unataka sheria ya firewall inayoruhusu crawler za utafutaji wa AI, unahitaji vitu viwili: mfuatano halisi ambao kila crawler hutuma, na njia ya kuthibitisha kwamba ombi lilitoka kwa kampuni inayotajwa. Hapa chini kuna mifuatano kama kila mtoa huduma anavyoichapisha leo, orodha rasmi ya IP au mbinu ya DNS kwa kila moja, na hati mbili unazoweza kuendesha kukagua anwani.
Kwenye ukurasa huu
- Mifuatano ya user agent ya GPTBot, ClaudeBot, PerplexityBot na crawler nyingine za AI ni ipi?
- Crawler ipi inafanya nini, na majina yapi ni tokeni za robots.txt pekee?
- Watoa huduma wa crawler za AI huchapisha safu zao za IP wapi?
- Nitakaguaje anwani ya IP dhidi ya orodha rasmi za crawler?
- Nitathibitishaje Googlebot, bingbot au Applebot kwa reverse DNS?
- Kwa nini ukaguzi wa user agent hautoshi kutambua GPTBot?
- AISeeWhy inawezaje kusaidia kuthibitisha trafiki ya crawler za AI?
Mifuatano ya user agent ya GPTBot, ClaudeBot, PerplexityBot na crawler nyingine za AI ni ipi?
Hizi zimenakiliwa kutoka nyaraka za watoa huduma wenyewe. OpenAI na Perplexity zinaeleza kwamba namba za toleo zinaweza kubadilika, Google na Microsoft huweka toleo la sasa la kivinjari pale W.X.Y.Z inapoonekana, na Apple inasema mara kwa mara husasisha sehemu ya kivinjari. Linganisha kwa tokeni ya crawler, kama GPTBot au bingbot, si kwa mfuatano mzima.
# OpenAI (version numbers may change)
GPTBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OAI-SearchBot: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OAI-AdsBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
# Perplexity
PerplexityBot: Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-User: Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
# Google (W.X.Y.Z is the current Chrome version)
Googlebot Smartphone: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot Desktop: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Googlebot Desktop: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
# Microsoft Bing (W.X.Y.Z is the current Edge version)
bingbot desktop: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
bingbot: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
bingbot mobile: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
# Apple (the browser part changes from time to time)
Applebot desktop: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Applebot mobile: Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)
# Common Crawl
CCBot: CCBot/2.0 (https://commoncrawl.org/faq/)
# Anthropic and Meta: match the token, see the notes below
ClaudeBot, Claude-SearchBot, Claude-User
meta-externalagent/1.1, meta-webindexer/1.1, meta-externalfetcher/1.1 Makala ya msaada ya Anthropic inataja ClaudeBot, Claude-SearchBot na Claude-User lakini haichapishi mifuatano kamili ya user agent, kwa hivyo linganisha kwa tokeni hizo. Ukurasa wa crawler wa Meta unaorodhesha mifuatano inayoanza na tokeni zilizoonyeshwa, ikifuatiwa na kiungo cha nyaraka zake, na unasema mifuatano inayoonekana katika kumbukumbu zako inaweza kutofautiana kidogo.
Crawler ipi inafanya nini, na majina yapi ni tokeni za robots.txt pekee?
GPTBothukusanya maudhui yanayoweza kutumika kufunza miundo ya msingi ya OpenAI.OAI-SearchBothuonyesha tovuti katika utafutaji wa ChatGPT.ChatGPT-Userhupakua kurasa kwa vitendo ambavyo mtumiaji anaanzisha, na OpenAI inasema sheria za robots.txt huenda zisiihusu.OAI-AdsBothukagua kurasa zilizowasilishwa kama matangazo ya ChatGPT.ClaudeBothukusanya data kwa miundo ya Anthropic,Claude-SearchBothuboresha matokeo ya utafutaji, naClaude-Userhupakua kurasa mtu anapomuuliza Claude swali.PerplexityBothuonyesha na kuunganisha tovuti katika utafutaji wa Perplexity na haitumiki kufunza miundo ya msingi.Perplexity-Userhupakua kurasa mtumiaji anapouliza swali.meta-externalagenthupakua kwa ajili ya kufunza miundo ya AI au kuorodhesha maudhui,meta-webindexerhuorodhesha maudhui kwa utafutaji wa Meta AI, nameta-externalfetcherhupakua viungo kwa ombi la mtumiaji.CCBotni crawler ya Common Crawl, hifadhi ya umma ya wavuti.
Majina mawili hayaonekani kamwe katika kumbukumbu zako. Google-Extended ni tokeni ya robots.txt inayodhibiti kama maudhui ambayo Google inapakua yanaweza kutumika kwa miundo ya Gemini na kwa grounding; Google inasema haiathiri kujumuishwa katika Google Search. Applebot-Extended ni, kwa maneno ya Apple, user agent ya pili isiyopakua kurasa za wavuti; inaamua tu jinsi Apple inavyoweza kutumia data ambayo Applebot ilikusanya. Ombi linalodai kuwa mojawapo kati ya hizo si halisi.
Watoa huduma wa crawler za AI huchapisha safu zao za IP wapi?
Watoa huduma wengi huchapisha faili ya JSON yenye umbo lilelile: safu (array) ya prefixes ya vitu vyenye ipv4Prefix au ipv6Prefix, pamoja na creationTime inayokuambia ni ya karibu kiasi gani.
- OpenAI:
https://openai.com/gptbot.json,https://openai.com/searchbot.json,https://openai.com/chatgpt-user.jsonnahttps://openai.com/adsbot.json, moja kwa kila wakala. - Anthropic:
https://claude.com/crawling/bots.json, orodha moja kwa crawler zake. Anthropic inasema anwani iliyo kwenye orodha hii inaonyesha crawler inatoka Anthropic, na pia kwamba kuzuia kwa IP huenda kusihakikishe kwa uhakika kujiondoa. - Perplexity:
https://www.perplexity.ai/perplexitybot.jsonnahttps://www.perplexity.ai/perplexity-user.json. - Google:
https://developers.google.com/static/crawling/ipranges/common-crawlers.jsonkwa Googlebot, na faili tofauti kwa crawler za kesi maalum na fetchers zinazoanzishwa na mtumiaji. Reverse DNS pia inafanya kazi. - Apple:
https://search.developer.apple.com/applebot.json, au reverse DNS katikaapplebot.apple.com. - Common Crawl:
https://index.commoncrawl.org/ccbot.json. Kwa IPv4 pia inapendekeza reverse DNS iliyothibitishwa kwa forward chini yacrawl.commoncrawl.org; anwani za IPv6 hazina reverse DNS. - Microsoft:
https://www.bing.com/toolbox/bingbot.json, ambayo Bing inakuomba uisasishe kila siku kwa sababu inaweza kubadilika wakati wowote, au reverse DNS inayoishia nasearch.msn.com. Bing pia inatoa zana ya umma ya Verify Bingbot. - Meta: ukurasa wake wa crawler hauchapishi orodha ya IP.
Nitakaguaje anwani ya IP dhidi ya orodha rasmi za crawler?
Hati hii inapakia kila orodha na kuripoti ipi ina anwani hiyo. Inahitaji Python 3 pekee, na inashughulikia IPv4 na IPv6.
python3 - 20.171.206.5 <<'EOF'
import ipaddress, json, sys, urllib.request
LISTS = {
"GPTBot": "https://openai.com/gptbot.json",
"OAI-SearchBot": "https://openai.com/searchbot.json",
"ChatGPT-User": "https://openai.com/chatgpt-user.json",
"ClaudeBot, Claude-SearchBot, Claude-User": "https://claude.com/crawling/bots.json",
"PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
"Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
"Googlebot": "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
"bingbot": "https://www.bing.com/toolbox/bingbot.json",
"Applebot": "https://search.developer.apple.com/applebot.json",
"CCBot": "https://index.commoncrawl.org/ccbot.json",
}
ip = ipaddress.ip_address(sys.argv[1])
found = False
for name, url in LISTS.items():
req = urllib.request.Request(url, headers={"User-Agent": "ip-list-check"})
try:
prefixes = json.load(urllib.request.urlopen(req, timeout=15))["prefixes"]
except Exception as err:
print(f"{name}: list not loaded ({err})")
continue
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
if any(ip in net for net in nets if net.version == ip.version):
print(f"{ip} is listed for {name}")
found = True
if not found:
print(f"{ip} is not in any of these lists")
EOF Orodha hubadilika. Kila faili ina creationTime yake, Perplexity inapendekeza kusasisha sheria zako za firewall kiotomatiki, na Bing inakuomba usasishe orodha yake kila siku. Ukitumia orodha hizo katika firewall, zipakue kwa ratiba badala ya kuzinakili mara moja.
Nitathibitishaje Googlebot, bingbot au Applebot kwa reverse DNS?
Google, Microsoft, Apple na Common Crawl zinaeleza ukaguzi uleule wa hatua mbili: tafuta jina la anwani ya IP, hakikisha linaishia na kikoa cha mtoa huduma, kisha tafuta jina hilo na uhakikishe linarudisha IP ileile. Hatua zote mbili ni muhimu, kwa sababu yeyote anayedhibiti safu ya IP anaweza kuweka jina lolote la reverse DNS analotaka, lakini hawezi kufanya kikoa cha mtoa huduma kielekeze kwake.
verify_crawler_ip() {
ip="$1"
name=$(host "$ip" | awk '/domain name pointer/ {print $NF}' | sed 's/\.$//' | head -n 1)
if [ -z "$name" ]; then echo "$ip: no reverse DNS name"; return 1; fi
case "$name" in
*.googlebot.com|*.search.msn.com|*.applebot.apple.com|*.crawl.commoncrawl.org) ;;
*) echo "$ip -> $name: not a crawler hostname"; return 1 ;;
esac
if host "$name" | grep -qwF -- "$ip"; then
echo "$ip -> $name -> $ip: verified"
else
echo "$ip -> $name: forward lookup does not return $ip"; return 1
fi
}
verify_crawler_ip 66.249.66.1
# 66.249.66.1 -> crawl-66-249-66-1.googlebot.com -> 66.249.66.1: verified Google inakubali googlebot.com, google.com na googleusercontent.com kwa aina zake zote za crawler; kipengele (function) kilicho hapo juu kinakubali googlebot.com pekee, ambacho ndicho Googlebot yenyewe hutumia katika mifano ya Google. Kwa crawler na fetchers nyingine za Google, tumia orodha zake za JSON. Utafutaji wa DNS huchukua muda, kwa hivyo uendeshe unapopitia kumbukumbu au uhifadhi matokeo, badala ya kila ombi la moja kwa moja.
Kwa nini ukaguzi wa user agent hautoshi kutambua GPTBot?
User agent ni kichwa (header) ambacho mteja huchagua, kwa hivyo mtu yeyote anaweza kutuma GPTBot au Googlebot. Common Crawl inaonya kwamba baadhi ya crawler hudai kwa uongo kuwa CCBot, na Bing inachapisha mwongozo wa kukagua kama trafiki inayodai kuwa bingbot kweli inatoka Bing. Kutambua GPTBot kwa uhakika kunamaanisha kukagua vyote viwili: tokeni katika user agent na IP ya chanzo katika gptbot.json. Ombi lenye tokeni kutoka anwani isiyoorodheshwa ni la mwigaji, na kulizuia hakuathiri OpenAI.
Kinyume chake pia hutokea. Kama firewall yako inaruhusu user agent pekee, waigaji wanaingia; kama inaruhusu safu ya IP pekee, wakala yeyote kutoka mtandao wa mtoa huduma huyo anapata matibabu yaleyale, hata yule uliyetaka kumzuia. Kulinganisha vyote viwili ndiyo sheria ya kutegemewa.
AISeeWhy inawezaje kusaidia kuthibitisha trafiki ya crawler za AI?
Ukaguzi wa bure wa IP ya crawler za AI wa AISeeWhy unakuambia, bila akaunti, kama anwani ya IP ni ya GPTBot au crawler nyingine ya AI. Ukaguzi wa bure wa tovuti kisha unaomba ukurasa wako kama crawler 16 za AI na za utafutaji kando ya kivinjari halisi, ili uone kile ambacho crawler halisi zingepokea.
Hatua zinazofuata
- Kikagua na kijaribu cha robots.txt kwa crawler za AI
- Ukaguzi wa IP ya crawler za AI
- Tazama mfano wa ripoti kamili
- Ufuatiliaji wa crawler za AI
- Mipango na bei
Crawler katika makala hii
Maswali yanayoulizwa mara kwa mara
User agent ya GPTBot ni ipi?
OpenAI inachapisha Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot na inaeleza kwamba toleo linaweza kubadilika. Linganisha kwa GPTBot na uthibitishe IP dhidi ya https://openai.com/gptbot.json.
User agent ya OAI-SearchBot ni ipi?
Mfuatano wa mtindo wa Chrome unaoishia na compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Anwani zake zimeorodheshwa katika https://openai.com/searchbot.json. OpenAI huitumia kuonyesha tovuti katika utafutaji wa ChatGPT, kazi tofauti na ya GPTBot.
Je, Anthropic inachapisha anwani za IP za ClaudeBot?
Ndiyo. Anthropic inaunganisha na https://claude.com/crawling/bots.json na inasema IP ya chanzo iliyo kwenye orodha hiyo inaonyesha crawler inatoka Anthropic. Makala yake ya msaada inataja tokeni za bots lakini si mifuatano kamili ya user agent.
Je, naweza kuona Google-Extended au Applebot-Extended katika kumbukumbu zangu?
Hapana. Zote mbili ni tokeni za robots.txt, si crawler. Google na Apple hupakua kurasa kwa Googlebot na Applebot na hutumia tokeni hizi tu kuamua jinsi maudhui yanavyoweza kutumika kwa AI.
Orodha ya anwani za IP za bingbot iko wapi?
Bing inaichapisha kwenye https://www.bing.com/toolbox/bingbot.json na inakuomba uisasishe kila siku. Mwongozo wake unapendekeza ukaguzi wa reverse na forward DNS, ambapo jina lazima liishie na search.msn.com.
Vyanzo
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl the web? What is Claude Bot?
- Perplexity: Perplexity crawlers
- Google: Google's common crawlers
- Google: Verifying Googlebot and other Google crawlers
- Bing Webmaster Tools: Which crawlers does Bing use?
- Bing Webmaster Tools: How to verify Bingbot
- Microsoft Bing: Verify Bingbot tool
- Apple: About Applebot
- Meta: Meta web crawlers
- Common Crawl: CCBot