AI crawler user agents and IP ranges: GPTBot, ClaudeBot, PerplexityBot and more
GPTBot identifies itself with a user agent containing GPTBot/1.4; +https://openai.com/gptbot, and OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple and Common Crawl each publish a JSON list of the IP ranges their crawlers use. A request is real only when both match: the user agent names the crawler and the source IP is in that vendor's list, or, for Googlebot, bingbot, Applebot and CCBot, passes a reverse and forward DNS check. A user agent on its own proves nothing, because anyone can send it.
When you see GPTBot or ClaudeBot in your logs, or want a firewall rule that lets AI search crawlers through, you need two things: the exact string each crawler sends, and a way to prove a request came from the company it names. Below are the strings as each vendor publishes them today, the official IP list or DNS method for each, and two scripts you can run to check an address.
On this page
- What are the user agent strings of GPTBot, ClaudeBot, PerplexityBot and other AI crawlers?
- Which crawler does what, and which names are only robots.txt tokens?
- Where do AI crawler vendors publish their IP ranges?
- How do I check an IP address against the official crawler lists?
- How do I verify Googlebot, bingbot or Applebot with reverse DNS?
- Why is a user agent check not enough to detect GPTBot?
- How can AISeeWhy help verify AI crawler traffic?
What are the user agent strings of GPTBot, ClaudeBot, PerplexityBot and other AI crawlers?
These are copied from the vendors' own documentation. OpenAI and Perplexity note that version numbers can change, Google and Microsoft put the current browser version where W.X.Y.Z appears, and Apple says it occasionally updates the browser part. Match on the crawler token, such as GPTBot or bingbot, not on the whole string.
# OpenAI (version numbers may change)
GPTBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OAI-SearchBot: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OAI-AdsBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
# Perplexity
PerplexityBot: Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-User: Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
# Google (W.X.Y.Z is the current Chrome version)
Googlebot Smartphone: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot Desktop: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Googlebot Desktop: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
# Microsoft Bing (W.X.Y.Z is the current Edge version)
bingbot desktop: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
bingbot: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
bingbot mobile: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
# Apple (the browser part changes from time to time)
Applebot desktop: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Applebot mobile: Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)
# Common Crawl
CCBot: CCBot/2.0 (https://commoncrawl.org/faq/)
# Anthropic and Meta: match the token, see the notes below
ClaudeBot, Claude-SearchBot, Claude-User
meta-externalagent/1.1, meta-webindexer/1.1, meta-externalfetcher/1.1 Anthropic's help article names ClaudeBot, Claude-SearchBot and Claude-User but does not print full user agent strings, so match on those tokens. Meta's crawler page lists strings that start with the tokens shown, followed by a link to its documentation, and says the strings your logs show may vary slightly.
Which crawler does what, and which names are only robots.txt tokens?
GPTBotcollects content that may be used to train OpenAI's foundation models.OAI-SearchBotsurfaces sites in ChatGPT search.ChatGPT-Userfetches pages for actions a user starts, and OpenAI says robots.txt rules may not apply to it.OAI-AdsBotchecks pages submitted as ChatGPT ads.ClaudeBotgathers data for Anthropic's models,Claude-SearchBotimproves search results, andClaude-Userfetches pages when a person asks Claude a question.PerplexityBotsurfaces and links sites in Perplexity search and is not used for training foundation models.Perplexity-Userfetches pages when a user asks a question.meta-externalagentcrawls for training AI models or indexing content,meta-webindexerindexes content for Meta AI search, andmeta-externalfetcherfetches links at a user's request.CCBotis the crawler of Common Crawl, a public web archive.
Two names never appear in your logs. Google-Extended is a robots.txt token that controls whether content Google crawls may be used for Gemini models and grounding; Google says it does not affect inclusion in Google Search. Applebot-Extended is, in Apple's words, a secondary user agent that does not crawl webpages; it only decides how Apple may use data that Applebot collected. A request that claims to be either one is not genuine.
Where do AI crawler vendors publish their IP ranges?
Most vendors publish a JSON file in the same shape: a prefixes array of objects with ipv4Prefix or ipv6Prefix, plus a creationTime that tells you how fresh it is.
- OpenAI:
https://openai.com/gptbot.json,https://openai.com/searchbot.json,https://openai.com/chatgpt-user.jsonandhttps://openai.com/adsbot.json, one per agent. - Anthropic:
https://claude.com/crawling/bots.json, one list for its crawlers. Anthropic says an address on this list shows the crawler comes from Anthropic, and also that IP blocking may not reliably guarantee an opt-out. - Perplexity:
https://www.perplexity.ai/perplexitybot.jsonandhttps://www.perplexity.ai/perplexity-user.json. - Google:
https://developers.google.com/static/crawling/ipranges/common-crawlers.jsonfor Googlebot, with separate files for special-case crawlers and user-triggered fetchers. Reverse DNS also works. - Apple:
https://search.developer.apple.com/applebot.json, or reverse DNS inapplebot.apple.com. - Common Crawl:
https://index.commoncrawl.org/ccbot.json. For IPv4 it also recommends forward-confirmed reverse DNS undercrawl.commoncrawl.org; IPv6 addresses have no reverse DNS. - Microsoft:
https://www.bing.com/toolbox/bingbot.json, which Bing asks you to refresh daily because it can change at any time, or reverse DNS ending insearch.msn.com. Bing also offers a public Verify Bingbot tool. - Meta: its crawler page does not publish an IP list.
How do I check an IP address against the official crawler lists?
This script loads each list and reports which one contains the address. It needs only Python 3, and it handles IPv4 and IPv6.
python3 - 20.171.206.5 <<'EOF'
import ipaddress, json, sys, urllib.request
LISTS = {
"GPTBot": "https://openai.com/gptbot.json",
"OAI-SearchBot": "https://openai.com/searchbot.json",
"ChatGPT-User": "https://openai.com/chatgpt-user.json",
"ClaudeBot, Claude-SearchBot, Claude-User": "https://claude.com/crawling/bots.json",
"PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
"Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
"Googlebot": "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
"bingbot": "https://www.bing.com/toolbox/bingbot.json",
"Applebot": "https://search.developer.apple.com/applebot.json",
"CCBot": "https://index.commoncrawl.org/ccbot.json",
}
ip = ipaddress.ip_address(sys.argv[1])
found = False
for name, url in LISTS.items():
req = urllib.request.Request(url, headers={"User-Agent": "ip-list-check"})
try:
prefixes = json.load(urllib.request.urlopen(req, timeout=15))["prefixes"]
except Exception as err:
print(f"{name}: list not loaded ({err})")
continue
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
if any(ip in net for net in nets if net.version == ip.version):
print(f"{ip} is listed for {name}")
found = True
if not found:
print(f"{ip} is not in any of these lists")
EOF The lists change. Each file carries its own creationTime, Perplexity recommends automating updates to your firewall rules, and Bing asks you to refresh its list daily. If you use the lists in a firewall, fetch them on a schedule instead of copying them once.
How do I verify Googlebot, bingbot or Applebot with reverse DNS?
Google, Microsoft, Apple and Common Crawl document the same two-step check: look up the name of the IP address, confirm it ends in the vendor's domain, then look up that name and confirm it returns the same IP. Both steps matter, because whoever controls an IP range can set any reverse DNS name they like, but cannot make the vendor's domain point back to it.
verify_crawler_ip() {
ip="$1"
name=$(host "$ip" | awk '/domain name pointer/ {print $NF}' | sed 's/\.$//' | head -n 1)
if [ -z "$name" ]; then echo "$ip: no reverse DNS name"; return 1; fi
case "$name" in
*.googlebot.com|*.search.msn.com|*.applebot.apple.com|*.crawl.commoncrawl.org) ;;
*) echo "$ip -> $name: not a crawler hostname"; return 1 ;;
esac
if host "$name" | grep -qwF -- "$ip"; then
echo "$ip -> $name -> $ip: verified"
else
echo "$ip -> $name: forward lookup does not return $ip"; return 1
fi
}
verify_crawler_ip 66.249.66.1
# 66.249.66.1 -> crawl-66-249-66-1.googlebot.com -> 66.249.66.1: verified Google accepts googlebot.com, google.com and googleusercontent.com across its crawler types; the function above accepts only googlebot.com, which is what Googlebot itself uses in Google's examples. For Google's other crawlers and fetchers, use its JSON lists. DNS lookups take time, so run them when you review logs or cache the result, rather than on every live request.
Why is a user agent check not enough to detect GPTBot?
The user agent is a header the client chooses, so anyone can send GPTBot or Googlebot. Common Crawl warns that some crawlers falsely claim to be CCBot, and Bing publishes a guide for checking whether traffic that claims to be bingbot really comes from Bing. Detecting GPTBot reliably means checking both: the token in the user agent and the source IP in gptbot.json. A request with the token from an unlisted address is an imitator, and blocking it does not affect OpenAI.
The reverse also happens. If your firewall allows only a user agent, imitators get in; if it allows only an IP range, any agent from that vendor's network gets the same treatment, even one you meant to block. Matching both is the dependable rule.
How can AISeeWhy help verify AI crawler traffic?
The free AISeeWhy AI crawler IP check tells you, without an account, whether an IP address belongs to GPTBot or another AI crawler. The free site check then requests your page as 16 AI and search crawlers next to a real browser, so you can see what the genuine crawlers would receive.
Next steps
- robots.txt tester and checker for AI crawlers
- AI crawler IP check
- See an example full report
- AI crawler monitoring
- Plans and prices
Crawlers in this article
Frequently asked questions
What is the GPTBot user agent?
OpenAI publishes Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot and notes that the version may change. Match on GPTBot and confirm the IP against https://openai.com/gptbot.json.
What is the OAI-SearchBot user agent?
A Chrome-style string ending in compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Its addresses are listed in https://openai.com/searchbot.json. OpenAI uses it to surface sites in ChatGPT search, which is a different job from GPTBot.
Does Anthropic publish ClaudeBot IP addresses?
Yes. Anthropic links to https://claude.com/crawling/bots.json and says a source IP on that list shows the crawler is from Anthropic. Its help article names the bot tokens but not full user agent strings.
Can I see Google-Extended or Applebot-Extended in my logs?
No. Both are robots.txt tokens, not crawlers. Google and Apple fetch pages with Googlebot and Applebot and use these tokens only to decide how the content may be used for AI.
Where is the bingbot IP address list?
Bing publishes it at https://www.bing.com/toolbox/bingbot.json and asks you to refresh it daily. Its own guide recommends the reverse and forward DNS check, where the name must end in search.msn.com.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl the web? What is Claude Bot?
- Perplexity: Perplexity crawlers
- Google: Google's common crawlers
- Google: Verifying Googlebot and other Google crawlers
- Bing Webmaster Tools: Which crawlers does Bing use?
- Bing Webmaster Tools: How to verify Bingbot
- Microsoft Bing: Verify Bingbot tool
- Apple: About Applebot
- Meta: Meta web crawlers
- Common Crawl: CCBot