AI crawler user agents and IP ranges: GPTBot, ClaudeBot, PerplexityBot and more

Published

Short answer

GPTBot identifies itself with a user agent containing GPTBot/1.4; +https://openai.com/gptbot, and OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple and Common Crawl each publish a JSON list of the IP ranges their crawlers use. A request is real only when both match: the user agent names the crawler and the source IP is in that vendor's list, or, for Googlebot, bingbot, Applebot and CCBot, passes a reverse and forward DNS check. A user agent on its own proves nothing, because anyone can send it.

When you see GPTBot or ClaudeBot in your logs, or want a firewall rule that lets AI search crawlers through, you need two things: the exact string each crawler sends, and a way to prove a request came from the company it names. Below are the strings as each vendor publishes them today, the official IP list or DNS method for each, and two scripts you can run to check an address.

On this page
  1. What are the user agent strings of GPTBot, ClaudeBot, PerplexityBot and other AI crawlers?
  2. Which crawler does what, and which names are only robots.txt tokens?
  3. Where do AI crawler vendors publish their IP ranges?
  4. How do I check an IP address against the official crawler lists?
  5. How do I verify Googlebot, bingbot or Applebot with reverse DNS?
  6. Why is a user agent check not enough to detect GPTBot?
  7. How can AISeeWhy help verify AI crawler traffic?

What are the user agent strings of GPTBot, ClaudeBot, PerplexityBot and other AI crawlers?

These are copied from the vendors' own documentation. OpenAI and Perplexity note that version numbers can change, Google and Microsoft put the current browser version where W.X.Y.Z appears, and Apple says it occasionally updates the browser part. Match on the crawler token, such as GPTBot or bingbot, not on the whole string.

# OpenAI (version numbers may change)
GPTBot:          Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OAI-SearchBot:   Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User:    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OAI-AdsBot:      Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

# Perplexity
PerplexityBot:   Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-User: Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

# Google (W.X.Y.Z is the current Chrome version)
Googlebot Smartphone: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot Desktop:    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Googlebot Desktop:    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

# Microsoft Bing (W.X.Y.Z is the current Edge version)
bingbot desktop: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
bingbot:         Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
bingbot mobile:  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

# Apple (the browser part changes from time to time)
Applebot desktop: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Applebot mobile:  Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)

# Common Crawl
CCBot:           CCBot/2.0 (https://commoncrawl.org/faq/)

# Anthropic and Meta: match the token, see the notes below
ClaudeBot, Claude-SearchBot, Claude-User
meta-externalagent/1.1, meta-webindexer/1.1, meta-externalfetcher/1.1

Anthropic's help article names ClaudeBot, Claude-SearchBot and Claude-User but does not print full user agent strings, so match on those tokens. Meta's crawler page lists strings that start with the tokens shown, followed by a link to its documentation, and says the strings your logs show may vary slightly.

Which crawler does what, and which names are only robots.txt tokens?

Two names never appear in your logs. Google-Extended is a robots.txt token that controls whether content Google crawls may be used for Gemini models and grounding; Google says it does not affect inclusion in Google Search. Applebot-Extended is, in Apple's words, a secondary user agent that does not crawl webpages; it only decides how Apple may use data that Applebot collected. A request that claims to be either one is not genuine.

Where do AI crawler vendors publish their IP ranges?

Most vendors publish a JSON file in the same shape: a prefixes array of objects with ipv4Prefix or ipv6Prefix, plus a creationTime that tells you how fresh it is.

How do I check an IP address against the official crawler lists?

This script loads each list and reports which one contains the address. It needs only Python 3, and it handles IPv4 and IPv6.

python3 - 20.171.206.5 <<'EOF'
import ipaddress, json, sys, urllib.request

LISTS = {
    "GPTBot": "https://openai.com/gptbot.json",
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "ClaudeBot, Claude-SearchBot, Claude-User": "https://claude.com/crawling/bots.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
    "Googlebot": "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
    "bingbot": "https://www.bing.com/toolbox/bingbot.json",
    "Applebot": "https://search.developer.apple.com/applebot.json",
    "CCBot": "https://index.commoncrawl.org/ccbot.json",
}

ip = ipaddress.ip_address(sys.argv[1])
found = False
for name, url in LISTS.items():
    req = urllib.request.Request(url, headers={"User-Agent": "ip-list-check"})
    try:
        prefixes = json.load(urllib.request.urlopen(req, timeout=15))["prefixes"]
    except Exception as err:
        print(f"{name}: list not loaded ({err})")
        continue
    nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
    if any(ip in net for net in nets if net.version == ip.version):
        print(f"{ip} is listed for {name}")
        found = True
if not found:
    print(f"{ip} is not in any of these lists")
EOF

The lists change. Each file carries its own creationTime, Perplexity recommends automating updates to your firewall rules, and Bing asks you to refresh its list daily. If you use the lists in a firewall, fetch them on a schedule instead of copying them once.

How do I verify Googlebot, bingbot or Applebot with reverse DNS?

Google, Microsoft, Apple and Common Crawl document the same two-step check: look up the name of the IP address, confirm it ends in the vendor's domain, then look up that name and confirm it returns the same IP. Both steps matter, because whoever controls an IP range can set any reverse DNS name they like, but cannot make the vendor's domain point back to it.

verify_crawler_ip() {
  ip="$1"
  name=$(host "$ip" | awk '/domain name pointer/ {print $NF}' | sed 's/\.$//' | head -n 1)
  if [ -z "$name" ]; then echo "$ip: no reverse DNS name"; return 1; fi
  case "$name" in
    *.googlebot.com|*.search.msn.com|*.applebot.apple.com|*.crawl.commoncrawl.org) ;;
    *) echo "$ip -> $name: not a crawler hostname"; return 1 ;;
  esac
  if host "$name" | grep -qwF -- "$ip"; then
    echo "$ip -> $name -> $ip: verified"
  else
    echo "$ip -> $name: forward lookup does not return $ip"; return 1
  fi
}

verify_crawler_ip 66.249.66.1
# 66.249.66.1 -> crawl-66-249-66-1.googlebot.com -> 66.249.66.1: verified

Google accepts googlebot.com, google.com and googleusercontent.com across its crawler types; the function above accepts only googlebot.com, which is what Googlebot itself uses in Google's examples. For Google's other crawlers and fetchers, use its JSON lists. DNS lookups take time, so run them when you review logs or cache the result, rather than on every live request.

Why is a user agent check not enough to detect GPTBot?

The user agent is a header the client chooses, so anyone can send GPTBot or Googlebot. Common Crawl warns that some crawlers falsely claim to be CCBot, and Bing publishes a guide for checking whether traffic that claims to be bingbot really comes from Bing. Detecting GPTBot reliably means checking both: the token in the user agent and the source IP in gptbot.json. A request with the token from an unlisted address is an imitator, and blocking it does not affect OpenAI.

The reverse also happens. If your firewall allows only a user agent, imitators get in; if it allows only an IP range, any agent from that vendor's network gets the same treatment, even one you meant to block. Matching both is the dependable rule.

How can AISeeWhy help verify AI crawler traffic?

The free AISeeWhy AI crawler IP check tells you, without an account, whether an IP address belongs to GPTBot or another AI crawler. The free site check then requests your page as 16 AI and search crawlers next to a real browser, so you can see what the genuine crawlers would receive.

Free tools

Next steps

Crawlers in this article

Frequently asked questions

What is the GPTBot user agent?

OpenAI publishes Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot and notes that the version may change. Match on GPTBot and confirm the IP against https://openai.com/gptbot.json.

What is the OAI-SearchBot user agent?

A Chrome-style string ending in compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Its addresses are listed in https://openai.com/searchbot.json. OpenAI uses it to surface sites in ChatGPT search, which is a different job from GPTBot.

Does Anthropic publish ClaudeBot IP addresses?

Yes. Anthropic links to https://claude.com/crawling/bots.json and says a source IP on that list shows the crawler is from Anthropic. Its help article names the bot tokens but not full user agent strings.

Can I see Google-Extended or Applebot-Extended in my logs?

No. Both are robots.txt tokens, not crawlers. Google and Apple fetch pages with Googlebot and Applebot and use these tokens only to decide how the content may be used for AI.

Where is the bingbot IP address list?

Bing publishes it at https://www.bing.com/toolbox/bingbot.json and asks you to refresh it daily. Its own guide recommends the reverse and forward DNS check, where the name must end in search.msn.com.

Sources

More articles