AISeeWhy
EN
Start free trial

How to check your server logs for AI crawlers

Published

Short answer

Search your access log's user agent field for tokens such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot, then look at the status codes they received. A user agent alone proves nothing, so check each source IP against the vendor's published IP list, or by reverse and forward DNS for Google and Apple.

Robots.txt says what you ask crawlers to do. Your access log shows what actually happened: which AI crawlers came, which pages they requested, and what your server answered. It is the most direct evidence you have, and it needs nothing more than the log files and a terminal. The examples below use the combined log format that nginx and Apache use by default.

On this page
  1. What does an AI crawler visit look like in an access log?
  2. How do I count AI crawler visits?
  3. What did each crawler actually receive?
  4. How do I verify that a visit really came from OpenAI, Anthropic or Perplexity?
  5. How do I verify Googlebot, Applebot and CCBot?
  6. What do fake AI crawlers look like?
  7. What can logs not tell you?

What does an AI crawler visit look like in an access log?

In the combined format, each line holds the client IP, the time, the request line, the status code, the response size, the referrer and the user agent. nginx defines it as $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent", and Apache's combined format has the same fields. A visit from OpenAI's training crawler looks like this:

20.171.206.5 - - [12/Nov/2026:10:00:01 +0000] "GET /pricing HTTP/1.1" 200 5120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

The tokens to look for are the ones the vendors document: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot and Claude-User (Anthropic); PerplexityBot and Perplexity-User (Perplexity); Googlebot, Applebot, meta-externalagent, meta-webindexer and CCBot. You will never see Google-Extended or Applebot-Extended. Google says Google-Extended has no separate user agent string, and Apple says Applebot-Extended does not crawl; both are robots.txt tokens only.

How do I count AI crawler visits?

Splitting each line on the double quote character puts the user agent in field 6. Matching only that field avoids false hits from URLs that happen to contain a bot's name. This counts visits per crawler across the current log and rotated, compressed ones:

zcat -f /var/log/nginx/access.log* \
  | awk -F'"' '{print $6}' \
  | grep -Eo 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot|bingbot|Applebot|meta-externalagent|meta-webindexer|CCBot' \
  | sort | uniq -c | sort -rn

Adjust the path for Apache, often /var/log/apache2/access.log* or /var/log/httpd/access_log*. zcat -f reads both plain and gzipped files. If a crawler you expect is missing, that alone is not proof of a block: it may simply not have visited in the period your logs cover.

What did each crawler actually receive?

Visits matter less than answers. A crawler that gets a 403 on every request has visited but read nothing. These commands show status codes, the most requested paths and whether the crawler fetched robots.txt:

# Status codes returned to PerplexityBot
zcat -f /var/log/nginx/access.log* | awk -F'"' '$6 ~ /PerplexityBot/ {split($3, s, " "); print s[1]}' | sort | uniq -c

# Most requested paths for OAI-SearchBot
zcat -f /var/log/nginx/access.log* | awk -F'"' '$6 ~ /OAI-SearchBot/ {split($2, r, " "); print r[2]}' | sort | uniq -c | sort -rn | head -20

# Did ClaudeBot fetch robots.txt, and what did it get?
zcat -f /var/log/nginx/access.log* | awk -F'"' '$6 ~ /ClaudeBot/ && $2 ~ /robots\.txt/ {split($3, s, " "); print s[1]}' | sort | uniq -c

Mostly 200s is what you want. Many 403s or 429s point to a firewall rule or rate limit. 5xx answers on robots.txt are serious: under RFC 9309 a crawler that gets a server error for robots.txt must assume the whole site is disallowed.

If your site sits behind a CDN, two things change. Requests the CDN blocks never reach your origin, so they do not appear in its logs at all; check the CDN's own security events instead. And unless you restore the visitor IP, the origin logs the CDN's address. Cloudflare, for example, sends the original IP in the CF-Connecting-IP header and logs a Cloudflare IP by default.

How do I verify that a visit really came from OpenAI, Anthropic or Perplexity?

These vendors publish the IP ranges their agents use: OpenAI at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json; Anthropic at claude.com/crawling/bots.json; Perplexity at perplexity.ai/perplexitybot.json and perplexity.ai/perplexity-user.json. Anthropic says that a source IP on its list indicates the crawler is coming from Anthropic. None of these three documents reverse DNS verification, so use the lists.

The files share one shape, a prefixes list of ipv4Prefix or ipv6Prefix entries, so one small script checks an IP against any of them:

python3 - 20.171.206.5 https://openai.com/gptbot.json <<'EOF'
import ipaddress, json, sys, urllib.request
ip = ipaddress.ip_address(sys.argv[1])
req = urllib.request.Request(sys.argv[2], headers={"User-Agent": "log-check"})
prefixes = json.load(urllib.request.urlopen(req))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
print(ip, "listed" if any(ip in n for n in nets) else "NOT listed")
EOF

To list the IPs that claimed a given crawler, run awk -F'"' '$6 ~ /GPTBot/ {split($1, a, " "); print a[1]}' over the log and pipe it through sort | uniq -c | sort -rn. Check each against the matching list. The lists change, so fetch them fresh rather than keeping an old copy.

How do I verify Googlebot, Applebot and CCBot?

Google, Apple and Common Crawl document a reverse and forward DNS check. Look up the IP's hostname, check its domain, then look up that hostname and confirm it points back to the same IP:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For Google the hostname must end in googlebot.com, google.com or googleusercontent.com, depending on the crawler; Google also publishes its ranges, with Googlebot in common-crawlers.json. For Apple it must be in applebot.apple.com, with ranges in applebot.json. For Common Crawl it must be under crawl.commoncrawl.org, and Common Crawl warns that other crawlers falsely claim the CCBot name.

What do fake AI crawlers look like?

Scrapers often borrow a well-known crawler's user agent to get past filters. Typical signs:

The reverse problem also exists: a crawler that uses an ordinary browser user agent cannot be identified from the log line at all. In 2025 Cloudflare said it had observed Perplexity crawling with undeclared, browser-like user agents from IPs outside its published ranges; Perplexity disputed that account and attributed part of the traffic to a third-party service. Logs alone cannot settle cases like this, which is why verification by published IP ranges matters in both directions.

What can logs not tell you?

Logs show what reached your server. They do not show what your CDN blocked before the origin, whether a page that returned 200 contained the text or only a script shell, or whether a crawler would be allowed if it came today. They also cover only the period you keep, and quiet sites may see AI crawlers rarely.

In an AISeeWhy account you can upload an access log in combined format (plain or .gz, up to 50 MB); it is analysed and discarded, and each AI crawler's requests are counted with their status codes and checked against the IP ranges the vendors publish.

Frequently asked questions

Why don't I see Google-Extended in my logs?

Google says Google-Extended has no separate user agent string. Crawling is done by Google's existing crawlers, and the token only controls how the content may be used. The same applies to Applebot-Extended, which Apple says does not crawl.

Can I trust the user agent in my logs?

No. Anyone can send any user agent. Treat it as a claim and confirm it with the vendor's published IP ranges or, for Google, Apple and Common Crawl, a reverse and forward DNS lookup.

My logs show no AI crawler visits. Am I blocked?

Not necessarily. The crawler may not have visited yet, your CDN may have blocked it before the origin, or your logs may record only the CDN's IP addresses. Check your CDN's security events and test the page with each crawler's user agent.

Is ChatGPT-User traffic a crawler?

OpenAI says ChatGPT-User is used for certain user actions and is not used for crawling the web in an automatic fashion. Its visits usually follow a person asking ChatGPT about a page, so they come in small numbers.

Sources

More articles