How do I test robots.txt rules for a URL and a specific crawler?

Published

Short answer

To test a URL for one crawler, find the group that names the crawler's token (or User-agent: * if none does), collect every Allow and Disallow rule in it that matches the URL path, and apply the longest one; on a tie, Allow wins. You can do this by hand, with Google's open-source parser, or with a checker that shows a verdict per crawler. Google's old robots.txt Tester in Search Console is gone, and the robots.txt report that replaced it shows what Google fetched, not a verdict for a URL.

A robots.txt file looks simple until you need a definite answer: may GPTBot fetch this page, and may OAI-SearchBot? The answer depends on which group the crawler picks and which rule inside that group wins, and both steps are easy to get wrong. Here is how the matching works under RFC 9309, four worked examples, and three ways to test a file yourself.

On this page
  1. What happened to the robots.txt Tester in Google Search Console?
  2. How does a crawler decide whether a URL is allowed?
  3. Can you walk through a robots.txt test example?
  4. How do I run Google's open-source robots.txt parser?
  5. Can I test robots.txt with Python's urllib.robotparser?
  6. Why does a robots.txt test pass while the crawler is still blocked?
  7. How do I check robots.txt for every AI crawler at once?

What happened to the robots.txt Tester in Google Search Console?

In November 2023 Google added a robots.txt report to Search Console and said it was sunsetting the robots.txt tester. The report sits under Settings and covers Google's crawlers only. It shows:

The report does not take a URL and return allowed or blocked. To check one URL for Googlebot, Google points to the URL Inspection tool, and for local testing to its open-source robots.txt library. Neither tells you anything about AI crawlers such as GPTBot or ClaudeBot.

How does a crawler decide whether a URL is allowed?

RFC 9309, the Robots Exclusion Protocol, defines the matching. Google documents the same behaviour for its crawlers.

The response code of robots.txt matters as much as its rules. A 4xx response means the crawler may fetch everything; Google treats every 4xx except 429 as if no file existed. A 5xx response means the crawler must assume the whole site is disallowed. Google stops crawling for the first 12 hours, then uses the last good copy for up to 30 days. RFC 9309 requires crawlers to parse at least 500 KiB, and Google ignores anything after 500 KiB.

Can you walk through a robots.txt test example?

Here is a small robots.txt with a default group and two named AI crawlers:

User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /search/help

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
Disallow: /drafts/
Disallow: /*.pdf$

Ties are rarer but follow the same logic. Google's documentation gives the case of allow: /folder and disallow: /folder for the URL /folder/page: both are the same length, so the URL is allowed.

How do I run Google's open-source robots.txt parser?

Google publishes the C++ library it uses in production to match robots.txt rules, with a small command-line program for testing one URL and one user agent against a local file. With Bazel installed:

git clone https://github.com/google/robotstxt.git robotstxt
cd robotstxt/
bazel build :robots_main
# arguments: local robots.txt file, crawler token, full URL
bazel run robots_main -- ~/robots.txt OAI-SearchBot https://example.com/files/report.pdf
# prints ALLOWED or DISALLOWED; exit code 0 means allowed, 1 means disallowed

The repository also documents a CMake build, which produces a binary named robots with the same arguments. The tool only parses and matches; it does not download robots.txt, so save the live file first. Its README also notes that crawler-specific behaviour outside parsing and matching is not part of the library.

Can I test robots.txt with Python's urllib.robotparser?

Python's standard library includes a parser that answers allowed or blocked for a user agent and a URL. This script tests the example above:

from urllib.robotparser import RobotFileParser

# Paste the robots.txt you want to test, or fetch it yourself first
ROBOTS_TXT = """
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /search/help

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
Disallow: /drafts/
Disallow: /*.pdf$
"""

tests = [
    ("Googlebot", "https://example.com/search/help/billing"),
    ("GPTBot", "https://example.com/blog/post"),
    ("OAI-SearchBot", "https://example.com/admin/settings"),
    ("OAI-SearchBot", "https://example.com/files/report.pdf"),
]

rp = RobotFileParser()
rp.parse(ROBOTS_TXT.splitlines())
for agent, url in tests:
    print(f"{agent:15} {url}  ->", "allowed" if rp.can_fetch(agent, url) else "blocked")

Check your Python version before trusting the output. Support for RFC 9309 matching was added to CPython in 2026 and first shipped in Python 3.13.14 and 3.14.5. Older releases, including all of 3.12 and earlier, apply the first rule in file order that matches, not the longest, and do not understand * or $ inside paths. On those versions the script reports the first and fourth tests the wrong way round: Googlebot blocked from /search/help/billing and OAI-SearchBot allowed to fetch the PDF.

Two more differences apply to every version. RobotFileParser.read() treats a 401 or 403 response for robots.txt as disallow all, while RFC 9309 says a 4xx means everything may be fetched. And read() downloads the file with Python's default user agent, which some firewalls block, so a test can report blocked for reasons unrelated to your rules. Fetching the file yourself and passing it to parse(), as above, avoids both.

Why does a robots.txt test pass while the crawler is still blocked?

A tester answers one question: do the rules allow this URL? Several things sit outside that answer:

How do I check robots.txt for every AI crawler at once?

The free AISeeWhy robots.txt checker takes a page address or a pasted robots.txt and shows, for each AI and search crawler, whether that page may be fetched. The free robots.txt generator writes a file from your choices, and the free check fetches the page as 16 crawlers to show whether a firewall blocks what robots.txt allows.

Fixes for this problem

Free tools

Next steps

Crawlers in this article

Frequently asked questions

Is there still a robots.txt tester in Google Search Console?

No. Google replaced it in November 2023 with the robots.txt report, which shows the files Google fetched, their status and any errors. To check whether Googlebot may crawl a specific URL, use the URL Inspection tool.

Does Allow override Disallow in robots.txt?

Only when the Allow rule is longer, or exactly as long. The rule with the longest matching path wins, whichever kind it is, and Allow wins a tie.

Is robots.txt case-sensitive?

Partly. The User-agent value is matched without regard to case, so gptbot and GPTBot name the same group. Paths are case-sensitive, so Disallow: /Private/ does not block /private/.

What does a robots.txt validator check?

A validator checks that the file is reachable, served as text and written in valid syntax. A tester goes further and gives a verdict for a specific URL and crawler. You usually need both, because a valid file can still block a page you meant to allow.

If a crawler has its own group, does it still follow User-agent: *?

No. A named group replaces the * group completely for that crawler. Any rule you want to apply to everyone has to be repeated in each named group.

Sources

More articles