How do I test robots.txt rules for a URL and a specific crawler?
To test a URL for one crawler, find the group that names the crawler's token (or User-agent: * if none does), collect every Allow and Disallow rule in it that matches the URL path, and apply the longest one; on a tie, Allow wins. You can do this by hand, with Google's open-source parser, or with a checker that shows a verdict per crawler. Google's old robots.txt Tester in Search Console is gone, and the robots.txt report that replaced it shows what Google fetched, not a verdict for a URL.
A robots.txt file looks simple until you need a definite answer: may GPTBot fetch this page, and may OAI-SearchBot? The answer depends on which group the crawler picks and which rule inside that group wins, and both steps are easy to get wrong. Here is how the matching works under RFC 9309, four worked examples, and three ways to test a file yourself.
On this page
- What happened to the robots.txt Tester in Google Search Console?
- How does a crawler decide whether a URL is allowed?
- Can you walk through a robots.txt test example?
- How do I run Google's open-source robots.txt parser?
- Can I test robots.txt with Python's urllib.robotparser?
- Why does a robots.txt test pass while the crawler is still blocked?
- How do I check robots.txt for every AI crawler at once?
What happened to the robots.txt Tester in Google Search Console?
In November 2023 Google added a robots.txt report to Search Console and said it was sunsetting the robots.txt tester. The report sits under Settings and covers Google's crawlers only. It shows:
- The robots.txt files Google found for the top 20 hosts of your property.
- The fetch status of each file (
Fetched,Not Fetched - Not found (404)orNot Fetched - Any other reason), when Google last checked it, and its size in bytes. - Issues in the file. Google says errors prevent a rule from being used, while warnings do not.
- The versions Google fetched in the last 30 days, and a button to request a recrawl after you fix the file.
The report does not take a URL and return allowed or blocked. To check one URL for Googlebot, Google points to the URL Inspection tool, and for local testing to its open-source robots.txt library. Neither tells you anything about AI crawlers such as GPTBot or ClaudeBot.
How does a crawler decide whether a URL is allowed?
RFC 9309, the Robots Exclusion Protocol, defines the matching. Google documents the same behaviour for its crawlers.
- Pick the group. User-agent matching is case-insensitive. If one or more groups name the crawler's token, their rules are combined and the
User-agent: *group is ignored entirely. Only a crawler with no named group follows*. - Pick the rule. Among the
AllowandDisallowrules whose path matches the URL, the one with the most characters wins. If anAllowand aDisalloware equally long,Allowshould win. - Read the patterns.
*matches any run of characters and$marks the end of the URL. Paths are case-sensitive, so/Admin/does not match/admin/. - No matching rule means allowed.
/robots.txtitself is always allowed.
The response code of robots.txt matters as much as its rules. A 4xx response means the crawler may fetch everything; Google treats every 4xx except 429 as if no file existed. A 5xx response means the crawler must assume the whole site is disallowed. Google stops crawling for the first 12 hours, then uses the last good copy for up to 30 days. RFC 9309 requires crawlers to parse at least 500 KiB, and Google ignores anything after 500 KiB.
Can you walk through a robots.txt test example?
Here is a small robots.txt with a default group and two named AI crawlers:
Googlebotand/search/help/billing: allowed. There is no Googlebot group, so the*group applies.Disallow: /searchandAllow: /search/helpboth match, and theAllowrule is longer.GPTBotand/blog/post: blocked. GPTBot has its own group, andDisallow: /matches every path.OAI-SearchBotand/admin/settings: allowed. OAI-SearchBot has its own group, soDisallow: /admin/in the*group does not apply to it. This is the most common surprise; repeat shared rules in every named group.OAI-SearchBotand/files/report.pdf: blocked.Allow: /matches with one character,Disallow: /*.pdf$matches with seven, and the longer rule wins.
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /search/help
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Disallow: /drafts/
Disallow: /*.pdf$ Ties are rarer but follow the same logic. Google's documentation gives the case of allow: /folder and disallow: /folder for the URL /folder/page: both are the same length, so the URL is allowed.
How do I run Google's open-source robots.txt parser?
Google publishes the C++ library it uses in production to match robots.txt rules, with a small command-line program for testing one URL and one user agent against a local file. With Bazel installed:
git clone https://github.com/google/robotstxt.git robotstxt
cd robotstxt/
bazel build :robots_main
# arguments: local robots.txt file, crawler token, full URL
bazel run robots_main -- ~/robots.txt OAI-SearchBot https://example.com/files/report.pdf
# prints ALLOWED or DISALLOWED; exit code 0 means allowed, 1 means disallowed The repository also documents a CMake build, which produces a binary named robots with the same arguments. The tool only parses and matches; it does not download robots.txt, so save the live file first. Its README also notes that crawler-specific behaviour outside parsing and matching is not part of the library.
Can I test robots.txt with Python's urllib.robotparser?
Python's standard library includes a parser that answers allowed or blocked for a user agent and a URL. This script tests the example above:
from urllib.robotparser import RobotFileParser
# Paste the robots.txt you want to test, or fetch it yourself first
ROBOTS_TXT = """
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /search/help
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Disallow: /drafts/
Disallow: /*.pdf$
"""
tests = [
("Googlebot", "https://example.com/search/help/billing"),
("GPTBot", "https://example.com/blog/post"),
("OAI-SearchBot", "https://example.com/admin/settings"),
("OAI-SearchBot", "https://example.com/files/report.pdf"),
]
rp = RobotFileParser()
rp.parse(ROBOTS_TXT.splitlines())
for agent, url in tests:
print(f"{agent:15} {url} ->", "allowed" if rp.can_fetch(agent, url) else "blocked") Check your Python version before trusting the output. Support for RFC 9309 matching was added to CPython in 2026 and first shipped in Python 3.13.14 and 3.14.5. Older releases, including all of 3.12 and earlier, apply the first rule in file order that matches, not the longest, and do not understand * or $ inside paths. On those versions the script reports the first and fourth tests the wrong way round: Googlebot blocked from /search/help/billing and OAI-SearchBot allowed to fetch the PDF.
Two more differences apply to every version. RobotFileParser.read() treats a 401 or 403 response for robots.txt as disallow all, while RFC 9309 says a 4xx means everything may be fetched. And read() downloads the file with Python's default user agent, which some firewalls block, so a test can report blocked for reasons unrelated to your rules. Fetching the file yourself and passing it to parse(), as above, avoids both.
Why does a robots.txt test pass while the crawler is still blocked?
A tester answers one question: do the rules allow this URL? Several things sit outside that answer:
- The live file differs from the one you tested, or the crawler still uses a cached copy. RFC 9309 says crawlers should not use a cached copy for more than 24 hours, and Google caches for up to 24 hours in general.
- The robots.txt URL returns an HTML page or an error. An HTML page with status 200 contains no valid rules, and a 5xx response blocks the whole site.
- robots.txt is a request, not access control. A firewall, CDN or bot protection can still answer the crawler with a 403 or a challenge page.
How do I check robots.txt for every AI crawler at once?
The free AISeeWhy robots.txt checker takes a page address or a pasted robots.txt and shows, for each AI and search crawler, whether that page may be fetched. The free robots.txt generator writes a file from your choices, and the free check fetches the page as 16 crawlers to show whether a firewall blocks what robots.txt allows.
Fixes for this problem
- robots.txt allows GPTBot, but the server blocks it
- robots.txt disallows OAI-SearchBot
- robots.txt is an HTML page
- robots.txt cannot be fetched
Next steps
- robots.txt tester and checker for AI crawlers
- See an example full report
- AI crawler monitoring
- Plans and prices
Crawlers in this article
Frequently asked questions
Is there still a robots.txt tester in Google Search Console?
No. Google replaced it in November 2023 with the robots.txt report, which shows the files Google fetched, their status and any errors. To check whether Googlebot may crawl a specific URL, use the URL Inspection tool.
Does Allow override Disallow in robots.txt?
Only when the Allow rule is longer, or exactly as long. The rule with the longest matching path wins, whichever kind it is, and Allow wins a tie.
Is robots.txt case-sensitive?
Partly. The User-agent value is matched without regard to case, so gptbot and GPTBot name the same group. Paths are case-sensitive, so Disallow: /Private/ does not block /private/.
What does a robots.txt validator check?
A validator checks that the file is reachable, served as text and written in valid syntax. A tester goes further and gives a verdict for a specific URL and crawler. You usually need both, because a valid file can still block a page you meant to allow.
If a crawler has its own group, does it still follow User-agent: *?
No. A named group replaces the * group completely for that crawler. Any rule you want to apply to everyone has to be repeated in each named group.
Sources
- Google Search Console Help: robots.txt report
- Search Engine Roundtable: New Google Search Console robots.txt report replaced robots.txt tester tool
- Google Search Central: How Google interprets the robots.txt specification
- Google Search Central: Create and submit a robots.txt file
- RFC 9309: Robots Exclusion Protocol
- GitHub: google/robotstxt
- Python documentation: urllib.robotparser
- CPython source: Lib/urllib/robotparser.py