robots.txt checker for AI crawlers

Enter a page address: we read the site's robots.txt and show, for every AI and search crawler, whether it may fetch that page. Or paste a robots.txt to test it before you publish it.

robots.txt is only a request. A firewall or bot protection can still block a crawler that robots.txt allows; the full check tests that. Run the full check →

How AI crawlers read robots.txt

Each crawler follows the group that names its token (for example GPTBot or PerplexityBot); only if none does, it follows User-agent: *. Within a group the longest matching rule wins, and Allow wins a tie. Blocking training crawlers such as GPTBot, ClaudeBot or Google-Extended keeps your content out of model training; blocking search and user crawlers such as OAI-SearchBot, ChatGPT-User or PerplexityBot keeps your pages out of AI answers.