robots.txt generator for AI crawlers
Choose which AI crawlers may read your site and copy the generated robots.txt. A common choice is to block AI training crawlers such as GPTBot and ClaudeBot but allow AI search crawlers such as OAI-SearchBot and PerplexityBot, so your pages can still appear and be cited in AI answers.
Blocking user-requested fetchers has limited effect: some vendors say these agents may not follow robots.txt, because a person asked for the page. Test the result with the robots.txt checker before you publish it. robots.txt checker for AI crawlers →
How to use it
Put the file at the root of your domain (https://example.com/robots.txt). If you already have rules for other crawlers, keep them and add only the AI groups. Changes take effect the next time each crawler reads the file, usually within a day.