GPTBot
GPTBot is OpenAI's crawler for content that may be used to train its generative AI foundation models. Disallowing it in robots.txt tells OpenAI not to use your content for training. Blocking it does not remove you from ChatGPT search answers, which depend on OAI-SearchBot.
What it does
OpenAI says GPTBot is used to make its generative AI foundation models more useful and safe. It crawls content that may be used in training those models, and disallowing GPTBot indicates that a site's content should not be used in training.
Whether to allow it is a policy decision, not a visibility one. ChatGPT search uses a separate crawler, OAI-SearchBot, and OpenAI documents the two settings as independent. When both are allowed, OpenAI says it may use the results of one crawl for both purposes to avoid crawling twice.
In our 2026-11 study, 21.4% of popular sites with a robots.txt blocked it. →
robots.txt token
GPTBot
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotHow to allow it
User-agent: GPTBot
Allow: /How to block it
User-agent: GPTBot
Disallow: /How to verify real visits
OpenAI publishes the IP ranges this agent uses as a JSON file. Match the source IP of the request against those ranges; a request that names GPTBot but comes from another address is not from OpenAI. OpenAI notes the version number in the user agent may change, so match the GPTBot token rather than the full string.
Check if your site lets it in →
OpenAI crawlers explained: GPTBot, OAI-SearchBot and ChatGPT-User →