OpenAI crawlers explained: GPTBot, OAI-SearchBot and ChatGPT-User
OpenAI uses GPTBot to collect content that may train its models, OAI-SearchBot to index pages for ChatGPT search, and ChatGPT-User for visits a person asks for. Blocking GPTBot opts you out of training only. Blocking OAI-SearchBot removes you from ChatGPT search answers. robots.txt may not apply to ChatGPT-User.
Three different user agents from one company, each with its own robots.txt token, cause a lot of confusion. Many sites blocked GPTBot when it appeared, and some later found they had also blocked the crawler that decides whether ChatGPT can cite them, often through a broad firewall rule. This article goes through each agent using OpenAI's own documentation.
On this page
What does GPTBot do?
GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. OpenAI's documentation says that disallowing GPTBot indicates a site's content should not be used in training those models.
Its user agent contains the token GPTBot, for example Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. OpenAI notes the version number may change, so match on the token, not on the full string. OpenAI publishes its IP ranges at openai.com/gptbot.json.
Whether to allow it is a policy decision. Some publishers want their content in training data, others do not. Blocking it does not affect ChatGPT search, because that is a different crawler. If you change your mind later, removing the Disallow line is enough; there is nothing to register or request.
What does OAI-SearchBot do?
OAI-SearchBot is OpenAI's search crawler. It surfaces websites in ChatGPT's search features. OpenAI states plainly that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
If you want ChatGPT to cite your pages, this is the crawler to allow, both in robots.txt and in your firewall. Its IP ranges are published at openai.com/searchbot.json, which lets a firewall verify that a request really comes from OpenAI.
What does ChatGPT-User do?
ChatGPT-User is not a crawler in the usual sense. It fetches a page when a ChatGPT user or a Custom GPT asks for it, for example when someone pastes your URL into a chat and asks for a summary. OpenAI says it is not used for automatic crawling of the web.
Two details from OpenAI's documentation matter here. First, because these actions are initiated by a user, robots.txt rules may not apply. Second, ChatGPT-User is not used to determine whether content may appear in search. For search, the setting that counts is OAI-SearchBot.
What happens when I block each one?
- Block
GPTBot: your content is marked as not to be used for training. ChatGPT search can still show and cite you ifOAI-SearchBotis allowed. - Block
OAI-SearchBot: your pages will not be shown in ChatGPT search answers. Training is controlled separately byGPTBot. - Block
ChatGPT-Userin robots.txt: OpenAI says robots.txt may not apply to these visits, so the effect is not guaranteed. Blocking it at the firewall stops the fetch, and users who paste your link get an error instead of your content. - Block all three at the firewall: ChatGPT can neither index nor fetch your pages, so it has nothing of yours to quote.
- Allow all three: your pages can be indexed for search, fetched on request and, through
GPTBot, used for training.
OpenAI also notes that when both GPTBot and OAI-SearchBot are allowed, it may use the results of one crawl for both purposes to avoid crawling twice.
What robots.txt should I use for OpenAI's crawlers?
For most sites that want to appear in ChatGPT answers, allow the search crawler and decide separately about training. This file allows search and opts out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Disallow: /admin/Note a detail of RFC 9309: a crawler follows the group that names it and ignores the * group. In the example above, OAI-SearchBot would be allowed into /admin/ because its own group has no Disallow: /admin/ line. Repeat any paths you want kept private in each named group. OpenAI says search changes can take about 24 hours to apply.
How do I confirm the firewall lets them through?
robots.txt only states your policy. The firewall or CDN decides what each request actually receives. Request a page with each crawler's user agent and compare the status code and body size with a browser request. A 403, a 429 or a challenge page for the crawler, with a normal page for the browser, means the firewall overrides your robots.txt.
AISeeWhy runs this comparison for all three OpenAI agents at once, marks blocks of OAI-SearchBot and ChatGPT-User as P1 because they cost citations, and lists a blocked GPTBot as a P3 policy choice rather than an error.
Frequently asked questions
Is GPTBot the same as ChatGPT?
No. GPTBot is a crawler for training data. ChatGPT search uses OAI-SearchBot, and live visits requested by users come from ChatGPT-User.
Can I allow ChatGPT search but block training?
Yes. Allow OAI-SearchBot and disallow GPTBot in robots.txt. OpenAI documents the two settings as independent.
Should I block OpenAI by IP address?
Blocking by IP stops the crawler from reading your robots.txt and makes your policy harder to change. It is usually better to state the policy in robots.txt and use the published IP ranges to verify, not to block, OpenAI's crawlers.
Why match on the token and not the full user agent?
OpenAI says the version number in its user agents may change, and the example strings it publishes differ between crawlers. Rules that match GPTBot or OAI-SearchBot keep working after an update; rules that match the exact string can silently stop matching.