Can ChatGPT see my website? How to check

Q: Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI treats each crawler setting as independent. GPTBot relates to training, while OAI-SearchBot decides whether pages can be shown in ChatGPT search answers. You can disallow one and allow the other.

Published 2026-11-12

Short answer

ChatGPT can cite your page only if OpenAI's search crawler, OAI-SearchBot, may fetch it and gets the same HTML a browser gets. Check three things: your robots.txt does not disallow OAI-SearchBot, your firewall or CDN does not answer it with a 403 or a challenge page, and the main text is in the initial HTML.

Many site owners ask ChatGPT about their own product, see a competitor cited instead, and wonder whether ChatGPT can reach their site at all. The question has a concrete, testable answer. OpenAI publishes the names of its crawlers and what each one does, so you can check each link in the chain: the robots.txt rules, the response your server sends, and the content in that response.

On this page

Which OpenAI crawlers matter for ChatGPT answers?
How do I check my robots.txt for ChatGPT?
Can a firewall block ChatGPT even when robots.txt allows it?
How can I test what OAI-SearchBot gets?
What if the page loads but ChatGPT still says nothing useful?
How does AISeeWhy check this?

Which OpenAI crawlers matter for ChatGPT answers?

OpenAI documents three agents that site owners usually meet. They have different jobs, and blocking one does not block the others. OpenAI states that each robots.txt setting is independent.

OAI-SearchBot finds and indexes pages for ChatGPT's search features. OpenAI says sites opted out of it will not be shown in ChatGPT search answers. This is the one that decides whether you can be cited.
GPTBot collects content that may be used to train OpenAI's foundation models. Disallowing it says your content should not be used for training. It does not remove you from search answers.
ChatGPT-User visits a page when a person asks ChatGPT to, for example by pasting a link. OpenAI says robots.txt rules may not apply to these user-initiated visits, and that this agent is not used to decide whether content appears in search.

So the short version: OAI-SearchBot controls citations, GPTBot controls training, and ChatGPT-User handles live visits on a user's request.

How do I check my robots.txt for ChatGPT?

Open https://yourdomain.com/robots.txt in a browser. Look for a group that names OAI-SearchBot. If there is none, the crawler follows the User-agent: * group. A Disallow: / in whichever group applies blocks the whole site.

Also check that the file answers at all. Under RFC 9309, if robots.txt returns a server error (5xx) or cannot be reached, a crawler must assume the whole site is disallowed. A 404 is fine: it means no rules. A robots.txt address that returns an HTML page with status 200 is a common mistake, because crawlers then find no valid rules in it.

A setup that keeps ChatGPT search and opts out of training looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

OpenAI says changes for search can take about 24 hours to take effect after you update robots.txt.

Can a firewall block ChatGPT even when robots.txt allows it?

Yes, and it is a common cause. robots.txt is a request to well-behaved crawlers, not an access control. Your firewall, CDN or bot protection decides what is actually sent back. If it treats an AI crawler as an unwanted bot, the crawler gets a 403, a 429 or a challenge page such as Cloudflare's "Just a moment..." screen, while people in a browser see the page normally.

From the outside this looks like a site that is open to everyone. Only a request that identifies itself as the crawler shows the difference. In that case robots.txt says yes, the server says no, and the server wins.

How can I test what OAI-SearchBot gets?

Send two requests to the same URL, one with a browser user agent and one with the crawler's user agent, and compare the status code and size. OpenAI publishes an example user agent for each crawler; the version number in it may change.

# As a browser
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
  https://example.com/page

# As OAI-SearchBot
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://example.com/page

If the browser gets 200 with a large body and the crawler gets 403, 429, 503 or a much smaller body, your firewall is treating the crawler differently. One limit applies: your test comes from your own IP address, not from OpenAI's. Firewalls that verify crawlers against the published IP ranges (OpenAI lists them at openai.com/searchbot.json) may let the real crawler through while blocking your test. A block in this test still tells you how the firewall treats the user agent, which is worth fixing.

What if the page loads but ChatGPT still says nothing useful?

Access is only the first step. If the HTML that arrives is an empty shell filled in later by JavaScript, the crawler may receive almost no text. Look at the raw HTML with curl or your browser's view-source: if the main text is not there, the crawler probably does not see it either.

After that come the content signals: a clear short answer near the top of the page, headings that match the questions people ask, a named publisher and author, and visible dates. These do not guarantee a citation, but they make a page easier to quote correctly.

How does AISeeWhy check this?

AISeeWhy fetches your page as OAI-SearchBot, GPTBot, ChatGPT-User and 11 other crawlers next to a normal browser, reads your robots.txt by RFC 9309 rules and names the firewall when one answers with a block or challenge. Each finding comes with the fix to paste, and the free check shows your score and the first finding.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI treats each crawler setting as independent. GPTBot relates to training, while OAI-SearchBot decides whether pages can be shown in ChatGPT search answers. You can disallow one and allow the other.

Does ChatGPT-User follow robots.txt?

OpenAI says that because these visits are initiated by a user, robots.txt rules may not apply. If you want to stop those visits, a robots.txt rule is not a reliable control.

How long until a robots.txt change reaches ChatGPT?

OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust for search.

Is being reachable enough to be cited?

No. Access makes a citation possible; it does not guarantee one. What the page says, and how clearly it says it, still decides whether it is used in an answer.