Can Claude see my website? How to check
Claude can read your page only if your robots.txt allows Anthropic's agents and your server sends them the same HTML a browser gets. Check that robots.txt does not disallow Claude-SearchBot or Claude-User, that your firewall does not answer them with a 403 or a challenge page, and that the main text is in the initial HTML. Then test Claude itself: paste your URL into a chat and ask your buyers' questions with web search on.
If you ask Claude about your product and it cites other sites, the first question is whether it can reach yours at all. That has a testable answer. Anthropic publishes the robots.txt tokens of its agents and the IP addresses they come from, and Claude shows the sources it used. So you can check each step in order: what your robots.txt says, what your server sends, what your logs record, and what Claude does with a page when you give it one.
On this page
- The 5-minute check, step by step
- How does Claude read websites?
- How do I check if Claude can access my website through robots.txt?
- How do I test what Claude's agents receive from my server?
- How can I see whether Claude actually visits my site?
- How do I test Claude itself with my page?
- What if Claude can read my page but still does not cite it?
- How does AISeeWhy check this?
The 5-minute check, step by step
- Open your robots.txt and make sure no group for
Claude-SearchBot,Claude-UserorUser-agent: *disallows the page.ClaudeBotonly matters for training. - Run a free check of the page on AISeeWhy or send one request with the
Claude-Useruser agent: it must get status 200 and the same text a browser gets. - Search your access log for
Claude-SearchBotandClaude-Userand look at the status codes they received. - Paste your URL into Claude with web search on, ask it to summarize the page, then ask a question your buyers ask and check the citations.
If step 1 or 2 fails, fix access first; the sections below show how. If Claude reads the page but does not cite it, work on the content.
How does Claude read websites?
Claude reaches the web in two ways. With web search, it searches when a question needs current information, reviews several sources and answers with citations. Anthropic's help center says every response that uses web search includes citations. When someone gives Claude a specific link, it can open that page directly.
Anthropic documents three agents for this, each with its own robots.txt token. Claude-SearchBot crawls to improve Claude's search results, Claude-User fetches pages when a user's question needs them, and ClaudeBot collects content that may be used for training. Our separate guide to Anthropic's three crawlers covers what each one does in detail. For being read and cited, the first two are the ones to check.
One honest limit: Anthropic does not document which search index Claude's web search uses or how it chooses sources. It says blocking Claude-SearchBot may reduce your visibility in search results and blocking Claude-User prevents retrieval for user questions. Allowing both makes a citation possible; it does not guarantee one.
How do I check if Claude can access my website through robots.txt?
Open https://yourdomain.com/robots.txt and look for groups that name Claude-SearchBot or Claude-User. If neither is named, both follow the User-agent: * group. Token matching is case-insensitive under RFC 9309, and a Disallow: / in whichever group applies closes the whole site to that agent.
Check the response itself too. A robots.txt that returns a server error tells crawlers to assume everything is disallowed. A 404 means no rules. Two commands show the status and every group in the file:
# Status and content type of robots.txt (expect 200 and text/plain, or 404)
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/robots.txt
# Every user-agent, allow and disallow line, with line numbers
curl -s https://example.com/robots.txt | grep -n -i -E "^(user-agent|allow|disallow):" Anthropic says rules must be in the robots.txt at the top level of each subdomain, so blog.example.com needs its own file. These blocks are not rare. In our study of the 1,000 most popular domains, 67 of the 518 sites that served a robots.txt (12.9%) disallowed the home page for Claude-SearchBot and 68 (13.1%) for Claude-User, while 110 (21.2%) did so for the training crawler ClaudeBot.
How do I test what Claude's agents receive from my server?
robots.txt is a request, not access control. A firewall, CDN or bot protection can answer Anthropic's agents with a 403, a 429 or a challenge page while your own browser sees the page normally. Anthropic says its bots will not try to bypass CAPTCHAs, so a challenge page ends the visit.
Request the same URL with a browser user agent and with each Anthropic token, then compare status and size:
URL=https://example.com/page
for UA in "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
"Mozilla/5.0 (compatible; Claude-SearchBot)" \
"Mozilla/5.0 (compatible; Claude-User)"; do
curl -s -o /dev/null -w "%{http_code} %{size_download} $UA\n" -A "$UA" "$URL"
done Anthropic's help page names the tokens but does not publish full user agent strings, so these are test strings, not Anthropic's real ones. They show how rules that match the name treat the request. A 403, 429 or 503 for the Claude lines only, or a 200 that is much smaller than the browser's, points to your firewall.
Your test comes from your own IP address. Anthropic says requests from addresses in its list at claude.com/crawling/bots.json are from Anthropic. If your firewall allows verified crawlers by IP, the real agents may get through while your test is blocked, and the reverse is also possible. Use the list to allow Anthropic, not to block it: Anthropic advises against IP blocking because it also stops its bots from reading your robots.txt.
How can I see whether Claude actually visits my site?
Your access logs answer this directly. Search them for the three tokens, then check that the source addresses fall inside Anthropic's published ranges, because anyone can copy a user agent name:
# Requests that name an Anthropic agent: count by client IP and status code
grep -i -E "claude-searchbot|claude-user|claudebot" access.log \
| awk '{print $1, $9}' | sort | uniq -c | sort -rn | head -20
# Anthropic's published IP ranges
curl -s https://claude.com/crawling/bots.json | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix' The first command assumes the common combined log format, where field 1 is the client address and field 9 the status code. Many 403 or 429 codes next to Anthropic's agents confirm a firewall problem. No visits at all from Claude-SearchBot mean only that it has not crawled those pages yet, or that something earlier stops it.
How do I test Claude itself with my page?
Open a chat and paste the URL of one of your pages with a request such as "Summarize this page". Anthropic's help center says that with web search on, Claude can open links you provide. If web search is off, turn it on from the + menu; in Anthropic's newer Claude experience there is no toggle and Claude searches when it helps. On Team and Enterprise plans an owner must first enable web search for the organization. If Claude cannot open your page while a browser can, go back to the firewall test.
Then ask the questions your buyers ask, without naming your site, and open the cited sources. Anthropic says Claude may use a location inferred from your IP address for localized results, so answers can vary by place and between runs.
Developers can repeat this through the Claude API. The web search tool takes an allowed_domains list, so a request limited to your domain shows whether Claude's search returns any of your pages for a question:
curl https://api.anthropic.com/v1/messages \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2026-10-21" \
-H "content-type: application/json" \
-d '{
"model": "claude-sonnet-5-5",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "What does example.com offer?"}],
"tools": [{
"type": "web_search_20250305",
"name": "web_search",
"max_uses": 3,
"allowed_domains": ["example.com"]
}]
}' Empty search results mean Claude's search found nothing on that domain for the question. Anthropic charges for API web searches in addition to tokens. The API's web fetch tool also reports why a page could not be read: url_not_allowed covers blocks such as robots.txt, and url_not_accessible means an HTTP error. API answers can differ from what the Claude apps show.
What if Claude can read my page but still does not cite it?
First make sure the text it needs is in the HTML your server sends. Anthropic's API documentation says its web fetch tool does not support websites rendered with JavaScript, and Anthropic does not say that its crawlers render scripts. Fetch your page with curl and search the output for a sentence from your main content. If it is missing, server-side rendering or prerendering fixes this for Claude and for most other AI crawlers.
After that, the page itself decides. Put the answer to the page's main question in its first sentences, use headings that match the questions people ask, name the publisher and show dates. Then compare your page with the one Claude cites for the same question: the cited page often answers that exact question more directly.
How does AISeeWhy check this?
The free AISeeWhy check fetches your page as Claude-SearchBot, Claude-User, ClaudeBot and the other AI and search crawlers next to a real browser, applies your robots.txt to each, recognises challenge pages and shows how much text arrives without JavaScript. The AI visibility check, on paid plans, asks your questions through Claude with web search on your own API key and shows whether your brand or domain is mentioned or cited.
Next steps
- robots.txt tester and checker for AI crawlers
- AI crawler IP check
- See an example full report
- AI crawler monitoring
- Plans and prices
Crawlers in this article
Frequently asked questions
Can I check whether Claude can see my site for free?
Yes. Your robots.txt, a curl request, your server log and a question in Claude cost nothing. AISeeWhy's free check tests Anthropic's agents and 13 other crawlers for one page a day without an account.
Why can't Claude open my website when I paste the link?
The usual causes are a firewall or bot protection that answers Anthropic's agents with a 403 or a challenge page, a robots.txt rule that disallows Claude-User, or a page whose text only appears after JavaScript runs. Test the URL with curl as a browser and with the Claude tokens, and check your logs for blocked requests.
Does Anthropic publish a user agent string I can test with?
Anthropic's crawler help page lists the tokens ClaudeBot, Claude-SearchBot and Claude-User but no full user agent strings. To tell real Anthropic traffic from imitators, compare the source IP with the list at claude.com/crawling/bots.json.
How long does a robots.txt change take to reach Claude?
Anthropic does not give a time. RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours, so allow at least a day before you judge the result in your logs or in Claude's answers.
Does the Claude API give the same answers as the Claude app?
Not necessarily. The app may use your approximate location and its own settings, while an API request uses the tools and parameters you send. Treat API results as a repeatable test, not as a copy of what each user sees.
How do I know whether Claude mentions my brand?
Ask Claude, with web search on, the questions your buyers ask, and note whether it names or cites your site and which competitors appear instead. Repeat each question over several weeks, because answers vary. AISeeWhy's AI visibility check does this weekly through the Claude API with your key, next to up to four competitors.
Sources
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Anthropic: Enable and use web search
- Claude Platform Docs: Web search tool
- Claude Platform Docs: Web fetch tool
- Anthropic: crawler IP address list (bots.json)
- RFC 9309: Robots Exclusion Protocol
- AISeeWhy: AI Crawler Access Report, 2026-11