Can Gemini see my website? How to check

Published · Updated

Short answer

Gemini can use your page in a grounded answer only if the page is in Google's Search index and your robots.txt does not disallow the Google-Extended token. Google says grounding in the Gemini app gives the model content from the Google Search index, and that Google-Extended controls whether your content may be used for it. So check indexing in Search Console, check robots.txt for Googlebot and Google-Extended, then ask Gemini your buyers' questions and open its sources.

Gemini does not have a crawler you can watch for in your logs the way you can for other assistants. Google crawls with its existing crawlers, and a robots.txt token decides what Gemini may do with the content. That makes the check different: most of it happens in Google Search Console and in your robots.txt, and the last step happens in Gemini itself.

On this page
  1. The 5-minute check, step by step
  2. How does Gemini get information from websites?
  3. How do I check if Google has indexed my page?
  4. Is Google-Extended stopping Gemini from using my site?
  5. Can a firewall block Google while browsers get through?
  6. How do I see whether Gemini uses or cites my site?
  7. Do AI Overviews, AI Mode and snippet rules affect Gemini?
  8. How does AISeeWhy check this?

The 5-minute check, step by step

  1. Search Google for site: plus your page address, or use URL Inspection in Search Console, to confirm the page is in Google's index.
  2. Open your robots.txt and make sure Googlebot may fetch the page and Google-Extended is not disallowed, unless you chose to opt out.
  3. Run a free check of the page on AISeeWhy or send one request with the Googlebot user agent: it must get status 200 and the same text as a browser.
  4. Ask Gemini a question your page answers and open the sources it shows under the answer.

If the page is not indexed, Gemini's grounding cannot use it, so start with step 1. The sections below explain each step and what to do when it fails.

How does Gemini get information from websites?

Google's Gemini Apps Privacy Hub says some Gemini responses are grounded on Search results. Google's crawler documentation defines grounding as providing content from the Google Search index to the model at prompt time, to improve factuality and relevance. The same page says Google-Extended controls whether content Google crawls may be used for training Gemini models and for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.

Google-Extended has no user agent string of its own. Google says crawling is done with its existing user agents, and the token only works in robots.txt as a control. In practice this gives two conditions for Gemini to use your page: Google must have it in its index, and your robots.txt must not disallow Google-Extended for it.

Google does not document how Gemini chooses which indexed pages to use or cite. Meeting both conditions makes your page available; it does not guarantee that Gemini picks it.

How do I check if Google has indexed my page?

Use the URL Inspection tool in Google Search Console. For the indexed version it shows:

Test live URL fetches the page in real time, and View tested page shows a screenshot and the HTML Google received. If the screenshot is a challenge page or the HTML lacks your main text, Google, and with it Gemini, does not see your content. Request indexing asks Google to crawl the URL again; Google says it usually takes about a day, with no guarantee of indexing.

Is Google-Extended stopping Gemini from using my site?

Open your robots.txt and look for a group that names Google-Extended. A Disallow: / there tells Google not to use your pages for Gemini training or for grounding in Gemini Apps. Google says the token does not affect inclusion in Google Search and is not a ranking signal, so this block is easy to miss: your search traffic looks normal while Gemini cannot ground answers in your pages. Our separate article on AI Overviews and Google-Extended covers what the token does and does not control in Search.

Because the token never appears in requests, you cannot test it with curl or find it in logs. Reading the file is the only check:

# Show the Google-Extended and Googlebot groups with the lines after them
curl -s https://example.com/robots.txt | grep -n -i -A4 -E "^user-agent: *(google-extended|googlebot)"

# The * group applies to any token without its own group
curl -s https://example.com/robots.txt | grep -n -i -A4 "^user-agent: \*"

Matching is case-insensitive, and if no group names Google-Extended, the * group applies to it. Many sites added this block when AI training was first discussed. In our study of the 1,000 most popular domains, 114 of the 518 sites with a robots.txt named Google-Extended and 94 (18.1%) disallowed their home page for it. Decide whether that still matches what you want, given that it also covers grounding.

Can a firewall block Google while browsers get through?

Yes. If bot protection or a rate limit answers Google's crawlers with a 403, a 429 or a challenge page, Google cannot crawl the page, and over time it can fall out of Search and, with it, out of Gemini's grounding. URL Inspection is the best test here, because Google fetches from its own addresses; a curl test from your computer shows only how your rules treat the user agent.

To allow Google safely, verify the visitor rather than trusting the name. Google documents a reverse DNS lookup followed by a forward lookup, and publishes IP ranges for its common crawlers in common-crawlers.json:

# Reverse lookup: the name must end in googlebot.com, google.com or googleusercontent.com
host 66.249.66.1

# Forward lookup: must return the original address
host crawl-66-249-66-1.googlebot.com

Replace the example address with one from your logs. If the two lookups do not match, the request only claims to be Google.

How do I see whether Gemini uses or cites my site?

Ask Gemini the questions your buyers ask, without naming your site, and open the sources. Google's help center says the Sources button appears at the bottom of a response or inline, and opens a panel of related links that can include public websites. If there is no Sources button, Gemini did not provide links for that response, and Google notes that not all responses include them. Record which domains appear and repeat the same questions over a few weeks.

Developers can run the same test through the Gemini API with Grounding with Google Search. Google says it connects the model to real-time web content, and grounded answers carry citations with the URL and title of each source, so you can check whether your domain appears. API answers can differ from what the Gemini app shows.

Some Gemini-related fetches do reach your server. Google lists user-triggered fetchers such as Google-GeminiNotebook, which requests URLs that users add as sources to a Gemini Notebook project, and Google-Agent, used by agents on Google's infrastructure that act on a user's request. Google says these fetchers generally ignore robots.txt because a user asked for the fetch. Their visits show that people open your pages through those products; they are not the grounding path described above.

Do AI Overviews, AI Mode and snippet rules affect Gemini?

AI Overviews and AI Mode are Google Search features with their own controls. Google lists nosnippet, data-nosnippet, max-snippet and noindex as ways to limit what is shown from your pages. Its robots meta tag documentation says nosnippet also prevents content from being used as a direct input for AI Overviews and AI Mode, and that max-snippet limits how much may be used. Search Console also has a site-wide Search generative AI setting that can exclude a site from AI Overviews, AI Mode and generative AI features in Discover.

None of these pages mentions the Gemini app. Do not assume that a snippet rule or the Search Console setting changes Gemini's grounding, or that Google-Extended changes AI Overviews. noindex is the exception: by Google's definition of grounding, a page outside the Search index cannot be used for it. To find snippet rules you may have forgotten, check the HTML and the headers:

# Robots meta tags and data-nosnippet attributes in the HTML
curl -s https://example.com/page | grep -i -o -E '<meta[^>]*name="(robots|googlebot)"[^>]*>|data-nosnippet'

# The same rules sent as an HTTP header
curl -s -I https://example.com/page | grep -i "^x-robots-tag"

How does AISeeWhy check this?

The free AISeeWhy check fetches your page as Googlebot and the other AI and search crawlers next to a real browser, applies your robots.txt to each, including the Google-Extended token, recognises challenge pages and shows how much text arrives without JavaScript. The AI visibility check, on paid plans, asks your questions through the Gemini API with Google Search grounding on your own key and shows whether your brand or domain is mentioned or cited. It does not query AI Overviews or AI Mode.

Free tools

Next steps

Crawlers in this article

Frequently asked questions

Can I check whether Gemini can see my site for free?

Yes. Search Console, a site: search, your robots.txt and a question in Gemini cost nothing. AISeeWhy's free check tests Googlebot, the Google-Extended rule and 14 other crawlers for one page a day.

Does Gemini have its own crawler?

Not for grounding. Google says Google-Extended has no user agent string and that crawling is done with its existing crawlers. Separate user-triggered fetchers such as Google-GeminiNotebook and Google-Agent visit pages when a user asks, and Google says they generally ignore robots.txt.

If I block Google-Extended, can Gemini still mention my brand?

Google says it will not use your content for grounding in Gemini Apps. That rule covers your own pages. Google does not say anything about content on other sites that write about you, so Gemini may still name your brand from those sources.

How long until a robots.txt change reaches Gemini?

Google does not give a time for Gemini. Its robots.txt documentation says Google generally caches the file for up to 24 hours, so allow at least a day before testing again, and longer for pages Google recrawls rarely.

Do AI Overviews and Gemini use the same sources?

Google does not document how the two relate. Both draw on Google's index, but they have different controls: snippet rules and the Search Console setting for AI Overviews and AI Mode, and Google-Extended for Gemini grounding. Check them separately.

How do I track Gemini answers over time?

Keep a fixed list of buyer questions, ask them in Gemini on a schedule and record which domains appear in the sources. AISeeWhy's AI visibility check runs your questions weekly through the Gemini API with Google Search grounding and shows mentions next to up to four competitors. Results from the API can differ from the Gemini app.

Sources

More articles