Googlebot
Googlebot is Google's main crawler for building its search indexes. Google says robots.txt rules for Googlebot are the control for how a site is crawled for Search, including AI features such as AI Overviews and AI Mode. Blocking it removes your pages from Google Search and those AI answers.
What it does
Googlebot is used to find information for building Google's search indexes. Crawling preferences addressed to the Googlebot token affect Google Search and products such as Google Images, Video, Google News and Discover. It runs as a smartphone and a desktop crawler; both obey the same Googlebot rules, and Google says its common crawlers always obey robots.txt when crawling automatically.
Google's documentation on AI features describes Googlebot rules as the control for Search, including AI Overviews and AI Mode, and points to nosnippet, data-nosnippet, max-snippet and noindex for limiting what is shown. Use for Gemini training is controlled separately by Google-Extended.
In our 2026-11 study, 2.7% of popular sites with a robots.txt blocked it. →
robots.txt token
Googlebot
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36How to allow it
User-agent: Googlebot
Allow: /How to block it
User-agent: Googlebot
Disallow: /How to verify real visits
Google says user agent strings can be spoofed and recommends a two-step DNS check. Run a reverse DNS lookup on the requesting IP: the hostname should end in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and confirm it returns the same IP. Google also publishes the IP ranges of its common crawlers as JSON, so you can match addresses automatically.
Check if your site lets it in →
AEO vs SEO: what is the same and what changes for AI answers →