CCBot
CCBot is the crawler of Common Crawl, a non-profit that maintains a free, open repository of web crawl data. The archive is widely used by researchers and AI developers, so blocking CCBot is mainly a training decision. It does not affect any search engine or AI search product directly.
What it does
Common Crawl is a non-profit that produces and maintains an open repository of web crawl data, available to anyone at no cost. CCBot collects that data. Common Crawl says it checks robots.txt before fetching a page, obeys Crawl-delay, and does not execute JavaScript.
To block it, Common Crawl documents a robots.txt group for the CCBot token with Disallow: /. To slow it down instead, it suggests a Crawl-delay line in the same group.
In our 2026-11 study, 24.1% of popular sites with a robots.txt blocked it. →
robots.txt token
CCBot
User-agent string
CCBot/2.0 (https://commoncrawl.org/faq/)How to allow it
User-agent: CCBot
Allow: /How to block it
User-agent: CCBot
Disallow: /How to verify real visits
Common Crawl runs CCBot on dedicated IP ranges with reverse DNS, so you can check whether a logged request came from the real CCBot. It publishes the IPv4 and IPv6 ranges as JSON and warns that other crawlers falsely identify themselves as CCBot.