AI crawler monitoring
A firewall rule, a CDN setting or a new robots.txt can lock AI crawlers out overnight, and nothing on your site looks broken to people. Monitoring rechecks your important pages and emails you when something new blocks them.
Free · No signup · Results in 30 seconds
Plans and prices → See an example full report →
What monitoring checks
Each run is the same check as the free check on our home page. Your page is fetched as 16 AI and search crawlers, as curl and as a desktop browser, each with its published user agent. robots.txt is evaluated for every crawler token, and the structured data, content, dates, sitemaps, hreflang and llms.txt of the page are checked. The result is a 0 to 100 score and a list of findings with priorities P1 to P3.
When you get an email
Monitored pages are rechecked daily or weekly, as you choose. You get an email when a new blocking issue appears, for example a crawler that now gets a 403 or a challenge page, or when the score drops by 10 points or more. The email says what changed and links to the report with the fix.
What it cannot see
Requests come from our server with each crawler's user agent, not from the crawler operators' own IP addresses. A firewall that trusts real crawlers by IP may treat them differently from us, and every block finding says so. To see how the real crawlers were answered, the server-log check on paid plans reads your access log and counts each AI crawler's visits and status codes.
Plans with monitoring
- Pro, $22 a month: 3 monitored pages, for a personal, non-commercial site.
- Business, $88 a month: 25 monitored pages, for company sites and client work.
- Agency, $288 a month: 60 monitored pages for many client sites.
- Enterprise, $888 a month: 100 monitored pages.
The free plan has no monitoring: it gives one check a day with the score and the first finding. Plans are billed monthly, can be cancelled at any time, and every payment can be refunded in full within 14 days.
How to start
- Run the free check on your most important page and look at the first finding.
- Choose a plan on the pricing page and sign in with your email.
- Add the pages to watch under Monitoring in your account: the home page, pricing, top product or service pages and key articles.
- Choose daily or weekly. After a fix, recheck the page to see the score before and after.
Frequently asked questions
Which changes trigger an alert?
A new blocking issue on a monitored page, such as a crawler that is now refused, or a score drop of 10 points or more since the previous check.
Does monitoring use the real GPTBot or ClaudeBot?
No. It sends their published user agents from our server. That shows how your firewall and server treat those user agents; the server-log check shows the real visits.
Can I monitor client sites?
Yes, on Business, Agency and Enterprise. Pro is for personal, non-commercial use.
Can I try it before paying?
The free check shows what one run finds on any page. Monitoring itself needs a paid plan, which you can cancel at any time and get refunded within 14 days.