Changed robots.txt? See how each AI crawler reads it
One line in the wrong group can block ChatGPT search while you meant to block only AI training. Check the result for 16 crawlers.
Free · No signup · Results in 30 seconds
Which group applies
A crawler follows the group that names it, not the * group, which surprises many site owners.
Search versus training
Block training crawlers if you wish, while keeping the ones that put you in AI search answers.
The exact fix
The lines to change, ready to paste, and a recheck to confirm.
After moving to Cloudflare · After a redesign · After moving to Shopify