AI crawlers and AEO, explained
Practical guides for site owners, marketers and developers: which AI crawlers exist, what they are allowed to read, and how to make sure AI answers can read and cite your pages.
Can ChatGPT see my website? How to check
How to check whether ChatGPT can read and cite your site: OAI-SearchBot, GPTBot and ChatGPT-User, robots.txt, firewalls and a quick curl test.
Read article →OpenAI crawlersOpenAI crawlers explained: GPTBot, OAI-SearchBot and ChatGPT-User
What OpenAI's three crawlers do, which robots.txt token controls what, and what blocking GPTBot, OAI-SearchBot or ChatGPT-User changes for your site.
Read article →Firewalls and CDNsIs Cloudflare blocking AI crawlers? How to find out and fix it
Why Cloudflare and other WAFs answer AI crawlers with 403s or challenge pages, how to spot it, and how to let AI search crawlers through safely.
Read article →robots.txtrobots.txt for AI crawlers: what to allow, what to block
How to write robots.txt for AI crawlers: allow AI search, choose about training, the token for each crawler, RFC 9309 matching rules and common mistakes.
Read article →llms.txtWhat is llms.txt, and is it worth adding?
llms.txt explained: what the proposal is, its Markdown format, who reads it today and how to write one in ten minutes, with an honest look at adoption.
Read article →AEO basicsAEO vs SEO: what is the same and what changes for AI answers
AEO compared with SEO: what stays the same, what changes for AI answers (access, answer-first content, entities, freshness) and a checklist.
Read article →Claude crawlersAnthropic's crawlers: ClaudeBot, Claude-SearchBot and Claude-User
Anthropic's three web agents: what ClaudeBot, Claude-SearchBot and Claude-User do, their robots.txt tokens and what blocking each one changes.
Read article →Perplexity crawlersPerplexity's crawlers: PerplexityBot and Perplexity-User
Perplexity's two agents: what PerplexityBot and Perplexity-User do, which one robots.txt controls, and how to allow them or block them at the firewall.
Read article →Structured dataStructured data for AI answers: what to mark up and how
Which schema.org markup helps AI answers read your page: Organization with sameAs, Article dates and author, FAQPage and Product with price, as JSON-LD.
Read article →JavaScript renderingCan AI crawlers read content rendered by JavaScript?
Why content rendered only by JavaScript is often missing for AI crawlers, how to check what they receive, and how to fix it with SSR or static HTML.
Read article →