12 AEO checks to run before you publish a page (2027)
Before you publish, confirm the URL returns 200 without redirects, robots.txt allows each AI search crawler, your firewall serves them the real page, and the answer is in the raw HTML. Then check for stray noindex or nosnippet, one canonical, an answer under the H1, valid matching JSON-LD, agreeing dates, a sitemap entry with internal links, hreflang, and request indexing.
Most pages that never show up in AI answers fail on something boring: a leftover noindex, a firewall rule, a canonical pointing at staging. These twelve checks catch those before launch. Each one takes a few minutes, most need nothing more than a terminal and the page URL, and they are listed in the order a crawler meets them.
On this page
- How long does each check take?
- 1. The URL returns 200 without a redirect chain
- 2. robots.txt allows the page for each AI search crawler
- 3. The firewall serves AI crawlers the real page
- 4. The answer is in the raw HTML
- 5. No noindex or nosnippet left over
- 6. One canonical URL, absolute and pointing at itself
- 7. The title and H1 say what the page answers, with the answer right below
- 8. JSON-LD parses and matches the visible text
- 9. Visible date, markup dates and sitemap lastmod agree
- 10. The page is in the sitemap and linked from related pages
- 11. hreflang is reciprocal, with x-default
- 12. Ask search engines to crawl the new page
- How we chose these checks
How long does each check take?
A rough guide for one page, with the tool you need. Run checks 1 to 5 on every page; the rest once per template, then spot-check.
- 1. Status 200: 1 minute, curl.
- 2. robots.txt per crawler: 3 minutes, browser or curl.
- 3. Firewall response: 3 minutes, curl.
- 4. Text in raw HTML: 2 minutes, curl.
- 5. No stray noindex or nosnippet: 2 minutes, curl.
- 6. One canonical: 2 minutes, view-source.
- 7. Title, H1 and short answer: 5 minutes, reading the page.
- 8. JSON-LD valid and matching: 5 minutes, Rich Results Test or Schema Markup Validator.
- 9. Dates agree: 3 minutes, page, markup and sitemap.
- 10. Sitemap and internal links: 5 minutes, sitemap.xml and your CMS.
- 11. hreflang: 5 minutes, only for multilingual sites.
- 12. Request indexing: 2 minutes, Search Console and IndexNow.
1. The URL returns 200 without a redirect chain
A crawler should get the page itself with status 200 on the exact URL you will share and list in the sitemap. Each redirect is one more step that can fail, and a page that answers 200 with a "not found" message is a soft 404 that search engines may drop.
# Final status, number of redirects and final URL
curl -s -o /dev/null -L -w "%{http_code} redirects=%{num_redirects} %{url_effective}\n" https://example.com/new-page/ Pass: 200 redirects=0 and the same URL back. Fix: link and list the final URL, and keep at most one redirect from old addresses.
2. robots.txt allows the page for each AI search crawler
Read robots.txt as each crawler would. A group that names a crawler replaces the User-agent: * group for it, and under RFC 9309 the longest matching rule wins, with Allow winning a tie.
Check at least OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Googlebot. A new section such as /guides/ can be caught by an old Disallow: /g rule, because rules match path prefixes. Also confirm robots.txt itself returns 200 or 404: under RFC 9309 a server error means crawlers should assume the whole site is disallowed.
Pass: no rule for any of these tokens disallows the path. Our guide on how to test robots.txt shows how to test one URL for one crawler.
3. The firewall serves AI crawlers the real page
Request the page with a browser user agent and with crawler user agents, then compare status and size. Bot protection that matches the name can return a 403, a 429 or a challenge page to crawlers only.
URL=https://example.com/new-page/
for UA in "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
"Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
"Mozilla/5.0 (compatible; Claude-SearchBot)" \
"Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"; do
curl -s -o /dev/null -w "%{http_code} %{size_download} $UA\n" -A "$UA" "$URL"
done Pass: every line shows 200 and a similar size. A much smaller size usually means a challenge page. These strings test rules that match the name; real crawlers also come from published IP ranges, which a test from your own machine cannot reproduce, so also check your firewall's allow rules for those ranges.
4. The answer is in the raw HTML
Search the HTML the server sends for a sentence from your answer. The browser's element inspector shows the page after scripts ran, which is not what most crawlers get.
curl -s https://example.com/new-page/ | grep -c "a sentence from your short answer" Pass: 1 or more. A 0 means the text is added by JavaScript, which OpenAI, Anthropic and Perplexity do not document that their crawlers run. Render it on the server or at build time; our guide on JavaScript content and AI crawlers lists the options.
5. No noindex or nosnippet left over
Staging sites often ship with noindex, and a template can carry it to production. Check both the robots meta tag and the X-Robots-Tag HTTP header, since either one counts.
curl -s -D - -o /tmp/page.html https://example.com/new-page/ | grep -i "x-robots-tag"
grep -i -o '<meta[^>]*name="robots"[^>]*>' /tmp/page.html Pass: nothing printed, or only index, follow and similar. Google says a page must be indexed and eligible for a snippet to appear in AI Overviews and AI Mode, and that nosnippet also blocks use as a direct input for them.
6. One canonical URL, absolute and pointing at itself
The page should declare exactly one rel="canonical", as an absolute URL, pointing at the URL you publish. Google calls the canonical a strong signal, not a command, and warns against relative URLs and against conflicting signals such as a sitemap that lists another address.
Common launch mistakes are a canonical copied from a template page, one that still points at a staging host, and two canonical tags from a theme and a plugin at once.
Pass: one tag, the live URL, same protocol and host as in the sitemap.
7. The title and H1 say what the page answers, with the answer right below
Read the top of the page as a stranger would. The title and H1 should name the question or topic in the words buyers use, and the first two or three sentences under the H1 should answer it in full.
Keep the title to about 60 characters so it is less likely to be cut off in search results. Write the short answer so it still makes sense when quoted alone, with numbers, units and the product name spelled out.
Pass: someone who reads only the first paragraph has the answer. Our list of ways to get cited in AI answers covers writing for quotes in more depth.
8. JSON-LD parses and matches the visible text
Broken JSON-LD is ignored as a whole, so one missing comma removes all of it. Paste the URL into Google's Rich Results Test or the Schema Markup Validator and fix every error.
Then compare the values with the page. The headline, dates, price, currency and organization name in the markup should be the ones a reader sees. Google says no special markup is needed for its AI features, and that structured data should match the visible text.
Pass: no errors, and every value is visible on the page. Skip if: the page type has no fitting markup; a plain page is better than wrong markup.
9. Visible date, markup dates and sitemap lastmod agree
Show a labeled date on the page, use the same dates in datePublished and dateModified, and set the sitemap lastmod to the last significant change. Google asks for visible and marked-up dates to agree, and says it uses lastmod only if it is consistently and verifiably accurate.
A sitemap generator that stamps every URL with today's date on each build makes lastmod useless for all pages, not only the new one.
Pass: three places, one date.
10. The page is in the sitemap and linked from related pages
Crawlers find pages through links and sitemaps. Add the new URL to your XML sitemap, using the exact canonical URL, and link to it from at least two related pages and a hub or index page.
A sitemap file can hold up to 50,000 URLs and 50 MB uncompressed; larger sites need a sitemap index. Google says it ignores priority and changefreq, so do not spend time on them. A page with no internal links is an orphan that crawlers may find late or not at all.
Pass: the URL is in the sitemap and at least two pages link to it with descriptive anchor text.
11. hreflang is reciprocal, with x-default
If the page exists in several languages, each version should list all versions, including itself, and each must link back. Google ignores hreflang annotations that are not confirmed from the other side. Add x-default for the version shown when no language matches.
Skip if: the site has one language. Our guide on hreflang x-default explains when you need it.
12. Ask search engines to crawl the new page
Once the page is live and passes the checks above, tell search engines about it. In Google Search Console, use URL Inspection on the new URL and choose Request indexing. For Bing and other engines that support IndexNow, submit the URL to the IndexNow endpoint with your key; Bing notes that this does not guarantee crawling or indexing.
AI assistants have no submission form. Their crawlers find the page through links and search, which is why checks 2, 3 and 10 matter. If you publish an llms.txt file, add the page there too; Google says its Search does not use it, so treat it as optional.
Pass: URL Inspection shows the live page can be indexed.
How we chose these checks
The list follows the path a crawler takes: reach the URL, read robots.txt, pass the firewall, read the HTML, then interpret it. Each check is based on what OpenAI, Anthropic, Perplexity and Google document about their crawlers, on RFC 9309 for robots.txt, and on Google's guidance for canonicals, dates, sitemaps and hreflang. We left out checks with no documented effect on AI answers, such as keyword density or word count targets.
The checks show that a page can be read; they cannot show that it will be cited. That depends on how good the answer is compared with other sources. Repeat checks 2 to 5 after every firewall, CDN, theme or CMS change, because those are the changes that silently undo them.
AISeeWhy runs most of these checks in one pass: it fetches the page as 16 AI and search crawlers next to a real browser, applies robots.txt per crawler and reads the HTML, markup, canonical and dates. The free check shows the score and the first finding; the full report lists each problem with a ready fix.
Fixes for this problem
- OAI-SearchBot is blocked
- Page content needs JavaScript
- Lists written as plain text
- Sections do not open with a short answer: 2 of 3
Next steps
- robots.txt tester and checker for AI crawlers
- llms.txt checker and validator
- See an example full report
- AI crawler monitoring
- Plans and prices
Crawlers in this article
Frequently asked questions
Do I need to run these checks on every page?
Run checks 1 to 5 on every new page, because a single URL can be blocked or carry a stray noindex. Checks 6 to 11 depend mostly on the template, so run them once per page type and spot-check new pages.
Why can't I submit a page to ChatGPT or Claude like I can to Google?
OpenAI, Anthropic and Perplexity do not offer a URL submission tool for web pages. Their crawlers find pages by following links and through search, so internal links, a sitemap and open access for OAI-SearchBot, Claude-SearchBot and PerplexityBot are what you control.
My page passes all twelve checks. Why is it still not cited?
The checks make a page readable, not preferred. An assistant picks the sources that answer a question best, so compare your page with the pages it does cite: whether they answer sooner, state prices or specs in text, or offer data yours lacks.
How soon after publishing do AI crawlers visit?
No vendor publishes a schedule. Your server logs show when OAI-SearchBot, Claude-SearchBot or PerplexityBot first request the URL; our guide on checking server logs for AI crawlers shows how to find them.
Sources
- RFC 9309: Robots Exclusion Protocol
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google Search Central: AI features and your website
- Google Search Central: Robots meta tag, data-nosnippet and X-Robots-Tag
- Google Search Central: How to specify a canonical URL
- Google Search Central: Influence your byline dates
- Google Search Central: Build and submit a sitemap
- Sitemaps.org: Sitemaps XML format
- Google Search Central: Tell Google about localized versions of your page
- Google Search Console Help: URL Inspection tool
- Bing: IndexNow, get started
- Google Search Console Help: Rich Results Test
- Schema.org: Schema Markup Validator
- Google Search Central: JavaScript SEO basics