2 pages refused the crawl

Short answer

These pages answered 401, 403, 429 or 503, or timed out, when the audit fetched them as a browser. That is often a firewall, bot protection or rate limit rather than a broken page. Check whether search and AI crawlers get the same answer.

crawl.unreachable · Priority: P1–P3, by crawler type

What it means

These pages answered 401, 403, 429 or 503, or timed out. A firewall or rate limit may be stopping crawlers too; check whether search and AI crawlers get the same answer.

Why it matters for AI search

The audit fetches many pages in a short time from one server, so a rate limit can answer 429 or 503 even when people get the page. If the same rule applies to search and AI crawlers, these pages are not fetched for their indexes and cannot be cited in AI answers.

A 401 means the page needs a login, which is expected for account areas but not for public content. A 503 on every page may mean maintenance mode was left on. These pages are kept apart from broken links because the fix depends on who is being refused.

How to fix it

  1. Open a few of the listed URLs in a private browser window. If they load, the block is aimed at automated requests.
  2. Look for the refused requests in your firewall, CDN or hosting logs. Make sure verified search crawlers such as Googlebot and Bingbot, and the AI search crawlers you want, are allowed; many CDNs, Cloudflare among them, have a setting for verified bots.
  3. If a rate limit caused it, raise the limit for normal page requests or leave HTML pages out of it.
  4. If pages answer 401, remove the login from public pages, or stop linking to private pages from public ones.
  5. Run the site audit again from your account to confirm the result.

How to confirm the fix

After the change, press "Check again" on the report: it lists what was fixed and how the score changed.

Check my page →

All findings and how the score works