AISeeWhy
EN
Start free trial

Perplexity's crawlers: PerplexityBot and Perplexity-User

Published

Short answer

Perplexity uses PerplexityBot to surface and link websites in its search results, and Perplexity-User to visit pages when a user's question needs them. Perplexity says PerplexityBot follows robots.txt and is not used to train foundation models, while Perplexity-User generally ignores robots.txt because a user requested the fetch.

Perplexity is built around citations: answers come with numbered links to the pages they draw on. Its crawler documentation is short, and it states clearly which agent follows robots.txt and which does not. That difference decides which tool you use if you want to control either one.

On this page
  1. What does PerplexityBot do?
  2. What does Perplexity-User do?
  3. What does robots.txt control, and what not?
  4. How is Perplexity different from ChatGPT and Claude?
  5. What robots.txt should I use for Perplexity?
  6. How do I make sure the firewall does not block Perplexity?

What does PerplexityBot do?

PerplexityBot is designed to surface and link websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models. To appear in Perplexity's search results, Perplexity recommends allowing PerplexityBot in robots.txt.

Perplexity documents this user agent:

Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

The version number may change, so firewall rules should match the token PerplexityBot. Perplexity publishes the IP ranges at perplexity.ai/perplexitybot.json.

What does Perplexity-User do?

Perplexity-User supports user actions. When someone asks Perplexity a question, it may visit a web page to help give an accurate answer and include a link to that page. Perplexity says it is not used for crawling the web or collecting training data.

Its documented user agent is Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user), and its IP ranges are published at perplexity.ai/perplexity-user.json.

What does robots.txt control, and what not?

This is the key point. Perplexity states that because a user requested the fetch, Perplexity-User generally ignores robots.txt rules. A robots.txt group for PerplexityBot affects indexing for search; it does not reliably stop live fetches on a user's behalf.

Perplexity says each robots.txt setting works independently, and that changes may take up to 24 hours to be reflected.

How is Perplexity different from ChatGPT and Claude?

All three vendors separate a search crawler from a user-requested fetcher. Two details set Perplexity apart. First, it does not run a separate training crawler: it says PerplexityBot is not used to crawl content for AI foundation models, so there is no training token to choose about. Second, it is explicit that its user fetcher generally ignores robots.txt, where OpenAI says robots.txt may not apply to ChatGPT-User and Anthropic says its bots, including Claude-User, honor robots.txt.

In practice this means a single decision for Perplexity's search crawler in robots.txt, and a firewall decision if you ever want to stop user-requested fetches.

What robots.txt should I use for Perplexity?

If you want to be cited, allow the search crawler explicitly. Remember that a named group does not inherit rules from *, so repeat any private paths:

User-agent: PerplexityBot
Allow: /
Disallow: /account/

User-agent: *
Allow: /
Disallow: /account/

Since PerplexityBot is not a training crawler, blocking it to opt out of model training would cost you search visibility without serving that goal.

If you list a sitemap in robots.txt, keep the Sitemap: line outside any group or at the end of the file; it applies to all crawlers. Also make sure the sitemap itself is not blocked by your firewall, since a crawler that cannot fetch it cannot use it to find new or changed pages.

How do I make sure the firewall does not block Perplexity?

Bot protection often treats any non-browser request as suspicious. Request your page with each Perplexity user agent and compare the response with a browser request. If the crawler gets a 403, a 429 or a challenge page, add an allow rule that matches the token and, ideally, the published IP ranges, so that only real Perplexity traffic gets through. In Cloudflare, AI Crawl Control lists AI crawlers with an Allow or Block action for each.

Verify rather than trust the user agent. Anyone can send a request that says PerplexityBot, so an allow rule based on the user agent alone admits imitators too, and a block based on it can be dodged. Checking the source IP against Perplexity's published lists makes both kinds of rule reliable.

AISeeWhy tests PerplexityBot and Perplexity-User alongside 12 other crawlers. Because both decide whether Perplexity can use your page in an answer, a block of either is reported as P1, with the rule to fix it.

Frequently asked questions

Does Perplexity use my content to train AI models?

Perplexity says neither PerplexityBot nor Perplexity-User is used to collect content for training AI foundation models.

Why does Perplexity-User ignore robots.txt?

Perplexity's reasoning is that a user asked for the page, so the fetch acts on that user's behalf rather than as automated crawling. Its documentation says the fetcher generally ignores robots.txt for that reason.

How can I block Perplexity-User if I need to?

Use your firewall or CDN, matching the user agent token and Perplexity's published IP ranges. robots.txt is not a dependable control for this agent.

How long does a robots.txt change take to apply?

Perplexity says it may take up to 24 hours for its systems to reflect changes.

Will blocking PerplexityBot stop Perplexity from using my content at all?

No. It keeps your pages out of the crawl for Perplexity's search index, but Perplexity-User can still fetch a page when a user asks about it, because that fetcher generally ignores robots.txt.

Sources

More articles