PerplexityBot

PerplexityBot is Perplexity’s automated crawler for surfacing and linking websites in search results. Perplexity states that it is not used to collect content for AI foundation-model training.

It is separate from Perplexity-User, which may visit a page when a user asks Perplexity a question. Perplexity says this user-initiated fetcher generally ignores robots.txt, so the two tokens should not be treated as interchangeable.

Access choices

To allow search crawling explicitly:

User-agent: PerplexityBot
Allow: /

To request a site-wide block:

User-agent: PerplexityBot
Disallow: /

There is no need to invent a “Perplexity training bot” name. Use the current official crawler list and review it periodically.

robots.txt is only one layer

A permissive rule does not help if a CDN, WAF, rate limiter, authentication wall, or JavaScript challenge rejects the request. Perplexity recommends combining the documented user-agent with its published IP ranges when configuring a WAF.

Verification should cover:

The live robots.txt group and the exact target path.
Source IP membership in the current official JSON list.
Edge and origin response codes for representative pages.
Redirect chains, canonical destinations, and asset access.
Referrals and citations after the crawl, measured separately from access.

Automate IP-list refreshes if firewall rules depend on them. A stale copied range can silently block legitimate requests, while a user-agent match alone can be spoofed.

What an allowed crawl means

It means the crawler is permitted to request the URL. It does not prove that Perplexity indexed the page, selected it for an answer, or sent a click. Track AI referral traffic and citations by landing page, answer topic, and date. Keep “crawler seen,” “page cited,” and “visit received” as separate events.

For paywalled or private content, use authentication and authorization. robots.txt is public and is not a security control.

Primary sources

See also: Perplexity AI, AI crawlers, and robots.txt.