AI crawler
PerplexityBot: what it does and how to control it.
This is the Perplexity bot that finds pages for its search results. Here is its user agent, how to verify it, and what blocking it costs you.
Short answer
What is this crawler?
PerplexityBot is Perplexity's crawler. Perplexity says it is designed to surface and link websites in Perplexity search results and is not used to crawl content for AI foundation models. Perplexity-User is a separate agent, used when someone asks a question.
In brief
Four things to know first.
It powers search results
Perplexity says this bot surfaces and links websites in Perplexity search. It is not for foundation-model training.
Two agents, two jobs
The first bot crawls for search. Perplexity-User fetches a page when a person asks a question.
Verify by user agent and IP
Perplexity publishes the IP ranges for each agent as JSON files, so you can confirm a request is real.
Blocking has a cost
Perplexity recommends allowing this bot if you want your site to appear in its search results.
Who runs it
PerplexityBot versus Perplexity-User
Perplexity documents two agents. The first builds search results. Perplexity-User visits a page to help answer a specific question from a user.
| Perplexity's search crawler | Perplexity-User | |
|---|---|---|
| Purpose | Surface and link sites in search results | Visit a page to answer a user's question |
| Model training | Not used for foundation models | Not used for crawling or training |
| robots.txt | Follows your rules | Generally ignores them |
Recognise it
Spot PerplexityBot in your logs
Check two things: the user agent string and the source IP. Perplexity publishes its IP addresses as JSON files. Match the IP against perplexitybot.json or perplexity-user.json.
| Agent | User agent | IP list |
|---|---|---|
| Perplexity's search crawler | Mozilla/5.0 AppleWebKit/537.36 user agent naming Perplexity's search bot, version 1.0 | Perplexity's published IP list for its search crawler |
| Perplexity-User | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) | perplexity.com/perplexity-user.json |
Control access
Allow PerplexityBot step by step
Perplexity recommends allowing this bot in robots.txt and permitting requests from its published IP ranges. Firewalls often block it without you noticing.
Allow it in robots.txt
Add a group for this bot in your robots.txt file that does not disallow the pages you want shown.Fetch the IP ranges
Download the IP lists Perplexity publishes: perplexity.com/perplexitybot.json for PerplexityBot and perplexity.com/perplexity-user.json for Perplexity-User. Perplexity updates these ranges regularly.Allow it in your WAF
Add Allow rules for the user agent and IP ranges. In AWS WAF, give them higher priority than blocking rules.Combine both checks
Perplexity advises matching user agent and IP together, for security while admitting real bot traffic.Refresh and monitor
Automate periodic IP updates. Changes can take time to propagate, so watch your logs to confirm requests get through.
Not sure if AI crawlers can reach your site?
We check crawler access, rendering and schema as part of our Technical GEO work, so your key pages are easier to read and cite.
The trade-off
Should you block PerplexityBot?
Blocking this bot works against being listed in Perplexity search. The sources do not say how a block changes citations, so treat that as unmeasured and test it yourself.
Blocking Perplexity's search crawler does not cover Perplexity-User. Perplexity lists them as two separate crawlers with their own user-agent strings, so write a rule for each.
Allow it
Perplexity recommends it if you want your site to appear in its search results.
Block it
Add a robots.txt group for Perplexity's search crawler that disallows your site. Perplexity recommends allowing this crawler if you want your site to appear in its search results.
User fetches
Perplexity-User may still visit a page when someone asks about it. Use a WAF rule to stop it.
FAQ
PerplexityBot: common questions.
What is its user agent?
The crawler's user agent starts with Mozilla/5.0 AppleWebKit/537.36. The string names the bot with version 1.0 and links to Perplexity's page about it. For firewall rules, Perplexity recommends combining user agent matching with IP address verification.
Does this bot train AI models?
Perplexity says this bot is not used to crawl content for AI foundation models. It states the bot is designed to surface and link websites in Perplexity search results. Perplexity-User is also not used to collect content for training.
Does Perplexity-User follow robots.txt?
Perplexity says Perplexity-User generally ignores robots.txt rules, because a person requested the fetch. It visits a page to help answer that user's question and may link to it. To restrict it, use a firewall rule instead of robots.txt.
How do I verify that a request from this bot is real?
Compare the request's source IP with Perplexity's published IP list for its search crawler. Perplexity publishes a separate file at perplexity.com/perplexity-user.json for Perplexity-User. It advises combining user agent matching with IP verification, and refreshing the lists regularly.
Will a firewall block it?
A Web Application Firewall can block it. Perplexity says sites using a WAF may need to explicitly allow its bots. In Cloudflare, set the rule action to Allow. In AWS WAF, give the allow rules higher priority than blocking rules.
Should I block it in robots.txt?
Block it only if you do not want your pages shown in Perplexity search. Perplexity recommends allowing this bot so your site can appear in its results. The sources do not say whether this also governs Perplexity-User.
Check that AI crawlers can read your site.
We run your buyers' questions through ChatGPT, Claude, Perplexity and Gemini and show you the gap.