AI crawler
Claude-SearchBot: what it does and how to control it.
Claude searchbot is one of three Anthropic robots. Learn what it does with your pages, how to recognise it in your logs, and what blocking it costs.
Short answer
What is this crawler?
Claude-SearchBot is an Anthropic crawler that navigates the web to improve search result quality in Claude. It analyzes content to make search responses more relevant and accurate. It is separate from ClaudeBot, which collects content for model training.
In brief
Four things to know first.
A search crawler
Anthropic says this bot improves search result quality for users. It is not the training crawler.
Verified by IP list
Anthropic publishes its crawler IP addresses at claude.com/crawling/bots.json for you to check against.
Controlled in robots.txt
Anthropic says its bots honor robots.txt. Use this bot's token to allow or block it.
Blocking has a cost
Anthropic says blocking it may reduce your visibility and accuracy in user search results.
Who and what
Anthropic's crawler for Claude search.
Anthropic runs three robots: one for model development, one for searching the web, and one for retrieving content at users' direction. The search robot is the one covered here.
Anthropic uses separate robots so website owners get transparency and can choose which uses to allow.
| Robot | What it does | If you block it |
|---|---|---|
| ClaudeBot | Collects content that could contribute to model training | Future materials are excluded from training datasets |
| Anthropic's search crawler | Improves relevance and accuracy of search results | Content is not indexed for search optimization |
| Claude-User | Visits sites when a Claude user asks a question | Content is not retrieved for a user's query |
Recognise it
How to spot Claude-SearchBot.
Look for the Claude-SearchBot name in your server logs. Anthropic says this robot analyzes online content to make search responses more relevant and accurate.
Anthropic publishes a list of its crawler IP addresses. If a crawler's source IP is on that list, Anthropic says it comes from Anthropic.
User agent name
Search your logs for Claude-SearchBot. Anthropic says this robot navigates the web to improve search result quality for users.
IP list
Compare the source IP with claude.com/crawling/bots.json.
Crawl speed
Anthropic says it aims to avoid disruption and respects Crawl-delay where appropriate.
Malfunction reports
Email claudebot@anthropic.com from an address at your domain so the report can be verified.
Are AI crawlers reaching your pages?
Technical GEO checks crawler access, rendering and schema, so the pages you want cited can be reached and read.
Control it
Allow or block it in robots.txt.
Anthropic says its bots honor industry-standard robots.txt directives and will not try to bypass CAPTCHAs. Edit robots.txt rather than blocking IP addresses.
Open robots.txt
Edit the file in your site's top-level directory.Name the bot
Add the User-agent line that names Anthropic's search robot.Choose your rule
Use Disallow: / to block the whole site. Leave it out to allow crawling.Repeat per subdomain
Anthropic says the block must be added for every subdomain you want to opt out.Skip IP blocking
Anthropic warns this may not hold, because it stops its bots reading your robots.txt.
The trade-off
Should you block it?
Anthropic says disabling this bot prevents it from indexing your content for search optimization, which may reduce your visibility and accuracy in user search results.
Blocking ClaudeBot is a separate choice about training. You can allow the search bot while blocking ClaudeBot, because each robot has its own robots.txt token.
| Allow it | Block it | |
|---|---|---|
| Search visibility | Content can be indexed for Claude search | May reduce visibility and accuracy in results |
| Your control | Rules stay in robots.txt | Takes effect only through robots.txt |
| Training | Unaffected; ClaudeBot is separate | Unaffected; ClaudeBot is separate |
FAQ
Claude-SearchBot: common questions.
Is this bot the same as ClaudeBot?
ClaudeBot and Anthropic's search robot are different bots. Anthropic says ClaudeBot collects web content that could contribute to model training. The search robot analyzes content to improve the relevance and accuracy of search results.
Does this bot respect robots.txt?
Anthropic's bots respect robots.txt, according to Anthropic's help centre. It says they honor industry-standard do-not-crawl directives, support the non-standard Crawl-delay extension and will not attempt to bypass CAPTCHAs. It warns that blocking IP addresses may not work as an opt-out.
How do I block this bot in robots.txt?
To opt out, add a User-agent line naming Anthropic's search robot, followed by Disallow: /, to the robots.txt file in your top-level directory. Anthropic says to repeat this for every subdomain you want to opt out. It also warns that blocking its IP addresses may not work.
What happens if I block this bot?
Blocking this bot prevents Anthropic from indexing your content for search optimization. Anthropic says this may reduce your site's visibility and accuracy in user search results. It does not change what ClaudeBot collects for training, which has its own rule.
How can I verify a request is from Anthropic?
Check the request's source IP against the list Anthropic publishes at claude.com/crawling/bots.json. Anthropic says a crawler with a source IP on that list is coming from Anthropic. A user agent name alone is not proof.
Can I slow this crawler down?
You can slow it with the non-standard Crawl-delay extension in robots.txt, which Anthropic says it supports. Add the bot's User-agent line followed by a Crawl-delay value. Anthropic also says it tries to avoid intrusive crawling.
Find out if Claude can find and cite your site.
We run your buyers' questions through ChatGPT, Claude, Perplexity and Gemini and show you the gap.