AI crawler
OAI-SearchBot: the crawler behind ChatGPT search.
OpenAI's search crawler surfaces sites in ChatGPT search. Learn to spot it in your logs and allow or block it in robots.txt, separately from GPTBot.
Short answer
What is this search crawler?
OAI-SearchBot is OpenAI's crawler for ChatGPT's search features. Per OpenAI, it surfaces websites in search results. Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
In brief
Four things to know first.
It powers ChatGPT search
OpenAI says this crawler is used to surface websites in search results in ChatGPT's search features.
It is separate from GPTBot
OpenAI says the two settings are independent. You can allow the search crawler and disallow GPTBot.
Blocking removes you from answers
Opted-out sites will not be shown in ChatGPT search answers. They can still appear as navigational links.
OpenAI recommends allowing it
Allow it in robots.txt and allow requests from its published IP ranges so your site can appear in search results.
Who and what
What OAI-SearchBot does with your content.
OpenAI uses several crawlers and user agents for its products. The one tied to search surfaces websites in ChatGPT's search features.
OpenAI runs several crawlers and user agents. Some run automatically, and others run when a user asks. Its search crawler and GPTBot use separate robots.txt tags, so you can control each one.
| Bot | OpenAI's own account | robots.txt |
|---|---|---|
| The search crawler | Surfaces sites in ChatGPT search results | Controls search inclusion |
| GPTBot | Crawls content that may train foundation models | Disallow to opt out of training |
| ChatGPT-User | Visits pages when a user asks | Rules may not apply |
| OAI-AdsBot | Checks safety of pages submitted as ads | Only visits submitted ad pages |
Recognise it
OAI-SearchBot user agent and IPs
OpenAI gives an example user agent string and says the version number may change. Match on the crawler's name, not a fixed version.
A user agent can be copied by anyone. OpenAI publishes the IP ranges for this crawler, so compare the requesting IP with that list.
| Signal | What OpenAI publishes |
|---|---|
| User agent token | The search crawler |
| Example string | Crawler name, version 1.4, link to https://openai.com/searchbot |
| IP list | https://openai.com/searchbot.json |
| robots.txt requests | May carry an extra "robots.txt" marker |
Control it
Allow or block OAI-SearchBot
Control this crawler through its own robots.txt tag.
Decide your goal
Want to appear in ChatGPT search answers? Allow this crawler. Want to stay out? Disallow it.Edit robots.txt
Add a group for this crawler's own token with an Allow or Disallow rule. Keep GPTBot as a separate group.Allow the IP ranges
If a firewall or CDN filters bots, allow requests from the ranges listed at openai.com/searchbot.json.Wait about 24 hours
OpenAI says search systems can take about 24 hours to adjust after a robots.txt change.Check your logs
Look for requests from the published IPs, and confirm they reach your key pages.
Not sure what the crawlers can read?
We fix crawler access, schema and rendering so assistants can reach and cite your key pages.
Trade-off
Should you block OAI-SearchBot?
If you want visibility in ChatGPT search, do not block it. OpenAI says opted-out sites will not be shown in ChatGPT search answers, though they can still appear as navigational links.
To keep content out of model training, OpenAI points to GPTBot instead. Disallowing GPTBot signals that your content should not be used for training. It does not remove you from search.
| Allow it | Block it | |
|---|---|---|
| ChatGPT search answers | Your pages can be surfaced | Not shown in answers |
| Navigational links | Can appear | Can still appear |
| Model training | Controlled by GPTBot, separately | Controlled by GPTBot, separately |
FAQ
OAI-SearchBot: common questions.
What is OpenAI's search crawler?
OpenAI's search crawler surfaces websites in ChatGPT's search features. OpenAI keeps it separate from GPTBot. A site can allow the search crawler while disallowing GPTBot, which signals that its content should not train OpenAI's foundation models.
What is the user agent of OpenAI's search crawler?
OpenAI's example user agent string names the search crawler, gives version 1.4 and links to https://openai.com/searchbot. OpenAI says the version number may change. Match on the crawler's name, not a fixed version.
How do I verify a request really comes from OpenAI's search crawler?
Compare the requesting IP address with the list OpenAI publishes at https://openai.com/searchbot.json. A user agent string alone is easy to fake, so the IP match is the stronger check for a request claiming to be this crawler.
Is this search crawler the same as GPTBot?
OpenAI's search crawler and GPTBot are separate. OpenAI says the settings are independent. A site can allow the search crawler so it appears in search results. The same site can disallow GPTBot to signal that its content should not train OpenAI's foundation models.
Does blocking this search crawler hide me from ChatGPT?
Per OpenAI, opting out of this crawler keeps your site out of ChatGPT search answers. Opted-out sites can still appear as navigational links. Blocking GPTBot instead only signals that content should not train OpenAI's models, not that it should leave search.
Does ChatGPT-User follow my search-crawler rules?
ChatGPT-User does not decide whether content appears in Search. OpenAI says to manage Search opt-outs through its search crawler's robots.txt rules. Users initiate ChatGPT-User actions, so robots.txt rules may not apply to it.
See whether ChatGPT can reach and cite your pages.
We run your buyers' questions through ChatGPT, Claude, Perplexity and Gemini and show you the gap.