ChatGPT crawlers are four distinct web agents operated by OpenAI: GPTBot (training crawler), OAI-SearchBot (search index crawler), ChatGPT-User (live session fetcher), and OAI-AdsBot (ad landing page validator). Each has a different job, a different user-agent string, and a different access rule. Blocking the wrong one removes you from a surface you probably want to be on. Allowing all four without understanding them means losing control over training data use. This guide covers what each crawler does, what it reads, how to control it, and what the consequences of each decision look like in practice, using OpenAI's official documentation as the primary source throughout.

In our crawler access work with brands across industries, we have run this audit on hundreds of sites and find two failure modes with consistent frequency: teams block GPTBot to avoid training data use and unknowingly eliminate themselves from ChatGPT search results, and teams assume a correct robots.txt guarantees access when a WAF or CDN rule is silently returning 403 to every AI bot request. Both are fixable once identified. Both are invisible in standard analytics. And both are fully preventable with the technical knowledge in this guide.

The Four OpenAI Crawlers That Matter for AI Visibility

According to OpenAI's crawler documentation, "OpenAI uses web crawlers ('robots') and user agents to perform actions for its products, either automatically or triggered by user request." The documentation lists four named user agents, each with an independent access rule.

The four named user agents in OpenAI's crawler ecosystem are: GPTBot, OAI-SearchBot, ChatGPT-User, and OAI-AdsBot. Each token has a distinct role: GPTBot is OpenAI's training data crawler; OAI-SearchBot is the search index crawler that determines ChatGPT Search citation eligibility; ChatGPT-User is the live-session fetcher triggered by real user queries during active ChatGPT conversations; OAI-AdsBot is the ad validator that visits landing pages submitted for ChatGPT advertising. These four identifiers are the tokens that appear in robots.txt rules and in your server logs.

The independence is important. OpenAI confirms that "each setting is independent of the others," meaning a decision to block one bot has no effect on the others. Most teams that lose ChatGPT visibility do so because they conflated training data collection with search indexing and blocked the wrong agent.

GPTBot: The Training Crawler

GPTBot is OpenAI's training crawler, collecting content that may be used to build future model versions. OpenAI's crawler documentation states that GPTBot "is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."

Its example user-agent string, as published in OpenAI's documentation:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

OpenAI publishes its IP addresses at https://openai.com/gptbot.json. This is the verification anchor for auditing server logs, because user-agent strings can be spoofed, but IP ranges cross-checked against published lists cannot.

GPTBot's crawl volume is significant. "GPTBot alone generated 569 million requests across Vercel's network in a single month, representing a 305% increase in raw requests," and GPTBot's "market share among AI crawlers" grew "from 5% to 30% between May 2024 and May 2025."

OAI-SearchBot: The Search Index Crawler

OAI-SearchBot builds and refreshes the index that underlies ChatGPT's search features. This is the crawler that determines whether your page enters the candidate pool for citation. OpenAI's documentation states: "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

Its example user-agent string:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

OpenAI also uses a variant with a robots.txt marker when fetching your robots.txt file specifically, to help site owners distinguish those requests in server logs:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

IP addresses are published at https://openai.com/searchbot.json. OpenAI notes that "it can take ~24 hours from a site's robots.txt update for our systems to adjust" for search results.

ChatGPT-User: The Live Fetcher

ChatGPT-User is not a crawler in the traditional sense. OpenAI's documentation states that ChatGPT-User is not used for crawling the web in an automatic fashion, and that it fires when users ask ChatGPT or a CustomGPT a question.

Its user-agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

IP ranges are published at https://openai.com/chatgpt-user.json.

A critical policy detail: OpenAI states that "because these actions are initiated by a user, robots.txt rules may not apply." This is a deliberate policy position. OpenAI also notes that "ChatGPT-User is not used to determine whether content may appear in Search," directing use of OAI-SearchBot in robots.txt for managing search opt-outs and automatic crawl.

OAI-AdsBot: The Ads Validator

OAI-AdsBot validates landing pages for advertisers running ChatGPT ads. OpenAI's documentation states: "OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models."

Its user-agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

OpenAI's documentation lists OAI-AdsBot, which it describes as "used to validate the safety of web pages submitted as ads on ChatGPT."

For advertisers, the OpenAI Help Center is explicit: "You must allow OAI-AdsBot. We recommend allowing both OAI-AdsBot and OAI-SearchBot." The same page confirms that "OAI-AdsBot is officially verified and allowlisted by Cloudflare."

How the ChatGPT Search Pipeline Actually Works

Understanding the crawler roles becomes more useful when you see how they connect to the live answer pipeline. Our diagnosis: the content access mistake we see repeatedly is teams treating ChatGPT as a single entity with a single crawler, when in fact it is a two-stage pipeline with distinct technical requirements at each stage. Getting indexed by OAI-SearchBot (stage one) and getting read by ChatGPT-User during a live session (stage two) are separate problems requiring separate solutions.

OpenAI's documentation makes the indexing and live-fetch functions explicit: OAI-SearchBot handles surfacing "websites in search results in ChatGPT's search features," while ChatGPT-User fires "when users ask ChatGPT or a CustomGPT a question, it may visit a web page." The practical framework is: OAI-SearchBot determines whether your page is eligible; ChatGPT-User determines whether it is read.

The Search Indexing and Live Retrieval Path

When ChatGPT determines a query benefits from current web data, it draws on the index built by OAI-SearchBot. Our reading of OpenAI's bot documentation gives the following sequence. User asks a question the model determines needs live data.

ChatGPT issues a search query against its internal index (OAI-SearchBot's output).
The model picks 1 to 5 candidate URLs.
ChatGPT-User fetches each URL with a normal HTTP GET.
Each response is converted to clean text: HTML is stripped and navigation is removed.
The cleaned text feeds the model as context, along with the URL.
The model writes an answer and emits inline citations.
The final response reaches the user with clickable source links. that "ChatGPT-User is the live-fetch bot. If your robots.txt blocks ChatGPT-User, you can never appear in ChatGPT's browse results, regardless of GPTBot." It also confirms that "JS rendering is not part of the fetch path. The response body is processed as text."

What OAI-SearchBot Reads vs What ChatGPT-User Reads

These two bots access your content at different moments and for different purposes. OAI-SearchBot reads your content periodically to maintain the search index. ChatGPT-User reads your content in real time, at the exact moment a user session requires it.

This distinction matters for content freshness. Content updated after OAI-SearchBot's last visit exists in ChatGPT's index in its older form, but ChatGPT-User will fetch the current version when triggered by a live query. Server-side rendering is critical for both: "GPTBot and ClaudeBot cannot execute JavaScript, while Googlebot has full rendering capabilities through its browser-based infrastructure." Content loaded only via client-side JavaScript is invisible to both crawlers regardless of when they visit.

robots.txt: The Primary Control Mechanism

Your robots.txt file is the standard way to express access policy to OpenAI's crawlers. OpenAI uses "OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI."

The Two Decisions You Are Actually Making

"search crawling and training crawling are separate decisions, controlled by separate bots. You can allow citation while blocking training, or the reverse. Conflating the two is how well-meaning teams accidentally make themselves invisible."

Your two primary decisions are independent:

Do you want ChatGPT to cite your content in search answers? If yes, allow OAI-SearchBot. If no, block OAI-SearchBot. OpenAI's documentation confirms: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
Do you want your content used in model training? If no, disallow GPTBot. Blocking GPTBot does not remove you from search citations. OpenAI's documentation confirms this is by design: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models."

A Working robots.txt Configuration

The most common configuration for sites that want ChatGPT search visibility without contributing to training data:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: OAI-AdsBot
Allow: /

If you are comfortable with training data use, replace the GPTBot rule with Allow: /. For teams managing visibility across all AI search engines, extend the file to cover ClaudeBot, PerplexityBot, Google-Extended, and Bingbot. Submitting your sitemap to Bing Webmaster Tools is part of the same access work, since both ChatGPT and Copilot draw on Bing's index.

Token Spelling Is Where This Quietly Breaks

A robots.txt rule only applies when the user-agent token matches exactly what the crawler announces. We see repeatedly that OpenAI's training crawler "uses the token GPTBot," and that "write GptBot or GPT-Bot and the rule does nothing." In their assessment, "the single most common 'we blocked it but it still crawled' finding is a typo in the token, not a logic error." Do not copy tokens from memory. Copy them from OpenAI's official crawler documentation directly.

The ChatGPT-User Exception: Why robots.txt Has Limits

The most important gap to understand in ChatGPT's crawler model is this: robots.txt was designed to govern automated, systematic crawls. It was not designed for user-triggered, real-time fetching. OpenAI treats these as fundamentally different categories, which creates a tier where one crawler plays by different rules.

The Three-Tier System OpenAI Uses

Our diagnosis: the root problem teams encounter here is that they apply the same mental model to all four OpenAI bots, then discover ChatGPT-User behaves differently and interpret that as a bug. It is not. It is a policy choice with a specific rationale.

OpenAI's documentation states that "because these actions are initiated by a user, robots.txt rules may not apply" for ChatGPT-User. The result is a three-tier access model:

Tier 1 (GPTBot): Respects robots.txt. Used for training. Block it to opt out of training data collection.
Tier 2 (OAI-SearchBot): Respects robots.txt. Used for search indexing. Block it to remove yourself from ChatGPT search answers.
Tier 3 (ChatGPT-User): Does not reliably respect robots.txt. Fires on user demand. Control requires server-side methods.

"ChatGPT-User no longer respects robots.txt directives for user-initiated browsing actions," explicitly confirming this three-tier structure.

What This Means for Access Control

The framework for making the right decision here: ask what you are actually trying to prevent. If you want to prevent training data collection, block GPTBot. If you want to prevent search citations, block OAI-SearchBot. If you want to prevent live session fetching, you need server-side controls, because a Disallow rule for ChatGPT-User in robots.txt is not reliable.

The risk of treating all three as the same: you add a robots.txt block for ChatGPT-User, believe you have control, and your content continues to be read during live sessions. The false security is more dangerous than knowing the control does not exist.

For most sites, blocking ChatGPT-User is counterproductive, since ChatGPT drives more AI referral traffic than any other engine.

If reliable blocking is required, server-side user-agent detection against ChatGPT-User combined with IP verification against OpenAI's published ranges at https://openai.com/chatgpt-user.json is the correct technical approach.

Layers Beyond robots.txt: Why Access Control Fails Silently

The diagnosis most teams miss: a clean robots.txt does not mean the crawler can reach your content. There are three additional layers that can intercept a request before it ever reaches your server, and each one is invisible to the team reading robots.txt and concluding "everything is fine."

We see this pattern in our crawler access audits with regularity. A brand invests in content quality, robots.txt is correctly configured, and citation counts do not move. The first diagnostic step we run is always the same: can the crawler actually reach the page? In our experience, the WAF block is the single most common silent failure on sites that have recently migrated hosting, changed CDN provider, or enabled a new security tier.

The OpenAI Help Center identifies the access stack explicitly, noting that "most websites have multiple layers of protection before a crawler can successfully access a webpage." The guide names three: robots.txt, web protection and bot mitigation, and human verification and anti-bot logic.

WAF and CDN Blocks: The Silent Killer

Many sites run web application firewalls (WAF) or CDN-level bot protection that intercepts crawler requests before they reach your server. The guide specifically names "Cloudflare, Akamai, or other web protection providers" as sources of this problem, noting that "these systems can mistakenly block legitimate crawlers, often returning 403 Forbidden errors."

The risk is concrete: your robots.txt says Allow, your content team publishes strong material, and Cloudflare's bot-protection toggle returns 403 to every GPTBot or OAI-SearchBot request. Your analytics shows nothing unusual because AI bots do not execute JavaScript and never appear in GA4. The failure is completely silent.

To test this, CheckVibe recommends fetching your site with a crawler user-agent:

curl -A "GPTBot/1.2 (+https://openai.com/gptbot)" -I https://yourdomain.com

A 403, 429, or challenge page means the crawler is blocked at the edge. Repeat this test with OAI-SearchBot and ChatGPT-User user-agents, since WAF rules often treat each identifier differently.

JavaScript Rendering: The Invisible Content Problem

The rendering gap is the second silent failure. "GPTBot and ClaudeBot cannot execute JavaScript, while Googlebot has full rendering capabilities through its browser-based infrastructure." Vercel's analysis, referenced in the same source, shows that while the ChatGPT crawler does make JavaScript file requests, "they're only fetching these files as text for training purposes, not executing them."

This creates a diagnostic trap: content ranks well in Google (which renders JavaScript) while being completely invisible to OpenAI's crawlers. Your site can be page one in Google search and a blank response to GPTBot simultaneously, and nothing in your dashboard will tell you this.

The recommendation: server-side render or statically generate every page you want AI crawlers to index. Next.js does this by default for server-rendered pages, but client-only useEffect data fetching and dynamically loaded marketing content remain common sources of the problem.

IP Verification: Distinguishing Real Crawlers From Spoofed Traffic

A third layer shapes how you interpret your access data. Not every request claiming to be GPTBot is actually from OpenAI: a user-agent string is a header a client sets for itself, so anything can send one. OpenAI publishes three crawler range files: gptbot.json, chatgpt-user.json, and searchbot.json. Cross-referencing a visiting IP against those published ranges is the only reliable way to confirm the visit is from OpenAI.

The commercial consequence of skipping this step: your AI analytics may show GPTBot visiting certain pages frequently, leading you to conclude those pages are valued by OpenAI, when you are actually tracking scraper traffic wearing OpenAI's name. The gap between detecting a visit and verifying it is where the analysis goes wrong, and it goes wrong quietly, because a spoofed hit looks exactly like a real one in a log.

How to Audit ChatGPT Crawler Access in Four Passes

Pagechecks.com identifies that "access is not one switch. It is four separate layers, and a block can hide in any of them." Their methodology works through the layers in order, stopping when a block is found.

A Four-Pass Audit Sequence

Run these passes in order. A failure at an outer layer makes inner layers irrelevant:

Robots.txt fetch. Confirm your robots.txt file returns HTTP 200, parses without errors, and uses the exact token spellings published in OpenAI's bot documentation.
Path-level rules. Confirm the rule that applies to the specific path you care about, not just the homepage. A global Allow: / can be overridden by a more specific Disallow: /blog/ rule below it. The failure that hides longest is the one where the homepage is reachable and a high-value folder is not, because every spot check on the domain root comes back clean.
Page-level response. Fetch the page directly using curl -A with a crawler user-agent and inspect the HTTP response code and headers. A noindex or nosnippet directive can suppress citation even when robots.txt is correct.
CDN and edge. Check whether your WAF or CDN returns a 403 or 429 to the crawler user-agent before the request reaches your server. CheckVibe notes that some bot-protection products ship "one-click 'block AI bots' toggles" that are enabled by default on new plans.

What a Clean Audit Result Actually Means

Passing all four passes confirms the crawler can reach the content. It does not guarantee citation. As CheckVibe frames it, "there are exactly three places the failure can live: access (the crawler can't reach you), extraction (it reaches you but sees nothing), or grounding (it sees you but won't cite you)." This four-pass audit addresses only the first layer. For extraction and grounding, the work moves to content structure and authority, which is the domain of AI citation optimization.

In our audits, access failures are among the most common reasons brands with strong content still do not appear in ChatGPT citations. These are silent failures: no error message reaches your content team, and standard analytics shows nothing unusual. Access must be confirmed before content quality becomes the variable. A brand that fixes its content but leaves a WAF block in place will see zero improvement in recommendation share.

The Consequences of Each Control Decision

The four OpenAI crawlers require separate decisions, and the consequences are direct. The table below maps each crawler to its primary control mechanism and the effect of blocking it.

Crawler	Job	Primary control	Effect of blocking
GPTBot	Training data collection	robots.txt Disallow	Content excluded from future model training. You remain eligible for ChatGPT search citations via OAI-SearchBot.
OAI-SearchBot	Search index for ChatGPT	robots.txt Disallow	Pages do not appear in ChatGPT search answers. May still appear as navigational links.
ChatGPT-User	Live fetch during user sessions	Server-side blocking (robots.txt may not apply)	ChatGPT cannot read and cite your pages during active conversations.
OAI-AdsBot	Ad landing page validation	robots.txt Disallow or WAF block	Ad landing pages cannot be validated; ads may not serve.

Blocking GPTBot

Blocking GPTBot removes your content from potential use in OpenAI's model training. It does not remove you from ChatGPT's search index. OpenAI's documentation is explicit on this, "If you block GPTBot in robots.txt, you're telling OpenAI not to use your content for model training. You're not blocking yourself from appearing in ChatGPT answers, that's a different bot. This is the most commonly misunderstood point about bot management."

The choice to block GPTBot carries no direct cost to your search visibility. Many publishers make this choice while keeping OAI-SearchBot allowed.

Blocking OAI-SearchBot

Blocking OAI-SearchBot removes your site from the candidate pool for ChatGPT search citations. OpenAI's documentation states that "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." The effect is reversible, but the delay is roughly 24 hours per OpenAI's published timeline, and your content team will not receive any notification that the block is in place.

In our experience, this is the block that most commonly happens by accident. A wildcard Disallow: * rule intended to block scrapers, or a robots.txt file copied from a privacy-focused configuration, catches OAI-SearchBot along with everything else. The site appears fine in Google Search Console while being completely absent from ChatGPT's index.

Blocking ChatGPT-User

Blocking ChatGPT-User via robots.txt may have limited practical effect. OpenAI's documentation notes robots.txt rules "may not apply" to ChatGPT-User because "these actions are initiated by a user." If reliable blocking is required, server-side detection using the published user-agent string and IP ranges is the appropriate approach.

Blocking ChatGPT-User has a direct visibility cost. In our own client analytics, live-session traffic is the part clients notice first Removing yourself from the engine generating the most AI-sourced traffic, while believing robots.txt gave you control, is a commercially significant mistake.

Crawler Traffic You Can See (and Traffic You Cannot)

AI bot traffic is almost entirely invisible to standard analytics tools. GA4, Adobe Analytics, Matomo, all of them operate by loading a JavaScript snippet in the user's browser. Bots don't run JavaScript. They make direct HTTP requests to your server, read the HTML, and leave. The analytics tag never fires.

To find AI crawler activity, filter raw server logs or CDN analytics for these user-agent strings, then verify IPs against OpenAI's published ranges:

GPTBot (training crawler, scheduled visits)
OAI-SearchBot (search index crawler, continuous)
ChatGPT-User (live fetcher, user-triggered)

User-agent filtering alone conflates real OpenAI crawlers with spoofed traffic. IP verification is what separates them.

llms.txt: A Complementary Signal, Not a Replacement

The llms.txt convention, a markdown file at the root of your site giving AI models a curated index of your most important content, is increasingly mentioned alongside robots.txt. addresses it directly: "You will see advice to add an llms.txt file to your root. It is harmless, and some practitioners include it, but there is no solid evidence that major engines currently use it for ranking or citation. Treat it as optional, and do not let it distract from the things that demonstrably matter: crawler access, page structure, and sourcing. Spend the hour on your robots.txt and schema instead."

We take the same position. If your site is blocked to OAI-SearchBot, an llms.txt file will not compensate. The path to improving visibility in AI answers starts with confirmed crawler access, then moves to content structure and entity authority.

Frequently Asked Questions

If I block GPTBot, will ChatGPT stop citing me?

No. GPTBot is OpenAI's training crawler, not its search crawler. OpenAI's documentation confirms that OAI-SearchBot governs search indexing independently of GPTBot. You can block GPTBot and remain fully citable in ChatGPT search answers, as long as OAI-SearchBot is allowed. Babypenguin.ai confirms: "Block GPTBot if you don't want training, but leave OAI-SearchBot alone."

Does ChatGPT-User respect robots.txt?

Not reliably. OpenAI's crawler documentation states that "because these actions are initiated by a user, robots.txt rules may not apply." This is a deliberate policy. If you need to block ChatGPT-User access reliably, server-side user-agent detection combined with IP verification against OpenAI's published ranges at https://openai.com/chatgpt-user.json is the appropriate method.

How do I verify that a GPTBot visit in my logs is actually from OpenAI?

Cross-reference the visiting IP against OpenAI's published IP ranges at https://openai.com/gptbot.json. User-agent strings can be copied by any scraper. IP verification against OpenAI's published ranges is what separates real traffic from impersonators. For OAI-SearchBot, use https://openai.com/searchbot.json.

How long does it take for robots.txt changes to affect ChatGPT search?

OpenAI states that "it can take ~24 hours from a site's robots.txt update for our systems to adjust" for search results. This applies specifically to OAI-SearchBot's effect on ChatGPT search citations. Changes affecting GPTBot operate on a different schedule, depending on when the training crawl next revisits your domain.

What is OAI-AdsBot and do I need to allow it?

OAI-AdsBot validates landing pages for ChatGPT ads. If you are not running ChatGPT ads, OAI-AdsBot will not visit your site: OpenAI's documentation confirms it "only visits pages submitted as ads." If you are running ads, blocking it prevents ad landing pages from validating.

Can I allow OAI-SearchBot while blocking GPTBot at the same time?

Yes, and this is explicitly supported. OpenAI's documentation states that "each setting is independent of the others," giving this exact example: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models." The two rules sit in separate User-agent groups in your robots.txt and do not interact.

Next Step: Audit Your Access and Close the Visibility Gap

Getting robots.txt right is the baseline, but it is only the first gate. A blocked WAF rule, a JavaScript-rendered page, or a misconfigured CDN policy can eliminate you from ChatGPT citations even when robots.txt is correct. Because AI bot traffic is invisible to standard analytics, you may not know you are blocked until you look specifically at server logs and CDN data.

Start with a free AI visibility audit. We will check your crawler access across all four OpenAI bots, identify the exact rules blocking or allowing each one, and show you where your content currently sits in ChatGPT's citation pool.

For teams that need enterprise AI visibility tracking across a large site, crawler access is the starting point for understanding why recommendation share does not reflect content investment. The access layer and the content quality layer both need to be right, and they need to be audited separately.

To get cited by ChatGPT consistently, crawler access is the prerequisite. Content structure and authority determine whether an accessible page becomes a cited one. Our guide on working with a GEO agency walks through how the access and content layers are typically managed together.

Book a call to walk through your current crawler access configuration and identify where your biggest gap is.

ChatGPT Crawlers: The Complete Technical Guide (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot)