Apple crawler
Applebot: what Apple's crawler does with your site.
Apple's crawler feeds Siri, Spotlight and Safari. The Extended token is a separate opt-out for AI model training. Here is how to recognise both and control them.
Short answer
What is Apple's search crawler, and what is the Extended token?
Applebot is Apple's web crawler. Its data powers search in Spotlight, Siri and Safari. Applebot-Extended is a robots.txt token, not a crawler. Disallowing it opts your content out of training Apple's generative AI models.
In brief
Four things to know first.
One crawler, several uses
Data from Apple's crawler powers Spotlight, Siri and Safari search. It may also help train Apple's generative AI models.
The Extended token is an opt-out
It does not crawl pages. It only controls whether data from Apple's search crawler trains Apple's AI models.
Blocking AI training keeps you in search
Apple says pages that disallow the Extended token can still appear in search results.
Another control stops AI answers
The nosnippet tag keeps your content out of Apple's AI-generated answers.
Who and what
What Applebot does with your content
Apple runs this crawler. According to Apple, the data it crawls powers search features across its ecosystem, including Spotlight, Siri and Safari.
Apple also says the data may help train its foundation models, and may give AI models context and current content for answers shown in Apple products.
| Use | Where it appears | How you control it |
|---|---|---|
| Search | Spotlight, Siri, Safari | Allow or disallow Apple's crawler |
| Model training | Apple Intelligence, Services, Developer Tools | Disallow the Extended token |
| AI answer context | Broad world knowledge answers in Siri and Search | Use the nosnippet meta tag |
Recognise it
How to spot Applebot in your logs
Apple says the user agent string contains the crawler's name plus other browser details, followed by a version number and a link to Apple's crawler help page.
Apple adds that its crawler occasionally updates the browser version it advertises. Match on the crawler-name token, not the full string.
Match the token
Look for the crawler-name token in the user agent. The surrounding browser details change.
The Extended token never appears
Apple says it does not crawl webpages, so you will not see it in logs.
The iTMS agent is different
It crawls only URLs tied to registered Apple Podcasts content and ignores robots.txt.
Control it
Allow or block Applebot in robots.txt
Apple says its search crawler respects standard robots.txt directives. If your file names Googlebot but not Apple's crawler, it follows the Googlebot rules. It ignores crawl-delay.
Decide what you want
Choose search presence, AI training, AI answers, or any mix of the three.Opt out of training only
Name Apple's extended training token in a User-agent line, then add Disallow: /. Apple says search listings stay.Keep content out of AI answers
Add the nosnippet meta tag to specific content. The crawler still fetches the page.Block search entirely
Name Apple's search crawler in a User-agent line, then add Disallow: /. Your pages leave Apple's search results.Do not block page resources
Apple says blocking JavaScript or CSS may stop its crawler rendering your content properly.
Not sure which crawlers can reach you?
We check crawler access, rendering and schema on the pages you want cited, then fix them in order of impact.
The trade-off
Should you block it?
Allowing Apple's crawler lets your content appear in search for Apple users worldwide, Apple says. Disallowing the Extended token costs you no search visibility, and Apple says its rules are not considered in Search ranking.
Apple says allowing the Extended token helps improve its generative AI models. That is a gain for Apple, not a promised benefit to you. The choice is yours.
| Allow the Extended token | Disallow it | |
|---|---|---|
| Apple search listings | Stay | Stay, per Apple |
| Training Apple's models | Your content may help | Your content is opted out |
| Crawling by Applebot | Continues | Continues |
FAQ
Apple's crawler: common questions.
Is the Extended token a separate crawler?
The Extended token is not a crawler. Apple says it does not crawl webpages. It is a robots.txt token that controls whether data Apple's search crawler has already collected may train Apple's generative AI models. You will not see it in your server logs.
What is Apple's crawler user agent?
Apple's user agent string carries the crawler's name alongside browser details, a version number and a link to Apple's help page. Apple says it occasionally updates the advertised browser version, so match on the crawler-name token rather than the whole string.
Does blocking the Extended token remove me from Siri and Spotlight?
Blocking the Extended token does not remove you from Apple search. Apple says pages that disallow it can still appear in search results, and that your content stays discoverable through Spotlight, Siri and Safari. It only opts you out of generative model training.
Does Apple's crawler obey crawl-delay?
Apple's crawler does not follow the crawl-delay directive. Apple says its crawl rate adjusts automatically when a site slows down or returns errors, and that it caches crawled content to reduce unnecessary crawling.
How do I keep my content out of Apple's AI-generated answers?
Apply the nosnippet meta tag to the specific content. Apple says it will not use nosnippet data as extra context when AI models generate output in its products. The crawler can still fetch the page, so the content stays discoverable in search.
Does Apple's search crawler respect robots.txt?
Apple's search crawler respects standard robots.txt directives in general search crawls aimed at it. If your file does not mention it but does mention Googlebot, it follows the Googlebot rules. The iTMS agent for Apple Podcasts content is the exception and ignores robots.txt.
Find out which AI systems can read your site.
We run your buyers' questions through ChatGPT, Claude, Perplexity and Gemini and show you the gap.