AI bot directory
Every one of the AI crawlers in your logs
Search engines, AI assistants and data collectors all send bots to your pages. Open the profile for the bot you spotted and see how to allow or block it.
Every bot, one page each
Find the bot you need
- GooglebotGoogle's Search crawler: verify it and what blocking costs.
- Google-ExtendedA robots.txt token for Gemini, not a crawler.
- GPTBotOpenAI's model-training crawler, and how it differs from OAI-SearchBot.
- OAI-SearchBotOpenAI's search crawler: what it does, how to spot it, how to allow it.
- ChatGPT-UserOpenAI's agent that opens pages when a ChatGPT user asks.
- ClaudeBotAnthropic's training crawler: what it collects and how to block it.
- Claude-SearchBotAnthropic's crawler for Claude search results.
- PerplexityBotPerplexity's search crawler, and how it differs from Perplexity-User.
- BingbotBing's main crawler: user agent, verification and robots.txt.
- AmazonbotAmazon's crawler: what it does and how to control it.
- ApplebotApple's crawler for Siri, Spotlight and Apple Intelligence.
- PetalBotHuawei's Petal Search crawler: user agent, verification, robots.txt.
- BytespiderByteDance's crawler for LLM training data, per Cloudflare.
- CCBotCommon Crawl's open-dataset crawler, used to train AI models.
- Meta-ExternalAgentMeta's crawlers for AI training and on-request fetching.
- facebookexternalhitMeta's link-preview crawler: what it does, how to spot and control it.
- YouBotYou.com's reported crawler: thin evidence, no verification.
- DuckAssistBotDuckDuckGo's crawler for AI-assisted answers: what it does and how to block it.
- MistralAI-UserMistral's agent that fetches pages for Vibe users.
Free AI Audit
Not sure which bots help you?
We check your robots.txt, server logs and key pages, then show which AI crawlers can reach your content and which ones you should allow or block.