Which Agency Audits Sites for LLM Crawlability and Citation-Readiness?
By Karim MezitiSeptember 20, 2026Updated June 2026
Plenty of sites that rank well on Google are structurally invisible to AI search engines. Not because the content is weak or the brand unknown, but because nobody has checked whether the crawlers behind ChatGPT, Perplexity and Claude can reach the pages at all.
That is what an LLM crawlability and citation-readiness audit is for. It asks two questions a traditional SEO audit never does: can AI systems access your site, and when they can, is the content structured so it can be extracted and attributed?
The traffic alone makes the case. Cloudflare compared more than 30 AI and search crawlers across May 2024 and May 2025 and found GPTBot requests up 305%, with PerplexityBot posting the steepest rise of the cohort at 157,490% from a very small base. In a separate part of the same analysis, Cloudflare looked at robots.txt on 3,816 domains from the top 10,000 and found most had no explicit position on AI crawlers either way.
An LLM audit is not an SEO audit
An SEO audit asks whether Googlebot can crawl, index and rank. An LLM audit asks whether AI retrieval systems can access, extract and cite. Different inputs, different outputs, and different failure modes.
| Audit | What it checks | What failure looks like |
|---|---|---|
| SEO | Googlebot access, indexing, ranking | Pages not indexed, low rankings |
| LLM crawlability | AI agent access, render parity, schema, entity signals | Pages invisible to the engines entirely |
| Citation-readiness | Answer structure, extractability, corroboration | Crawled, read, and never named |
A site can pass every SEO check and fail every LLM check, because the AI crawlers have their own user-agents, their own access rules and their own rendering limits.
One practitioner benchmark gives a sense of scale. Yaniv Goldenberg scored 61 SaaS and AI company sites against a published rubric using his own open-source auditing tool and found 56% scoring 60 out of 100 or below, with recurring gaps in schema coverage, entity signals and explicit crawler permissions. Treat the number as directional rather than definitive: it is one person's rubric, scored with their own tool, and their own site sits at the top of the leaderboard as a reference deployment. What it illustrates is the shape of the problem, which matches what turns up in practice.
Layer one: can the crawlers reach you
Crawler access in robots.txt
The first check is whether each agent is explicitly addressed. OpenAI alone runs three with different jobs, and the names are not interchangeable:
OAI-SearchBotsurfaces sites in ChatGPT's search features. This is the citation agent.GPTBotcollects content that may train foundation models.ChatGPT-Userfetches a page because a user asked for it.Claude-SearchBotandClaudeBotsplit the same way for Anthropic, alongsideClaude-User.PerplexityBotindexes for Perplexity results, and Perplexity states it is not used for model training.Googlebotfeeds the index that AI Overviews draws on;Google-Extendedcontrols Gemini training only and has no effect on indexing.
A blanket User-agent: * with Disallow: / blocks all of them at once. Plenty of sites set something like that years ago for reasons that no longer apply and have never revisited it.
Checking at the edge, not just in robots.txt
Cloudflare, Akamai and AWS WAF bot rules are enforced before robots.txt is ever read, so a permissive robots.txt proves nothing on its own. The check is to fetch a page as each agent and read the status code. A 403 where robots.txt says allow means the block is at the edge.
JavaScript render parity
No vendor documents a rendering guarantee for its retrieval agent, so the safe assumption is that content existing only after a client-side framework runs is not seen. The test is direct: compare what curl returns with the agent's user-agent against what the browser shows. If your key paragraph is missing from the raw response, that is the problem.
Sitemaps, canonicals and status codes
A sitemap should contain canonical, indexable URLs returning 200 and nothing else. Canonicals should be self-referencing or point at the real destination rather than at something that redirects. Deleted pages should return 404 or 410 rather than a 200 with an apology on it, which gets indexed as content.
Speed, honestly
Load time gets quoted in GEO content with more precision than the evidence supports. The correlational data that exists comes from tooling vendors rather than the engines, and at least one of those vendors places paint metrics outside its own top twenty influencing factors. Slow pages are worth fixing for reasons that predate AI search. Treating a specific paint threshold as a citation lever is not supported.
Layer two: can they safely cite you
Access is the prerequisite. A page an engine can reach but cannot cleanly extract gets read and passed over.
Answer-first structure
Engines extract from the top of sections. A page that spends its opening paragraph setting context loses to one that answers immediately. The working test for each section: does it stand alone, is it short, and does it carry a specific number or named entity?
Schema, and the rule that matters more than the markup
The types that earn their place are Organization with sameAs, Article with author and publisher tied to that Organization by a stable @id, and FAQPage where there are real questions. Every question and answer in the JSON-LD must also exist in visible HTML on the same page. Schema describing content that is not there gives an engine nothing to lift.
Entity consistency
Engines cross-reference a brand across independent sources. If the name, description and category differ between your site, LinkedIn, Crunchbase and the directories you appear in, that cross-referencing degrades. The audit checks those against each other and flags the mismatches.
What exists about you off your own domain
A brand that only exists on its own website has nothing corroborating it. Third-party mentions, community threads and earned media are what an engine uses to confirm you are real and relevant.
Community presence is worth auditing specifically, and worth understanding correctly. Ahrefs, studying 1.4 million ChatGPT prompts, found Reddit made up 67.8% of non-cited URLs and was cited just 1.93% of the time through its dedicated retrieval channel. ChatGPT reads it heavily and names it rarely. That makes community signal a context input on ChatGPT rather than a citation source, while on other engines it behaves differently. Either way, what the community says about you is worth knowing before you try to change anything else.
LLMReach does not use fake accounts, automated comments, vote manipulation, undisclosed promotion, or content designed to imitate genuine community discussion.
llms.txt: it turns up on gap lists, including the benchmark cited above, but none of the four major engines documents reading it and none has confirmed it changes retrieval or citation behaviour. It is cheap to add and there is no evidence it moves anything. Do it last, if at all, and be sceptical of anyone selling it as a lever.Finding out whether you have a problem
Before committing to anything, the first question is whether a gap exists at all. The LLMReach free AI visibility audit answers it. During a guided review meeting, LLMReach walks you through your priority buyer prompts, current AI visibility, competitor citations, source patterns, and the technical or content gaps that matter most. You leave the call knowing where the gap is, what is causing it, and which changes would matter first.
Your audit is reviewed live on the call. It is not emailed as a PDF.
An audit without execution is a list of problems you still have
Most agencies offering a GEO audit deliver a document. The findings above are not hard to produce; acting on all of them, and keeping them current as engines change what they weight, is the part that does not happen on its own.
| Discipline | What it addresses |
|---|---|
| Prompt research | The queries your buyers actually run, which everything else is measured against |
| Technical | Crawler access at robots.txt and at the edge, rendering, schema, entity signals, crawl hygiene |
| Content | Answer-first restructuring of the pages the prompt set says matter |
| Community | Accurate brand signal where buyers discuss the category |
| Citation tracking | Citation frequency per prompt per engine against the baseline |
Where LLMReach runs the full Citation Stack, the guarantee is specific. No single input is guaranteed on its own: no individual page, platform, prompt or AI response is promised. The Citation Stack is guaranteed as a whole: a 30% increase in total citations across your site, measured against your own baseline, within 90 days. If that is not reached, you choose between continued work at no charge and a full refund.
The category is still early. Most brands have not checked whether their robots.txt blocks OAI-SearchBot, whether their CDN blocks it anyway, or whether their key pages survive a fetch without JavaScript. Those are afternoon-sized problems with compounding consequences, and the audit is where the list starts.
Frequently asked questions
What is an LLM crawlability audit?
It checks whether AI crawlers can reach your pages and whether those pages survive without JavaScript. That means robots.txt rules for each retrieval agent, the CDN or WAF rules enforced before robots.txt is read, render parity, sitemap and canonical health, and status codes. It is the access layer, separate from whether content is worth citing.
What is citation-readiness?
Citation-readiness is whether a page an engine can reach is structured so it can be extracted and attributed. It covers answer-first section openings, schema that mirrors visible content, consistent entity naming across independent sources, freshness, and whether anything off your own domain corroborates that you exist.
How is an LLM audit different from an SEO audit?
An SEO audit asks whether Googlebot can crawl, index and rank. An LLM audit asks whether AI retrieval agents can access and cite. A site can pass every SEO check and fail every LLM check, because the AI crawlers have their own user-agents, their own access rules and their own rendering limits.
Is a permissive robots.txt enough to allow AI crawlers?
No. CDN and WAF bot rules from Cloudflare, Akamai or AWS are enforced before robots.txt is read, so an agent can be allowed in the file and blocked at the edge. The only reliable check is to fetch a page as each agent and read the status code returned.
Does llms.txt improve AI citations?
There is no evidence that it does. None of the four major engines documents reading llms.txt, and none has confirmed it changes retrieval or citation behaviour. It appears on gap lists because it is easy to check, not because it has been shown to matter. Add it last, if at all.
During a guided review meeting, LLMReach walks you through your priority buyer prompts, current AI visibility, competitor citations, source patterns, and the technical or content gaps that matter most. You leave the call knowing where the gap is, what is causing it, and which changes would matter first.