Technical AEO Checklist

AI citation eligibility is decided before a word of your content is read. Every engine runs a technical filter first: can it reach the page, render it, and resolve who published it? Pass that filter and your content competes on quality. Fail it and the engine moves to the next candidate, however well the page is written.

Crawler access: who is knocking and what you must let in

Each engine sends more than one crawler, and they do different jobs. Getting the names right is the whole exercise, because a rule written against the wrong token does nothing while looking like it works. The table below is taken from current vendor documentation: OpenAI, Anthropic, Perplexity and Google, checked in September 2026.

Training and retrieval are separate decisions

Blocking GPTBot removes your content from OpenAI's training pipeline and does nothing to OAI-SearchBot, which is the agent that fetches pages for live ChatGPT answers. Blocking Google-Extended opts you out of Gemini training and has no bearing on whether Googlebot indexes you for AI Overviews. Two decisions, two sets of rules, and conflating them is the most common way a site loses citations it did not mean to give up.

A robots.txt that keeps every citation agent open while declining to feed the training crawlers looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

That is a starting point, not a recommendation. Whether you feed the training crawlers is a data policy question, and plenty of B2B sites decide the opposite way for good reasons. What is not optional is the top half.

Common misconfigurations

• A wildcard Disallow: / under User-agent: * blocks every crawler that has no rule of its own. If you use one, every retrieval agent above needs an explicit allow.
• Blocking /assets/ or /wp-content/ stops crawlers loading the CSS and JavaScript needed to render the page. The HTML arrives, the rendered content does not.
• Confusing ClaudeBot with Claude-SearchBot. They are separate tokens in Anthropic's documentation and blocking one does nothing to the other. The same trap exists for GPTBot and OAI-SearchBot.
• Allowing the agent in robots.txt and blocking it at the edge. Cloudflare, Akamai and AWS WAF bot rules are enforced before robots.txt is ever read. A clean robots.txt file proves nothing on its own.

How to verify: fetch the page as each retrieval agent and read the status code.

curl -A "OAI-SearchBot" -I https://yourdomain.com/your-page

A 200 confirms access. A 403 means something is blocking it, and if robots.txt looks correct, that something is your CDN or WAF. Repeat for Claude-SearchBot, PerplexityBot and Googlebot. For Googlebot, also run the URL Inspection tool in Search Console, which shows the rendered output rather than just the response code.

JavaScript rendering: what crawlers see when the content loads late

Part 2 covered how to write a passage that extracts cleanly. This section is about whether a crawler can see that passage at all.

On a client-rendered page, the HTML in the initial response may be close to empty, and the visible content only exists once a JavaScript bundle has executed. Google's JavaScript SEO documentation describes indexing as a two-phase process: Googlebot crawls the raw HTML, then queues the page for rendering in a separate pass with a headless Chromium. That second pass can be deferred, so a URL can be discovered well before its content is.

Which crawlers render JavaScript

Googlebot does. For the others, none of the four vendors documents a rendering guarantee, and Google's own page on dynamic rendering notes that "other search engines may choose to ignore JavaScript and won't see JavaScript-generated content." Absent a vendor commitment, the safe assumption for OAI-SearchBot, Claude-SearchBot and PerplexityBot is that JavaScript-only content is invisible. Do not take that on faith either way: the test below tells you what each agent actually receives from your site.

How to tell which category your pages are in

1. Open the page and use View Source, not DevTools. DevTools shows the rendered DOM, which is the thing you are trying to look past. View Source shows what the server sent.
2. Search that source for a sentence you can see on the page. If it is not there, it is JavaScript-rendered.
3. Run curl -A "OAI-SearchBot" https://yourdomain.com/your-page and search the response body for the same sentence. This is the definitive check for non-Google agents.
4. For Googlebot, compare the HTML tab against the Screenshot tab in Search Console URL Inspection. A gap between them is a rendering problem.

The fix

For pages where citation matters, the answer is server-side rendering or static generation. Both put complete HTML in the first response and the question disappears. Dynamic rendering, where the server detects a crawler and serves it pre-rendered HTML, is explicitly not the recommended path: Google's documentation calls it "a workaround and not a long-term solution for problems with JavaScript-generated content in search engines," and it adds a second code path you now have to keep correct forever.

Schema markup: what still earns something and what is cargo cult

Schema is not dead, and the story that it is comes from over-reading two deprecations. Google has withdrawn exactly two schema-backed rich results in recent memory: HowTo in September 2023, and FAQ on May 7, 2026. The search gallery still documents around thirty features, Article, Product, Review snippet, Breadcrumb and Organization among them.

What has changed for B2B content pages is the reason to bother. The SERP feature was never the main prize for most of these page types. Schema now does two jobs: it tells an engine who published this, and it hands over structured content that does not have to be inferred from prose.

Types that earn their place

• Organization with sameAs. The highest-leverage block on the list, and the subject of the next section.
• Article, for content type, author and dates. datePublished and dateModified are the properties that carry the freshness signal.
• FAQPage. The rich result is gone, but the type is still valid Schema.org, still crawled by anything that reads structured data, and still the cleanest way to hand over discrete question and answer pairs. Every pair in the JSON-LD must also exist in visible HTML on the same page.
• BreadcrumbList, which tells an engine where the page sits in the site's structure.
• HowTo, for genuinely procedural content. There is no search feature left to win, but the markup still describes the sequence to any parser that reads it.

Types that are cargo cult

• Review and AggregateRating about yourself. Google's policy is explicit: "If the entity that's being reviewed controls the reviews about itself, their pages that use LocalBusiness or any other type of Organization structured data are ineligible for star review feature," and the same page extends that to embedded third-party review widgets.
• Product on an informational page. It belongs where price, availability and SKU are real and maintained, not on a solutions page.
• Event, Recipe, JobPosting. Valid types, wrong site.

Connect the blocks with @id

Isolated schema blocks are weaker than connected ones. Define the Organization once with a stable @id, then point at that same @id from every Article block's publisher and every Person block's worksFor. That gives an engine a graph it can walk from the content to the entity that published it, instead of a pile of unrelated assertions.

"publisher": { "@id": "https://yourdomain.com/#organization" }

How to verify: a single JSON syntax error silently voids the whole block with nothing visible on the page. Run every implementation through validator.schema.org, which checks the markup itself rather than eligibility for a specific Google feature.

Entity signals: how an engine resolves who you are

When an engine weighs a citation candidate it also weighs the publisher. If it cannot confidently resolve who that is, it defaults to lower trust or skips the citation. This is entity disambiguation, and structured data is how you solve it.

sameAs, and why Wikidata comes first

The sameAs property on your Organization block is an array of URLs pointing at external profiles describing the same entity. Schema.org defines it as the "URL of a reference Web page that unambiguously indicates the item's identity." Each entry is effectively a merge instruction: this profile and this domain are one thing.

For a B2B organization the priority order is:

1. Wikidata. Structured, machine-readable, and feeding the Knowledge Graph that Google's own retrieval draws on. An entry whose official website property is set and whose label matches your Organization name exactly is the fastest route to a resolved entity. Search wikidata.org for your brand first; notability requirements are looser than Wikipedia's, but they are not absent, and an entry for a company with no independent coverage will be deleted.
2. LinkedIn company page. Name, founding year and URL matching your schema exactly.
3. Crunchbase. Funding and founder fields completed.
4. GitHub organization. Worth real effort for SaaS and technical brands. Put the canonical URL in the bio.

Three fields that must match everywhere

Across every sameAs target and your own markup, three things have to be identical: the organization name, the canonical URL, and the description. An inconsistency in any of them blocks the merge. A sameAs URL that redirects or returns anything other than a 200 is worse than omitting it, because it hands the engine a resolution failure instead of a fact.

How to verify: run the Schema Markup Validator to confirm the block parses, then open every sameAs URL by hand and check for a direct 200 with no redirect hop. For the Wikidata entry, confirm the Q-number resolves at query.wikidata.org.

llms.txt: a straight answer on whether it is worth the hour

llms.txt is a plain-text file proposed by Answer.AI in September 2024, offering AI crawlers a structured index of a site's content by analogy with robots.txt. The proposal and its reference material live in the AnswerDotAI/llms-txt repository.

It is not a standard. No standards body has adopted it, and as of September 2026 it does not appear in the crawler documentation of OpenAI, Anthropic, Perplexity or Google. None of the four has publicly confirmed that it reads the file or that its presence changes retrieval or citation behaviour. The adoption that exists is concentrated in developer documentation sites, where it works well as a navigation aid for AI coding assistants. That is a real use case and a different one from B2B marketing pages.

So: if you publish a large documentation library and want coding tools to navigate it, llms.txt is cheap and sensible. For a marketing site under a hundred pages, the hour buys more as schema validation, redirect cleanup or sameAs completion. Implement it last, and do not expect a measurable citation lift from it.

Crawl and index hygiene: the failures that keep good pages out

A page can have flawless content and flawless schema and still never reach an answer, because the crawl layer never delivered it to the candidate pool. These are the specific failures that do it.

Sitemaps

Your XML sitemap should contain canonical, indexable URLs that return 200, and nothing else. Every noindex page, redirect or 404 in there actively misdirects crawl effort. The lastmod value signals when a page last meaningfully changed and crawlers use it to prioritise re-crawls, so an unchanged lastmod on a page you have just rewritten throws away a freshness signal you earned.

Canonicals

Every page needs a self-referencing canonical unless it is deliberately pointing elsewhere. A canonical that points at a URL which then redirects creates a resolution hop the crawler has to unwind, and at scale that is crawl effort spent on nothing. Google's crawl budget documentation describes what gets crawled as the product of a crawl capacity limit and crawl demand, and notes that low-value URLs consume capacity that would otherwise reach your good pages.

Redirect chains

A chain is any path where A redirects to B which redirects to C. Each hop costs latency and crawl effort, and it leaves ambiguity about which URL is authoritative, so an engine may end up citing an intermediate URL that still appears in your sitemap or navigation. Collapse every chain to a single 301 straight to the destination. Run a full crawl before and after any migration, because migrations are where chains are born.

Status codes

Pages that should not exist return 404 or 410. A deleted page that returns 200 with an apology on it is a soft 404, and the crawler indexes the apology as content. Authenticated pages return 401. Temporarily unavailable pages return 503 with a Retry-After header, which tells a crawler to come back instead of dropping the URL.

Orphaned pages

A page with no internal links pointing at it is an orphan. It may be indexed if the sitemap carries it, but nothing in your link graph says it matters, so it is crawled rarely. Export every page with zero inbound internal links and either link to it from somewhere relevant or redirect it to its nearest topical parent.

How to verify: in Search Console, open Settings, then Crawl stats. Three numbers matter: average response time, where sustained figures above 1,000ms are worth investigating; the proportion of responses that are 200, where a large error share means crawl effort is being wasted; and the split between HTML and other file types. Then open the Pages report and filter for "Discovered, currently not indexed". A large count there is a crawl problem, not a content problem, and the fixes are in this section rather than the previous ones.

The full technical checklist

Every control in this part, one row each. Run it against each domain.