llms.txt is a plain-text file placed at the root of your domain that gives AI crawlers a curated map of your most important pages. It helps large language models understand what your site does, which content matters most, and how to retrieve it efficiently.

How is llms.txt different from robots.txt?

robots.txt controls access by telling crawlers which parts of a site they can or cannot fetch. llms.txt does not block access. It provides semantic context and content prioritization so AI systems can identify relevant pages faster and with less ambiguity.

Which AI crawlers use llms.txt?

Known AI crawlers that can use llms.txt include GPTBot, ClaudeBot, PerplexityBot, and Googlebot for AI Overviews. They use it as a retrieval guide, not as a ranking signal in the traditional search sense.

What pages should I include in llms.txt?

Include your most authoritative, crawlable, and up-to-date pages. That usually means core service pages, detailed guides, original research, case studies, public pricing pages, and other pages that directly answer your audience's questions.

Will llms.txt guarantee AI citations?

No. llms.txt improves the odds that AI crawlers can find and understand your best content, but citations still depend on content quality, topical authority, crawlability, and whether your pages directly answer the user's query.

What is llms.txt? Complete Guide for 2026

Most websites are invisible to AI. Not because their content is bad, but because AI crawlers have no efficient way to understand what the site is about, which pages matter, or how the information is structured. They land on a homepage full of navigation menus, JavaScript, and marketing copy, and they move on.

llms.txt is the fix for that. It is a plain-text file you place at the root of your domain that tells AI crawlers exactly what your organization does, which pages contain your most important content, and how that content is organized. Think of it as a briefing document written specifically for large language models, not for human readers and not for traditional search bots.

The adoption gap is striking: as of 2026, fewer than 10% of websites have deployed an llms.txt file. That means 90% of brands are leaving their AI visibility entirely to chance, hoping that GPTBot, ClaudeBot, and PerplexityBot can reverse-engineer their site structure from raw HTML. Most of the time, they cannot do it well.

This guide covers everything you need to know to deploy llms.txt correctly: the official specification, real syntax examples, which AI crawlers read it, how it fits into a broader technical AEO infrastructure, and the implementation mistakes we see most often across client deployments.

What llms.txt is and how it differs from robots.txt
Which AI crawlers read it and how they use the information
The exact file format with a copy-paste template
How to decide which pages to include and which to exclude
Common mistakes that undermine the file's effectiveness
Step-by-step implementation you can complete today

What is llms.txt?

llms.txt is a Markdown-formatted plain-text file located at yourdomain.com/llms.txt. It was proposed by Jeremy Howard (founder of fast.ai) in September 2024 as a standard way for website owners to communicate structured, LLM-readable information directly to AI systems at inference time.

The core idea is straightforward. When a user asks ChatGPT or Perplexity a question that involves your brand or topic area, the AI crawler that fetches information from the web has a problem: your website was built for humans. It has navigation menus, cookie banners, JavaScript rendering, and HTML markup that obscures the actual content. Converting all of that into something a language model can efficiently process is lossy and unreliable.

llms.txt sidesteps that problem entirely. Instead of making the crawler parse your full site, you give it a curated map: here is what we do, here are the pages that matter, and here is how to interpret them.

Key distinction: llms.txt is not about training data. It is primarily useful at inference time, meaning when a user is actively asking an AI a question and the model is retrieving current information from the web. This is the moment that determines whether your brand gets cited.

What the File Contains

A properly structured llms.txt file includes three core components:

Component	Purpose	Required?
H1 title	Names the project or organization	Yes
Blockquote summary	Concise description of what the site does	Recommended
H2 sections with link lists	Curated URLs with descriptions, grouped by topic	Recommended

The H2 sections are where most of the strategic value lives. You group your most important pages into named categories (Documentation, Products, Case Studies, etc.) and provide a brief description of what each URL contains. AI crawlers use these descriptions to decide which pages are worth retrieving for a given query.

Understanding how AI engines decide what to cite is essential context here: the crawlers are not reading everything. They are making retrieval decisions based on signals like these descriptions, page structure, and semantic relevance to the query at hand.

llms.txt vs. robots.txt: Not the Same Thing

This is the most common point of confusion, and it matters because the two files serve completely different purposes. Conflating them leads to implementation errors that leave AI visibility gaps.

robots.txt is an access control file. It tells crawlers which parts of your site they are and are not allowed to index. It is a gatekeeper: you are either allowed in or you are not. It says nothing about what your content means or which pages are most important.

llms.txt is a semantic context file. It assumes the crawler already has access to your content and gives it a structured guide to understanding that content. It answers questions like: what does this organization do, which pages represent our best thinking on a topic, and how should the information be interpreted?

The Practical Difference

Think of it this way: robots.txt is the bouncer at the door. llms.txt is the concierge inside who tells you where to go once you are in.

File	Controls	Read by	Primary purpose
`robots.txt`	Crawl access	All bots	Block or allow indexing
`llms.txt`	Content prioritization	AI crawlers	Semantic context + page hierarchy
`sitemap.xml`	URL discovery	Search + AI bots	Complete list of indexable pages

A sitemap lists every indexable page on your site. That is the opposite of what llms.txt does. A good llms.txt is deliberately selective: it surfaces the 10 to 30 pages that best represent your expertise and are most likely to be relevant when an AI is answering questions in your topic area. Giving an AI crawler 500 URLs is not helpful. Giving it 20 well-described, well-chosen URLs is.

The three files work as a system. robots.txt controls access. sitemap.xml provides discovery. llms.txt provides meaning. All three should be in place and aligned for a complete technical AEO infrastructure.

Which AI Crawlers Read llms.txt?

The short answer: the crawlers behind the AI platforms that are actively displacing traditional search as the primary way people find information.

As of 2026, the following crawlers are known to read and use llms.txt data during web retrieval:

GPTBot (OpenAI / ChatGPT): OpenAI's crawler fetches live web content for ChatGPT's browsing and web search features. A well-structured llms.txt helps GPTBot identify your highest-value pages quickly rather than crawling broadly.
ClaudeBot (Anthropic / Claude): Anthropic's retrieval crawler uses llms.txt to understand site structure when Claude is answering questions that require current web information.
PerplexityBot (Perplexity): Perplexity is built entirely around real-time web retrieval. It is one of the most active consumers of llms.txt data because its entire answer generation pipeline depends on efficient content extraction from the web.
Googlebot (Google AI Overviews): Google's crawler reads llms.txt as part of its signals for AI Overviews, the AI-generated answer blocks that now appear above organic results for a large share of queries. If your brand is not appearing in AI Overviews, llms.txt is one of the technical signals worth checking.

What These Crawlers Actually Do With the File

This is where most guides stop short. The crawlers do not just "read" your llms.txt file and move on. They use the structured link list to make selective retrieval decisions.

When a user asks Perplexity "what is the best approach for [your topic area]," PerplexityBot does not crawl your entire site. It looks at your llms.txt, reads the section descriptions, identifies which URLs are semantically relevant to the query, fetches those specific pages, and extracts content to inform the answer.

This is why description quality matters. A URL listed as [Our Blog](https://example.com/blog) gives the crawler almost no information. A URL listed as [B2B SaaS Pricing Strategy Guide](https://example.com/blog/b2b-saas-pricing): Comprehensive breakdown of usage-based vs. seat-based pricing models with ROI benchmarks gives the crawler exactly what it needs to decide whether that page is relevant to a given query.

The description is not metadata. It is the decision signal.

The llms.txt File Format: Exact Syntax and Structure

The specification, maintained at llmstxt.org, defines a precise Markdown structure. Here is the full format with annotations:

# Organization or Project Name

> One to three sentence summary of what this organization does, who it serves,
> and what makes its content authoritative on this topic. Be specific and factual.

Optional: One or two paragraphs of additional context that help LLMs interpret
the linked content correctly. Use this for important caveats, methodology notes,
or scope clarifications.

## Section Name (e.g. Core Services, Key Resources, Documentation)

- [Page Title](https://yourdomain.com/page-url): Brief description of what this page contains and why it is relevant. Be specific about the content type and key topics covered.
- [Page Title](https://yourdomain.com/page-url): Description.

## Another Section

- [Page Title](https://yourdomain.com/page-url): Description.

## Optional

- [Page Title](https://yourdomain.com/page-url): Description of lower-priority content that can be skipped if context window is limited.

The ## Optional section has a specific technical meaning in the spec: crawlers that need to reduce context window usage will skip URLs listed here first. Use it for supplementary content (older blog posts, archive pages, secondary resources) rather than your core pages.

A Real-World Example

Here is what a well-built llms.txt looks like for a B2B SaaS company in the workflow automation space:

# Acme Workflow

> Acme Workflow is an enterprise workflow automation platform that helps operations
> teams at mid-market and enterprise companies eliminate manual approval processes,
> integrate disparate tools, and reduce process cycle times by an average of 67%.
> Acme serves over 2,400 companies across financial services, healthcare, and logistics.

Acme's platform is built on a no-code visual workflow builder with native integrations
to Salesforce, SAP, ServiceNow, and 200+ other enterprise tools. All data is processed
within SOC 2 Type II and ISO 27001 certified infrastructure.

## Core Product

- [Workflow Automation Platform Overview](https://acmeworkflow.com/platform): Complete overview of the visual workflow builder, trigger types, integration capabilities, and enterprise security features.
- [Pricing and Plans](https://acmeworkflow.com/pricing): Seat-based and usage-based pricing tiers for teams of 10 to 10,000+, including enterprise custom pricing details.
- [Security and Compliance](https://acmeworkflow.com/security): SOC 2 Type II, ISO 27001, GDPR, and HIPAA compliance documentation and certifications.

## Use Cases

- [Approval Workflow Automation](https://acmeworkflow.com/use-cases/approval-workflows): How operations teams automate multi-step approval chains across finance, HR, and legal departments.
- [Cross-Tool Integration Workflows](https://acmeworkflow.com/use-cases/integrations): Connecting Salesforce, SAP, and ServiceNow in automated end-to-end processes without custom code.

## Resources

- [2026 State of Workflow Automation Report](https://acmeworkflow.com/research/2026-report): Original research on automation adoption rates, ROI benchmarks, and implementation timelines across 500 enterprise companies.
- [ROI Calculator](https://acmeworkflow.com/roi-calculator): Interactive tool for estimating time savings and cost reduction from workflow automation based on team size and process volume.

## Optional

- [Customer Stories](https://acmeworkflow.com/customers): Case studies from enterprise clients in financial services, healthcare, and logistics.
- [Blog](https://acmeworkflow.com/blog): Articles on workflow automation best practices, tool comparisons, and implementation guides.

Notice what this example does well: the H1 and blockquote give the AI a precise, factual understanding of who Acme is and what they do. The section descriptions are specific enough that a crawler can match them to query intent. The ## Optional section deprioritizes content that is useful but not essential.

This is the level of specificity that moves the needle on AI citations.

How to Decide Which Pages to Include

This is the strategic decision that most brands get wrong. They either include too many pages (turning llms.txt into a second sitemap) or too few (making it so sparse it provides no useful signal).

The selection framework we use across client deployments comes down to four questions:

1. Does this page directly answer questions your target audience asks AI?

Start with query intent. What are the questions your potential customers are typing into ChatGPT, Perplexity, or Claude? Your llms.txt should prioritize the pages that best answer those questions. If you sell enterprise security software, a page titled "What is Zero Trust Architecture?" belongs in your llms.txt. Your "About the Founders" page does not.

2. Does this page demonstrate topical authority?

AI engines weight sources that demonstrate depth of expertise in a specific domain. Include your most comprehensive, evidence-backed content: original research, detailed guides, data-driven analyses. These pages signal that your domain is a credible source on the topic. Understanding what is generative engine optimization helps contextualize why authority signals matter so much in this environment.

3. Is this page technically accessible to crawlers?

A page blocked by robots.txt, behind a login wall, or requiring JavaScript rendering to display its content is not a useful inclusion. If you list a URL in llms.txt that the crawler cannot actually access, you are wasting the signal. Verify that every included URL is crawlable before publishing.

4. Is the content current and accurate?

AI engines that cite outdated information and get corrected by users learn to deprioritize that source. Only include pages where the content is actively maintained. If a page has not been updated in two years and contains statistics that are now wrong, exclude it.

Pages to Include vs. Exclude

Include	Exclude
Core product/service pages	Legal pages (ToS, Privacy Policy)
Authoritative long-form guides	Thin category or tag pages
Original research and data	Pagination pages
Case studies with specific results	Checkout and transactional flows
FAQ and comparison pages	Staff bios and internal pages
Pricing pages (if public)	Outdated or unmaintained content

Target 15 to 30 URLs for most sites. Larger sites with deep content libraries can go up to 50, but only if each URL is genuinely distinct in topic and value. Quality of selection matters far more than quantity.

llms.txt as Part of a Complete Technical AEO Stack

llms.txt is a powerful signal, but it is one layer of a multi-layer system. Brands that deploy llms.txt in isolation without addressing the rest of their technical AEO infrastructure see limited results. Here is how the layers interact.

The Three-Layer Technical AEO Infrastructure

Layer 1: Access and Discovery

robots.txt: Ensures AI crawlers are not blocked from your key pages
sitemap.xml: Provides a complete, up-to-date URL inventory
Page speed and server response: Slow pages get abandoned by crawlers under time pressure

Layer 2: Semantic Signals

llms.txt: Curated content map with topical context and page prioritization
Schema markup: Structured data (Organization, Article, FAQ, HowTo, Product) that tells AI engines exactly what type of content each page contains and how to interpret it
Heading structure: Clean H1/H2/H3 hierarchy that mirrors the logical structure of your content

Layer 3: Content Authority

E-E-A-T signals: Author credentials, citations, original research, and expertise indicators
Citation-worthy content: Specific statistics, original data, and well-sourced claims that AI engines can confidently quote
Internal linking: Clear topical clusters that reinforce your domain's authority in a subject area

Schema markup deserves special attention here. A well-implemented llms.txt tells the crawler which pages to visit. Schema markup on those pages tells the crawler what it is looking at when it gets there. The two signals compound each other. For a detailed implementation guide, see how to implement structured data for AI.

The compounding effect: Brands with all three layers in place consistently outperform brands with only one or two in AI citation audits. In our deployments across financial services, SaaS, and healthcare clients, the combination of llms.txt plus schema markup plus crawl-accessible content produces measurably higher citation rates than any single element alone.

Why the 90% Gap is a Strategic Opportunity

Fewer than 10% of websites have deployed llms.txt. That number is not a sign that the standard has not been adopted. It is a sign that most marketing and SEO teams have not yet recognized that AI engines require a different kind of technical optimization than traditional search.

The brands that move first in establishing this infrastructure are building a compounding advantage. AI engines develop source preferences based on consistent, high-quality retrieval experiences. A domain that reliably serves well-structured, accurate, and relevant content to AI crawlers gets cited more, which trains the model to treat it as a trusted source, which leads to more citations. The feedback loop is real, and it starts with getting the technical foundation right.

Common llms.txt Mistakes (and How to Avoid Them)

After deploying llms.txt configurations across more than 20 industries, these are the errors we see most consistently.

Mistake 1: Generic or Vague Descriptions

The most common and most damaging mistake. Descriptions like "Our main blog" or "Product information" give AI crawlers nothing to work with. The crawler cannot match a vague description to a specific query intent, so the page gets skipped.

Fix: Write descriptions that include the specific topic, the content format, and the key value the page delivers. Treat each description as a 15-to-25-word pitch to an AI: what is on this page and why does it matter for someone asking about this topic?

Mistake 2: Listing Blocked Pages

Including URLs in llms.txt that are blocked by robots.txt or require authentication is a contradiction that wastes the signal. The crawler sees the URL, tries to fetch it, and hits a wall. This does not help your visibility and may create a negative crawl experience signal.

Fix: Before publishing llms.txt, verify every URL is crawlable by checking robots.txt rules and testing the URL without authentication. Use Google Search Console or a crawl audit tool to confirm accessibility.

Mistake 3: Treating llms.txt Like a Sitemap

Some brands dump 200+ URLs into llms.txt. This defeats the purpose. The file is meant to be a curated, high-signal guide, not an exhaustive index. More URLs means less signal per URL, and crawlers with limited context windows will skip lower-priority entries.

Fix: Keep the file to 15 to 30 core URLs. If you have a large content library, use the ## Optional section for secondary content and ensure your primary sections contain only your highest-value pages.

Mistake 4: Ignoring the Blockquote Summary

The blockquote immediately below the H1 is the most important text in the file. It is the first thing a crawler reads and it frames everything that follows. Brands that write a generic tagline here ("We help businesses grow") miss the opportunity to establish topical authority from the first line.

Fix: Write the blockquote as if you are answering the question "What does this organization do, who does it serve, and why should an AI trust it as a source?" Be specific about your industry, your audience, and what makes your content credible.

Mistake 5: Setting It and Forgetting It

llms.txt is not a one-time deployment. As your content evolves, the file needs to evolve with it. A file that still references pages that have been deleted, moved, or significantly changed is actively hurting your AI visibility.

Fix: Schedule a quarterly llms.txt audit. Verify all URLs still return 200 status codes, update descriptions when page content changes significantly, and add new high-value content as it is published.

How to Implement llms.txt: Step-by-Step

This is a technical task that a developer can complete in under an hour once the content decisions are made. The content decisions take longer.

Step 1: Audit Your Existing Content

Before writing a single line of the file, map your content landscape. List every page on your site that:

Directly addresses questions your target audience asks AI engines
Demonstrates topical authority with depth, data, or original research
Is currently ranking in traditional search (a signal that it has relevance)
Has been published or significantly updated within the past 18 months

This audit typically surfaces 40 to 60 candidate pages. You will cut this down to 15 to 30 in the next step.

Step 2: Prioritize and Select

Score each candidate page on three dimensions:

Query relevance: How directly does this page answer a question your audience asks AI?
Content depth: Does this page go beyond surface-level to provide specific, citable information?
Crawl accessibility: Is this page fully accessible to AI crawlers without authentication or JavaScript dependency?

Keep only pages that score well on all three. Be ruthless. A shorter, higher-quality list outperforms a longer, diluted one.

Step 3: Write the File

Use the template below as your starting point. Replace every placeholder with specific, accurate information about your organization.

# [Your Organization Name]

> [Organization name] is a [specific description of what you do] that helps [specific audience]
> [achieve specific outcome]. [One sentence on your scale, credentials, or what makes your
> content authoritative on this topic.]

[Optional: 1-2 sentences of additional context that helps LLMs interpret your content correctly.
Use this for methodology notes, scope clarifications, or important caveats.]

## [Primary Content Category - e.g. Core Services, Products, Solutions]

- [Page Title](https://yourdomain.com/url): [Specific description of content, topics covered, and why it is relevant to queries in this space.]
- [Page Title](https://yourdomain.com/url): [Description.]

## [Secondary Content Category - e.g. Resources, Research, Guides]

- [Page Title](https://yourdomain.com/url): [Description.]
- [Page Title](https://yourdomain.com/url): [Description.]

## Optional

- [Page Title](https://yourdomain.com/url): [Description of lower-priority supplementary content.]

Step 4: Publish the File

Save the file as llms.txt (plain text, UTF-8 encoding, no BOM required but allowed)
Upload it to the root directory of your domain so it is accessible at https://yourdomain.com/llms.txt
Verify it is accessible by visiting the URL directly in a browser
Confirm it is not blocked by your robots.txt file (check that no rule blocks /llms.txt)
Check that the file returns a 200 HTTP status code, not a redirect

Step 5: Verify and Monitor

After publishing:

Test the URL with multiple AI platforms: ask ChatGPT or Perplexity a question about your brand or topic area and observe whether your pages are cited
Use server logs to check whether GPTBot, ClaudeBot, and PerplexityBot are visiting llms.txt (they typically crawl it within days of publication)
Set a calendar reminder to audit and update the file quarterly

One additional recommendation: create .md versions of your most important pages at the same URL with .md appended (e.g., yourdomain.com/guide/topic.md). This is a companion recommendation from the llms.txt specification and gives AI crawlers a clean, markup-free version of your content to process. It is not required, but it meaningfully improves the quality of content extraction.

Frequently Asked Questions

Is llms.txt an official standard?

Not yet in the sense of being ratified by a standards body like the W3C or IETF. It was proposed by Jeremy Howard in September 2024 and is maintained as an open specification at llmstxt.org with community input via GitHub. However, it has achieved meaningful de facto adoption: major AI crawlers including GPTBot, ClaudeBot, and PerplexityBot recognize and use the file. In practice, it functions as a working standard even without formal ratification.

Will implementing llms.txt guarantee that AI engines cite my brand?

No, and anyone who tells you otherwise is overpromising. llms.txt is a technical signal that improves the probability of AI crawlers finding, understanding, and retrieving your content. Whether your content actually gets cited depends on additional factors: the quality and specificity of your content, your domain's established authority, whether your pages directly answer the queries being asked, and the competitive landscape in your topic area. llms.txt is a necessary foundation, not a silver bullet.

Does llms.txt affect traditional Google search rankings?

Not directly. The file is designed for AI inference-time retrieval, not traditional search indexing. However, Googlebot does read llms.txt as part of its AI Overviews signals, so there is indirect relevance to Google's AI-powered features. For traditional organic rankings, robots.txt, schema markup, and content quality remain the primary technical signals.

How long does it take for AI crawlers to discover and process a new llms.txt file?

Based on server log analysis across client deployments, GPTBot and PerplexityBot typically discover and crawl a new llms.txt file within 24 to 72 hours of publication. ClaudeBot tends to follow within a week. Observable changes in citation behavior take longer, typically two to four weeks, as the crawlers need to fetch and index the referenced pages and the citation patterns need time to reflect the new signals.

Should I use llms.txt if my site is small or if I only have a few pages?

Yes, arguably more so. A small site with a clear, well-structured llms.txt can punch well above its weight in AI citations because the file does the disambiguation work that a large site's content volume normally provides. If your site has 10 to 20 high-quality pages on a specific topic, a well-written llms.txt makes it immediately clear to AI crawlers that your domain is a focused, authoritative source on that topic. That clarity is an advantage, not a limitation.

Find Out if Your Site is Being Cited by AI Engines

llms.txt is one of the most impactful technical changes a brand can make right now to improve AI visibility. It takes hours to implement, requires no ongoing cost, and positions your site to be found and cited by the AI platforms that are increasingly where your audience goes for answers.

But before you implement it, you need to know where you stand. Is your site currently being crawled by GPTBot and PerplexityBot? Are your pages being cited in ChatGPT, Claude, or Perplexity responses, or are they being ignored? Are your competitors already building an AI citation advantage while you are still optimizing for traditional search?

Get a free AI visibility audit from LLMReach. We will analyze whether your site is currently being crawled and cited by AI engines, identify the technical gaps in your AEO infrastructure, and show you exactly what it would take to start appearing in AI-generated answers for your target queries.

The 90% of brands that have not deployed llms.txt are not your competition. The 10% that have are.

What is llms.txt? The Complete Guide to AI Crawler Optimization