Implementing llms.txt: A Practical Guide to the AI-Crawl Standard

What llms.txt is and why it appeared

llms.txt is a plain-text file, written in Markdown, placed at the root of a domain (for example, example.com/llms.txt). It offers large language models a curated, human-authored map of a site: what the organization is, what it does, and which pages carry the information most worth reading. It was proposed in September 2024 by Jeremy Howard of Answer.AI, and it spread quickly through developer-facing documentation before reaching wider software, publishing, and commerce sites.

The problem it addresses is concrete. AI systems operate inside finite context windows, and raw web pages are expensive to consume. A typical HTML page buries its substance under navigation, cookie banners, advertising, tracking scripts, and styling. When a model or an agent fetches that page, much of the token budget is spent parsing markup that carries no meaning. llms.txt proposes a shortcut: a single, ad-free, script-free document that states the important things directly and links to clean sources.

The comparison to robots.txt is common but imprecise. robots.txt tells crawlers what they may not touch. A sitemap enumerates every URL for completeness. llms.txt does neither. It is an act of curation, not exclusion or exhaustive listing. It says, in effect: if you want to understand this site quickly and correctly, start here, and read these things first.

The format specification

The specification is deliberately minimal and requires a specific structure. The file opens with an H1 containing the site or project name. It is followed by a blockquote summary, a single sentence or two describing what the site is and who it serves. Optional prose may follow to add context. The body is organized into H2 sections, each holding a Markdown bullet list of links, where every link is written as a title in brackets, the URL in parentheses, then a colon and a short description of what that page contains.

Two rules matter more than they appear to. First, the file must be valid Markdown that parses with standard libraries and no custom extensions, because deviating from the structure reduces how reliably a model can read it. Second, the descriptions after each colon do real work: they are the signal a model uses to decide which linked page answers a given question. Vague or promotional descriptions waste the format's one advantage.

• H1: the name of the site or organization
• Blockquote: a short summary of purpose and audience
• Optional prose: context that does not fit a link list
• H2 sections: grouped topics such as Documentation, Products, Policies
• Bullet links: [Page title](https://example.com/page): concise description of the page's contents

llms-full.txt and the Markdown URL convention

Two companion practices grew alongside the base file. The first is llms-full.txt, a longer document that embeds the actual content of the linked pages rather than only pointing to them. The intent is a single fetch that gives an agent everything it needs without following links across a site. The trade-off is size: a full file can grow large enough to exceed the very context limits llms.txt was meant to respect, so it suits compact documentation better than sprawling sites.

The second is the Markdown URL convention, where a site serves a clean .md version of each page alongside its HTML, for instance page.html and page.html.md. This lets a system that finds a link in llms.txt retrieve the content without stripping markup at all. Together the three pieces form a small system: llms.txt for the map, .md endpoints for clean sources, and llms-full.txt for the one-shot bundle.

None of these are mandatory. A useful and complete implementation can be a single well-written llms.txt. The companions are optimizations for specific situations, not requirements of the standard.

How to write and place one

Implementation is mechanically simple. Author the file by hand or generate it from an existing sitemap or documentation tree, then place it at the domain root so it resolves at /llms.txt. Serve it as text/plain or text/markdown. Because it is static, it carries almost no performance cost and no risk to the rendered site.

The judgment is in selection, not syntax. Include the pages that define the entity and answer real questions: what the organization is, its core products or services, authoritative documentation, pricing or terms where relevant, and primary contact or policy pages. Exclude thin, duplicative, or purely navigational URLs. A shorter file of high-value links outperforms a long file that mirrors the sitemap, because the format's entire purpose is to reduce noise, not reproduce it.

Maintenance is the step most often skipped. An llms.txt that drifts out of sync with the live site becomes a source of confident, stale answers. Treat it as a published artifact: update it when the site's structure or core facts change, and keep the descriptions truthful. A file that overstates or misstates what a page contains does active harm, because it is read as a direct assertion rather than inferred from context.

What AI systems actually do with it in 2026

The honest state of adoption is mixed, and implementers should understand it before investing effort. As of early 2026, no major AI provider, including OpenAI, Google, Anthropic, Meta, and Mistral, has publicly committed to reading or acting on llms.txt in production search and answer systems. In July 2025 Google's Gary Illyes stated that Google does not support the file and does not plan to, and John Mueller likened it to the long-discredited keywords meta tag.

Independent measurement supports the skepticism about crawling. A large study of roughly 137,000 sites found that the overwhelming majority of llms.txt files attract no model readers at all, and that named crawlers such as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended generally skip the file and fetch HTML directly. A striking share of the traffic that does hit these files comes from auditing and monitoring tools checking whether the file exists, a largely self-referential pattern rather than genuine consumption.

Adoption on the publishing side, meanwhile, is real but modest and partly automatic. Studies across large domain samples place adoption in the high single digits to low double digits, and platform defaults inflate the raw count: in 2026 Shopify began shipping an llms.txt to stores by default, which raises presence without reflecting deliberate authorship. The gap between files that exist and files that are read is the central fact of the current moment.

Where the real value is right now

Given that AI search engines largely ignore it, the defensible case for implementing llms.txt in 2026 is not organic AI search ranking. It is the assistant and agent surface. Tools that operate with an explicit URL, such as AI-assisted code editors, documentation assistants, and agents a user points at a specific site, can and do consume a clean llms.txt to orient themselves. In these contexts the file measurably improves how well the assistant understands the source, because the human has already directed it to fetch.

This reframes the decision. llms.txt is best understood today as a developer-experience and agent-readiness measure with low cost and low risk, not as a search tactic with proven returns. For a documentation-heavy product, an API, or any site whose users routinely load it into an assistant, the file earns its place. For a site betting on discovery through general AI search, it will not move the outcome on its own, because the systems that drive that discovery are not reading it.

The prudent posture is to implement it well where it helps, keep it accurate, and treat it as insurance against a future where support broadens, while not mistaking its presence for visibility. The convention may yet be adopted more formally; a standardization effort has been discussed but has not materialized. Building a correct file now costs little and positions the site to benefit if that changes.

Common anti-patterns to avoid

As the convention has spread, a recognizable set of mistakes has formed. The most damaging is treating llms.txt as a marketing surface. Descriptions written to persuade rather than to describe degrade the file's usefulness, because a model reads them as claims about content and either surfaces inaccurate summaries or learns to distrust the source. The format rewards plain, verifiable statements of what each page contains.

The second common error is scale for its own sake: dumping the entire sitemap into the file, or generating an enormous llms-full.txt that no context window can hold. Both defeat the purpose. A third is neglect, letting the file fall out of date until it describes pages that have moved or changed. Because the file is authoritative by design, a stale entry produces a confident wrong answer rather than a harmless omission.

• Promotional descriptions instead of factual ones
• Mirroring the full sitemap rather than curating
• Oversized llms-full.txt files that exceed context limits
• Stale links and descriptions that contradict the live site
• Treating the file as proof of AI visibility rather than a readiness measure

llms.txt within an AIO strategy

AI Optimization, or AIO, is the discipline that succeeds SEO as discovery shifts from search to AI recommendation. It is the umbrella practice; Generative Engine Optimization and Answer Engine Optimization are subsets of it, concerned with how content is generated into answers and surfaced by answer engines respectively. Within that structure, llms.txt is a tactic, not a strategy. It is one mechanism for making a site legible to machines, and its weight in the overall picture should match its actual reach.

The reason llms.txt cannot stand alone is that AI systems build their understanding of an entity from many corroborating signals, not from a single self-published file. Structured data, consistent facts across the open web, third-party validation, and clear on-page content all feed the model's picture, and they do so whether or not a crawler ever reads /llms.txt. A site that is well structured for machines will be understood even by systems that ignore the file; a site that relies on the file alone will not.

The practical conclusion is to implement llms.txt as one accessibility measure among several, do it accurately, and keep expectations calibrated to how it is actually used. It contributes most when the broader foundation is already sound, and it substitutes for none of that foundation.