robots.txt and the AI Crawlers

Why an old file suddenly matters again

robots.txt is one of the oldest conventions on the web. It is a plain-text file placed at the root of a domain, at /robots.txt, that tells automated clients which parts of a site they may fetch. For most of its life it governed search engine spiders and the occasional archiver. The stakes were modest: a blocked path meant a page did not appear in a results list.

That has changed. The same file now stands at the entrance to a second discovery layer, the one where AI assistants read the web to answer questions and recommend businesses. When someone asks ChatGPT, Claude, Perplexity, or Google's AI answers for a supplier, a comparison, or a definition, the response is assembled from content those systems were allowed to crawl. If your robots.txt closes the door on the crawlers that feed those systems, you are not merely lower in a list. You are absent from the answer.

This is the practical face of AIO, the discipline of AI Optimization that succeeds SEO as discovery shifts from search to recommendation. Before an AI system can trust or cite a business, it has to be able to read it. robots.txt is the first gate, and getting it wrong is one of the few AIO mistakes that removes you from consideration entirely rather than merely weakening your position.

How the file actually works

robots.txt follows the Robots Exclusion Protocol, a convention created in 1994 and formalized by the IETF as Proposed Standard RFC 9309 in 2022. The syntax is deliberately simple. A file lists one or more groups, each beginning with a User-agent line that names a crawler, followed by Disallow and Allow lines that specify paths. A User-agent of asterisk matches every crawler not addressed by name. A Disallow of slash blocks the entire site. An empty Disallow allows everything.

Two properties of the file are essential to understand. First, it operates at the level of the named user-agent, so you can grant one crawler full access while refusing another in the same file. Second, and more consequentially, robots.txt is advisory. RFC 9309 codified the format, but it did not create an enforcement mechanism. The file is a request that well-behaved crawlers honor voluntarily. It has no legal force and stops nothing on its own. Compliance depends entirely on the operator of the crawler choosing to obey.

For the major AI companies, that choice is generally made in favor of compliance, because a public reputation for ignoring robots.txt is a liability. But the gap between a request and a guarantee is real, and it defines both the power and the limits of the file. Where robots.txt succeeds, it succeeds because the reader agrees to be governed by it.

The AI crawler fleet: three jobs, not one

The single biggest source of error in configuring robots.txt for AI is treating each vendor as a single bot. The major providers run fleets, and the bots inside a fleet do different jobs with different consequences for your visibility.

The first job is training. A training crawler performs bulk ingestion, collecting large volumes of text to build the corpus a future model learns from. It does not drive traffic and it does not cite you in the moment. Its value to you is indirect and long term: content absorbed during training shapes what a model knows and how confidently it can speak about your domain. The second job is search indexing. A search crawler builds and refreshes an index that an assistant queries when it needs current information to answer a question. The third job is the live user fetch: when a person asks an assistant about a specific page, an agent bot retrieves that page in real time to ground the answer, often with a citation and a link back.

These jobs map to different user-agent tokens, and blocking one does not block the others. Refusing a training crawler removes you from a model's learned corpus but leaves you visible in that vendor's live search and citation flow, provided you allow the search and agent bots. Confusing the two is how businesses accidentally disappear from AI answers while believing they only opted out of training.

The bots you will actually see, by name

OpenAI runs a three-bot fleet. GPTBot is the training crawler. OAI-SearchBot builds the index that powers ChatGPT's search feature. ChatGPT-User fetches a page live when a user's prompt requires it. Blocking GPTBot opts you out of training only. It does not remove you from ChatGPT search, which depends on OAI-SearchBot as a separate token. Anthropic follows the same pattern: ClaudeBot is the training crawler, while a separate in-product search crawler supports Claude's web search and a user-agent handles live fetches on behalf of Claude users. Perplexity operates PerplexityBot for indexing and a user-facing fetcher for real-time retrieval.

Google and Apple handle the distinction differently, through control tokens rather than separate crawlers. Google-Extended is not a crawler at all. It is a token you place in robots.txt to withhold your content from training Gemini and related generative products. The crawler itself remains Googlebot, and blocking Google-Extended does not affect your presence in Google Search or its AI answers. Applebot-Extended works the same way for Apple Intelligence: it opts you out of Apple's AI training without removing you from Apple's search results.

Beyond these, the landscape includes CCBot, which feeds the Common Crawl dataset that many models draw from, along with crawlers from Meta, Amazon, ByteDance, and others. The list changes quickly. OpenAI introduced its dedicated search bot in 2024, Anthropic split off a separate search crawler in 2025, and new tokens continue to appear. A robots.txt written for AI in 2024 is already partly out of date, which makes periodic review part of the discipline rather than a one-time task.

The cost of blocking

The instinct to block AI crawlers is understandable. Training bots consume bandwidth without sending visitors, and many publishers resent their work being absorbed into a model for free. By August 2025, Cloudflare reported that more than 2.5 million websites had chosen to disallow AI training through its managed controls. The impulse is real and, for some content owners, entirely rational.

But the cost is asymmetric, and it is easy to underestimate. Blocking a training crawler withholds your content from a corpus. Blocking a search or agent crawler withholds you from the answer a user is receiving right now, at the moment of highest intent, when an assistant is actively assembling a recommendation. The economics of the two are not comparable. Training crawlers fetch enormous volumes of pages for every visitor they eventually send, with crawl-to-referral ratios that for some vendors run into the thousands. Search and agent crawlers, by contrast, fetch a page because a specific person is asking about your specific topic. Those are the requests that produce citations and clicks.

A blanket block that treats all AI crawlers as one category therefore trades a small, diffuse cost, some absorbed training data, for a large, concentrated one: exclusion from the live recommendation surface that is replacing search. In AIO terms, an unreadable business cannot be a recommended one. Accessibility, one of the seven pillars, is not an optimization to layer on later. It is the precondition for every other pillar to matter.

Opting out of training without going dark

The reason the fleet distinction matters is that it lets you make a precise choice instead of a crude one. A defensible default for most businesses that want to protect their content while remaining discoverable is to allow the search and retrieval crawlers, allow the live user-fetch agents, and decline the pure training crawlers where you object to bulk ingestion.

Concretely, that means allowing OAI-SearchBot, ChatGPT-User, Perplexity's crawlers, and Anthropic's search and user agents, while using Google-Extended and Applebot-Extended to cleanly opt out of Google and Apple training with no effect on your search inclusion. GPTBot and ClaudeBot can be disallowed if you specifically want to stay out of those training corpora, with the understanding that this may modestly reduce how well future models know your domain from memory. The point is that each decision is now separable. You are not forced to choose between total exposure and total invisibility.

There is a countervailing argument worth stating plainly. Presence in a model's training data contributes to entity strength, the pillar concerned with whether an AI system recognizes your business as a distinct, well-defined entity. A business that appears consistently across the open web, and therefore across training corpora, is more likely to be understood and recalled without a live lookup. Blocking all training is a legitimate choice, but it is a trade, not a free defensive move.

The limits of a voluntary standard

Because robots.txt is advisory, it protects you only against crawlers that choose to comply, and it makes you visible only to crawlers that choose to read it. Both halves of that sentence have been tested in public. In August 2025, Cloudflare documented Perplexity using undeclared crawlers that impersonated an ordinary Chrome browser, rotating IP addresses and network origins to reach content on sites that had explicitly blocked Perplexity's declared bot. Cloudflare responded by delisting Perplexity as a verified bot and adding heuristics to block the stealth traffic. Perplexity countered that an assistant fetching a page on a user's behalf is closer to a person opening a browser than to a bulk crawler, and should not be bound by the same rules.

That dispute is unresolved, and it exposes the structural weakness of the file: robots.txt is a norm, not a wall. If your goal is genuine enforcement rather than a stated preference, robots.txt has to be backed by something with teeth, typically a web application firewall or a bot-management layer that verifies crawler identity and blocks at the network level. robots.txt states your intent. The enforcement layer makes it real against actors who ignore intent.

The standards landscape is also still moving. The IETF convened an AI Preferences working group in 2025 to define machine-readable rules for how AI systems may use content, and a separate proposal, llms.txt, circulates as a way to guide assistants toward a site's most important pages. Adoption of these newer signals is uneven, and major providers have not committed to all of them. For now, robots.txt remains the one control that every reputable AI crawler reads, which is precisely why it deserves deliberate configuration rather than neglect.

A working configuration and how to maintain it

Treat robots.txt as a living document with three responsibilities. First, address the AI crawlers by name rather than relying on a single wildcard rule, so that training, search, and agent bots can be governed independently. A wildcard Disallow is the most common way businesses accidentally block the very crawlers that would have cited them. Second, keep the file current. Because vendors add and rename tokens frequently, schedule a review rather than writing the file once and forgetting it. Third, confirm that the crawlers you intend to allow are actually reaching your important content, by checking server logs for their user-agent strings.

Two verification habits separate a configured file from an assumed one. Read your access logs and confirm that GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and the others are fetching the pages you expect and receiving successful responses rather than errors or redirects. And remember that robots.txt governs only fetching. If a page is allowed but slow, blocked behind JavaScript that a crawler cannot render, or buried where no crawler can find a link to it, the door is open but the room is empty. Accessibility is fetchability and readability together.

Configured well, robots.txt does something quietly powerful. It removes the single failure that no amount of good content can survive, being unreadable, and it lets the rest of the AIO discipline do its work. The pillars of clarity, evidence, and expertise only influence an AI recommendation once the system is permitted to read the pages that carry them. robots.txt is where you grant that permission, on purpose.