AI Optimization Terminology
A Disallow line governs the fetch, not the use: robots.txt under RFC 9309
Google-Extended has no user agent of its own, and AI Overviews have no robots.txt token at all. Both facts follow from what RFC 9309 says a Disallow line is, and from what the RFC leaves to vendors.
Under RFC 9309, the Robots Exclusion Protocol the IETF published in September 2022, a Disallow line means one thing: the named crawler should not request the path. What a system does with content after it fetches it falls outside the RFC. A guide that tells a site it is "blocking AI" with robots.txt is describing a usage outcome in a protocol whose only verbs concern access. The standard that would supply a usage verb is a draft in the IETF's AIPREF working group, and Google's AI Overviews show what the gap costs a site until that draft is done.
RFC 9309: Allow and Disallow are fetch rules
RFC 9309 organizes robots.txt into groups. A user-agent line names a crawler, and the Allow and Disallow lines beneath it name paths that crawler may or may not request. The RFC states that these rules are not a form of access authorization: a compliant crawler honors them, and the file by itself stops nothing. No directive in RFC 9309 names training, grounding, summarizing or citing. Google's robots.txt specification calls itself Google's interpretation of the protocol and points readers to RFC 9309 as the original standard. Gary Illyes of Google gave the short version in a Google Search Central video that Search Engine Journal covered on October 22, 2026: "as defined by RFC 9309, robots.txt is an access preference protocol."
Section 2.2.4 of RFC 9309, "Other Records," lets crawlers interpret records the protocol does not define, with Sitemaps as its example, as long as parsing them does not interfere with the records the protocol does define. The RFC gives those other records no meaning. A line other than Allow or Disallow means whatever the crawler reading it decides it means.
Google-Extended: a usage control written in access syntax
Google-Extended is a robots.txt user-agent token that controls use, not fetching. Google's list of common crawlers states that Google-Extended has no separate HTTP request user agent string, that crawling is done with existing Google user agent strings, and that "the robots.txt user-agent token is used in a control capacity." Read against a server log, that statement has a consequence: no request in any access log carries Google-Extended as its user agent, so a site cannot confirm the control by watching its own traffic. Read against RFC 9309, a Disallow under Google-Extended is addressed to a crawler that never makes a request.
Google's crawler page supplies the token's meaning. Google-Extended manages whether content Google crawls may be used to train future generations of Gemini models, and whether it may be used for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
The same syntax therefore carries two meanings. Under Googlebot, Disallow means do not fetch. Under Google-Extended, Disallow means content Google has already fetched may not be used for the purposes Google names. The first meaning comes from the RFC. The second comes from Google's documentation, and Google can revise that page while a site's robots.txt file stays byte for byte the same.
AI Overviews: where access and use come apart in Google Search
AI Overviews and AI Mode are the clearest case of one fetch serving two uses. Google's Search Central guidance on AI features says a page must be indexed and eligible to appear in Google Search with a snippet to be shown as a supporting link in either feature, so both draw on the index Googlebot builds. Google's crawler page lists no separate token for them and says Google-Extended does not affect a site's inclusion in Google Search or act as a ranking signal.
Here is a file a site might ship after reading a "block AI" guide: ``` User-agent: * Content-Signal: search=yes, ai-train=no Disallow: /drafts/ User-agent: Google-Extended Disallow: / ```
Read line by line, that file does three different kinds of work. The Disallow under the wildcard group is an RFC 9309 fetch rule: Googlebot and every other compliant crawler without its own group skips /drafts/ and may fetch everything else. The Disallow under Google-Extended withholds content from Gemini training and Vertex AI grounding, because Google's documentation says so. The Content-Signal line states a preference under Cloudflare's policy, a record RFC 9309 gives no meaning. None of the three reaches AI Overviews or AI Mode.
Google's page-level controls are a separate channel. The same Search Central guidance names nosnippet, data-nosnippet, max-snippet and noindex as the ways to limit what AI features show from a page. They live in the page's HTML or its HTTP response headers, not in robots.txt, and they govern what Search shows, not whether Googlebot fetches. Each applies to classic results and AI features alike: a nosnippet that removes a page's text from AI Overviews removes its snippet from the blue link too.
No robots.txt token separates Google Search from Google's AI answers in Search. The only line that touches them is a Disallow aimed at Googlebot, and that line also stops the fetch classic Search depends on. A site that disallowed Google-Extended made a real usage decision about Gemini and no decision at all about AI answers in Search.
The IETF AIPREF drafts: Content-Usage and the train-ai vocabulary
The IETF's AI Preferences working group, AIPREF, is drafting the usage layer that RFC 9309 lacks. Its draft draft-ietf-aipref-attach, co-written by Gary Illyes of Google and Martin Thomson of Mozilla, is a proposed update to RFC 9309. It would add a Content-Usage rule to robots.txt and a matching HTTP header. Its current revision is dated August 19, 2026. A separate vocabulary draft, draft-ietf-aipref-vocab, dated September 14, 2026, defines the "train-ai" label and states that it reflects no consensus within the Working Group.
Under the attach draft, Allow and Disallow would continue to govern which URLs can be crawled, and Content-Usage would state preferences about how crawled content should be used. Those preferences would apply only to paths a crawler is permitted to fetch. The draft's example file allows crawling and adds a Content-Usage value of "train-ai=n". The authors of the usage layer did not redefine Disallow to mean "do not train." They added a second field.
As of Search Engine Journal's October 22, 2026 report, the attach draft was still a Working Group Document. It had not been submitted to the IESG, the IETF's steering group, though the working group's target date for that step was August 2026. The current revision expires on February 20, 2027, unless a new one replaces it.
The strongest objection: for training crawlers, Disallow already is a usage decision
The strongest case against treating a robots.txt Disallow as access only comes from crawlers built solely to collect training data. On this view, OpenAI, Anthropic and Common Crawl already treat a Disallow under their training crawlers, GPTBot among them, as a usage decision, because those crawlers fetch for no other purpose. "Blocking AI" with robots.txt is then accurate for most vendors, and Google, which fetches once with Googlebot and serves both Search and AI answers from that fetch, is the exception.
The objection is right about the outcome and wrong about where the meaning lives. A Disallow under GPTBot does what RFC 9309 says: GPTBot does not request the path. That the request would have fed training is a fact about the crawler, supplied by the vendor's description of it, the same kind of source that gives Google-Extended its meaning. A vendor that widens what its crawler's fetches are used for changes the outcome of every Disallow aimed at it, and no robots.txt file records the change. The line also reaches only future requests from the token it names. It does not reach a copy fetched before the line was written, a copy fetched under another token, or a copy held in an archive. A Disallow under Common Crawl's crawler stops that crawler's next visit and says nothing to anyone training on what the archive already holds. For every vendor, the file says do not fetch, and only the vendor's page says what not fetching prevents.
A second objection holds that robots.txt already carries usage through Section 2.2.4 of RFC 9309. Cloudflare's Content Signals Policy, announced in September 2025, uses that room: a line such as "Content-Signal: search=yes, ai-train=no" already appears in robots.txt files. Allow and Disallow mean the same thing to every compliant crawler because the RFC fixes their meaning. Content-Signal means what Cloudflare's policy says it means, and only to crawlers that adopt the policy. Cloudflare's announcement draws the line itself. It says robots.txt lets a site instruct which crawlers can access which parts of it, then adds: "It does not, however, let them know what they are able to do with your content after accessing it." If extension records already made robots.txt a standard usage channel, the AIPREF working group would have nothing to standardize.
The authors of the usage layer did not redefine Disallow to mean "do not train." They added a second field.
If draft-ietf-aipref-attach becomes an RFC, Allow and Disallow keep their meaning, and robots.txt gains a usage field defined by a standard rather than by a vendor. Until then, every usage meaning in a robots.txt file is a sentence on a vendor's page, and a definition of a usage token is citable only with the date of the page behind it. A robots.txt file shipped last year still parses exactly as it did. What its Disallow lines prevent is whatever the vendor's page says today.
Further reading
- Search Engine Journal (Matt G. Southern), report on Gary Illyes's Search Central video and the status of the IETF Content-Usage draft; the grounding report for this essay searchenginejournal.com/google-illyes-robots-txt-crawling-ai-training
- Cloudflare (Will Allen), announcement of the Content Signals Policy, a published usage policy layered on robots.txt; used in the objection section blog.cloudflare.com/content-signals-policy
Every source above was fetched and a verbatim phrase confirmed on the page before this essay published. Nothing here is paraphrased from memory.