The Future Website: Building for Machines and Humans at Once

What this entry means by a future website

The phrase future website usually arrives attached to a redesign. This entry uses it narrowly, and the narrow version is the useful one: a website that has to serve two kinds of client at once. One is a person, arriving with a browser, reading, scanning, deciding. The other is an automated client, arriving on behalf of an AI assistant or a search feature, fetching the document, extracting what it can, and leaving. The design question is whether those two requirements pull in the same direction or in opposite ones.

The second audience is now large enough to design for deliberately. Cloudflare Radar publishes a continuously updated measure of the share of requests it classifies as automated rather than human. That share moves day to day, and Cloudflare measures several different things under related labels, so a single figure quoted out of context is misleading. What the dashboard supports is the weaker and more durable statement: automated retrieval is a routine, ongoing portion of traffic to public pages, not an occasional event.

None of this establishes that building for machines produces citations, visibility, or traffic. It establishes only that a document published on the open web is now read under two very different conditions, and that a publisher who has never looked at the second condition does not know what the second reader sees.

The clients are named, separable, and documented

The most concrete change of the last two years is that the major operators stopped shipping one anonymous crawler and started publishing named user agents with stated purposes. This matters because it converts a vague question, should we let AI read the site, into several specific ones that can be answered differently.

OpenAI documents four. GPTBot is described as being used to make its generative AI foundation models more useful and safe, and OpenAI states that disallowing it indicates a site's content should not be used in training foundation models. OAI-SearchBot is described as used to surface websites in search results in ChatGPT's search features, and OpenAI states plainly that sites opted out of it will not be shown in ChatGPT search answers. OAI-AdsBot validates the safety of pages submitted as ads. ChatGPT-User covers certain user actions in ChatGPT and Custom GPTs, and here OpenAI's documentation carries an important caveat: because these actions are initiated by a user, robots.txt rules may not apply.

Anthropic documents three with a similar split. ClaudeBot collects web content that could potentially contribute to training. Claude-User accesses websites when individuals ask questions to Claude. Claude-SearchBot navigates the web to improve search result quality. Anthropic states that its bots respect do not crawl signals by honoring industry standard directives in robots.txt, supports the non-standard Crawl-delay extension, and warns that blocking its IP addresses may not work reliably because doing so impedes its ability to read the robots.txt file at all.

The practical consequence: training access, search access, and user-initiated fetching are three separate decisions with three separate levers, and a site that blocks a training agent has not thereby removed itself from an assistant's answers. Publishers who set one rule for everything are, on the operators' own published descriptions, making a coarser decision than the controls allow.

The rendering gap, and the one finding that most changes how a site is built

In December 2024, Vercel and MERJ published a network-scale analysis of AI crawler behavior across Vercel's infrastructure, covering GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and crawlers operated by Meta, ByteDance, Perplexity, Apple and Google. Over one month the study observed 569 million GPTBot requests and 370 million Claude requests. Its central finding was that none of the major AI crawlers it examined rendered JavaScript. Several downloaded JavaScript files, ChatGPT in 11.50 percent of requests and Claude in 23.84 percent, without executing them.

The study also recorded a striking inefficiency: ChatGPT spent 34.82 percent of its fetches on 404 pages and 14.36 percent following redirects, against 8.22 percent and 1.49 percent respectively for Googlebot. Whatever else that indicates, it suggests these clients were working from less reliable URL knowledge than a mature search crawler.

The design implication, if the finding still holds, is direct: content that exists only after client-side JavaScript runs is not present in the document those clients receive. Server-rendered or statically generated HTML is what they read. This is the single largest divergence between the human reader, whose browser executes everything, and the automated one, whose fetch may stop at the raw response body.

The limit on that implication should be stated clearly. The study is dated December 2024. Crawler capability is not fixed, operators do not announce rendering changes, and Google's own AI surfaces draw on infrastructure that has rendered JavaScript for years. Treat the finding as the best available public measurement of a moment, not as a permanent property of these systems, and re-measure against server logs rather than assuming.

Structured data is a description of the page, not a substitute for it

Google describes structured data as a standardized format for providing information about a page and classifying page content, and says it recommends JSON-LD as easiest to implement and maintain while treating all three supported formats as equally acceptable. That is a modest and well-bounded claim about what markup does: it makes explicit, in a parseable form, things a document already says.

The constraint attached to it is the part most often skipped, and it is the hinge of this entry. Google's documentation states: do not create blank or empty pages just to hold structured data, and do not add structured data about information that is not visible to the user, even if the information is accurate. That rule forces the two audiences back into alignment. A machine-readable layer that describes something the human reader cannot find on the page is out of policy on Google's own terms, and it is also a semantic consistency failure in the AIOFacts sense: the document says two different things depending on who reads it.

So the honest framing of structured data is not that it is a channel to the machines. It is a second, more explicit encoding of the same assertions the prose already makes. Where the two disagree, the disagreement is the defect.

The parallel-file question: one document or two

There is real pressure toward serving machines something other than the human page, and it comes from a genuine efficiency argument. Cloudflare shipped Markdown for Agents in February 2026, converting HTML to markdown for clients that request it with an Accept header, and reported one of its own posts dropping from 16,180 tokens as HTML to 3,150 as markdown, roughly an 80 percent reduction, with an x-markdown-tokens response header so a client can size its context window before reading.

That approach has a property worth naming: it is content negotiation. The same document, at the same URL, in a different representation. It cannot drift from the human page because it is derived from it.

The other approach is a separate authored file, of which llms.txt is the best known proposal. Here the relevant primary evidence is unusually direct. Google's AI features documentation states: you do not need to create new machine readable files, AI text files, or markup to appear in these features, and there is also no special schema.org structured data that you need to add. It further states that to be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, and that there are no additional requirements or special optimizations necessary.

AIOFacts treats that as a position worth stating rather than a settled verdict on the proposal: a derived representation of the page is structurally safer than a hand-maintained parallel corpus, because the second one is a document that can be wrong about the first. Whether any operator will eventually consume such a file is not established here, and the absence of a documented consumer today is not proof of permanent irrelevance.

Access became a negotiation, and the negotiation has no standard yet

All of the controls above rest on RFC 9309, the Robots Exclusion Protocol, published by the IETF in 2022. It is a real standard, and its limits are structural: it expresses a request, it has no enforcement mechanism, and it was written to govern crawling, not the downstream uses that publishers now care most about. The vocabulary of training, grounding, search indexing and user-initiated retrieval does not exist in it. Operators have filled that gap with their own conventions, which is why every operator's control surface looks slightly different.

The IETF chartered the AI Preferences working group to address exactly this, describing the current landscape as a confusing array of non-standard signals in the robots.txt file, and noting that the resulting loss of confidence pushes publishers toward blunter measures such as blocking IP ranges. Its remit is a common vocabulary for expressing preferences about use of content for AI training, plus means of attaching that vocabulary to content through mechanisms including robots.txt and HTTP headers. That work is in progress, and a chartered working group is not a deployed standard.

The economics running underneath the negotiation are also now partly measurable. Cloudflare's August 2025 crawl-to-click analysis reported crawl-to-refer ratios that differed by orders of magnitude between operators, and Cloudflare is explicit that the figures come from a fixed set of customers in specific weeks, seen from its own network, with crawler purpose classified from operator disclosures rather than independent verification. Read as a rough shape rather than a precise measurement, it indicates that retrieval volume and referred visits are largely decoupled, which is why access is being renegotiated at all.

What the two readers actually have in common

Set the mechanisms side by side and the overlap is larger than the conflict. What the documented behavior of automated clients rewards is, with few exceptions, what a careful human reader has always been served by.

Content present in the served HTML helps a reader on a slow connection and is what a non-rendering client receives. Accurate headings give a person a scannable structure and give a parser a document outline. Stable, resolvable URLs prevent a reader hitting a dead link and prevent the 404 waste the Vercel and MERJ data recorded. An opening paragraph that answers the question before the context helps a scanning reader and yields an extractable claim. Descriptive link text and image alternative text are accessibility obligations first, and machine-readable description second. Structured data that mirrors visible content keeps the two encodings honest with each other.

The genuine divergences are narrow and mostly operational rather than editorial: which named agents are permitted to fetch, whether an alternative representation is negotiated, how snippet controls such as nosnippet, data-nosnippet and max-snippet are applied, and how fast a document can be served under crawler load. Google directs publishers to those existing snippet controls to limit what its features show, rather than to anything AI-specific.

The failure mode to avoid is building a second site. A machine-facing layer maintained separately from the human page becomes stale, and a stale parallel document is worse than no parallel document, because it asserts things the canonical page no longer says.

What cannot be seen from outside

Everything above describes access and representation: what a client is documented to fetch, what it is documented to honor, and what arrives in the response. None of it describes selection. No operator publishes how a retrieved document is chosen, weighted, summarized or attributed inside a model or a retrieval pipeline, and nobody outside that operator can observe it. A page that is perfectly accessible to a retrieval client may never appear in an answer, and the reason will not be visible from the publisher's side.

This is why the correct posture toward the whole subject is engineering rather than optimization. Serving a complete, accurate, fast, well-structured document to whoever asks for it is defensible on its own terms, independent of any claim about downstream effect. Publishers should be suspicious of anyone, including AIOFacts, who describes the step after retrieval with more confidence than the published evidence supports.