AI Search and Traditional Search: What Carries Over, and What Does Not

Why this comparison is being made now

AI answer surfaces now sit between a person and a website in a growing share of information seeking. Google has placed AI Overviews and AI Mode inside its results pages, ChatGPT carries search features, and Claude carries web search. Each of these returns a composed answer with citations rather than a ranked list of documents, which is a different product shape from the one search practice was built around.

The question site owners keep asking is whether two decades of search practice still applies. Reading the current documentation, the answer separates into three layers that behave differently: retrieval, which largely carries over; access control, which has fragmented; and presentation and measurement, which have changed materially. Most disagreement about this topic comes from arguing about all three at once.

This piece keeps them apart, cites the primary document for each claim, and states plainly where the available evidence runs out.

What carries over: retrieval still runs on an index

Google is direct about the relationship between its generative features and its existing systems. Its guidance for site owners states that for a page to appear in these features, "a page must be indexed and eligible to be shown in Google Search with a snippet," and that there are no additional technical requirements beyond standard Search eligibility. Its optimization guide goes further: "The best practices for SEO continue to be relevant because our generative AI features on Google Search are rooted in our core Search ranking and quality systems."

That is one vendor describing one product, and it should not be generalized into a statement about every assistant. It is, however, an explicit statement about the largest such surface, and it is the clearest available evidence that crawlability, indexability, and snippet eligibility are current prerequisites rather than legacy concerns.

A weaker version of the same shape appears elsewhere. OpenAI documents OAI-SearchBot as the agent "used to surface websites in search results in ChatGPT's search features," and states that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." Anthropic documents Claude-SearchBot as navigating the web "to improve search result quality for users." In each case the documented mechanism is retrieval of live web documents, not recall from model training.

• Indexed and snippet eligible, per Google's stated requirement for its AI features
• Reachable by the named retrieval agent in robots.txt, per OpenAI and Anthropic crawler documentation
• Not withheld by nosnippet, data-nosnippet, max-snippet, or noindex, which Google lists as the controls that limit appearance

What is genuinely new: access is negotiated per purpose

Under traditional search, one crawler identity mostly settled one question: may this site be indexed. The current documentation splits that decision by purpose, and this is a real structural change rather than a reframing.

OpenAI documents GPTBot for content "that may be used in training our generative AI foundation models," OAI-SearchBot for search surfacing, and ChatGPT-User for user-initiated fetches, adding that "because these actions are initiated by a user, robots.txt rules may not apply." Anthropic documents the same three roles as ClaudeBot, Claude-SearchBot, and Claude-User, each with its own robots.txt token. Google separates Googlebot from Google-Extended.

One robots.txt file therefore now expresses several separate commercial positions at once: whether content may contribute to training, whether it may be cited in an assistant's answer, and whether it may be fetched live when a person asks a question. A site can be fully indexed in conventional search and absent from a given assistant because a retrieval agent was disallowed, often by a rule copied from a template written to block training crawlers. That failure mode did not exist before and is invisible in conventional rank reporting.

What does not carry over: the answer can end the session

The clearest measured difference is behavioral. Pew Research Center analyzed 68,879 Google searches from 900 US adults who shared browsing data during March 2025, with results pages collected in April 2025. Users clicked a traditional result on 8% of pages where an AI summary appeared, against 15% where none appeared. Clicks on a source link inside the summary itself occurred on 1% of such pages. Sessions ended after 26% of pages carrying an AI summary, against 16% of pages without one.

The limits belong in the same paragraph as the numbers. This is one country, one month, one search engine, and a panel that agreed to be tracked. Pew states its own constraints: technical limits on identifying AI-generated summaries on other search engines, and the fact that the summary for any given search may change over time. The study describes an association across a population, not a causal effect on any individual site.

Read conservatively, it still describes something traditional search did not have. The result surface can satisfy the information need without a visit. A practice measured entirely by click acquisition is therefore measuring an outcome the surface may no longer produce, which is why visibility and traffic have become two questions instead of one.

What does not carry over: the typed query is no longer the unit

Google documents that AI Overviews and AI Mode "may use a 'query fan-out' technique, issuing multiple related searches across subtopics," and says this lets the system display a wider and more diverse set of helpful links than a single result set would. The retrieval that produces one answer is therefore not only retrieval for the string the person typed.

Two consequences follow from the documented mechanism. A page can contribute to an answer without ranking for the phrase entered, and a page that ranks for that phrase may not be drawn on at all. Separately, conversational surfaces accumulate context across turns, so there is no single stable results page attached to a keyword in the way rank tracking assumes.

AIOFacts proposes, as a position rather than a finding, that the useful unit shifts from a page matching a query toward a passage that answers a sub-question completely enough to be quoted. This is an inference from a documented mechanism. It is not a claim about how any system weighs or selects content internally, and nobody outside a system's operator is in a position to make that claim.

What degrades: measurement, and this is the real gap

Traditional search supplied a shared instrument. Impressions, average position, and clicks, reported per query by the engine itself, were imperfect but common to everyone. AI answer surfaces are not yet measured to that standard. Google directs site owners to Search Console for performance in its AI features, which is a real reporting surface and a narrow one, covering a single platform.

Beyond that, answers vary by user, by session, and over time. There is no public cross-platform record of what was cited, to whom, and when. Third-party visibility tools reconstruct an estimate by sampling prompts, which is a legitimate method carrying a specific limitation: a sample of prompts is not a census of answers, and two vendors sampling differently will disagree without either being dishonest.

This measurement gap is why conflicting statistics circulate about how far AI citations overlap with conventional organic rankings. Widely repeated overlap percentages are frequently attributed to studies that, on inspection, do not contain them. Checking the most commonly cited overlap figure against the Semrush AI Overviews study it is usually credited to, AIOFacts could not locate that figure in the study. Treat any single overlap number as unverified until the primary document has been read.

Proposed AI-specific requirements the documentation does not support

A category of practice has appeared promising technical requirements specific to AI surfaces. The strongest current test case is llms.txt, a proposed file for exposing site content to language models.

Google's optimization guide addresses this directly, stating that "you don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and separately that there is no special schema.org structured data required for its AI features. Independently, an Ahrefs analysis of 137,210 domains with traffic in May 2026 found that 97% of llms.txt files received no requests at all. Ahrefs notes that its own sample skews technical and SEO aware, and that a request does not confirm the content was consumed.

A vendor statement and an independent measurement pointing the same direction is the strongest form of evidence available on a question like this. It does not establish that the format is worthless, and adoption could change. It does establish that as of this date no documented retrieval system requires it, so publishing one is a low-cost bet rather than completed visibility work, and it should be recorded as such.

The structured data point deserves care in the other direction. Google saying schema is not required for its AI features is not the same as saying structured data is unhelpful. It remains documented and required for many rich results, and it remains a machine readable statement of entity facts. The error to avoid is selling it as an AI requirement, because the documentation says otherwise.

A working position on the difference

AIOFacts does not treat AI search as the successor to traditional search, and does not treat the two as unrelated. The supportable description is a split across layers.

Access, indexability, and document quality carry over almost entirely, on the documentation available today. Access control has fragmented into separate decisions about training, search surfacing, and live fetching, each of which now needs an explicit answer. Presentation has changed, because a composed answer can resolve the need and end the session. Measurement has weakened, and no participant currently holds a complete view of it.

Practically, that means most of what an organization already does for search is not wasted, and the parts that need new work are the ones nobody previously had to think about: which agents are permitted, whether individual passages stand alone as answers, and how visibility is tracked when the click is no longer the evidence. Anyone describing this shift as a clean replacement of one discipline by another is describing something the current documentation does not show.