Hallucinations and Brand Safety

What a hallucination is, in plain terms

A hallucination is a confident statement that is not supported by the material the system had available. The National Institute of Standards and Technology avoids the popular word entirely. In its Generative AI Profile, published July 2024 as NIST AI 600-1, it uses the term confabulation and defines it as the production of confidently stated but erroneous or false content, colloquially called hallucinations or fabrications, by which users may be misled or deceived.

The distinction is worth keeping. The everyday word suggests a rare malfunction, something visibly broken. The documented failure is more ordinary and harder to spot: text that is fluent, well formed, plausibly sourced in appearance, and wrong. It arrives in the same register as text that is correct, which is precisely what makes it a brand problem rather than a curiosity.

There is also a published argument about why this persists. Kalai, Nachum, Vempala and Zhang, writing in September 2025, argue that common training and evaluation procedures reward guessing over admitting uncertainty, because a benchmark scored as simply right or wrong awards nothing for a stated non-answer and awards positive expected credit for a guess. That is an argument about how these systems are measured, offered by researchers who build them. It is presented here as their analysis, not as a description of how any deployed product behaves internally.

Why this became a brand problem rather than a research problem

When an interface answers a question instead of listing ten results, an error about a company can be the entire thing a person sees. There is no second listing underneath to contradict it, and in many cases no visible trail back to whatever the claim was drawn from.

The most instructive documented case involves a Minnesota solar contractor, Wolf River Electric, which sued Google in 2025 alleging that AI Overviews told searchers the company was facing a lawsuit from the state attorney general. As reported by Reason's Volokh Conspiracy and by the Star Tribune, the company was not a party to the attorney general action it was associated with. Google moved the case to federal court and disputes responsibility for the output. The matter is unresolved, and nothing here should be read as a finding against Google.

A second case shows the shape of the risk more clearly than any statistic. In March 2025 the privacy group noyb filed a complaint with the Norwegian data protection authority on behalf of Arve Hjalmar Holmen, describing a ChatGPT output that presented him as a convicted child murderer while correctly naming his home town and the number and genders of his children. True detail wrapped around a false core is considerably harder for a reader to discount than an obviously absurd answer.

Grounding reduces the exposure. It does not remove it.

Google documents the mechanism its AI features use. Its Search Central documentation describes retrieval augmented generation, also called grounding, as a technique that improves the quality, accuracy and freshness of AI responses by relying on core Search ranking systems to retrieve relevant, current web pages from the Search index. Retrieval narrows what a response is built from, which is a meaningful constraint.

It is a constraint, not a guarantee. Vectara maintains a public leaderboard that measures how often a model introduces unsupported content when summarizing a document it has been given. Reported rates differ by model and are revised as models change, so the leaderboard is worth consulting directly rather than quoting from memory. The structural point survives whatever the current numbers are: summarizing a supplied document is close to the easiest case, and it is not error free.

Operators have said as much in their own words. After widely shared bad results in May 2024, Google published a note describing more than a dozen technical improvements, including better detection of nonsensical queries and limits on the use of satire and of user generated content in certain responses. That is a useful primary record of both a failure and a fix.

Misattribution is a separate failure from factual error

A response can be broadly accurate and still attach the wrong name to it. In March 2025 the Tow Center for Digital Journalism tested eight AI search tools by supplying excerpts from real articles and asking each tool to identify the source. Collectively the tools answered more than 60 percent of those queries incorrectly, with individual error rates ranging from 37 percent for the strongest performer to 94 percent for the weakest.

For an organization, that finding has two directions. A claim you never made can be credited to you, and a claim you did make and can substantiate can be credited to someone else. Being cited is not the same as being cited correctly, and a citation carries an implication of verification that the underlying process may not have performed.

The limits of that study should be stated with it. It measured provenance of news article excerpts, at one point in time, across a specific set of products. It does not establish an error rate for statements about businesses, and the products tested have shipped many versions since.

Where liability has actually landed so far

For an assistant a company deploys itself, there is a decided case. In Moffatt v. Air Canada, 2024 BCCRT 149, issued 14 February 2024, the British Columbia Civil Resolution Tribunal found the airline liable in negligent misrepresentation after its website chatbot told a customer that bereavement fares could be applied for retroactively, which was not the airline's policy. The tribunal rejected the argument that the chatbot should be treated as a separate entity responsible for its own statements, and awarded CA$650.88 in damages.

The sum is small. The principle is not. In that jurisdiction, a company remains responsible for information presented on its own site, whether the sentence came from a static page or was generated on demand.

Responsibility for what a third party's AI system says about you is a different and unsettled question, currently being argued rather than answered. Treat the deployed-assistant case as instructive and the third-party question as open. This is a reference note on the public record, not legal advice.

What a publisher can and cannot control

No external party controls a model's weights, its retrieval ranking, or its refusal behavior. What a publisher does control is the supply side: what accurate material about the entity exists publicly, how unambiguous it is, and whether independent sources agree with it. Where a claim about a business is scarce, contested, or scattered across contradictory pages, the conditions for a confident wrong answer are better than they need to be.

Google states plainly that there are no additional requirements and no special optimizations needed to appear in AI Overviews or AI Mode beyond a page being indexed and eligible to be shown with a snippet. That is worth taking at face value, and it points the work away from tricks and toward clarity of record.

Practical measures that are within reach, and that are defensible whether or not any particular system uses them:

• Publish the facts most often misstated about the organization plainly, in one durable place, with a visible last-updated date.
• Keep legal name, ownership, locations, service area, credentials, licensing and pricing posture consistent across the site, profiles and directories.
• State what the organization is not, where confusion with a similarly named entity or an unrelated legal action is foreseeable.
• Carry the evidence with the claim: filings, registries, certifications and named third-party coverage, linked rather than asserted.
• Keep the correction path fast, so a public record can be updated in hours rather than in a quarterly cycle.

Monitoring, because outputs are not stable objects

Responses vary with phrasing, region, account history, product surface and model version. A single check is an observation of one moment, not a measurement of a fixed state, and treating it as the latter produces false confidence in both directions.

A workable practice is a fixed panel of prompts, run on a schedule, covering the questions a buyer, a journalist, a regulator and a competitor would actually ask. Record the date, the product and version where visible, the exact prompt, the verbatim response, and a screenshot. Capture matters more than volume, because a bad output that is not preserved is difficult to escalate later.

Interpret conservatively. One anomalous answer is an observation. The same false claim reproduced across separate sessions and separate products is a pattern, and only the pattern justifies the effort of a formal correction.

The correction paths that currently exist

Correction generally starts with the public record rather than with the platform, because a retrieval-based response is assembled from material that can be improved. If the false claim traces to an outdated directory entry, an unresolved news item, or a page that conflates two entities, that is the fixable layer.

Platform controls exist but are narrower than they are often assumed to be. Google Search Console offers a generative AI control governing whether a site helps ground responses in Search AI features, which is a decision about participation, not a mechanism for correcting an answer. Major assistants provide in-product feedback on individual responses. Search products provide their own removal and reporting routes for specific categories of content.

What is not established anywhere in public documentation is a general path for an organization to correct a specific generated statement about itself and verify that the correction took effect. That gap is the core of noyb's argument in the Norwegian complaint, which contends that there is no way for an individual to have false generated output about them corrected. The sequence to prepare in advance is straightforward: preserve the evidence, fix the underlying record, use the available platform feedback and removal routes, and escalate only where the claim is material and persistent.