Hallucination
AI & SearchAlso: AI Hallucination · Model Hallucination · Confabulation
Quick definition
Hallucination is when an AI language model produces output that sounds confident and coherent but is factually wrong, invented or misleading. The model does not know it is wrong. It generates plausible-sounding text based on patterns, not on verified knowledge.
How it varies across Australia
Hallucination rates vary sharply by model, task type and how well the prompt is grounded in source material. Retrieval-augmented approaches reduce the rate but do not eliminate it. The risk sits highest in tasks that ask models to recall specific facts, cite sources or produce numbers without being given those facts first.
See AI adoption patterns across Australian industries →Four types marketers encounter
The model states something false as a fact, such as a wrong statistic, date or company detail.
The model invents a plausible-sounding source, author or study that does not exist.
The model reaches a confident conclusion through logic that does not hold up.
The model describes a real company's product, pricing or history incorrectly, sometimes harmfully.
What it actually means
A hallucination is not a glitch. It is the model doing exactly what it was trained to do, which is produce the most plausible next token given the context. The model has no internal fact-checker. It has no access to ground truth. It generates text the way a confident person fills in a blank, by reaching for what sounds right.
For marketers, this matters because AI is now embedded in content production, search features, customer-facing chatbots and research workflows. When a language model is asked to summarise your brand, cite a competitor's pricing or report an industry statistic it was not given in the prompt, it will frequently invent something convincing.
The outputs that are most dangerous are not the obviously wrong ones. Those get caught. The dangerous hallucinations are plausible, well-formatted and match the tone of real information well enough to pass a quick read.
This is why AI-generated content in regulated categories, legal, financial, health, requires human review at the output stage, not just at the prompt stage. It is also why generative AI in search results, like AI Overviews in Google, creates new content accuracy risks for brands whose products or services are summarised without retrieval from a current, authoritative source.
Understanding hallucination is not optional for marketers using AI at scale. It is the foundational risk that shapes every other AI governance decision.
The problem with hallucination is not that the model lies. It is that the model does not know it is lying.
How it shows up
Hallucination shows up differently depending on the task. In content production it appears as invented statistics, wrong product details or fake competitor comparisons. In chatbots it appears as confident wrong answers to customer questions about pricing, policy or availability. In research summaries it appears as plausible-sounding citations that do not exist when you check.
In AI-driven search features like AI Overviews and Perplexity, hallucination shows up as incorrect brand summaries or wrong factual claims about your products appearing as the first thing a searcher sees. That is an answer engine optimisation (AEO) and brand-accuracy problem that is hard to correct after the fact.
The Australian context
Australian businesses in financial services, healthcare and legal services face specific regulatory exposure when AI-generated content contains hallucinated claims. The Australian Securities and Investments Commission (ASIC) and the Australian Health Practitioner Regulation Agency (AHPRA) both hold content publishers responsible for accuracy, regardless of the tool used to produce it.
Australian consumer law under the Australian Competition and Consumer Commission (ACCC) also applies to false or misleading representations even when they are AI-generated. The production method is not a defence. This makes hallucination risk a compliance risk for any regulated Australian business using generative AI in customer-facing content, not only a quality risk.
Where people get this wrong
Related terms
Common questions
Can you stop an AI from hallucinating?
Not completely. You can reduce hallucination by grounding the model in verified source material, using retrieval-augmented generation (RAG), restricting the task scope, and adding human review at the output stage. But no current model eliminates hallucination entirely, which is why human oversight remains necessary for factual or high-stakes content.
Does hallucination get worse with longer outputs?
Generally yes. Longer outputs give the model more opportunity to drift from source material and more surface area for compounding errors. Short, grounded tasks with explicit source material produce fewer hallucinations than open-ended long-form generation tasks.
How does hallucination affect my brand in AI search results?
AI search features like Google's AI Overviews and Perplexity may summarise your brand using hallucinated details if your owned content is thin, outdated or not structured clearly. Wrong pricing, features or claims can appear as the first thing a searcher sees. Strong answer engine optimisation (AEO) and well-structured owned content reduce this risk.
Is hallucination a legal risk for Australian businesses?
Yes, in regulated categories. Australian consumer law prohibits false or misleading representations regardless of how the content was produced. Financial services, health and legal content published by AI without adequate fact-checking can create ASIC, AHPRA or ACCC exposure. The AI being the author is not a defence.
Keep exploring
About New Rebellion
New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.
How we think →