What is an AI hallucination?
A hallucination is when AI gives an answer that sounds credible but is wrong or made up.
A language model is built to write text that sounds plausible. It has no built-in check that says it does not know. When the material is missing it sometimes fills the gap itself, in the same confident tone as when it is right.
It can be a reference to a routine that does not exist or a figure nobody can trace. The risk is highest when the model answers about things it has never seen, such as your internal rules.
The risk can be reduced. Let AI answer from your own documents, show the source of every answer and let the system say it found nothing when the material is missing.
How it works
The model chooses words by what is likely in context. It does not check the answer against a store of facts. When the question concerns something missing from the material, the most likely response is still an answer, often in the same shape as a correct one. That is why invented names, clauses and sources can look so real.
Why it matters
The risk for an organisation is that a wrong answer gets used as if it were right. An invented statement about a procedure, a rule or a contract term can lead to a poor decision. The more AI is used day to day, the more it matters to see what an answer rests on.
What to keep in mind
Hallucinations cannot be removed entirely with today's technology. The aim is to make them rare and easy to spot.
Pay particular attention to details such as dates, amounts and references. They are easy to skim past and hard to guess correctly. Check the source whenever the answer will be used for something that matters.
What it means for you
Ask the supplier what happens when the answer is not in your documents. A good system says so plainly.
Common questions
Why does AI make things up?
A language model is built to write text that sounds plausible. When material is missing, it sometimes fills the gap itself without saying so.
How can you spot a hallucination?
Rarely by the tone, since a wrong answer sounds just as confident as a right one. Check it against the source. An answer without a source citation is harder to review.
How do you reduce the risk?
Let the AI answer from your own documents using RAG, show sources and instruct the assistant to say when the material is missing.
Related terms
Read more
All terms Updated