What Is AI Hallucination and How Do You Prevent It?
AI hallucination is when a language model produces information that sounds confident but is false or unsupported, such as a made-up policy, citation, price or quote. It happens because models generate the most plausible next words rather than looking facts up. You cannot eliminate it, but you can reduce it sharply: ground answers in your own documents, let the model say "I don't know," require citations, keep tasks narrow, test against known answers, and have a person review anything high-stakes.
- Hallucinations come from how language models work: they predict plausible text, they do not check facts.
- NIST's Generative AI Profile calls this risk confabulation and lists it as a core generative AI risk.
- Grounding answers in retrieved documents is the single most effective fix for business use.
- Explicit permission to say "I don't know" and a requirement to quote sources both reduce made-up answers.
- Your business can be held responsible for what its AI tells customers, so test before launch.
Why AI Models Make Things Up
A large language model is trained to predict the next piece of text given everything before it. That makes it very good at producing fluent, plausible answers. It does not, by default, look anything up or check whether a statement is true. When the model lacks the right information, the most plausible-sounding continuation can be a confident fabrication.
Several things make it more likely: questions about obscure or very recent facts, requests for exact figures, citations or quotes, long multi-step reasoning, and prompts that pressure the model to give an answer no matter what. NIST's Generative AI Profile (AI 600-1) uses the term confabulation for this risk, a deliberate choice because the model is not perceiving anything; it is filling gaps.
What It Looks Like in a Business
- A website chatbot invents a refund window or a discount that does not exist.
- An assistant drafting a proposal cites a regulation section or case that is wrong or fictional.
- A data extraction step "reads" an invoice total that is not on the page.
- A summary of a call attributes a commitment to the wrong person.
Notice that most of these are not exotic questions. They are routine tasks where the model was missing one piece of information, or was asked for precision it could not supply, and filled the gap instead of saying so. That is why the fixes below focus on supplying information and permitting refusal rather than on choosing a cleverer model.
The stakes are real. In a widely reported 2024 decision, a Canadian tribunal held Air Canada responsible for incorrect bereavement-fare information its website chatbot gave a customer, rejecting the idea that the bot was separate from the company. Assume your business owns whatever its AI says.
How to Reduce Hallucinations
Ground the answers in your data
Give the model the facts instead of asking it to remember them. Retrieval-augmented generation finds relevant passages from your documents and puts them in the prompt. Google, OpenAI and Microsoft all offer grounding features for this, including grounding on search results or your own data.
Allow "I don't know"
Anthropic's guidance on reducing hallucinations starts here: explicitly tell the model it may say it does not have enough information. Without that permission, models tend to guess.
Require quotes and citations
Ask the model to quote the source passage that supports each claim, and to withdraw any claim it cannot support. In an automated system you can check that the quoted text really exists in the source.
Narrow the task and use structure
A model asked to "answer anything about our company" will drift. One asked to "answer service-area and scheduling questions from these five documents" will not drift as far. For data extraction, ask for structured output and validate it with code: does the total equal the line items, is the date valid?
Keep a human on high-stakes outputs
Anything legal, medical, financial or contractual should be reviewed by a qualified person before it is sent or relied on. Use AI to prepare, not to decide, and make the reviewer's job easy by showing the sources next to each draft.
Test Before Launch, Then Keep Testing
You cannot judge accuracy from a demo. Build a test set of real questions or documents with known correct answers, including tricky ones and questions the system should refuse. Run it before launch and every time you change the prompt, the content or the model. Track three numbers: correct answers, wrong answers and correct refusals. After launch, review a sample of real conversations each week and add any failures to the test set. Our guide to evaluating AI output quality lays out the process.
Choosing Tools With Hallucination in Mind
Newer models generally hallucinate less than older ones, and some are better at declining when unsure, but none are immune. When comparing vendors or building a system, ask: does it show sources? Can it be restricted to approved content? Does it log answers for review? Can you test it on your own questions before buying? A tool that answers slightly fewer questions but is reliably right on the ones it answers is usually the better business choice. If you want help building those controls, see DR3AM AI automation.
Sources and Further Reading
We link to the primary documentation so you can verify the details yourself.
- Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1), NIST
- Reduce hallucinations, Anthropic Docs
- Grounding overview, Google Cloud Vertex AI
- OWASP Top 10 for LLM Applications, OWASP Gen AI Security Project
Related Questions
Will AI hallucinations go away as models improve?
They are becoming less frequent, especially with retrieval and web grounding, but there is no sign they will disappear entirely, because the underlying method is still prediction. Plan systems that detect and contain errors rather than assuming the next model will be perfect.
Does turning down temperature stop hallucinations?
Lower temperature makes output more consistent and less random, which helps for extraction and classification. It does not give the model knowledge it lacks, so a low-temperature model can repeat the same wrong answer reliably. Grounding and testing matter more.
Are hallucinations a bigger problem in customer-facing AI?
Yes, because nobody reviews the answer before the customer acts on it. Internal tools usually have a staff member checking the output. Customer-facing assistants need tighter scope, approved content only, clear refusal behavior and an easy path to a person.