The first time someone gets caught out by a confident AI mistake — a fake legal citation, an invented quote, a made-up author, a slightly wrong tax rule — they learn an important lesson. The model did not get confused or have a bad day. It did exactly what it was designed to do, and the user happened to ask a question where that produced a wrong answer.
This is not hypothetical. In Mata v. Avianca (2023), lawyers submitted non-existent judicial opinions generated with ChatGPT. The court imposed sanctions after examining both the false submissions and counsel’s conduct; read the court’s 22 June 2023 opinion and order rather than a secondary retelling.
This is one of the few topics in AI where understanding the mechanism matters. Once you see how hallucinations work, you stop being surprised by them, and you stop being burned by them.
The official term
The common name for “AI confidently produces a false answer” is hallucination. The term is imperfect because the model is not seeing anything. What is happening is that the model produces text that is fluent, plausible, and wrong.
The practical point is worth absorbing: hallucinations are not only a temporary product defect. They come from the gap between generating plausible language and verifying truth. You should expect them, plan around them, and not assume the next model release removes the need for verification.
Why it happens
To predict the next word well, a language model has to have absorbed an enormous amount of language structure — grammar, facts, conversational patterns, how a doctor’s note differs from a tweet. When you ask it a question, it produces a continuation that statistically fits the patterns it has learned.
Here is the catch: the generated answer does not come with a dependable, inspectable truth signal. Models can be trained to express uncertainty and can use retrieval or tools, but a confident or cautious tone still does not prove whether a claim is correct. When reliable information is sparse — a niche legal case, a recent event, an obscure person, or an exact number — a model may still generate a plausible-sounding answer that is wrong.
The practical version: the model has a strong mechanism for producing a likely continuation and a weaker mechanism for proving that the continuation is true. On common material, those two often line up. On niche, recent, proprietary, or exact material, they diverge.
Do not treat confidence, polish, or length as evidence. A wrong AI answer can be formatted better than the right source document.
Where hallucinations show up most often
The pattern is consistent across all current AI assistants. They hallucinate most when asked for:
- Specific names, dates, numbers, or quotes. “Who said X?” “What year was Y published?” “Cite three studies about Z.” Statistical patterns are excellent at generating the shape of a citation and unreliable at the actual values.
- Niche or obscure topics. Anything where the training data was thin — small companies, rare diseases, regional regulations, individual people who are not famous. The thinner the data, the bigger the gap between “plausible” and “true.”
- Recent events. Models have a training cutoff. Ask about something that happened after that, and unless the model is actively searching the web, it will either say “I’m not sure” (the good outcome) or guess.
- Proprietary information about your specific company. It does not have your customer list, your codebase, your contracts, or your internal data. If you ask about them, it will produce what such information would probably look like.
- Math and exact reasoning. Multi-step arithmetic, unit conversions, financial calculations, and anything where a small slip cascades. Models do better than they used to, especially when they can use a calculator tool, but they are not yet reliable for high-stakes precision.
Notice the pattern: the model fails predictably when you ask for something specific without giving it a reliable source.
Where factual errors are less central — but still possible
Some tasks depend less on recalled facts, which lowers one kind of hallucination risk:
- Drafting and structuring writing. No “facts” required.
- Rewriting and editing your own text. The source of truth is what you gave it.
- Summarizing a document you provided. The model can quote and compress what is already in front of it, but it can still omit qualifications or misattribute a statement; compare important summaries with the source.
- Brainstorming and idea generation. There is no “true” answer to hallucinate.
- Explaining well-known concepts. Common material may be answered more reliably than obscure material, but it still needs verification when accuracy matters.
These are lower-risk uses, not fact-free guarantees. Specific facts about niche things deserve extra scrutiny, but ordinary summaries and explanations are not exempt.
Three habits that protect you
You do not need a PhD in machine learning to avoid the trap. You need three habits.
1. Treat any specific claim as a starting point, not the final word
If the model gives you a name, a date, a number, a citation, a regulation, or a quote — and the answer matters — verify it. A quick search, a check against the source document, or “where did you find this? quote the passage” as a follow-up will catch most issues. The shorter and more specific the answer, the more important the check.
A particularly useful follow-up: “What source would verify this, and which exact claims should I check?” This does not prove the answer, but it turns a broad response into a verification checklist.
2. Use search when the answer needs to be current or verifiable
All major AI assistants now have a web-search mode. ChatGPT has a Search button; Claude has search built into many tasks; Gemini has it natively. Turn it on when you are asking about anything time-sensitive or factual. The model becomes much more reliable because it can cite actual sources, and you get links you can check.
A common mistake is to use the base model for a research question and then trust the answer. The right pattern is “open a fresh chat, turn on search, then ask.” It takes one extra click.
3. Give it the source instead of asking it to remember
If you have the document, paste it in. If you have the data, attach the spreadsheet. If you have the original email, include it. The model is dramatically more reliable when summarizing or analyzing content you provide than when asked to recall from training.
Weak: “What does Estonian employment law say about probation periods?”
Better: “Here is the section of the Estonian Employment Contracts Act on probation [paste]. Explain what it means for a six-month contract.”
The first prompt is a hallucination trap. The second is grounded in real text and the model can stay accurate.
A source-check workflow
When an answer matters, do this before you use it:
- Highlight specific claims. Names, dates, prices, quotes, citations, legal claims, medical claims, and exact numbers.
- Ask for verification targets. “Which claims in your answer should I verify, and what source would verify each one?”
- Check the primary source. Prefer official docs, original regulations, product pages, filings, contracts, or source documents over summaries.
- Replace unsupported claims. If you cannot verify a claim quickly, remove it, mark it as uncertain, or rewrite it as a question.
- Save the source link. Future you should know where the answer came from.
The companion checklist linked from this article turns this into a quick go/no-go review.
A useful exception: when hallucination is fine
Sometimes you want the model to invent things — that is the entire point. Fiction, brainstorming, hypotheticals, edge cases for tests, creative variations on a theme, “what would the worst version of this argument be” prompts. In creative work, hallucination is not a flaw; it is a feature. Just notice the difference between “give me five plausible-sounding company names” (great use) and “give me five real companies that do X” (a trap).
What about model upgrades?
Newer models hallucinate less than older ones on average, especially on common topics. But on the niche end — the rare cases, the recent events, the proprietary details — even the best 2026 models still make things up confidently. Do not assume the next release solves this. The structural reason hallucinations exist is unchanged.
A small safety checklist
Before you rely on an AI answer, ask:
- Does this answer contain names, dates, numbers, quotes, citations, laws, prices, medical advice, or financial claims?
- Is the topic recent, niche, local, proprietary, or high-stakes?
- Did the model use search, a provided source, a database, or a calculator?
- Can I point to the source sentence, table, or record that supports the claim?
If the answer to 1 or 2 is yes and the answer to 3 or 4 is no, you have a hallucination risk. Verify before acting.
The habit that catches hallucinations
Hallucinations are not evenly distributed. They are more likely to cause harm around specific names, dates, numbers, obscure topics, recent events, private data, and exact calculations. Drafting and brainstorming reduce the need for recalled facts, but summaries and explanations can still distort a source.
Build the habit of catching them: verify specifics, use current sources for factual questions, and provide the source instead of relying on model recall. The NIST Generative AI Profile treats confabulation as a risk to manage, not a problem that fluent output resolves.
