Why Models Make Things Up
Hallucination isn't lying. It comes from the same mechanism that makes models work at all.
In Context Windows: The Model's Working Memory, you saw how a long conversation can drift as details get lost. This time we tackle the most famous failure of the same predictive engine: the model confidently inventing facts.
When a language model tells you that the Eiffel Tower is in Rome, it isn't lying. It is doing what it was built to do: produce a plausible next word, then the next. Hallucination, the name for these confident inventions, is not a separate fault that switches on. It is the same mechanism that makes the model fluent and useful, working in a place where plausible and true have drifted apart.
Some of it cannot be trained away by better prediction alone. Researchers have shown that a model trained to predict text well must still get some rare facts wrong, the ones it saw only once, because nothing in its training text lets it tell the true version from plausible alternatives. Once you see why, the whole thing stops feeling mysterious.
Why does a model that's so good at predicting words sometimes get facts completely wrong?
The model is not a database of facts. It is a next token predictor. When you ask it a question, it does not look up an answer. It writes a continuation that would be likely given the text it learned from. If the most likely continuation is wrong, it will produce that, fluently and confidently.
Think of a chef who has only ever read cookbooks. She can write a flawless recipe for a dish she has never tasted. If the cookbooks keep repeating a wrong tip, her recipe will repeat it confidently. She isn't lying. She is reproducing the pattern she learned.
A language model learns the same way. During pretraining, it reads trillions of tokens from websites, books and articles and learns to predict each next token. That goal is purely statistical: make the observed text likely. During pretraining there is no separate signal for truth. If a myth such as "we only use 10% of our brains" appears often enough, the model learns to repeat it in the contexts where people repeat it. The TruthfulQA benchmark found exactly this: models "generated many false answers that mimic popular misconceptions", and the largest models of the time were generally the least truthful (TruthfulQA: Measuring How Models Mimic Human Falsehoods).
So hallucination happens wherever what the model finds likely parts ways with what is true. Pretraining teaches it what sounds right, not what is right.
What does the training data have to do with hallucinations?
The training text is the model's only window onto the world. If a fact appears rarely, or in conflicting versions, the model cannot learn it reliably.
Imagine a chatbot asked about a cake called Lemon Velvet. Suppose its training text mentions this cake's oven temperature exactly once: 180 °C. Thousands of other cakes bake at 170, 175 or 190. With one mention against all those plausible neighbors, the model cannot tell the one true number apart from the believable ones, and it may confidently say 175.
Theory backs this up. A 2023 paper proved that a language model that is a good predictor (in a precise sense called calibrated) must hallucinate on "arbitrary" facts like this one, at a rate close to the share of such facts that appear exactly once in its training text, even if that text has no errors at all (Calibrated Language Models Must Hallucinate). The same paper notes that facts seen many times give no such statistical reason to hallucinate, and that fixing the rare ones may need training beyond prediction.
That also explains why models struggle with recent events, niche topics and personal questions like "What did I eat for lunch yesterday?" The training text simply holds too little about them, so the model falls back on patterns from similar situations, and those may not match reality.
Do small errors snowball?
During training, the model always predicts the next token from real text. When it writes, it builds on its own words instead, so a wrong detail early in an answer stays there for every later word to build on. Researchers disagree about how much this matters in general. One study found that models often recover from small slips instead of drifting further (Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?), while another argues that errors do accumulate and hurt quality (Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation).
What is well documented is a narrower effect. Once a model has committed to a wrong answer, it may go on to justify it with more false claims, even claims it can recognize as false when asked about them separately. In one study, ChatGPT and GPT-4 could identify 67% and 87% of their own mistakes when shown them on their own, yet still made them while defending an earlier answer (How Language Model Hallucinations Can Snowball). Newer reasoning models partly work around this by thinking before they answer, so they can catch a mistake before committing to it.
Why doesn't later training fix it?
As What an LLM Actually Is (and Why Predicting Words Works) explained, chat models get more training after pretraining, on example answers and on feedback about which answers are better. That training can reward correct answers, and it can reward saying "I don't know". It helps, but it does not fix the problem.
A 2025 paper argues that the way models are tested is a big part of the reason (Why Language Models Hallucinate). Most benchmarks score an answer as simply right or wrong. "I don't know" earns nothing, the same as a wrong answer, while a guess sometimes earns full marks. So models end up "optimized to be good test-takers", and on a test like that, guessing when unsure improves the score. Like a student facing a multiple choice exam with no penalty for wrong answers, the model learns to always fill in something. The authors argue that the fix is to change how the main benchmarks are scored, so that admitting uncertainty stops being punished.
Reasoning models do not escape this either. OpenAI's own system card for its o3 model reported a hallucination rate of 33% on a test of questions about people, against 16% for the older o1, because o3 "tends to make more claims overall", right and wrong ones alike (OpenAI o3 and o4-mini System Card).
Why does the model sound so confident even when it's wrong?
Confidence is a matter of style, and style is something the model learned. Answers in its training text are usually written confidently, and later training mostly rewards answers that read as complete and helpful. A hedged "I think it might be X" rarely scores better than a firm "It is X". So the wording of an answer tells you very little about whether it is right.
You can see this when you ask a chatbot to summarize a long article that was cut to fit its context window. The model never sees the missing part. Instead of saying so, it may fill the gap with plausible details that fit the parts it did see. It is not trying to deceive you. It is producing the most likely continuation, and the most likely continuation is a complete, fluent summary.
Researchers have found ways to measure the doubt hiding under the confident wording. One method, semantic entropy, asks the model the same question several times and groups the answers by meaning. If the answers mean different things, the model is unsure, and that turns out to predict wrong answers better than earlier methods (Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation). Another, called DoLa, compares what the model's later and earlier layers predict while it writes, and nudges it toward the facts it has stored; it raised scores on TruthfulQA by 12 to 17 points for one family of models without any retraining (DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models).
Is hallucination a bug or a feature?
It is neither. It comes from the same mechanism that makes the model work. The model produces text that is a plausible continuation of what came before. When plausible lines up with true, you get a correct answer. When it does not, you get a hallucination. The model is not switching modes. It is the same engine, working where its training gave it strong patterns or weak ones.
Think back to Part 1. The model's core skill is predicting the next token, learned in pretraining and then shaped by further training into a helpful assistant. Fluency, creativity and factual recall all come from that core, and so does the occasional confident invention. You cannot keep the first three and simply delete the fourth.
That changes the engineering question. You don't "fix" hallucination like a single bug. You design around it: retrieval to ground answers in real documents, uncertainty estimates to flag risky answers, tools such as web search for anything recent, and training and tests that reward admitting uncertainty.
| Cause | What it means | Why it leads to hallucinations |
|---|---|---|
| Pretraining on text | Learns to make the text it saw likely | No separate truth signal; popular myths can get high probability |
| Rare facts | A fact seen only once in the training text | The model cannot tell the true version from plausible alternatives |
| Building on its own words | Each token follows the ones it already wrote | An early wrong claim can be defended with more wrong claims |
| Test incentives | Benchmarks score "I don't know" the same as a wrong answer | Guessing pays, so models learn to guess |
| Missing information | The answer is not in the training text or the prompt | The model fills the gap with something plausible |
Frequently Asked Questions
Q: Can better prompts eliminate hallucinations? Good prompts reduce some of them by making the task clearer, giving the model the facts it needs, and telling it that "I don't know" is an acceptable answer. But they cannot add knowledge the model lacks. Prompting is a mitigation, not a cure.
Q: Why do models hallucinate more on long outputs? A long answer makes more specific claims, and each one is a chance to be wrong. Every token also builds on the ones before it, so an early wrong claim can pull later ones along with it.
Q: Does retrieval augmented generation (RAG) stop hallucinations? RAG puts relevant documents into the prompt, which greatly reduces hallucinations caused by outdated or missing knowledge. But the model can still ignore the retrieved text or add details that are not in it. RAG reduces the problem; it doesn't eliminate it.
Q: Can a model ever "know" it's hallucinating? In a sense. Methods such as semantic entropy can detect when a model is unsure from the spread of its answers, which shows the uncertainty is there to be found. But a model does not act on that signal unless it is trained or set up to do so.
Q: Will future models stop hallucinating? Rates on common facts keep falling as models, training and tools improve. But rare facts set a statistical floor for prediction alone, and as long as tests reward guessing, models will keep learning to guess. The realistic goal is to make hallucinations rarer and easier to catch, not to eliminate them.
Q: What can I do as a user? Ask for sources, and open them to check they exist and say what the answer claims. Turn on web search for anything recent. Tell the model it is fine to say it doesn't know. Be extra careful with names, numbers, dates, quotes and citations, which are the details models most often invent. And for anything that matters, check the answer somewhere else.
Test yourself
You're building a customer support chatbot that answers questions about your company's internal policies. The model sometimes invents plausible but wrong policy details when a question is slightly ambiguous. You've already improved the prompts. What's one change to the system that addresses the root cause of these hallucinations, and why would it help?
Answer: Add a retrieval step that finds the exact policy passages relevant to the question and puts them into the prompt along with it (RAG). The inventions most likely happen because your internal policies are not in the model's training text, so when a question is ambiguous it falls back on generic patterns of what policies usually say. With the real policy text in its context window, the most likely continuation becomes a faithful summary or quote of that text, not an invention. It does not change the model's tendency to guess, but it changes the input so that guessing isn't needed. Two extras help: tell the model to say when the passages do not answer the question, and show the user which passage each answer came from so they can check it.
Next: Open vs Closed Models: What 'Open Source AI' Really Means. It explains the difference between models you can download and models you can only call, and what "open" really promises.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (Lin et al. 2021)
- Calibrated Language Models Must Hallucinate (Kalai and Vempala 2023)
- Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (He et al. 2019)
- Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation (Arora et al. 2022)
- How Language Model Hallucinations Can Snowball (Zhang et al. 2023)
- Why Language Models Hallucinate (Kalai et al. 2025)
- OpenAI o3 and o4-mini System Card (OpenAI, April 2025), Table 4
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (Kuhn et al. 2023)
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (Chuang et al. 2023)
