I recently read “Why Language Models Hallucinate” by Kalai et al., a paper from researchers at OpenAI and Georgia Tech.
Hallucinations have often been defined as generations that contain one or more plausible falsehoods. The authors offer a more technically descriptive definition: generations that are not grounded in the training data or prompt.
There is already a substantial body of research on why hallucinations happen. Prior work points to overconfidence, randomness in sampling, early errors that snowball, information that appears rarely in training data, sycophancy, spurious correlations, exposure bias, the reversal curse, and irrelevant context that pulls the model away from the main subject. There is also the old GIGO rule: errors and partial errors in the training data can be reproduced in the output.
This paper narrows the focus to reasons tied to pre-training and post-training. The authors advance two main points.
- Hallucinations are a natural part of the pre-training process.
- Leaderboards tend to reward models that guess rather than acknowledge uncertainty.
Hallucinations are a natural part of pre-training
During pre-training, a base language model learns to approximate the distribution of text in its training data. The authors describe this as a density-estimation problem: the model is learning a probability distribution over text. Their argument is that hallucinations are not mysterious. They arise naturally from the statistical process of learning a language distribution.
That does not mean every hallucination is inevitable or that mitigation does not work. It means we should not expect hallucinations to go to zero in a broadly useful model that generalizes beyond its training data. The alternative would be a model that either says “I don’t know” all the time or memorizes information instead of generalizing it.
The authors use arbitrary facts, such as birthdays, as one example. If there is no learnable pattern in the data and a birthday fact appears only once in the pre-training data, the model may not have enough evidence to reproduce it reliably.
Leaderboards reward guessing behavior
The authors argue that hallucinations persist after post-training because current evaluation systems often reward guessing over acknowledging uncertainty. This is very similar to human test-taking pressure.
If a test gives zero points for a blank answer and zero points for a wrong answer, the rational strategy is to guess. Leaving the answer blank guarantees no credit. Guessing creates at least some chance of receiving credit.
Language models face a similar incentive structure. Many benchmarks use binary grading: a correct answer receives credit, while an incorrect answer and an “I don’t know” response both receive zero. Under that scoring system, a model that guesses when uncertain can outperform one that accurately signals uncertainty.
That matters because leaderboards shape development priorities. If influential benchmarks reward confident guessing, competitive pressure works against models that are more careful, calibrated, and willing to say when they do not know.
The researchers propose changing the scoring incentives: give credit for correct answers, penalize incorrect ones, and allow an “I don’t know” response to receive zero. That would reward models for knowing when not to answer.
I also found it useful that “I don’t know” is not the only way to signal uncertainty. A model can ask a clarifying question, hedge appropriately, or omit details it cannot support. The goal is not to make models less useful or overly cautious. It is to stop rewarding them for sounding certain when they are not.
The problem is not simply that models hallucinate. It is that many of the systems used to compare them reward hallucination-like guessing.
If we want more accurate models, we need evaluation systems that reward accuracy, calibrated uncertainty, and appropriate restraint—not just confident answers.