Source-linked AI summary
How Language Model Hallucinations Can Snowball
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, Noah A. Smith
TL;DR
Language models can hallucinate while sounding plausible, and such errors are often attributed to knowledge gaps. This paper studies cases where models commit to an incorrect answer, produce incorrect explanations, and can nevertheless recognize those explanations as wrong in isolation.
Problem
Language models frequently hallucinate plausible but incorrect facts or nonexistent sources, raising whether hallucinations always reflect knowledge gaps.
Method
The paper automatically constructs three zero-shot question-answering datasets spanning prime numbers, U.S. senators, and flight connectivity, then examines answer explanations and isolated-claim recognition.
Results
Over 95% of the time, ChatGPT and GPT-4 commit to a Yes/No answer within the first token, which is often followed by an incorrect explanation that the model can recognize as incorrect in isolation.
Takeaways & Limitations
Hallucination snowballing describes over-commitment to earlier hallucinations, producing simple mistakes that would not otherwise occur and motivating methods that help models acknowledge errors and backtrack.
Takeaways & Limitations
The experiments use only two proprietary models and focus on English question answering rather than other tasks such as summarization or code generation.
Abstract
from arXiv · showhide
A major risk of using language models in practical applications is their tendency to hallucinate incorrect statements. Hallucinations are often attributed to knowledge gaps in LMs, but we hypothesize that in some cases, when justifying previously generated hallucinations, LMs output false claims that they can separately recognize as incorrect. We construct three question-answering datasets where ChatGPT and GPT-4 often state an incorrect answer and offer an explanation with at least one incorrect claim. Crucially, we find that ChatGPT and GPT-4 can identify 67% and 87% of their own mistakes, respectively. We refer to this phenomenon as hallucination snowballing: an LM over-commits to early mistakes, leading to more mistakes that it otherwise would not make.
1 Introduction
The paper introduces hallucination snowballing: language models may produce incorrect explanations to preserve an earlier wrong answer, even when they can recognize those claims as false in isolation. Experiments and prompting analyses show that faulty context can induce simple mistakes that models would otherwise avoid.
- Language models hallucinate plausible false facts or nonexistent sources, and these errors are commonly attributed to knowledge gaps.
- Hallucination snowballing describes incorrect explanations generated after an earlier wrong answer, despite the model separately recognizing the explanation as incorrect.
- The authors construct three QA datasets spanning primality, U.S. senator constraints, and flight-based graph connectivity.
- 95% of incorrect cases still contain snowballed hallucinations under “Let’s think step by step” prompting.
- The paper argues that models should acknowledge initial mistakes and revise answers, motivating methods that enable backtracking.
2 Why do we expect hallucination snowballing?
The authors expect hallucination snowballing when questions force models to commit to an answer before explaining it and require inherently sequential reasoning. Early commitment then leaves the answer in context while later tokens generate its explanation.
- Snowballing is predicted for questions requiring initial committal and inherently sequential reasoning.
- Initial committal: GPT-4 and ChatGPT begin with Yes or No 95.67% and 98.40% of the time, respectively, on the authors’ datasets.
- Initial committal: After committing to a one-timestep answer, the model continues generating an explanation for that answer.
- Inherently sequential: Transformers cannot solve primality testing or graph connectivity within one timestep because these problems require multiple reasoning steps.
- Inherently sequential: The authors hypothesize that an incorrect early commitment leads the model to generate supporting incorrect facts afterward.
3 Experiments
The paper evaluates hallucination snowballing using three structured QA datasets and a two-stage procedure that tests both answer accuracy and recognition of incorrect explanations. ChatGPT and GPT-4 frequently answer incorrectly, yet can often identify isolated false claims in their own explanations.
- Evaluation: The evaluation first measures answer accuracy, then tests whether the model identifies an incorrect explanatory step in a separate verification interaction.Incorrect claims are extracted from explanations and checked against the same model; extraction procedures differ by task.
- Datasets: Three 500-question yes/no datasets probe primality testing, U.S. senator search, and graph connectivity with verifiable incorrect justifications.The tasks are designed to require more than one transformer timestep, while incorrect answers imply specific claims such as false factorizations, senator affiliations, or nonexistent flights.
- Results: 39.87% average accuracy for ChatGPT versus 16.6% for GPT-4, with all but ChatGPT on Senator Search below 50% accuracy.GPT-4 performs worse than ChatGPT across all three datasets despite its broader benchmark reputation.
- Results: 67.37% of ChatGPT’s and 87.03% of GPT-4’s incorrect claims are detected when presented alone.The paper treats these detectable false claims as snowballed hallucinations and reports that both models remain highly susceptible to the phenomenon.
4 Can we prevent snowball hallucinations?
The paper examines prompting and decoding or training strategies for reducing hallucination snowballing. Reasoning-before-answer prompts improve accuracy, but snowballed hallucinations remain common when the model makes mistakes, while several mitigation strategies are proposed or constrained by available APIs.
- Engineering Better Prompts: Reasoning-before-answer prompts substantially improve task accuracy, reaching perfect Senator Search performance, ≤10% Primality Testing error, and ≤30% Graph Connectivity error.The prompt appends “Let’s think step-by-step” to the original question.
- Engineering Better Prompts: 95% of incorrect-answer cases still contain snowballed hallucinations under “Let’s think step by step.”The model may hallucinate during its reasoning chain, causing later snowballed hallucinations.
- Engineering Better Prompts: 94.90% is GPT-4’s average snowballed hallucination rate across tasks under the reasoning prompt, with ChatGPT similarly high.An example shows an incorrect claim about available flights inducing a later false claim that the model can verify separately.
- Decoding and Training Methods: At temperatures 0.6 and 0.9, both error rates and snowballed hallucination rates remain similarly high for GPT-4 and ChatGPT.The experiments used greedy decoding, equivalent to temperature t = 0.
- Decoding and Training Methods: Top-k and nucleus sampling are not expected to help because narrowing candidate tokens can increase immediate commitment to an answer.Beam search could retain alternative sequences in principle, but its effect could not be tested because the OpenAI API lacks support.
- Decoding and Training Methods: Pretraining, instruction tuning, and backtracking data are proposed as broader ways to reduce snowballing by delaying commitment or encouraging correction.The paper suggests reasoning chains before answers and training examples that explicitly revise incorrect solutions.
5 Related Work
The paper situates hallucination snowballing within research on hallucinations, exposure bias, misleading context, and language-model inconsistency. Its distinctive contribution is showing that propagated errors can be recognized by the model itself, even from ordinary information-seeking questions.
- Hallucinations: Hallucinations are often linked to knowledge gaps, motivating retrieval-based mitigation strategies.Prior work studies hallucinations in text generation and plausible-sounding falsehoods.
- Hallucinations: Exposure bias provides a proposed explanation: models train on gold history but infer from potentially erroneous previous predictions.The paper extends prior work on compounding errors by showing that the model itself recognizes propagated errors.
- Misleading Context: Unlike misleading-question studies, this work finds the failure mode on innocent information-seeking queries that are not intentionally deceptive.The shared mechanism is that faulty context misguides the language model.
- LM (in)consistency: The findings add to evidence that language models can produce different answers across prompts, including after generating intermediate reasoning steps.Other prompt changes can improve performance despite adding no substantive information.
6 Conclusion
The paper defines hallucination snowballing and demonstrates that it is prevalent in state-of-the-art model generations, producing simple factual errors that would not otherwise occur. It warns that indiscriminate prioritization of fluency and coherence can come at the expense of factuality.
- Conclusion: Hallucination snowballing is demonstrated as prevalent in generations from state-of-the-art models, producing simple factual errors that would not otherwise occur.The paper encourages remedial work across model-development stages.
- Conclusion: The findings point to a risk that models prioritizing fluency and coherence indiscriminatively may sacrifice factuality.The paper calls for future study of remedial actions at all levels of model development.
Limitations
The study is limited to English question answering, two proprietary models, and API-constrained experiments. These constraints prevent evaluation on other tasks, access to output distributions, finetuning, and several mitigation strategies.
- Scope: The study focuses on English question answering and does not examine tasks such as summarization or code generation.Its conclusions therefore directly cover only the studied question-answering setting.
- Models and Access: Experiments use only ChatGPT and GPT-4, limiting evidence about other language models.The models were selected because of their state-of-the-art benchmark performance.
- Models and Access: API limitations block access to output probabilities and finetuning, restricting exploration of beam search and alternative training strategies.The paper notes that these capabilities would support studying sampling and instruction-tuning mitigations.
A.1 Graph Connectivity
The dataset represents flight information as directed graphs with a shared connection pattern and randomly assigned node names.
- Flight information is represented as a directed graph whose node names are randomly chosen from the 26 English letters.
- All graphs share a specific connection pattern, illustrated through the underlying graph structure in Figure 5.
A.2 Senator search
The dataset uses twelve colleges selected from top U.S. universities after excluding schools on a specified congressional-members list.
- The dataset contains twelve colleges selected from a list of top universities in the U.S.
- Universities appearing on The U.S. News & World Report’s Top 10 Colleges for Members of Congress list were excluded.
B Additional Results
Additional results provide detailed accuracy and hallucination-detection breakdowns, alongside examples and temperature- and prompting-based analyses. The Graph Connectivity examples reuse one graph structure while varying node labels and queried cities.
- Tables 6 and 7 report detailed question-answering accuracy and hallucination-detection accuracy breakdowns.
- Graph Connectivity examples use the same underlying graph structure while varying source and target cities and node labels.
- Tables 3–5 present GPT-4 verification examples for Graph Connectivity, Primality Testing, and Senator Alma Mater.
- Tables 8 and 9 report error rates and snowballed hallucinations under the “Let’s think step by step” prompt.
- Tables 10 and 11 report error rates under different temperature setups, while Table 12 gives a GPT-4 Senator Search verification failure example.