Source-linked AI summary
A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation
Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, Dong Yu
TL;DR
Large language models can produce fluent but factually incorrect or unfaithful text, limiting their reliability. The paper actively detects uncertain concepts, validates them with retrieved knowledge, repairs detected hallucinations during generation, and reports reduced hallucinations across GPT-3.5 experiments and broader studies.
Problem
Hallucinations are fluent, natural-looking outputs that can be factually incorrect, nonsensical, or unfaithful, undermining reliability and contributing to misinformation and privacy risks.
Method
The approach identifies potentially hallucinated concepts using model uncertainty, validates them with retrieved knowledge, repairs detected hallucinated sentences, and continues generation with the repairs.
Results
Active detection and mitigation reduced GPT-3.5 hallucinations from 47.5% to 14.5% on average, while detection achieved ∼88% recall and mitigation handled 57.6% of correctly detected hallucinations.
Takeaways & Limitations
The approach improves the reliability and trustworthiness of large language models and was evaluated with Vicuna and on multi-hop and false-premise questions.
Takeaways & Limitations
The approach increases inference cost, and validation depends on retrieved knowledge and the model’s ability to use it.
Abstract
from arXiv · showhide
Recently developed large language models have achieved remarkable success in generating fluent and coherent text. However, these models often tend to 'hallucinate' which critically hampers their reliability. In this work, we address this crucial problem and propose an approach that actively detects and mitigates hallucinations during the generation process. Specifically, we first identify the candidates of potential hallucination leveraging the model's logit output values, check their correctness through a validation procedure, mitigate the detected hallucinations, and then continue with the generation process. Through extensive experiments with GPT-3.5 (text-davinci-003) on the 'article generation task', we first demonstrate the individual efficacy of our detection and mitigation techniques. Specifically, the detection technique achieves a recall of ~88% and the mitigation technique successfully mitigates 57.6% of the correctly detected hallucinations. Importantly, our mitigation technique does not introduce new hallucinations even in the case of incorrectly detected hallucinations, i.e., false positives. Then, we show that the proposed active detection and mitigation approach successfully reduces the hallucinations of the GPT-3.5 model from 47.5% to 14.5% on average. We further demonstrate the effectiveness and wide applicability of our approach through additional studies including performance on different types of questions (multi-hop and false premise questions) and with another LLM from a different model family (Vicuna). In summary, our work contributes to improving the reliability and trustworthiness of large language models, a crucial step en route to enabling their widespread adoption in real-world applications.
1 Introduction
Large language models generate fluent text but can hallucinate, undermining reliability and adoption. The paper proposes actively detecting and mitigating hallucinations during generation through uncertainty-based validation and sentence repair.
- Hallucinations are fluent, natural-looking outputs that are factually incorrect, nonsensical, or unfaithful to the source, creating risks such as misinformation and privacy violations.
- The paper actively detects and mitigates hallucinations during generation because hallucinated sentences increase the likelihood of later hallucinations.
- Detection: The detection stage identifies key concepts, estimates uncertainty from model logits, and validates uncertain concepts using retrieved knowledge.
- Mitigation: The mitigation stage repairs potentially hallucinated sentences using retrieved knowledge as evidence, then continues generation with the repaired sentence appended to the context.
- Results: ∼88% recall and 57.6% mitigation of correctly detected hallucinations demonstrate separate efficacy of the detection and mitigation techniques.
- Results: 47.5% to 14.5%: active detection and mitigation reduced GPT-3.5 hallucinations on average in the article generation task.
2 Approach
The approach actively detects and mitigates hallucinations during generation by identifying potentially unreliable concepts, validating them with retrieved knowledge, repairing hallucinated sentences, and continuing generation. It combines model uncertainty signals with validation and evidence-based mitigation, while acknowledging validation and inference-cost constraints.
- Overview: The active approach iteratively detects hallucinations and mitigates them during generation to help prevent propagation into subsequent sentences.A hallucinated sentence increases the likelihood of later hallucinations, motivating intervention before continuing generation.
- Hallucination Detection: Detection identifies important concepts, estimates uncertainty from token-level logit probabilities, validates uncertain concepts, and verifies their information using retrieved knowledge.Models without logit outputs can pass all or heuristically selected concepts directly to validation.
- Hallucination Mitigation: Mitigation repairs potentially hallucinated sentences by removing or substituting unsupported information using retrieved knowledge as evidence, and may use that knowledge as context for the next sentence.The approach also studies simpler instructional techniques for achieving its detection and mitigation objectives.
- Calculate Model’s Uncertainty: Minimum token probability is preferred because one low-probability token can provide strong evidence of uncertainty that averaging may obscure.For multi-token concepts, the minimum score preserves uncertainty concentrated in a single token.
- Validate Concepts: Validation creates Yes/No questions about selected concepts and checks their answers against retrieved knowledge, with direct model instruction preferred for question generation.The validation result depends on both the retrieved knowledge and the model’s ability to use it when answering.
- Design Considerations: The approach improves reliability but increases inference cost, although parallel validation and smaller models may make the process more efficient.The authors frame this cost as a trade-off for reducing hallucinations and improving trustworthiness.
3 Experiments and Results
Experiments evaluate hallucination detection and mitigation on GPT-3.5 article generation across diverse topics. The active approach uses uncertainty signals and validation to detect and repair hallucinations, reducing their occurrence while examining propagation across sequentially generated sentences.
- Experimental Setup: 150 topics from diverse domains were used to prompt GPT-3.5 to generate articles, with the first five sentences manually annotated for correctness.The topic set spans domains such as sports, politics, music, literature, film, and history.
- Motivating Findings: Hallucinations in earlier sentences are associated with more hallucinations in subsequent sentences, with the gap increasing for later sentence positions.The reported comparisons are A > B and A > C, while D exceeds C and B when prior sentences contain no hallucination.
- Motivating Findings: Higher probability scores correspond to lower hallucination tendency at both sentence and concept levels, making low-probability concepts a detection signal.Sentence scores use the minimum probability across tokens of identified concepts, while concept scores use the minimum across the concept’s tokens.
- Hallucination Detection: Using probabilities over identified concept tokens produces a stronger sentence-level precision-recall signal than using all sentence tokens.The concept-token method achieves a higher area under the precision-recall curve than the all-token method.
- Hallucination Detection: The Minimum probability calculation technique achieves the highest area under the curve for concept-level hallucination detection.It is compared with Average and Normalized techniques.
- Active Detection and Mitigation: The active approach reduces GPT-3.5 hallucinations from 47.4% to 14.53% by detecting uncertain concepts, validating them, repairing flagged sentences, and continuing generation.The approach reduces hallucinations across all categories shown in Figure 8.
4 Additional Experiments
Additional studies show that the approach generalizes beyond article generation to Vicuna, multi-hop questions, and false-premise questions, while supporting question rectification and other applications.
- Vicuna-13B: The approach considerably reduces hallucinations for Vicuna-13B on an article-generation study using 50 generated sentences.The study sampled 10 randomly selected topics.
- Multi-hop Questions: For multi-hop questions, the method applies detection and mitigation to answers generated step by step.Each step uses low-probability concepts, web-search validation, and mitigation before the next step is generated.
- False Premise Questions: False-premise questions form a challenging evaluation setting because state-of-the-art models often struggle to respond appropriately.The evaluation used 50 adversarial questions on which GPT-3.5 initially generated incorrect responses, paired with true-premise questions.
- False Premise Questions: With retrieved knowledge as context, GPT-3.5 answered 24% of false-premise questions correctly, whereas the approach answered 76% correctly.The false-premise evaluation set was adversarially collected, with GPT-3.5 initially incorrect on every question.
- False Premise Questions: 76% of false-premise questions were successfully rectified, while no true-premise question was incorrectly modified.The approach evaluates complete-answer correctness and preserves the semantics of six modified true-premise questions.
- Other Applications: The approach may also improve abstractive summarization and claim verification, but these use cases are left for future work.For summarization, validation knowledge would come from the original document rather than the web.
5 Related Work
Related work studies both the causes of language-model hallucinations and methods for detecting or addressing them, including sampling divergence, internal-state classification, and factuality-oriented sampling.
- Causes of Hallucination: Prior research attributes hallucinations to training-data quality, source-target divergence, ill-suited modeling, and inference randomness.
- Addressing Hallucinations: One line of work addresses hallucinations through sampling-based detection, where inconsistent sampled responses signal potentially hallucinated facts.The approach contrasts consistent samples for well-known concepts with divergent or contradictory samples for hallucinated facts.
- Addressing Hallucinations: Another method trains a separate classifier on LLM activation values to predict statement truthfulness.
- Addressing Hallucinations: Factuality-oriented sampling has been proposed based on the hypothesis that sampling randomness is more harmful later than earlier in a sentence.
6 Conclusion
The paper presents active hallucination detection and mitigation during generation. Experiments report reduced GPT-3.5 hallucinations, effective individual components, applicability to other models and question types, and no new hallucinations from false-positive mitigation.
- The proposed approach actively detects and mitigates hallucinations of large language models during generation.
- GPT-3.5 hallucinations fell from 47.5% to 14.5% on average on the article-generation task.
- The detection technique achieved high recall, while mitigation addressed a large fraction of correctly detected hallucinations without introducing new hallucinations for false positives.
- The approach was also evaluated with Vicuna and on multi-hop and false-premise questions.
- The authors frame the work as contributing to improved reliability and trustworthiness of language models.
A.1 Identify Key Concepts
The study compares three methods for identifying key concepts in generated sentences and finds that instructing the model best captures important concepts.
- The instruction technique identifies most important concepts, whereas entity extraction misses many and keyword extraction includes many insignificant concepts.
A.2 Create Validation Question
Validation questions are created for each concept, with Yes/No and Wh formats represented. Yes/No questions are preferred because their answers are easier to check.
- Table 8 presents validation questions corresponding to each identified concept.
- The examples include both Yes/No and Wh-question formats.
- Yes/No questions are preferred because checking their answers is relatively easier.
B Evaluation Data
The evaluation data include generated-sentence statistics and annotations at both sentence and concept levels. GPT-3.5 outputs average about 18 words per sentence and 3.2 identified key concepts per sentence.
- GPT-3.5 generated sentences average approximately 18 words.
- Each generated sentence contains approximately 3.2 key concepts identified by the instruction technique.
- The evaluation includes sentence-level and concept-level hallucination annotations.
C Recall of Hallucination Detection vs Probability Threshold
Recall is compared across hallucination-detection techniques at different probability thresholds. Web search considerably outperforms self-inquiry at every threshold shown.
- Web search considerably outperforms self-inquiry in hallucination-detection recall at all probability thresholds.
- Figure 12 compares recall for self-inquiry and web-search techniques across different probability thresholds.
- The surrounding evaluation materials include instructional prompts, concept-identification examples, and mitigation failure cases.
E Multi-hop QA Experiment
The multi-hop QA experiment uses bridge questions from HotpotQA and prompts with in-context examples for step-by-step answering. Responses generated with the proposed approach are also presented.
- The experiment includes multi-hop bridge questions from the HotpotQA dataset.
- The model is prompted to answer multi-hop questions step by step using in-context examples.
- Examples of responses generated using the proposed approach are provided for the multi-hop bridge questions.
F False Premise QA Experiment
The false-premise QA experiment compares GPT-3.5 responses with responses using retrieved knowledge and the proposed approach. Supporting materials include false-premise question pairs, rectification prompts, and example outputs.
- False Premise QA Experiment: Table 16 compares GPT-3.5 responses, retrieved-knowledge responses, and responses produced by the proposed approach on false-premise questions.The comparison covers a few false-premise questions.
- False Premise QA Experiment: Validation-question examples correspond to keyphrases identified by the Instructing the Model technique.
- False Premise QA Experiment: Sentence- and concept-level annotations distinguish correct concepts from hallucinated concepts in generated articles.Correct concepts are represented in green, while hallucinated concepts are represented in red.
- False Premise QA Experiment: Additional materials cover multihop questions, step-by-step prompting for bridge questions, and examples of successful or failed hallucination mitigation.
- False Premise QA Experiment: False-premise questions are paired with corresponding true-premise questions in the experiment materials.
- False Premise QA Experiment: Instructional prompts are provided for rectifying false-premise questions.