Source-linked AI summary
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Chen Xu, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, Shuming Shi
TL;DR
Hallucination remains a critical challenge to the practical application and reliability of LLMs. This survey reviews recent detection, explanation, evaluation, and mitigation efforts, then discusses open challenges and future directions. It presents taxonomies, benchmarks, analyses of mitigation approaches, and outlooks for improving models’ ability to gauge their capability boundaries.
Problem
LLMs can confidently fabricate answers or generate content conflicting with context and established knowledge, making hallucination a critical challenge for practical application.
Method
The survey synthesizes recent research on hallucination definitions, benchmarks, metrics, sources, mitigation methods, challenges, and future directions.
Results
The survey provides taxonomies of hallucination phenomena and evaluation benchmarks while reviewing approaches to detect, explain, and mitigate hallucinations.
Takeaways & Limitations
Future progress requires addressing challenges in uncertainty estimation, retrieved-knowledge verification, and models’ calibration of their capability boundaries.
Takeaways & Limitations
Uncertainty-based hallucination mitigation remains constrained by inaccessible logits, verbalized overconfidence, and unresolved response-consistency measurement.
Abstract
from arXiv · showhide
While large language models (LLMs) have demonstrated remarkable capabilities across a range of downstream tasks, a significant concern revolves around their propensity to exhibit hallucinations: LLMs occasionally generate content that diverges from the user input, contradicts previously generated context, or misaligns with established world knowledge. This phenomenon poses a substantial challenge to the reliability of LLMs in real-world scenarios. In this paper, we survey recent efforts on the detection, explanation, and mitigation of hallucination, with an emphasis on the unique challenges posed by LLMs. We present taxonomies of the LLM hallucination phenomena and evaluation benchmarks, analyze existing approaches aiming at mitigating LLM hallucination, and discuss potential directions for future research.
1 Introduction
LLMs’ capabilities are accompanied by hallucinations that can diverge from input, prior context, or factual knowledge, undermining reliability in real-world use. This survey organizes recent work on detecting, explaining, and mitigating these failures.
- Hallucinations can produce plausible outputs that deviate from user input, contradict prior context, or conflict with factual knowledge.Such errors can include fabricated medical diagnoses or treatment plans with tangible risks.
- LLM hallucinations are difficult to address because web-scale training data may contain fabricated, outdated, or biased information.
- General-purpose LLMs create evaluation and mitigation challenges because they are expected to operate across tasks, languages, and domains.
- Hallucinated information can appear highly plausible, making errors difficult for both models and humans to detect.
- The survey reviews benchmarks, hallucination sources, mitigation methods, and future directions, based on recent work selected for insight, experiments, or evaluation reliability.
2 Hallucination in the Era of LLM
The paper defines LLM hallucination through three categories—input-, context-, and fact-conflicting outputs—while distinguishing hallucinations from related problems such as ambiguity. It emphasizes that LLM versatility and plausible errors complicate evaluation and detection.
- The survey categorizes hallucinations as input-conflicting, context-conflicting, or fact-conflicting outputs.The categories cover deviations from user input, contradictions with prior generated information, and inconsistency with or unverifiability against world knowledge.
- Input-conflicting hallucinations occur when generated content deviates from task instructions or task input, such as changing Hill to Lucas in a summary.
- Context-conflicting hallucinations arise when a model loses consistency across a response, such as referring first to Silver and later to Stern as NBA commissioner.
- Fact-conflicting hallucinations contradict established world knowledge or are treated as unverifiable, including the incorrect identification of Afonso II’s mother.The paper acknowledges that some unverifiable content may still be reasonable and not harmful.
- LLM versatility makes hallucination evaluation and mitigation difficult across multi-task, multilingual, and multi-domain use, while strong writing ability makes false information harder to detect.
- Ambiguity is distinct from hallucination: an ambiguous answer may be non-factual or incomplete in usefulness without directly contradicting established facts.
3 Evaluation of LLM Hallucination
LLM hallucination evaluation uses benchmarks spanning generation and discrimination formats, diverse task settings, and human or automatic metrics. Existing approaches face trade-offs between reliability, consistency, and evaluation cost.
- Evaluation formats: Existing benchmarks evaluate hallucination either by generating factual text or discriminating factual from non-factual statements.Generation treats hallucination as a generation-quality characteristic, whereas discrimination tests whether models identify factuality.
- Task formats: Benchmarks cover question answering, task instructions, and text completion through varied prompting formats.Representative datasets include TruthfulQA, HaluEval, FActScore, and FACTOR.
- Benchmark construction: Benchmark construction combines human validation with automatic pipelines to produce fluent, nonfactual, and diverse test examples.Some datasets manually verify generated items or atomic factual claims, while others use model-based sampling and filtering.
- Evaluation metrics: Human evaluation is reliable but labor-intensive and can be inconsistent because of annotator subjectivity.Guidelines may require qualitative labels, source verification, or fine-grained factual annotation.
- Evaluation metrics: Automatic evaluation improves efficiency but may diverge from human judgments across texts and evaluation settings.Model-based judges and rule-based metrics provide proxies rather than universally reliable substitutes for human assessment.
4 Sources of LLM Hallucination
The survey identifies hallucination sources across multiple stages of the LLM life cycle, including knowledge acquisition, alignment, and generation. These sources include false or missing knowledge, problematic instruction tuning, and sampling randomness.
- Pretraining: Missing or false knowledge in pretraining can cause LLMs to hallucinate when answering questions or completing tasks.Models may internalize misinformation or misinterpret spurious correlations as factual knowledge.
- Knowledge boundaries: Very large LLMs can be equally confident in correct and incorrect answers, producing over-confident hallucinations.Similar entropy distributions for correct and incorrect answers suggest unreliable confidence calibration.
- Alignment: Alignment can encourage hallucination when models are trained to follow instructions requiring knowledge they did not acquire during pretraining.Such training is characterized as misalignment because it rewards responses beyond the model’s prerequisites.
- Generation: Sampling-based decoding methods such as top-p and top-k increase hallucinations compared with greedy decoding on several benchmarks.The randomness introduced during sequential token generation is identified as a potential hallucination source.
5 Mitigation of LLM Hallucination
The survey reviews hallucination-mitigation studies according to when interventions occur in the LLM life cycle. It also summarizes popular LLM performance on major hallucination benchmarks.
- Mitigation framework: Mitigation approaches are categorized by their timing within the LLM life cycle.The review organizes recent work around when interventions are applied rather than treating mitigation as a single technique.
- Benchmark comparisons: The survey summarizes popular LLM performance on HaluEval, SimpleQA, and TruthfulQA.These benchmarks provide reference points for comparing hallucination-related performance across models.
5.1 Mitigation during Pre-training
Pretraining mitigation primarily targets the quality of training corpora because noisy or misleading data can corrupt parametric knowledge. At current corpus scales, practical systems rely mainly on automated filtering and selection, leaving room for more effective strategies.
- Rationale: Noisy pretraining data, including misinformation, can corrupt parametric knowledge and contribute to hallucinations.Factual knowledge acquired by language models can be traced back to their training data.
- Curation methods: Manual curation has reduced hallucinations in earlier data-to-text and table-to-text settings.Human annotators composed clean responses or refined existing dataset text.
- Scale constraint: Pretraining corpora now reach about two trillion tokens, making comprehensive manual curation increasingly difficult.Llama 2 is cited as an example of the scale that motivates automated approaches.
- Current practice: Current large-scale pretraining pipelines commonly use heuristic rules to select reliable data or filter noisy data.Examples include similarity to high-quality reference corpora and heuristic web-data extraction.
- Future direction: More effective data-selection and filtering strategies remain a proposed direction for pretraining mitigation.The survey identifies this as an open avenue beyond simple heuristics.
5.2 Mitigation during SFT
During SFT, carefully curated instruction-tuning data can improve factuality, but behavior cloning may also make models confidently answer beyond their knowledge boundaries. The survey therefore presents both data-curation strategies and honesty-oriented responses while noting important limitations.
- 5.2 Mitigation during SFT: SFT uses instruction-following data to elicit pretrained knowledge and teach models how to interact with users.The process fine-tunes pretrained foundational models on large instruction-following datasets using maximum likelihood estimation.
- 5.2.1 Mitigating hallucination introduced in pre-training during SFT: Carefully curated instruction-tuning datasets can improve factuality and truthfulness compared with less selective SFT data.Curation may be manual or automatic, and reported improvements include performance on TruthfulQA.
- 5.2.2 Mitigating hallucination introduced in SFT: SFT can introduce hallucinations when behavior cloning makes models respond positively to questions exceeding their knowledge boundaries.Models may confidently answer about unlearned knowledge instead of acknowledging incompetence.
- 5.2.2 Mitigating hallucination introduced in SFT: Honesty-oriented SFT adds samples such as “Sorry, I don’t know” to encourage models to admit incompetence.This approach targets hallucinations caused by answering questions beyond the model’s capacity.
- 5.2.2 Mitigating hallucination introduced in SFT: Honesty-oriented SFT has limited OOD generalization, and annotators cannot reliably represent models’ actual knowledge boundaries.The survey therefore characterizes solving this issue during SFT as sub-optimal.
5.3 Mitigation during RLHF
RLHF mitigates hallucination by optimizing models toward human preferences, including honesty, through reward modeling and reinforcement learning. The survey describes specialized honesty rewards and reports improved truthfulness, while cautioning that RL can produce over-conservative behavior.
- 5.3 Mitigation during RLHF: RLHF trains a reward model for human preferences and uses reinforcement learning to optimize the SFT model.The process commonly uses PPO or related RL algorithms to incorporate reward-model feedback.
- 5.3 Mitigation during RLHF: GPT4’s synthetic hallucination data for reward modeling increased TruthfulQA accuracy from about 30% to 60%.Process supervision also provides feedback for each intermediate reasoning step in reasoning tasks.
- 5.3 Mitigation during RLHF: Honesty-oriented RL rewards correct answers, uncertainty, premise-challenging, and safe incapability responses.The design distinguishes correct or wrong answers delivered with unhedged or hedged tones from uninformative responses such as “I don’t know”.
- 5.3 Mitigation during RLHF: Honesty-oriented RL can explore models’ knowledge boundaries and reduce reliance on human annotations that guess those boundaries.The survey presents these as advantages over honesty-oriented SFT, including improved generalization to OOD cases.
- 5.3 Mitigation during RLHF: RL-tuned LLMs may become over-conservative when helpfulness and honesty are imbalanced.The survey illustrates this trade-off with ChatGPT refusing to provide a clear answer it already knows.
5.4 Mitigation during Inference
Inference-time mitigation is emphasized because it is more cost-effective and controllable than training-time approaches. The survey reviews decoding strategies, external-knowledge methods, and uncertainty-based techniques, while noting practical access and verification challenges.
- Inference-time mitigation is typically more cost-effective and controllable than training-time approaches, making it the focus of many studies.
- Designing Decoding Strategies: Decoding strategies select output tokens from model probability distributions, and nucleus sampling can underperform greedy decoding in factuality.The authors attribute this underperformance to randomness introduced to increase diversity and creativity, which may lead to fabricated information.
- Designing Decoding Strategies: Chain-of-Verification plans independent verification questions, answers them, and uses the results to produce a revised response.The framework is based on the observation that independent verification questions typically yield more accurate facts than long-form answers.
- Designing Decoding Strategies: Context-aware decoding contrasts context-conditioned and query-only token distributions to increase reliance on retrieved knowledge and reduce factual hallucinations.The method addresses cases where LLMs inadequately attend to retrieved knowledge, especially when it conflicts with parametric knowledge.
- Summary & Discussion: Inference-time decoding methods are plug-and-play and easy to deploy, but many require token-level probabilities unavailable through limited APIs.
- Resorting to External Knowledge: External knowledge mitigation retrieves relevant information and then uses it to guide response generation through knowledge bases or external tools.The survey describes sources including corpora, databases, websites, the Internet, search engines, and task-specific tools such as code executors and scholarly search APIs.
- Resorting to External Knowledge: External knowledge avoids modifying LLMs, supports proprietary and real-time information, and enables tracing outputs to source evidence, but Internet sources may themselves be fabricated.Verifying the authenticity of retrieved Internet knowledge remains an open challenge.
- Exploiting Uncertainty: Uncertainty-based mitigation includes logit-based, verbalize-based, and consistency-based approaches, each facing unresolved applicability or reliability challenges.Logits are often inaccessible for closed-source models, models can be overconfident when verbalizing confidence, and response consistency is difficult to measure.
5.5 Other Methods
Other proposed mitigation directions include human-in-the-loop query alignment and architectural changes intended to reduce hallucinations. These approaches target retrieval misalignment and the use of bidirectional information.
- Human-in-the-loop: MixAlign uses LLMs and iterative user clarification to align queries with stored knowledge in retrieval-augmented generation.The framework addresses a proposed hallucination cause involving misalignment between knowledge and user questions.
- Optimizing model architecture: Architectural approaches include multi-branch and uncertainty-aware decoders, while bidirectional autoregressive modeling is claimed to reduce hallucinations by leveraging information from both directions.
6 Outlooks
The survey identifies unresolved challenges in evaluating and mitigating hallucinations across languages, modalities, model-editing settings, attacks, and reasoning models. It highlights future directions including more reliable evaluation, multilingual and multimodal analysis, agent settings, linguistic analysis, and calibration.
- Reliable evaluation: Automatic generation-style hallucination metrics may diverge from human annotations and vary across domains or generating models.Discrimination benchmarks can evaluate hallucination recognition more accurately, but their relationship to generation performance remains unclear.
- Multi-lingual hallucination: Hallucination research remains English-centric, while multilingual models can hallucinate in low-resource languages and produce cross-lingual inconsistencies.The survey reports accurate English answers alongside hallucinations for the same question in Chinese.
- Multi-modal hallucination: Large vision-language models inherit hallucination from LLMs and can exhibit more severe multimodal hallucinations, motivating benchmarks beyond object hallucination.Existing work also extends hallucination investigation to audio and video scenarios.
- Model editing: Model editing offers a data- and computation-efficient way to modify model behavior when hallucinations reflect false memorization or missing factual knowledge.The survey identifies model editing as one approach to mitigating these issues with minimal computational overhead.
- Attack/defense for inducing hallucination: Jailbreak prompts can manipulate even commercially strong LLMs into producing arbitrary responses, including hallucinations, making attack and defense strategies an open direction.The survey frames inducing and defending against hallucinations as a promising research topic.
- Future research directions: Future research should examine hallucination in LLM agents, linguistic prompt factors, and reasoning models’ improved calibration.The survey links hallucination occurrence to prompt readability, formality, and concreteness, while recent work reports calibration gains in reasoning models.
7 Conclusion
The survey reviews recent work on evaluating, tracing, and eliminating LLM hallucinations, then identifies challenges and future directions. It aims to support researchers and foster the practical application of LLMs.
- Conclusion: The survey comprehensively reviews recent advances in evaluating, tracing, and eliminating hallucinations, especially work published after ChatGPT’s release.It also discusses existing challenges and potential future research directions.
- Conclusion: The survey is intended as a resource for researchers studying LLM hallucinations and practical LLM applications.Its stated goal is to help foster the practical application of LLMs.
- Conclusion: The survey addresses hallucination because it remains a critical challenge to the practical application of LLMs despite their strong open-domain understanding and generation capabilities.The paper connects hallucination with the reliability of deploying LLMs in practice.
Ethic issues of Hallucination in LLMs
LLM hallucinations raise ethical concerns because users may disseminate misinformation and rely on unreliable outputs in consequential decisions. The paper emphasizes verification and responsible use to reduce potential harm.
- Ethical considerations: Responsible deployment requires careful consideration of ethical issues to mitigate potential harm and preserve trust in automated systems.The paper frames ethical analysis as necessary as LLMs become integrated into applications.
- Misinformation and public trust: Hallucinated responses can spread misinformation when users trust and disseminate model outputs.The concern is especially acute in healthcare, law, and finance, where inaccurate information can have serious consequences.
- Impact on decision making: Hallucinations can distort individual and institutional decision making when legal or medical professionals rely on unreliable LLM outputs.The paper describes these consequences as potentially deeply detrimental.
- Impact on decision making: Critical decisions should not be unduly influenced by unreliable AI outputs and require robust mechanisms for verifying provided information.The paper presents verification as an ethical safeguard for consequential use.