PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, Ting Liu

arXiv:2311.05232v2cs.CL

TL;DR

LLM hallucinations threaten the reliability of information retrieval because models can generate plausible but unsupported content. This survey develops a taxonomy and synthesizes causes, detection, benchmarks, mitigation, retrieval-augmented limitations, and future directions.

  • Problem

    LLM hallucinations threaten reliable information retrieval, while their open-ended behavior creates challenges distinct from those of task-specific models.

  • Method

    The survey categorizes hallucinations by factuality and faithfulness, then reviews their causes, detection methods, benchmarks, mitigation strategies, and retrieval-augmented limitations.

  • Results

    The survey identifies factuality and faithfulness as two primary hallucination types and synthesizes ongoing challenges across LLM and vision-language model hallucinations.

  • Takeaways & Limitations

    The survey provides insights intended to guide development of more robust information retrieval systems and trustworthy artificial intelligence.

  • Takeaways & Limitations

    Current post-training research gives limited attention to capability misalignment and knowledge boundaries in LLMs.

Abstract

from arXiv · show

The emergence of large language models (LLMs) has marked a significant breakthrough in natural language processing (NLP), fueling a paradigm shift in information acquisition. Nevertheless, LLMs are prone to hallucination, generating plausible yet nonfactual content. This phenomenon raises significant concerns over the reliability of LLMs in real-world information retrieval (IR) systems and has attracted intensive research to detect and mitigate such hallucinations. Given the open-ended general-purpose attributes inherent to LLMs, LLM hallucinations present distinct challenges that diverge from prior task-specific models. This divergence highlights the urgency for a nuanced understanding and comprehensive overview of recent advances in LLM hallucinations. In this survey, we begin with an innovative taxonomy of hallucination in the era of LLM and then delve into the factors contributing to hallucinations. Subsequently, we present a thorough overview of hallucination detection methods and benchmarks. Our discussion then transfers to representative methodologies for mitigating LLM hallucinations. Additionally, we delve into the current limitations faced by retrieval-augmented LLMs in combating hallucinations, offering insights for developing more robust IR systems. Finally, we highlight the promising research directions on LLM hallucinations, including hallucination in large vision-language models and understanding of knowledge boundaries in LLM hallucinations.

1 INTRODUCTION

This survey examines hallucinations in LLMs, which achieve major advances in language understanding, generation, reasoning, and factual knowledge use but can produce unreliable content. It proposes an LLM-specific taxonomy, analyzes causes, surveys detection and mitigation methods, and discusses limitations of retrieval-augmented generation.

  • 1 INTRODUCTION: LLMs have driven major progress in language understanding, generation, reasoning, and the use of extensive factual knowledge.The survey frames LLMs as enabling a significant paradigm shift in NLP.
  • 1 INTRODUCTION: The survey categorizes LLM hallucinations into factuality hallucination and faithfulness hallucination.Factuality concerns discrepancies from verifiable real-world facts, whereas faithfulness concerns divergence from user input or context.
  • 1 INTRODUCTION: LLM hallucinations arise from factors across data, training, and inference stages.This categorization is intended to provide a holistic view of their origins and mechanisms.
  • 1 INTRODUCTION: The survey reviews hallucination detection benchmarks and mitigation approaches organized around data-related, training-related, and inference-related causes.It also considers retrieval-augmented generation as a mitigation approach while examining its limitations and susceptibility to hallucinations.
  • 1 INTRODUCTION: The survey provides a comprehensive overview spanning taxonomy, contributing factors, detection, and mitigation of hallucinations in LLMs.Its organization proceeds from taxonomy and causes to detection strategies and benchmarks, followed by mitigation approaches.

2 DEFINITIONS

This section defines LLMs, outlines their three-stage training process as context for hallucination origins, and introduces a taxonomy distinguishing factuality and faithfulness hallucinations. The taxonomy reflects LLMs’ open-domain, user-centric use and their predominantly factual errors.

  • 2.1 Large Language Models: LLMs are general-purpose transformer-based models trained on massive textual corpora, with scaling producing emergent abilities such as in-context learning.Examples include GPT-3, PaLM, LLaMA, GPT-4, and Gemini.
  • 2.2 Training Stages: LLMs undergo pre-training, supervised fine-tuning, and reinforcement learning from human feedback, with each stage equipping them with distinct capabilities relevant to hallucination origins.Pre-training acquires language, world knowledge, and reasoning abilities through autoregressive prediction; SFT improves instruction following, while RLHF aligns outputs with human preferences.
  • 2.3 Hallucinations in LLMs: LLM hallucination denotes generated content that appears nonsensical or unfaithful to source content, extending the broader notion of perceiving entities or events absent from reality.The survey distinguishes intrinsic hallucinations, which conflict with provided context, from extrinsic hallucinations, which cannot be verified using context or external knowledge.
  • 2.3 Hallucinations in LLMs: Because LLMs support open-domain, user-centric interactions and hallucinations predominantly arise at the factual level, the survey introduces a more granular taxonomy than task-specific categorizations.The shift from task-specific toolkits to AI assistants with open-domain interactions motivates this taxonomy.
  • Factuality Hallucinations: Factuality hallucinations comprise factual contradiction and factual fabrication, covering respectively contradictory grounded facts and unverifiable claims.Factual contradiction further includes entity-error and relation-error hallucinations, while factual fabrication includes unverifiability and overclaim hallucinations.
  • Faithfulness Hallucinations: Faithfulness hallucinations concern inconsistency with user instructions, provided context, or the logical consistency of generated content.The survey identifies three subtypes of faithfulness hallucinations, although the supplied passage does not enumerate them.

3 HALLUCINATION CAUSES

The paper attributes LLM hallucinations to multifaceted causes spanning the full capability-acquisition process, organized into data, training, and inference factors.

  • 3 HALLUCINATION CAUSES: Hallucinations have multifaceted origins across the entire LLM capability-acquisition process.The section frames hallucination causes as spanning the full spectrum of how LLM capabilities are acquired.
  • 3 HALLUCINATION CAUSES: The causes are primarily categorized into data, training, and inference aspects.These aspects correspond to Sections 3.1, 3.2, and 3.3, respectively.

3.1 Hallucination from Data

Training data expand LLM capabilities while also driving hallucinations through misinformation, societal biases, and gaps in the knowledge acquired or legally available during pre-training. These gaps include uneven long-tail coverage, outdated information, copyright-sensitive content, and knowledge introduced during supervised fine-tuning.

  • Data Sources: Pre-training and alignment data expand LLM capabilities but become principal contributors to hallucinations through misinformation, biases, and other data-quality limitations.Pre-training data provide general capabilities and factual knowledge, while alignment data teach instruction following and human-preference alignment.
  • Misinformation and Biases: Misinformation from fake news and rumors can enter large pre-training corpora collected heuristically, producing imitative falsehoods.Heuristic collection enables scale but makes consistent data quality difficult to maintain.
  • Misinformation and Biases: Biases rooted in social media, especially around gender and nationality, can be amplified by LLMs as stereotypes and socially unequal hallucinations.Examples of biased source material include biased hiring, prejudiced news, and hate-spewing content.
  • Knowledge Boundaries: LLMs have knowledge boundaries because they cannot memorize all pre-training facts, especially long-tail knowledge, and because pre-training data omit rapidly changing or copyright-restricted content.The boundaries arise from both incomplete memorization and limitations inherent in the available corpus.
  • Knowledge Boundaries: Uneven document volume and entity popularity shape general-domain accuracy, while static model knowledge becomes outdated and may prompt fabricated or formerly correct answers.Once training ends, the model’s internal knowledge is never updated, creating difficulty with dynamic real-world information.
  • Knowledge Boundaries: Supervised fine-tuning can introduce new factual knowledge beyond the boundary established during pre-training.Instruction pairs labeled by human annotators may extend the model’s factual knowledge during SFT.

3.2 Hallucination from Training

Hallucinations can originate from shortcomings in either pre-training or alignment. Training-related failures include cascading token errors, responses beyond the model’s capability boundaries, factual overfitting, and sycophantic outputs that diverge from internal beliefs.

  • Hallucination from Training: Shortfalls in pre-training or alignment can inadvertently cause LLM hallucinations.Pre-training provides general-purpose representations and world knowledge, while alignment targets user instructions and preferences.
  • Pre-training: Erroneous generated tokens can cascade errors through subsequent sequence positions, producing a snowball effect.This failure arises from inconsistencies in causal, left-to-right token prediction.
  • Supervised Fine-Tuning: When annotated instructions exceed pre-defined capability boundaries, supervised fine-tuning trains responses beyond the model’s actual knowledge.Over-fitting on new factual knowledge can encourage fabricated content.
  • Alignment: Misalignment between internal truthfulness beliefs and generated outputs can produce sycophantic responses that appease human evaluators.Human-feedback refinement does not eliminate this divergence.

3.3 Hallucination from Inference

Hallucinations can arise during inference because decoding shortcomings, stochastic randomness, over-confidence, localized attention, and limited output expressivity impair faithful generation. Even when LLMs possess relevant knowledge, reasoning limitations can prevent accurate results in multi-hop settings.

  • 3.3 Hallucination from Inference: Decoding strategies can manifest LLM capabilities after pretraining and alignment, but shortcomings in decoding can also lead to hallucinations.Stochastic sampling is prevalent because highly likely sequences can produce surprisingly low-quality text.
  • 3.3 Hallucination from Inference: Stochastic sampling introduces randomness to support creative and diverse generation, yet high-likelihood sequences may still produce surprisingly low-quality text.The passage identifies stochastic sampling as the prevailing decoding strategy and links randomness to the problem of likelihood-quality mismatch.
  • 3.3 Hallucination from Inference: Over-confidence arises when generation prioritizes fluency and partially generated content over faithfully adhering to the source context.This phenomenon persists in causal language models, whose next-word predictions condition on both model context and partially generated text.
  • 3.3 Hallucination from Inference: Localized attention mechanisms prioritize nearby words during generation, producing a deficit in broader contextual consideration.The passage attributes this behavior to language models’ attention mechanisms and describes it as a limitation associated with over-confidence.
  • 3.3 Hallucination from Inference: The Softmax bottleneck constrains the expressivity of output probabilities when softmax operates with distributed word embeddings.The limitation affects softmax-based language models that compute word-prediction probabilities from final-layer representations and word embeddings.
  • 3.3 Hallucination from Inference: Reasoning limitations can cause inaccurate multi-hop answers despite the LLM possessing necessary knowledge, especially when multiple associations link questions.The passage also identifies the Reversal Curse as a specific reasoning failure in LLMs.

4 HALLUCINATION DETECTION AND BENCHMARKS

The section organizes hallucination detection into factuality detection, which tests alignment with real-world facts, and faithfulness detection, which evaluates consistency with provided context. It surveys retrieval-based verification, uncertainty estimation, and multiple faithfulness metrics while noting classifier-based limitations.

  • Detection taxonomy: Detection strategies divide into factuality hallucination detection and faithfulness hallucination detection.Factuality concerns real-world accuracy, whereas faithfulness concerns adherence to contextual information.
  • Factuality hallucination detection: Factuality detection uses fact-checking against trusted knowledge sources and uncertainty estimation from internal uncertainty signals.These approaches respectively rely on external verification and model-derived uncertainty.
  • Fact-checking: Fact-checking first extracts independent factual statements and then verifies them against trusted knowledge sources.FACTSCORE exemplifies fine-grained evaluation by decomposing long-form generations into atomic facts and computing the percentage supported by reliable sources.
  • Uncertainty estimation: Uncertainty-based detection supports zero-resource settings by estimating uncertainty from LLM internal states and behavior.Internal indicators include token probability and entropy, while behavioral methods include self-consistency and multi-debate.
  • Faithfulness hallucination detection: Faithfulness detection includes fact-based, classifier-based, QA-based, uncertainty-based, and LLM-based metrics.Classifier-based methods face a gap between related-task training and downstream evaluation, compounded by scarce annotated data; QA-based methods instead construct questions from output information units to test source consistency.

5 HALLUCINATION MITIGATION

This section reviews contemporary methods for mitigating LLM hallucinations, categorizing them by their underlying causes into data-, training-, and inference-related approaches.

  • 5 HALLUCINATION MITIGATION: The mitigation methods are organized according to hallucination causes, covering data-related, training-related, and inference-related hallucinations.Each category offers tailored solutions for its specific challenges.

5.1 Mitigating Data-related Hallucinations

The section presents data filtering, model editing, and retrieval-augmented generation as approaches for mitigating data-related hallucinations rooted in misinformation, bias, and knowledge gaps. It also highlights scalability, update, retrieval-quality, and performance limitations that motivate further research.

  • Data filtering: High-quality, human-curated pre-training data can reduce misinformation and social biases by improving source factuality.Radford et al. emphasized scraping web pages that had undergone rigorous human curation and filtration.
  • Data filtering: Deduplication identifies exact and semantic duplicates through substring matching, efficient data structures, and embedding-based methods such as SemDeDup.Semantic duplicates are data pairs with similar meanings but nonidentical strings.
  • Model editing: Model editing injects additional knowledge by locating and updating parameters or predicting weight updates with an external hyper-network.Locate-then-edit methods include ROME and MEMIT, while meta-learning methods include MEND and MAL-MEN.
  • Model editing: Model editing precisely addresses specific misinformation without extensive retraining but struggles with large-scale updates and may damage overall performance under continuous edits.MEND can also exhibit cancellation effects when parameter shifts for different keys counteract one another.
  • Retrieval-augmented generation: Retrieval-augmented generation alleviates knowledge-gap hallucinations by conditioning generation on externally retrieved knowledge, using one-time, iterative, or post-hoc retrieval.Post-hoc approaches research evidence and revise or verify generated reasoning against external knowledge.
  • Retrieval-augmented generation: Retrieval augmentation is domain-general and modular but can be harmed by irrelevant retrievals, which may reduce overall performance.External knowledge bases can function as replaceable or modifiable plug-ins.

5.2 Mitigating Training-related Hallucination

Training-related hallucinations arise from limitations in LLM architectures, training strategies, and alignment, motivating interventions across pretraining, supervised fine-tuning, and reinforcement learning from human feedback. Post-training methods offer direct mitigation, but capability misalignment and knowledge boundaries remain underexplored.

  • Training stages: Training-stage methods target architectural and objective limitations that produce fragmented representations, attention glitches, and inconsistent factual associations.Examples include bidirectional autoregressive modeling and factuality-enhanced preprocessing that converts document sentences into standalone facts.
  • Training stages: Pretraining interventions can substantially improve hallucination mitigation but often require computationally intensive architecture or objective changes with limited broad applicability.The section calls for adaptable and efficient strategies that avoid extensive system overhauls.
  • Alignment stages: Alignment-related hallucinations reflect capability misalignment and belief misalignment, with research focusing primarily on belief misalignment because LLM knowledge boundaries are difficult to define.Belief misalignment commonly appears as sycophancy when human preference judgments favor approval-seeking responses over truthful ones, potentially causing reward hacking.
  • Alignment stages: Post-training interventions mitigate sycophancy by improving preference judgments, fine-tuning on opinion-independent synthetic data, or steering activations using sycophantic and non-sycophantic prompt pairs.Activation steering subtracts an averaged difference vector from intermediate activations during inference to produce less sycophantic outputs.
  • Alignment stages: Post-training methods provide a direct and effective mitigation route, but future work should prioritize capability misalignment and the knowledge boundaries governing capability alignment.These methods bypass complexities associated with data sourcing and pretraining.

5.3 Mitigating Inference-related Hallucination

The section presents inference-time decoding strategies that mitigate hallucinations by improving factuality, faithfulness to context, and consistency in generated reasoning. It covers factuality-enhanced methods, context consistency, and logical consistency, while noting trade-offs involving diversity, adaptivity, and computational cost.

  • Factuality Enhanced Decoding: Factuality-enhanced decoding aligns LLM outputs with established real-world facts to reduce false or misleading information.These methods prioritize the factuality of generated content.
  • Factuality Enhanced Decoding: Factual-nucleus sampling dynamically adjusts nucleus probability during sentence generation using decay factors, lower boundaries, and sentence-level resets.The strategy addresses non-factual content introduced by randomness in open-ended sampling.
  • Factuality Enhanced Decoding: Post-editing decoding uses LLM self-correction to refine originally generated content without relying on an external knowledge base.Chain-of-Verification is introduced as a prompting-based self-correction method.
  • Factuality Enhanced Decoding: Factuality decoding assesses factuality at each decoding step and offers plug-and-play improvements without computation-intensive training, but may compromise diversity or informativeness.The central limitation is balancing factual accuracy against the range and informational value of generated content.
  • Faithfulness Enhanced Decoding: Faithfulness-enhanced decoding improves alignment with provided context through context consistency and strengthens generated reasoning through logical consistency.Context-aware decoding contrasts output distributions with and without context, while chain-of-thought decomposes complex problems into intermediate steps.
  • Faithfulness Enhanced Decoding: Faithfulness-enhanced strategies can improve contextual alignment and internal consistency, but often lack adaptive context mechanisms and may require computationally costly additional models.The section also notes that generated intermediate rationales may not faithfully capture the model’s underlying behavior.

6 HALLUCINATIONS IN RETRIEVAL AUGMENTED GENERATION

RAG can reduce hallucinations by supplying up-to-date external knowledge, but hallucinations persist when retrieval fails, sources are unreliable, context is poorly used, or generation is unfaithful. The survey organizes these weaknesses around query formulation, retrieval-source quality, retriever effectiveness, context utilization, and faithful decoding.

  • Overview: RAG reduces hallucination risk by incorporating up-to-date knowledge from large-scale external sources, yet retrieval-augmented LLMs can still hallucinate.The section frames RAG as a mitigation strategy rather than a complete solution.
  • Retrieval failures: Retrieval failures can propagate hallucinations through the pipeline because of query formulation, retrieval-source reliability and scope, or retriever effectiveness.These are identified as the three primary sources of retrieval-stage failure.
  • User queries: Ambiguous or complex queries increase hallucination risk by causing irrelevant, partial, or incorrect retrievals, while blind retrieval can mislead responses when external knowledge is unnecessary.Ambiguity can misdirect retrieval, whereas complex queries may exceed keyword- or semantic-matching capabilities; adaptive retrieval is proposed to address unnecessary retrieval.
  • Retrieval sources: Factually incorrect or outdated retrieval sources increase the risk of misleading outputs, while LLM-generated web content may be favored over human-authored content by modern retrieval models.Quality filters and credibility-aware generation are described as approaches for reducing the impact of flawed information.
  • Retriever and generation: Even with relevant evidence, RAG can fail through improper chunking, insufficient use of long-context information, and unfaithful generation from accurate sources.The passages highlight chunking, lost-in-the-middle context degradation, and failure to faithfully use relevant context as distinct limitations.

7 FUTURE DISCUSSION

The survey identifies two future research directions: hallucinations in large vision-language models and knowledge-boundary understanding in LLMs. It highlights inconsistent visual responses, ongoing mitigation efforts, and LLMs’ difficulty recognizing when their knowledge is insufficient.

  • 7.1 Hallucination in Large Vision-Language Models: Future work should investigate hallucinations in large vision-language models, which combine visual perception with advanced language understanding and generation.LVLMs extend earlier multimodal models by leveraging advanced LLMs for human interaction.
  • 7.1 Hallucination in Large Vision-Language Models: Evaluations show that LVLMs generate image-inconsistent responses involving nonexistent objects, incorrect object types or attributes, and faulty semantic relationships.Studies also report that LVLMs can be easily fooled.
  • 7.1 Hallucination in Large Vision-Language Models: Mitigation efforts fine-tune LVLMs for more truthful and helpful responses, rectify inconsistent outputs post hoc with expert models, or pursue approaches that avoid external tools.These directions target more robust large vision-language models.
  • 7.2 Knowledge Boundaries in LLM Hallucinations: LLMs’ inability to recognize their knowledge boundaries contributes to confident falsehoods produced without awareness of their knowledge limits.Research probes these boundaries using strategies including the probability of a correct response in multiple-choice settings.
  • 7.2 Knowledge Boundaries in LLM Hallucinations: Activation spaces contain latent structures related to truthfulness beliefs, while LLMs can encode question unanswerability despite overconfidence on unanswerable questions.This combination highlights a continuing challenge in understanding knowledge boundaries and hallucinations.

8 CONCLUSION

The survey examines the causes, detection methods and benchmarks, and mitigation strategies for hallucinations in large language models. Despite significant progress, LLM hallucination remains an ongoing concern requiring continuous investigation.

  • The survey investigates the underlying causes of hallucinations in large language models.
  • It reviews pioneering hallucination detection methodologies, related benchmarks, and effective mitigation strategies.
  • Despite significant strides, hallucination in LLMs remains an ongoing concern demanding continuous investigation.
Loading 2311.05232v2…