Source-linked AI summary
A Survey of Hallucination in Large Foundation Models
Vipula Rawte, Amit Sheth, Amitava Das
TL;DR
Hallucination in foundation models produces content that departs from factual reality, creating reliability concerns, particularly in high-stakes domains. This survey provides a cross-modal synthesis of hallucination research in large foundation models, covering its classification, detection, mitigation, tasks, datasets, evaluation, and future directions. It also reports that human evaluation found factual errors in 57.0% of model-generated video-captioning sentences.
Problem
Hallucination produces false or inaccurate information in foundation-model outputs, posing serious concerns in applications such as medicine, finance, law, and clinical settings.
Method
The survey classifies hallucination across text, image, video, and audio foundation models and examines detection, mitigation, tasks, datasets, evaluation metrics, and future directions.
Results
Human evaluation of video captioning found that 57.0% of model-generated sentences contain factual errors, while existing n-gram metrics align poorly with human assessments.
Takeaways & Limitations
Reliable hallucination assessment and mitigation require attention across foundation-model modalities and to evaluation approaches that capture factuality.
Abstract
from arXiv · showhide
Hallucination in a foundation model (FM) refers to the generation of content that strays from factual reality or includes fabricated information. This survey paper provides an extensive overview of recent efforts that aim to identify, elucidate, and tackle the problem of hallucination, with a particular focus on ``Large'' Foundation Models (LFMs). The paper classifies various types of hallucination phenomena that are specific to LFMs and establishes evaluation criteria for assessing the extent of hallucination. It also examines existing strategies for mitigating hallucination in LFMs and discusses potential directions for future research in this area. Essentially, the paper offers a comprehensive examination of the challenges and solutions related to hallucination in LFMs.
1 Introduction
This survey examines hallucination across large foundation models, where generated content can depart from factual reality. It organizes research across modalities and covers detection, mitigation, tasks, datasets, evaluation metrics, and future directions.
- 1 Introduction: Foundation models generate content across diverse tasks, including language understanding, text and image generation, conversation, image classification, and question-answering.They are trained on extensive volumes of largely unlabeled data, typically through self-supervised learning.
- 1 Introduction: Hallucination occurs when a foundation model produces fictional, misleading, fabricated, or otherwise inaccurate content rather than factual information.Plausible-sounding outputs may still diverge from reality because of training-data biases, unavailable up-to-date information, or contextual limitations.
- 1 Introduction: Hallucination is especially consequential in journalism, healthcare, legal contexts, medicine, banking, finance, and clinical settings, where factual accuracy and reliability are paramount.False or inaccurate outputs in these mission-critical domains can have serious effects on outcomes and operations.
- 1 Introduction: The survey addresses hallucination beyond language models by providing a comprehensive analysis across text, image, video, and audio foundation models.Its modality-based structure discusses hallucination and mitigation techniques for all four types and considers creative-domain cases separately.
- 1 Introduction: The paper categorizes existing research and examines detection, mitigation, tasks, datasets, and evaluation metrics for hallucination in large foundation models.It also presents the authors’ views and possible future directions while maintaining associated open-source resources.
2 Hallucination in Large Language Models
Hallucination in LLMs includes fabricated or self-contradictory responses, motivating detection, correction, benchmarking, and knowledge-grounding approaches across domains and languages.
- Phenomena: Hallucination occurs when LLMs produce fabricated responses or false statements, including explanations that rationalize earlier hallucinations.The survey also discusses self-contradictory outputs as a distinct reliability problem.
- Detection: SELFCHECKGPT provides zero-resource, black-box detection of inaccurate or unverified information in generative LLMs.It does not rely on additional resources or labeled data.
- Mitigation: PURR edits and corrects hallucinations by leveraging denoising language-model corruptions.The method aims to reduce hallucinated content and improve output accuracy.
- Evaluation: HaluEval benchmarks hallucination across domains and languages to help assess LLM reliability.The benchmark is presented as a systematic evaluation tool.
- Mitigation: External-knowledge methods align generated text with factual knowledge through interactive guidance, automated feedback, or structured knowledge bases.The cited approaches include LLM-AUGMENTER and Chain of Knowledge.
- Domain-specific LLMs: Hallucination research also covers high-stakes domains, including medicine and law, where false or inaccurate information can have serious consequences.Examples include the Med-HALT medical benchmark and retrieval-based methods for legal data screening.
3 Hallucination in Large Image Models
Large vision-language models face object hallucination: they may generate objects inconsistent with images, while instructions and generation styles affect both the problem and its evaluation. The section also describes contrastive augmentation and hallucination-detection data resources.
- Mitigation: The Hallucinator generates additional positive samples in feature space to strengthen contrastive learning.It is differentiable and incurs minimal computational overhead.
- Object hallucination: Object hallucination is common in LVLM descriptions, with visual instructions and frequently occurring or co-occurring objects influencing the issue.Existing evaluation methods are also affected by input instructions and generation styles.
- Evaluation: InstructBLIP produces hallucinatory text containing 30 percent non-existent objects in detailed visual responses.M-HalDetect contains 16,000 annotations on VQA examples for detecting such hallucinations.
4 Hallucination in Large Video Models
Video hallucination includes incorrect assumptions about frames and factual errors in generated captions. The section highlights scene-aware visual generation, video understanding systems, and human-aligned factuality evaluation.
- Video hallucination: Video hallucinations arise when models make incorrect or imaginative assumptions about frames and generate artificial or erroneous visual information.The paper illustrates this phenomenon with generated video captions.
- Visual generation: Scene-aware person insertion uses marked scene areas and person images to infer realistic poses and preserve scene context.Self-supervised training supports varied plausible poses and interactive editing.
- Video understanding: VideoChat combines foundational video models with LLMs through an adaptable neural interface for spatial, temporal, localization, and causal reasoning.Its instruction dataset contains thousands of videos paired with descriptions and conversations.
- Video generation: DMT addresses video inpainting when cross-frame information is lacking by using a dual-modality-compatible masked-transformer framework.The cited approach pretrains an image-inpainting model as a prior for video-model training.
- Evaluation: Human evaluation found that 57.0% of model-generated video-captioning sentences contain factual errors.Existing n-gram metrics align poorly with human assessments.
- Evaluation: FactVC is a model-based factuality metric that outperforms previous metrics for video captioning.It was introduced to address the mismatch between automatic metrics and human judgments.
5 Hallucination in Large Audio Models
Large audio-model work addresses data scarcity and explores automated caption generation for music and multimodal audio-text tasks.
- Automatic music captioning: LP-MusicCaps contains around 2.2 million captions paired with 0.5 million audio clips.The dataset was generated from extensive tag datasets using LLMs.
- Automatic music captioning: Researchers evaluated LP-MusicCaps with quantitative language metrics and human assessment.
- Automatic music captioning: A transformer-based music captioning model was evaluated in zero-shot and transfer-learning scenarios.
- Text-to-audio generation: Another approach used language-model data augmentation and an audio encoder to adapt a pre-trained text-to-image model for text-to-audio generation.
6 Hallucination is not always harmful: A different perspective
Hallucination is harmful when generated statements are inaccurate or violate norms, but unexpected outputs can support creative exploration and novel ideas.
- Creative uses: Hallucinating models can act as collaborative creative partners by producing valuable, factually ungrounded threads to explore.
- Creative uses: Unexpected responses may surprise people and stimulate the discovery of novel ideas in creative or artistic contexts.
- Risks and boundaries: Hallucinations become problematic when generated statements are factually inaccurate or contravene human, societal, or cultural norms.
7 Conclusion and Future Directions
The paper organizes hallucination research in LFMs across detection, mitigation, tasks, datasets, and evaluation metrics, then proposes future directions combining technical, human, and ethical approaches.
- Conclusion: The survey classifies existing hallucination research in LFMs across detection, mitigation, tasks, datasets, and evaluation metrics.
- Automated evaluation of hallucination: Future directions include specialized evaluation metrics for factual accuracy, coherence, and consistency.
- Automated evaluation of hallucination: Human assessments, adversarial inputs, and fact-focused fine-tuning are proposed to improve hallucination evaluation and model reliability.
- Curated sources of knowledge: Curated knowledge sources may support knowledge-graph integration, fact-checking, bias mitigation, active learning, and regulatory frameworks.
- Conclusion: Together, these directions target hallucination detection and mitigation while promoting responsible use of curated knowledge for accurate and trustworthy content.