PaperlayerView exact revision on arXiv

Source-linked AI summary

Hallucination of Multimodal Large Language Models: A Survey

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, Mike Zheng Shou

arXiv:2404.18930v2cs.CV

TL;DR

MLLMs perform strongly on multimodal tasks but often generate text inconsistent with visual content, raising reliability and safety concerns. This survey synthesizes hallucination causes, evaluation resources, mitigation methods, and open challenges, concluding that hallucination remains complex and persistent.

  • Problem

    MLLMs can generate plausible but visually inconsistent outputs, creating concerns about their practical reliability, safety, and trustworthiness.

  • Method

    The survey develops a granular taxonomy and reviews hallucination causes, evaluation metrics and benchmarks, mitigation strategies, and cross-modal alignment challenges.

  • Results

    The survey finds that MLLM performance is not always consistent across benchmarks because benchmarks emphasize different evaluation dimensions.

  • Takeaways & Limitations

    The survey provides an organized resource for understanding MLLM hallucinations and guiding research toward more robust and trustworthy models.

  • Takeaways & Limitations

    Existing mitigation methods often improve specific benchmarks empirically, while their underlying mechanisms and decision-making processes remain difficult to understand.

Abstract

from arXiv · show

This survey presents a comprehensive analysis of the phenomenon of hallucination in multimodal large language models (MLLMs), also known as Large Vision-Language Models (LVLMs), which have demonstrated significant advancements and remarkable abilities in multimodal tasks. Despite these promising developments, MLLMs often generate outputs that are inconsistent with the visual content, a challenge known as hallucination, which poses substantial obstacles to their practical deployment and raises concerns regarding their reliability in real-world applications. This problem has attracted increasing attention, prompting efforts to detect and mitigate such inaccuracies. We review recent advances in identifying, evaluating, and mitigating these hallucinations, offering a detailed overview of the underlying causes, evaluation benchmarks, metrics, and strategies developed to address this issue. Additionally, we analyze the current challenges and limitations, formulating open questions that delineate potential pathways for future research. By drawing the granular classification and landscapes of hallucination causes, evaluation benchmarks, and mitigation methods, this survey aims to deepen the understanding of hallucinations in MLLMs and inspire further advancements in the field. Through our thorough and in-depth review, we contribute to the ongoing dialogue on enhancing the robustness and reliability of MLLMs, providing valuable insights and resources for researchers and practitioners alike. Resources are available at: https://github.com/showlab/Awesome-MLLM-Hallucination.

1 INTRODUCTION

MLLMs extend LLM capabilities to multimodal tasks but can generate plausible content inconsistent with visual input. This survey organizes the field’s causes, evaluations, mitigation strategies, and open challenges.

  • MLLMs, also called LVLMs, show strong abilities in tasks such as image captioning and visual question answering but may produce factually spurious content.
  • MLLM hallucinations are studied through object categories, attributes, and relations, with counting and events included under attributes in this survey.
  • The survey analyzes hallucination origins across data, model, training, and inference stages, alongside dedicated metrics, benchmarks, and cause-linked mitigation methods.
  • The survey aims to deepen understanding of MLLM hallucination challenges and inform future development of more robust and trustworthy models.
  • Compared with surveys of pure LLMs, MLLM hallucination surveys are scarce, motivating this survey’s broader taxonomy, causal scope, literature coverage, and linked mitigation strategies.
  • The paper proceeds from foundational concepts to causes, evaluation metrics and benchmarks, mitigation approaches, and future challenges.

2 DEFINITIONS

The paper introduces LLMs and MLLMs, explains their training and architectural foundations, and defines multimodal hallucination primarily through visual-content inconsistency. It distinguishes category, attribute, and relation hallucinations.

  • LLMs are transformer-based models trained on vast text datasets, with capabilities including in-context learning, chain-of-thought prompting, and instruction following.
  • LLM training commonly comprises pre-training, supervised fine-tuning, and reinforcement learning from human feedback.
  • MLLMs enable LLMs to perceive and comprehend multiple modalities, especially vision and language, by combining visual capabilities with language reasoning and generation.
  • Interface-based MLLMs connect pretrained visual encoders and LLMs through learnable query-based or projection-layer interfaces, while some models train end to end from raw image patches.
  • Interface-based MLLM training typically uses cross-modal pre-training followed by multimodal instruction tuning.
  • Hallucination occurs when generated text does not align with visual content; object hallucination includes incorrect categories, attributes, or relations.

3 HALLUCINATION CAUSES

Hallucinations can arise throughout MLLM capability acquisition, with data-related causes involving insufficient quantity, poor quality, limited diversity, and statistical bias. These data conditions can undermine cross-modal learning and reliable responses.

  • Hallucination causes span four aspects of MLLM capability acquisition: data, model, training, and inference.
  • Data affects MLLM hallucination through quantity, quality, and statistical bias.
  • MLLM image-text and visual-question-answering datasets remain much less abundant than the text-only data used to train LLMs.
  • Heuristic collection methods increase data volume without guaranteeing quality, while inaccurate, misaligned, or corrupted samples can limit cross-modal feature alignment.
  • Instruction data often lacks negative examples and reject-answering responses, and models may consequently answer “Yes” even when “No” is appropriate.
  • Training-data object distributions create statistical biases through frequent objects and object co-occurrences, influencing model behavior during inference.

3.2 Model

Popular MLLMs connect pretrained vision, language, and alignment components rather than training end to end, allowing module-level errors to accumulate. A weak vision model can cause visual misclassification, misinterpretation, and information loss that undermine multimodal understanding.

  • Popular MLLMs usually combine a pretrained vision model, a pretrained LLM, and an alignment module instead of training end to end.
  • Errors from connected modules can accumulate, and inferior outputs from individual modules may lead to hallucinations.
  • A weak vision model can produce misclassification or misinterpretation of visual concepts, weakening visual perception.
  • Information loss during visual encoding may occur even with powerful vision models, undermining multimodal understanding.

3.3 Training

MLLMs generally use the same autoregressive next-token prediction objective as LLMs, but this objective may inadequately supervise visual content.

  • MLLM training typically applies autoregressive next-token prediction loss, as in LLMs.The loss is described as straightforward, effective, and scalable for language modeling.
  • The next-token objective may be unsuitable for visual learning because visual content has complex spatial structure.
  • Token-level optimization provides insufficient supervision for visual content.

3.4 Inference

During autoregressive inference, attention can shift away from visual content toward previously generated tokens. Uneven attention among visual tokens may further induce hallucinations in fine-grained visual understanding.

  • As generated sequences grow, self-attention increasingly focuses on previous text tokens, diluting attention on visual content.Attention visualizations show greater focus on special tokens such as punctuation rather than visual content.
  • Imbalanced attention among visual tokens can cause hallucinations when tasks require fine-grained understanding of visual objects.AvisC reports that a few image tokens receive excessive attention, while lower-weight tokens may contain essential nuanced object details.
  • A small portion of visual tokens can strongly affect final outputs and induce hallucination.

4 HALLUCINATION METRICS AND BENCHMARKS

MLLM hallucination evaluation centers mainly on object hallucination, using diverse metrics and benchmarks for generative and discriminative settings. However, benchmark results are not directly consistent because evaluation dimensions and emphases differ.

  • Evaluation resources primarily assess object hallucination in MLLM-generated content.Table 1 summarizes relevant hallucination benchmarks and metrics, ordered chronologically.
  • CHAIR measures hallucinated objects in image captions using image-grounded object references, with per-instance and per-sentence variants.Its object vocabulary is restricted to 80 MSCOCO objects, using tokenization and synonym mapping.
  • POPE converts hallucination evaluation into binary classification through short yes-or-no questions about whether probing objects appear in images.
  • Benchmarks cover multiple hallucination dimensions, including object existence, count, position, color, attributes, and spatial relations.MME includes four object-related perception subtasks, while MMHal-Bench covers eight question categories across 96 image-question pairs.
  • GAVIE evaluates relevancy and visual hallucination in 1,000 open-ended samples without human-annotated ground-truth answers.
  • Other benchmarks address negative-object imbalance, model-generated detection, bias and interference, relation-associated hallucination, and visual-context failure modes.NOPE distinguishes object hallucination from incorrectness, CIEM uses automatically generated QA pairs, and Bingo contains 190 failure instances plus 131 successes.
  • Mainstream MLLM performance is not always consistent across benchmarks because the benchmarks emphasize different evaluation dimensions.

5 HALLUCINATION MITIGATION

Hallucination-mitigation methods in MLLMs are systematically organized into four groups according to where they address the problem: data, model, training, and inference.

  • The survey categorizes hallucination-mitigation approaches into Data, Model, Training, and Inference methods.

5.1 Data

Data is identified as a primary factor inducing hallucination, motivating interventions that add negative, counterfactual, and reasoning examples or reduce dataset noise and errors.

  • Data is a primary factor inducing hallucination in MLLMs, so mitigation studies modify training data in several ways.Reviewed interventions introduce negative, counterfactual, or reasoning data and reduce noise or errors in existing datasets.
  • LRV-Instruction combines positive and negative instructions to counter models’ tendency to answer “Yes” consistently.Its negative instructions manipulate nonexistent objects, existent-object attributes, and knowledge.
  • HalluciDoctor calibrates instruction-tuning data by detecting hallucinated content through consistency crosschecking and eliminating it.The approach also examines long-tail distribution and object co-occurrence as factors in hallucination.
  • PerturboLLaVA trains with adversarially perturbed text that conflicts with visual content, reducing dependence on language priors.The perturbations align with general knowledge but intentionally mislead the model relative to the image, encouraging greater image dependence.
  • REVERIE adds rationale learning to visual instruction tuning so models predict why responses are correct or incorrect.Its curated dataset supplies intermediate reasoning supervision rather than training from responses alone.
  • ReCaption rewrites captions through keyword extraction followed by LLM-based caption generation to produce higher-quality image-caption pairs.The framework extracts verbs, nouns, and adjectives before generating rewritten sentences.

5.2 Model

Model-side mitigation targets visual information loss and imbalance between vision and language components. Approaches scale visual resolution, augment visual features, or regulate the influence of parametric language knowledge.

  • Vision Features: CLIP ViT features can lose visual details relative to pure vision models, motivating combinations with DINO ViT features.Experiments with additive and interleaved combinations reveal a trade-off between the two feature types.
  • Visual Experts: Visual expert-based models integrate latent information from multiple encoders and add structural knowledge enhancement modules.These designs aim to compensate for information loss from CLIP-based image encoding.
  • Vision Tools: Vision tools such as segmentation masks, depth maps, object detection, and OCR can augment object identification.VCoder uses additional perception formats, while another approach ensembles detection and OCR models.
  • Language Prior: HallE-Switch controls the extent of parametric knowledge used during detailed caption generation.The switch addresses hallucination attributed to generation based on language knowledge rather than visual content.
  • Architectural Alignment: PATCH inserts trainable virtual tokens between image features and enhanced prompt texts to align visual and textual representations.The tokens are designed to bridge the gap between encoded image features and augmented textual inputs.

5.3 Training

Training-side methods supplement language-modeling supervision with grounding, contrastive objectives, reasoning signals, and reinforcement learning to better align outputs with visual content.

  • Motivation: Language-modeling loss alone may be insufficient for processing the rich information encoded in visual content.This motivates auxiliary supervision and alternative optimization objectives.
  • Auxiliary Supervision: RAI-30k adds panoptic scene-graph relation annotations and mask prediction loss to guide MLLMs toward relevant image content.SAM supplies the expert vision model for mask supervision, encouraging more accurate responses and reduced vision hallucination.
  • Contrastive Learning: Contrastive-learning methods improve visual-text alignment or distinguish correct from hallucinatory responses during training.HACL targets the modality gap in embedding space, while HALVA trains on phrase-level correct-versus-hallucinated pairs.
  • Visual Information: VISTA counters gradual visual-information loss with Visual Steering Vector and Self-Logits Augmentation modules.The method reinforces visual cues in activation space and prioritizes semantically meaningful tokens using early excitation patterns.
  • Termination: EOS Decision adds Selective EOS Supervision and data filtering because training examples may exceed the MLLM’s perception limit.The objective modifies maximum-likelihood estimation to teach timely termination.
  • Positional Modeling: Concentric Causal Attention reduces the relative distance between visual and instruction tokens to mitigate positional effects from RoPE long-term decay.The method addresses hallucination when relevant visual cues are distant from instruction tokens.
  • Reinforcement Learning: Reinforcement-learning approaches include metric-based optimization, AI feedback, and human feedback.MOCHa uses NLI for fidelity and BERTScore for semantic adequacy in image-captioning optimization.
  • AI Feedback: Preference-based methods train models to favor accurate responses over hallucinatory ones using feedback generated by MLLMs or other AI models.Silkie distills preferences from GPT-4V, while POVID generates dispreferred feedback data with AI models.

5.4 Inference

Inference-time mitigation intervenes in decoding by amplifying visual conditioning, contrasting original and distorted distributions, grounding responses, selecting informative visual context, or correcting later-layer suppression.

  • Contrastive Decoding: Contrastive decoding reduces hallucination by comparing distributions produced from original and distorted visual or textual inputs.VCD uses distorted visual inputs to suppress statistical biases and language priors, while ICD and related methods manipulate prompts or visual layers.
  • Guided Decoding: Guided decoding uses object grounding or CLIP-based reliability scores to emphasize detected objects and favor less hallucination-prone responses.MARINE adds a vision encoder for grounding, and GCD recalibrates decoding using CLIPScore-derived scores.
  • Visual Context: Token-wise optimal visual context eliminated over 84.5% of hallucinations in an oracle study.HALC motivates selecting the most informative grounded visual tokens for each decoded token.
  • Dynamic Correction: MLLMs can recognize objects in preceding layers, but later-layer suppression of that recognition leads to hallucinations.DeCo uses preceding-layer knowledge to correct final output logits dynamically.
  • Visual Amplification: Visual dependence can weaken as generation lengthens, motivating decoding methods that explicitly amplify image conditioning.M3ID combines output distributions with and without the image prompt during generation.
  • Visual Amplification: Image-biased decoding contrasts original-model predictions with an image-focused model without changing model parameters.IBD modifies attention weights to create the image-biased model and derive a more reliable next-token distribution.
  • Visual Amplification: Prompt-relevant local attention generates an augmented image view that better supports object-related queries.AGLA matches image regions to the prompt to address attention deficiency.
  • Visual Amplification: Attention-based methods strengthen image-token contributions during inference, including residual visual decoding and enhanced attention heads.PAI updates decoder self-attention toward image tokens, while EAH broadcasts shallow-layer vision-sink attention maps across heads.

6 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies persistent challenges in data, cross-modal consistency, architecture, evaluation, interpretability, ethics, and the role of hallucination in user experience. It proposes future work spanning improved data practices, alignment, standardized evaluation, interpretability, responsible deployment, and human-centered objectives.

  • 6.1 Data Quality, Diversity, and Bias: Data quality, diversity, bias, scarcity, and calibration remain central challenges for reducing hallucinated content.Suggested directions include data collection, augmentation, and recalibration.
  • 6.2 Cross-modal Alignment and Consistency: Cross-modal consistency requires better modeling of relationships between modalities during both training and hallucination evaluation.Possible approaches include advanced architectures, additional learning objectives, diverse supervision, and unified consistency assessment.
  • 6.3 Designing Effective Model Architectures: Designing architectures specifically for hallucination remains difficult because models must capture linguistic structure while grounding outputs in visual content.Future architectures could be based on identified causes of hallucination.
  • 6.4 Establishing Standardized Benchmarks: Evaluation lacks unified standards: binary yes-or-no protocols may not reflect human use, while generative benchmarks often depend on external models.The survey calls for theoretically sound and accessible standardized benchmarks.
  • 6.5 Reframing Hallucination as a Feature: Future research may evaluate hallucination by its effect on human experience and explore cases where hallucination supports creativity without harming users.The survey distinguishes content that causes hallucination but does not affect user experience from content that does.
  • 6.6 Enhancing Interpretability and Trust: Current mitigation methods improve selected benchmarks but remain largely empirical, leaving underlying generation mechanisms and decisions insufficiently understood.Interpretability research could visualize internals, identify salient features, and trace generation from input to output.
  • 6.7 Addressing Ethical Concerns: Ethical concerns around misinformation, bias, privacy, and societal impact become increasingly important as MLLMs generate realistic text.The survey frames addressing these concerns as part of responsible AI development and deployment.

7 CONCLUSION

The survey examines hallucinations in MLLMs through their causes, evaluation metrics and benchmarks, and mitigation methods. It concludes that hallucination remains complex and persistent, motivating continued scrutiny, innovation, and future research.

  • 7 CONCLUSION: Hallucination remains a complex and persistent concern in MLLMs despite considerable progress.The survey presents its synthesis as a foundational resource for addressing hallucination phenomena.
Loading 2404.18930v2…