PaperlayerView exact revision on arXiv

Source-linked AI summary

Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens

Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, Xu Yang

arXiv:2411.16724v3cs.CV

TL;DR

Object hallucinations undermine LVLM reliability, while existing explanations have largely emphasized language-side mechanisms rather than visual processing. The paper analyzes visual attention across layers and heads, identifies middle-layer stages associated with hallucination, and uses those findings to guide inference-time mitigation. The method reduces hallucinations across mainstream LVLMs without additional training, while preserving descriptive detail.

  • Problem

    Object hallucinations undermine LVLM reliability, and the visual mechanisms underlying them remain poorly understood relative to language-side explanations.

  • Method

    The paper uses VAR and logit-lens analyses to study middle-layer visual processing, then integrates attention information across heads during inference to adjust visual attention.

  • Results

    Real object tokens receive higher visual attention than hallucinated tokens, hallucinations involve inconsistent cross-head object interactions, and detection reaches 74% AUROC and 88% mAP on LLaVA-1.5-7B.

  • Takeaways & Limitations

    Visual attention patterns provide interpretable signals for detecting object hallucinations and support training-free inference-time mitigation across mainstream LVLMs.

  • Takeaways & Limitations

    SVAR can weaken when a model maintains extremely high visual attention ratios across nearly all layers, and the boundaries of the two processing stages remain somewhat subjective.

Abstract

from arXiv · show

Hallucinations in Large Vision-Language Models (LVLMs) significantly undermine their reliability, motivating researchers to explore the causes of hallucination. However, most studies primarily focus on the language aspect rather than the visual. In this paper, we address how LVLMs process visual information and whether this process causes hallucination. Firstly, we use the attention lens to identify the stages at which LVLMs handle visual data, discovering that the middle layers are crucial. Moreover, we find that these layers can be further divided into two stages: ''visual information enrichment'' and ''semantic refinement'' which respectively propagate visual data to object tokens and interpret it through text. By analyzing attention patterns during the visual information enrichment stage, we find that real tokens consistently receive higher attention weights than hallucinated ones, serving as a strong indicator of hallucination. Further examination of multi-head attention maps reveals that hallucination tokens often result from heads interacting with inconsistent objects. Based on these insights, we propose a simple inference-time method that adjusts visual attention by integrating information across various heads. Extensive experiments demonstrate that this approach effectively mitigates hallucinations in mainstream LVLMs without additional training costs. Code is available at https://github.com/ZhangqiJiang07/middle_layers_indicating_hallucinations.

1. Introduction

The paper examines visual information processing as a source of object hallucinations in LVLMs, identifying middle-layer mechanisms and an inference-time mitigation method. Its analyses connect hallucinations to weak and inconsistent visual attention, while experiments show reduced hallucination rates without losing descriptive detail.

  • Object hallucinations occur when LVLMs generate objects absent from the visual input.
  • Existing work emphasizes language bias, leaving the visual mechanisms underlying object hallucinations poorly understood.
  • Middle layers contain two stages: visual information enrichment propagates image information to object tokens, while semantic refinement interprets it through text.
  • During visual information enrichment, real object tokens receive higher image-token attention than hallucinated tokens, while hallucinations show inconsistent attention across heads.
  • The proposed inference-time method integrates attention across heads and reduces average CHAIRI and CHAIRS by up to 6.3 and 24.1 points, respectively.Experiments on three mainstream LVLMs report mitigation while preserving detail in descriptions.

2. Related Work

Related work has addressed LVLM hallucinations through model adaptation, expert integration, decoding, and analyses of language-side biases. These approaches motivate examining visual information processing as another perspective on hallucination mechanisms.

  • LVLMs combine vision encoders, modality connectors, and pretrained language models for vision-language tasks but still suffer from severe object hallucinations.
  • Prior mitigation approaches include visual instruction fine-tuning, external expert models, and contrastive decoding strategies.
  • Recent analyses identify text-side contributors including anchor patterns and text inertia, whereas this paper investigates visual information processing.

3. Understanding Object Hallucinations

The analysis identifies middle Transformer layers as the main site of visual-information interaction, separating visual enrichment from semantic refinement. Real and hallucinated object tokens differ in attention behavior there, enabling hallucination detection.

  • Analytical tools: VAR measures each generated token’s interaction with image tokens by summing its attention weights over all image tokens.Higher VAR indicates greater contribution from image tokens during generation.
  • Experimental setup: On 2,000 COCO validation images, the study analyzes LLaVA-1.5-7B captions containing 4,397 real and 1,842 hallucinated object tokens.Real and hallucinated tokens are identified against object annotations, using only the first token for multi-token objects.
  • Middle-layer processing: Middle layers 5-26 show sustained high attention to image tokens, indicating that visual-information interaction primarily occurs there.The study finds this pattern across the analyzed real object tokens.
  • Middle-layer processing: Layers 5-18 enrich visual information through high VAR but low logit contribution, whereas layers 19-26 semantically refine image information for object prediction.Logit-lens analysis distinguishes the stages by whether image-token interpretations align with corresponding image patches.
  • Hallucination indicators: During visual enrichment, hallucinated tokens receive less image attention than real tokens, and SVAR5-18 significantly separates the two token types in both LLaVA-1.5-7B and 13B.The metric averages VAR across heads and sums it over layers 5-18.
  • Hallucination indicators: Compared with internal confidence, SVAR5-18 improves hallucination-detection AUROC by 8.82% and AP by 3.53% on LLaVA-1.5-7B.The metric treats real object tokens as positive samples and hallucinated tokens as negative samples.

4. Object Hallucination Mitigation

The method intervenes on middle-layer visual attention during inference, using cross-head information to reduce object hallucinations while preserving descriptive detail.

  • Attention intervention: The intervention adjusts image-token attention weights in middle layers during inference to correct visual information processing.It extracts pre-softmax attention scores and modifies them before normalization.
  • Attention intervention: Positive attention adjustments are computed from average absolute attention scores across heads, emphasizing regions consistently attended by multiple heads.This is intended to provide a more faithful, object-related direction for attention shifts.
  • Evaluation: CHAIRI measures the proportion of mentioned objects that are hallucinated, while CHAIRS measures the proportion of captions containing hallucinated objects.Lower values indicate fewer hallucinations.
  • Evaluation: The method is evaluated on three LVLMs using CHAIR hallucination metrics and F1 to assess hallucination reduction alongside description richness.Experiments include LLaVA-1.5, Shikra, and MiniGPT-4, with additional evaluations varying maximum output length.
  • Results: The method consistently reduces hallucinations across different token lengths and achieves the best average performance.The supplied passage reports robustness across output-length settings and qualitative results in Appendix C.7.

5. Ablations and Discussions

Ablations examine layer range and intervention strength, showing that the selected middle-layer range and an appropriate balance factor are important for jointly reducing hallucinations and preserving detail.

  • Baseline comparison: The method is compared with inference-intervention baselines on three LVLMs using CHAIR hallucination and F1 results.The supplied table caption states that the method outperforms other baselines while preserving description richness.
  • Balance factor: Lower α limits hallucination mitigation, whereas higher α compromises description richness.The study varies α over {0.3, 0.4, 0.5, 0.6, 0.7} using CHAIR metrics and F1.
  • Balance factor: An α value of 0.5 balances hallucination reduction with maintenance of detailed visual descriptions for the selected LVLMs.This value is reported as an example setting for both evaluated 7B models.

6. Conclusion and Insights

The paper frames visual information processing as a new perspective on object hallucination and proposes targeted middle-layer attention intervention as a simple inference-time mitigation direction.

  • Contributions: The study analyzes object hallucination through visual information processing and attention rather than only language-side mechanisms.It identifies three key findings and develops an inference-time mitigation method.
  • Future directions: Future work could categorize image tokens into object, color, and textual subgroups to study how LVLMs extract different visual information.The paper also suggests more granular stage-wise analyses.
  • Future directions: The paper suggests modifying targeted internal states, such as middle-layer attention weights, instead of intervening only on LVLM outputs.It presents this as a direction for calibration and more trustworthy LVLMs.

A. Limitations

The paper identifies limitations in both SVAR-based detection and the localization of visual-processing stages.

  • Detection limitation: SVAR detection may weaken when an LVLM exhibits extremely high visual attention ratios across nearly all layers.Shikra is given as an example of this attention behavior.
  • Stage localization: The specific ranges of visual information processing stages are identified somewhat subjectively using VAR and the logit lens.The paper leaves automatic stage localization with learnable layer weights for future work.

B.2. MLP Training Details

The hallucination detector uses a two-layer MLP trained on concatenated VAR scores, with its training setup documented alongside the relevant datasets and figures.

  • MLP detector: A two-layer MLP is trained to detect hallucinated object tokens from concatenated VAR scores.The classifier training pipeline is illustrated in Figure 10, with hyperparameters detailed in Table 7.
  • Object-token data: Table 6 provides statistical information for the case datasets used in the analysis.
  • Object-token data: Figure 9 reports real and hallucinated object-token distributions by their position in descriptions.
  • Training setup: The detector’s training hyperparameters are summarized in Table 7 for LLaVA-1.5-7B.

C.1. Case Study Results

Additional experiments examine VAR patterns, attention-sublayer contributions, hallucination detection, and statistical separation across LLaVA-1.5-13B, MiniGPT-4-7B, and Shikra-7B.

  • LLaVA-1.5-13B: LLaVA-1.5-13B exhibits the same two middle-layer patterns as LLaVA-1.5-7B, supporting generalization across model scale.The comparison uses VAR distributions and prediction contributions from MHSA sublayers.
  • Hallucination detection: Figures 15–17 compare object-hallucination detection curves for LLaVA-1.5-13B, MiniGPT-4-7B, and Shikra-7B.
  • Score distributions: SVAR3-13 and SVAR3-14 score distributions are shown across object-token types for Shikra-7B and MiniGPT-4-7B.
  • MiniGPT-4-7B and Shikra-7B: MiniGPT-4-7B and Shikra-7B show relatively higher VAR scores in middle layers than in other layers.Shikra-7B additionally shows two distinct middle-layer patterns based on MHSA contributions.
  • Statistical tests: SVAR scores are consistently significantly higher for real object tokens than hallucinated ones during visual information enrichment across all tested models.One-tailed t-tests are reported for LLaVA-1.5-7B, LLaVA-1.5-13B, Shikra-7B, and MiniGPT-4-7B.

C.3. Numerical Results of α Sensitivity

The section evaluates the balance factor α, attention-head behavior, layer roles, and AMBER performance for the hallucination-mitigation method.

  • α sensitivity: α controls the trade-off between hallucination mitigation and description richness in the attention intervention method.LLaVA-1.5-7B and LLaVA-1.5-13B are more sensitive to α than MiniGPT-4-7B.
  • α sensitivity: LLaVA-1.5 uses 576 image tokens versus 32 for MiniGPT-4, potentially magnifying α’s impact.
  • Attention-head behavior: During visual information enrichment, attention heads interact with inconsistent objects when generating hallucinated object tokens.Additional visualizations are provided for LLaVA-1.5-7B.
  • AMBER evaluation: The method is evaluated on the AMBER benchmark, which contains 1,004 images for the generative task, and the reported results demonstrate superior performance.
  • Layer roles: LLaVA-1.5-7B layers are organized into low-level image processing, visual information enrichment, and semantic refinement ranges.Visual information enrichment accumulates visual information with high VAR scores and low logit contribution.

C.7. Qualitative Results of Hallucination Mitigation

Qualitative examples illustrate hallucination mitigation across LLaVA-1.5-7B, MiniGPT-4-7B, and Shikra-7B by correcting attention distributions over image tokens.

  • Cross-model results: The method corrects attention distributions over image tokens in qualitative examples from three LVLMs.Examples are provided for LLaVA-1.5-7B, MiniGPT-4-7B, and Shikra-7B.
  • LLaVA-1.5-7B examples: The LLaVA-1.5-7B qualitative results include descriptions of a crowded city street and a dining-table scene.
  • Attention-head behavior: Figures 18 and 19 provide additional visualizations of attention-head behavior for LLaVA-1.5-7B.
  • Visualization conventions: Figures 20–22 mark real object words in blue and hallucinated object words in red.All examples use the prompt “Please help me describe the image in detail.”
Loading 2411.16724v3…