PaperlayerView exact revision on arXiv

Source-linked AI summary

Hallucination Begins Where Saliency Drops

Xiaofeng Zhang, Yuanchao Zhu, Chaochen Gu, Xiaosong Yuan, Qiyan Zhao, Jiawei Cao, Feilong Tang, Sinan Fan, Yaomin Shen, Chen Shen, Hao Tang

arXiv:2601.20279v1cs.CV

TL;DR

LVLM hallucination analyses have relied too heavily on forward attention patterns, leaving token-level grounding insufficiently explained. The paper introduces gradient-aware LVLMs-Saliency and two inference-time interventions, SGRS and LocoRE, and reports significant hallucination-mitigation performance across LVLMs and benchmarks. It also identifies a latency and fluency trade-off for SGRS and recommends α = 0.6 as the reported operating point.

  • Problem

    Existing approaches rely primarily on attention patterns and provide insufficient interpretability for distinguishing grounded from hallucinated LVLM outputs.

  • Method

    The paper combines gradient-aware token saliency with SGRS candidate rejection and LocoRE reinforcement of attention to recent output predecessors during inference.

  • Results

    The method demonstrates significant hallucination-mitigating performance across different LVLMs on image hallucination and generation benchmarks.

  • Takeaways & Limitations

    Low saliency among prior output tokens is a reported marker of contextual forgetting, while SGRS and LocoRE provide complementary mechanisms for preserving generation coherence.

  • Takeaways & Limitations

    SGRS increases latency from 30.8 ms/token to 41.2 ms/token and may reduce fluency through over-rejection; the paper recommends α = 0.6 as a trade-off.

Abstract

from arXiv · show

Recent studies have examined attention dynamics in large vision-language models (LVLMs) to detect hallucinations. However, existing approaches remain limited in reliably distinguishing hallucinated from factually grounded outputs, as they rely solely on forward-pass attention patterns and neglect gradient-based signals that reveal how token influence propagates through the network. To bridge this gap, we introduce LVLMs-Saliency, a gradient-aware diagnostic framework that quantifies the visual grounding strength of each output token by fusing attention weights with their input gradients. Our analysis uncovers a decisive pattern: hallucinations frequently arise when preceding output tokens exhibit low saliency toward the prediction of the next token, signaling a breakdown in contextual memory retention. Leveraging this insight, we propose a dual-mechanism inference-time framework to mitigate hallucinations: (1) Saliency-Guided Rejection Sampling (SGRS), which dynamically filters candidate tokens during autoregressive decoding by rejecting those whose saliency falls below a context-adaptive threshold, thereby preventing coherence-breaking tokens from entering the output sequence; and (2) Local Coherence Reinforcement (LocoRE), a lightweight, plug-and-play module that strengthens attention from the current token to its most recent predecessors, actively counteracting the contextual forgetting behavior identified by LVLMs-Saliency. Extensive experiments across multiple LVLMs demonstrate that our method significantly reduces hallucination rates while preserving fluency and task performance, offering a robust and interpretable solution for enhancing model reliability. Code is available at: https://github.com/zhangbaijin/LVLMs-Saliency

1 INTRODUCTION

The paper argues that attention-only analyses insufficiently explain LVLM hallucinations and introduces gradient-aware saliency to expose contextual grounding failures. It then proposes SGRS and LocoRE as complementary inference-time interventions that filter weakly grounded tokens and reinforce recent output context.

  • Motivation: Existing hallucination methods provide limited interpretability because attention maps alone do not clearly distinguish grounded from hallucinated outputs.The paper motivates a token-level tool that can reveal when and why hallucinations emerge during generation.
  • LVLMs-Saliency: LVLMs-Saliency multiplies attention weights by their corresponding gradients to quantify how strongly prior output tokens influence the next-token prediction.The metric provides an unsupervised, token-level view of contextual grounding that attention-only methods fail to capture.
  • Observed pattern: Hallucinations frequently occur when prior output tokens have low saliency for the next token, indicating collapsed contextual memory and weak token dependencies.Correct tokens retain high saliency toward related recent outputs, whereas hallucinated tokens show weak or collapsed prior-output saliency.
  • Mitigation: Saliency-Guided Rejection Sampling evaluates candidate-token grounding before commitment and resamples candidates with weak contextual dependencies.The mechanism is intended to prevent coherence-breaking tokens from entering the generation stream and triggering cascading hallucinations.
  • Combined framework: Together, SGRS and LocoRE form a closed-loop system that gates low-saliency tokens and stabilizes contextual dependencies across generation.Experiments across LVLMs and image hallucination and generation benchmarks are reported to show significant hallucination-mitigation performance.
  • Mitigation: Local Coherence Reinforcement strengthens attention from the current token to its most recent output predecessors, counteracting contextual forgetting after token acceptance.LocoRE operates within the output stream, while SGRS filters tokens before commitment.

2 ANALYSIS AND MOTIVATION

The paper develops a gradient-based saliency analysis to quantify how output tokens influence predictions and identifies statistically significant saliency differences between veridical and hallucinated tokens.

  • Hallucination Token Saliency Analysis: LVLMs-Saliency quantifies token-level hallucination saliency by combining attention weights with their corresponding gradients.The resulting measure captures token influence beyond forward-pass attention alone.
  • Hallucination Token Saliency Analysis: The framework processes model attention matrices and output logits to compute gradients of the loss with respect to attention.The loss uses the model’s output distribution and token-position indexing.
  • Hallucination Token Saliency Analysis: Per-head saliency is obtained through element-wise attention-gradient products with causal triangular masking, followed by head averaging and layer-wise normalization.The masking preserves autoregressive causal structure.
  • Hallucination Token Saliency Analysis: Quantitative analyses across Qwen2-VL-7B and LLaVA1.5-7B reveal statistically significant saliency differences between veridical and hallucinated tokens.The comparison is reported in Figures 1, 2, and 5.

3 METHODOLOGY

The methodology combines candidate-token filtering with local reinforcement of recent output dependencies. SGRS selects tokens using adaptive saliency thresholds, while LocoRE modifies the next-step attention to counter sequence-level context drift.

  • Saliency-Guided Rejection Sampling (SGRS): SGRS samples top-K candidate tokens and evaluates each candidate’s saliency against previously generated output tokens.The saliency uses target layers and output-token positions defined relative to system and image-token lengths.
  • Saliency-Guided Rejection Sampling (SGRS): A candidate is accepted only when its saliency reaches the adaptive threshold computed from the most recent W output tokens.The threshold scales historical average saliency through sensitivity parameter α.
  • Saliency-Guided Rejection Sampling (SGRS): If all SGRS candidates are rejected, the method falls back to the candidate with the highest saliency score.This fallback preserves a token-selection path when no candidate meets the threshold.
  • Local Coherence Reinforcement (LocoRE): LocoRE reinforces attention from the next-token query to previously generated output tokens within a local window of size ws.The distance-weighted gain uses reinforcement strength β and an indicator for membership in the recent window.
  • Local Coherence Reinforcement (LocoRE): LocoRE multiplies the relevant attention slice by its gain vector before the next self-attention softmax and weighted-sum operations.The update amplifies recent-context influence on the prediction at position P+1.
  • Synergistic Workflow: SGRS and LocoRE operate sequentially: SGRS selects the current token, then LocoRE reinforces recent-token dependencies for the next forward pass.The workflow combines token-level grounding with sequence-level protection against forgetting.

4 EXPERIMENTS

Experiments evaluate LocoRE and the combined SGRS–LocoRE framework across LVLMs and image benchmarks, including hallucination, VQA, and multimodal capability measures. Results show improved hallucination mitigation and preserved or improved benchmark performance, with a latency–suppression trade-off for stronger SGRS filtering.

  • Evaluation setup: Evaluations cover LLaVA-1.5, Qwen2-VL, and InternVL across comprehensive, general VQA, and hallucination benchmarks.The listed image benchmarks include LLaVAW, MM-Vet, MME, VizWiz, ScienceQA, POPE, and CHAIR.
  • Comparison with prior methods: Compared with methods targeting only visual attention or attention allocation, LocoRE addresses both visual information and contextual dependencies between text outputs.The cited comparison attributes weaker long-text CHAIR performance to ignoring either visual information or output-text context.
  • Benchmark results: LocoRE improves multimodal capability scores across benchmark categories, while combining SGRS with LocoRE further improves reasoning-intensive MME tasks.The strongest gains are reported for MME Existence and Position tasks.
  • Cross-model results: LocoRE improves VQA results and hallucination metrics across LLaVA-1.5, Qwen2-VL, and InternVL in integrated and generalized settings.The reported gains span structured and unstructured environments without model-specific restriction in the cited comparison.
  • Saliency analysis: LocoRE increases saliency assigned to recent output tokens, counteracting the baseline’s contextual forgetting behavior.Figure 3 contrasts an incorrect token with low prior-token saliency against a correct token with substantially higher recent-output saliency.
  • Ablation study: 28.3% and 22.8% CHAIR hallucination-rate reductions are reported for LLaVA-1.5 and Qwen2-VL at the full-method setting.The cited ablation reports α = 0.6 and β = 0.15 for LLaVA-1.5 and β = 0.20 for Qwen2-VL in Table 3’s caption, while the experiment passage states the full-method setting as β = 0.15.
  • Ablation study: Increasing α from 0.6 to 0.9 raises latency from 30.8 ms/token to 41.2 ms/token while risking reduced fluency through over-rejection.The authors recommend α = 0.6 as the practical trade-off between suppression, speed, and output quality.

5 RELATED WORK

Related work has used decoding changes, training, attention maps, Grad-CAM, and attention-sink analyses to study or mitigate LVLM hallucinations. These approaches motivate examining interactions between images and text while highlighting the need for complementary analysis of output generation.

  • Attention-based analyses: Prior work uses Grad-CAM and attention maps to visualize image–text interactions in complex reasoning tasks.These methods emphasize relevant areas through forward propagation.
  • Attention sinks: Attention-sink studies associate persistent high attention on particular tokens with hallucinated model outputs.The cited methods include OPERA, DOPRA, PAI, FastV, EAH, TAME, and Farsight.

6 CONCLUSION

The paper presents a saliency-based framework that complements attention-sink explanations by linking hallucinations to weak saliency in prior output tokens. Its inference-time interventions improve output accuracy across benchmarks without retraining.

  • Conclusion: The framework complements conventional attention-sink explanations by analyzing weak saliency in previously generated output tokens.The conclusion identifies SGRS and LocoRE as inference-time interventions that boost visual attention and reinforce local coherence.
  • Conclusion: LocoRE improves output accuracy across multiple benchmarks without requiring model retraining.The conclusion describes LocoRE as a plug-and-play intervention during text generation.

A.0.1 INFERENCE-TIME EFFICIENCY

Inference-time efficiency depends on the intervention: the full SGRS+LocoRE framework provides the strongest hallucination suppression but adds gradient-computation latency, whereas LocoRE alone offers a low-overhead alternative.

  • Efficiency trade-offs: 30–40% latency overhead per token is typically added by the full SGRS+LocoRE framework compared with standard greedy decoding.The overhead comes from the backward pass required for saliency computation in SGRS.
  • Efficiency trade-offs: LocoRE alone increases latency by <2% while significantly mitigating context-drift hallucinations through forward-only attention manipulation.LocoRE operates without gradient computation or parameter modification.
  • Efficiency trade-offs: LocoRE requires no auxiliary models, external detectors, or multi-pass decoding, operating entirely within the standard autoregressive loop.The method is presented as a plug-and-play approach with a favorable speed-efficiency trade-off.
  • Mechanism: LocoRE strengthens attention to recent outputs so the next-token prediction remains more strongly grounded in recent output history.This design directly counteracts the saliency decay associated with contextual forgetting.

B.1 STATISTICAL VALIDATION OF THE SALIENCY-HALLUCINATION RELATIONSHIP

Across three LVLMs and two benchmarks, token-level analyses consistently associate lower preceding-output saliency with hallucination, while intervention results and the continuous trend support saliency as a diagnostic signal.

  • Token-level validation: Across LLaVA-v1.5-7B, Qwen2-VL-7B, and InternVL-7B, hallucinated tokens have significantly lower saliency scores than correct tokens.The analysis covers approximately 12,000 generated tokens and reports mean ± standard deviation by label.
  • Saliency-hallucination relationship: Hallucination probability systematically decreases as the preceding output token’s saliency increases across all three models.Saliency scores are divided into 10 equally wide intervals, producing a strong negative correlation without non-monotonic jumps or plateaus.
  • Intervention analysis: The intervention experiment lowers selected correct-token saliency by scaling it with r ∈ 1.0, 0.8, 0.6, 0.4, 0.2 on LLaVA-v1.5-7B.The selected tokens had saliency greater than 0.45 and were evaluated on POPE and CHAIR.
  • Interpretation: Hallucinations emerge gradually as contextual saliency decays rather than through a single threshold event, supporting saliency as a continuous diagnostic signal.This conclusion follows the reported statistical analyses and intervention experiment.

B.2 FAILURE CASE: HIGH-SALIENCY HALLUCINATION

The low-saliency pattern is not universal: some hallucinated tokens retain high saliency, especially when contextual information is ambiguous or insufficient or when the model is highly confident in an incorrect output.

  • Failure modes: Some hallucinated tokens display relatively high saliency, contradicting the claim that hallucinated tokens overwhelmingly have low saliency.The paper illustrates this failure case on Qwen2-VL-7B, where the ground-truth answer is “a traffic cone”.
  • Failure modes: SGRS may reject context-independent content when the context is ambiguous or the input information is insufficient, even if the content follows language-generation rules.The limitation concerns the method’s reliance on relevance to previously generated content.
  • Failure modes: Some incorrect tokens may have high saliency because the model believes its current output is correct with high confidence.The paper attributes this observation to models being trained to produce seemingly reasonable answers and becoming overconfident after RLHF.

B.3 LONG SEQUENCE HALLUCINATION TOKEN AND LAYER EXPERIMENT

A long-sequence example shows that hallucination can arise locally within an otherwise partly correct sequence: the hallucinated token has lower saliency than nearby correct tokens.

  • Long-sequence example: In a generated sequence, “few” is hallucinated in the third sentence, while “preparing” and “significant” are correct outputs in the first and fourth sentences.The example examines hallucination and correct tokens across different sentences and adjacent positions.
  • Long-sequence example: The saliency of correct tokens before and after the hallucination remains greater than the saliency of the original hallucination token.The comparison highlights a local saliency drop around the hallucinated output.
Loading 2601.20279v1…