Source-linked AI summary
Visual Attention Faithfulness in Vision-Language Models is Heterogeneous
Xurui Song, Weishi Wang, Zhongqi Yue, Kuluhan Binici, Tao Bai, Hongxin Shao, Daniel Dahlmeier, Jun Luo
TL;DR
Whether visual attention faithfully reflects VLM reasoning remains underexplored. The paper uses causal perturbations of attention-ranked visual tokens to evaluate comprehensiveness and sufficiency, finding three heterogeneous processing modes and divergence from human-annotated regions across VQA and document tasks.
Problem
Whether attention weights faithfully reflect model reasoning remains an open question for the visual modality in Vision-Language Models.
Method
The paper applies token-level causal perturbations to visual tokens in representative VLMs and evaluates comprehensiveness and sufficiency gap.
Results
Visual attention faithfulness is heterogeneous, comprising Faithful-Sufficient, Faithful-Distributed, and Non-Focal modes across VQA and document tasks.
Takeaways & Limitations
Faithfulness is not binary: visual dependence and attention behavior vary with processing demands and VLM architecture, and human-annotated regions can diverge from model reliance.
Takeaways & Limitations
The findings focus on current dynamic-resolution VLMs and do not explicitly study reasoning-oriented models with multi-step deliberation.
Abstract
from arXiv · showhide
Whether attention weights faithfully reflect model reasoning has been actively debated in NLP, yet this question remains largely unexplored for the visual modality in Vision-Language Models (VLMs). We address this gap through causal perturbation analysis on current VLMs, evaluating both the comprehensiveness and sufficiency gap of attention-ranked visual tokens. Our analysis reveals that visual attention faithfulness is heterogeneous, manifesting in three distinct processing modes: Faithful-Sufficient, where top-$k$ attention tokens are both necessary and sufficient for prediction; Faithful-Distributed, where they are necessary but broader visual context remains required; and Non-Focal, where no localized attention region is individually necessary while visual information remains an essential trigger for prediction. Furthermore, human-annotated ground-truth regions satisfy comprehensiveness in only $\sim 60$% of cases compared with model attention rankings, revealing systematic divergence between model visual reliance and human intuition. We demonstrate these patterns across both general VQA on VQAv2 and document tasks on VRDU and ChartQA, showing that visual attention faithfulness varies systematically with processing demands and model architectures rather than being uniformly faithful or unfaithful.
1 Introduction
The paper evaluates whether visual attention faithfully reflects VLM reliance through causal token perturbations, finding heterogeneous processing modes and systematic divergence from human-annotated regions.
- 1 Introduction: Attention faithfulness is evaluated by removing or retaining top-k attention-ranked visual tokens and measuring comprehensiveness and sufficiency gap.The analysis uses token-level causal perturbations to test whether attended visual tokens are necessary or sufficient for prediction.
- 1 Introduction: The analysis identifies three processing modes: Faithful-Sufficient, Faithful-Distributed, and Non-Focal.These modes distinguish whether focal attention tokens are necessary, sufficient, or unable to localize necessary visual information.
- 1 Introduction: Faithful-Sufficient tokens are necessary and sufficient, whereas Faithful-Distributed tokens are necessary but require broader visual context.In Non-Focal processing, no concentrated attention region is necessary, although visual information remains an essential prediction trigger.
- 1 Introduction: ∼60% of human-annotated ground-truth regions satisfy comprehensiveness compared with model attention rankings.The comparison indicates systematic divergence between human-perceived task-relevant regions and model visual dependencies.
- 1 Introduction: The three modes extend beyond general VQA to document information extraction and document VQA, while varying across VLM architectures.On VRDU and ChartQA, Qwen2.5-VL behaves as an extreme Faithful-Distributed instance, whereas InternVL2.5 is closer to Faithful-Sufficient.
2 Related Work
Related work debates whether attention explains model behavior and develops quantitative faithfulness metrics, while VLM explanation methods often lack causal verification.
- 2 Related Work: NLP studies reach conflicting conclusions about whether attention weights correspond to feature importance or constitute valid explanations.Some work finds attention manipulable without changing predictions, while other work reports meaningful behavioral differences under constrained attention.
- 2 Related Work: Faithfulness research frames explanation quality as a graded, empirically verified property measured through comprehensiveness and sufficiency.Other approaches include recursive masking and retraining for faithfulness assessment.
- 2 Related Work: VLM interpretability methods generate spatial heatmaps or analyze attention mechanisms, but highlighted regions are not necessarily shown to causally drive outputs.Prior work also studies functional cross-attention heads, hallucination-related attention patterns, and attention sinks.
3 Method
The method tests whether attention-ranked visual tokens causally correspond to the visual information a VLM relies on. It ranks tokens by aggregated attention, intervenes by zero-ablating or retaining selected tokens, and evaluates necessity and sufficiency with normalized metrics.
- Causal evaluation: The evaluation treats visual attention faithfulness as a causal measurement problem linking attention-derived token rankings to prediction reliance.The framework combines attention-based token ranking, causal perturbation, and faithfulness metrics.
- Attention-based token ranking: Attention scores for each visual token aggregate weights across all transformer layers and heads.The aggregation averages over heads and sums across layers before ranking tokens in descending order.
- Attention-based token ranking: The method ranks visual tokens and selects the top-k set for intervention-based faithfulness analysis.The selected set is denoted S_k and contains the k highest-scoring visual tokens.
- Causal perturbation: Zero-ablation removes selected visual-token hidden states at the first decoder layer, excluding their contribution from subsequent computations.The intervention changes prediction probability without adding information to the representation space.
- Faithfulness metrics: Comprehensiveness measures whether removing top-k tokens degrades prediction, indicating whether they are necessary.The metric is normalized by the full-input prediction probability for comparability across samples.
- Faithfulness metrics: Sufficiency gap measures the normalized prediction drop when only the top-k tokens are retained; low or negative values indicate sufficiency.Together, comprehensiveness and sufficiency gap characterize the degree and nature of attention faithfulness.
4 Experiments and Analysis
Causal perturbation experiments show that visual attention faithfulness is heterogeneous across VQA and document tasks: localized tokens may be sufficient, necessary but incomplete, or non-focal. Human ground-truth regions often diverge from the visual evidence causally used by models, while processing modes vary with question demands and architecture.
- VQA perturbation results: 0.813 comprehensiveness and 0.469 sufficiency gap make overall top-k the strongest VQA perturbation strategy.Excluding five whole-image-understanding samples, overall top-k achieves the highest mean comprehensiveness and lowest mean sufficiency gap among four strategies.
- Three processing modes: Mode A covers 32.9% of samples, with Comp 0.978 and Sgap −0.026, indicating top-k tokens are necessary and sufficient.This mode typically involves localized object-centric evidence; ablating lane-marking tokens collapses prediction, while retaining them preserves it.
- Three processing modes: Mode B covers 47.1% of samples, with Comp 0.952 and Sgap 0.818, indicating top-k tokens are necessary but broader context remains required.Counting kites illustrates this pattern: focal kite tokens matter causally, but surrounding scene context is additionally needed.
- Three processing modes: Mode C covers 20.0% of samples, with Comp 0.213 and Sgap 0.460, while full-image ablation reaches Comp 0.973.No localized region is decisive, but visual input remains necessary as contextual activation for questions such as cellphone-use laws.
- Question-type associations: 55.6% of yes/no questions fall in Mode B, while number questions split mainly between Modes A and B and other questions reach 34.5% Mode C.The mode structure aligns with reasoning demands: localized counting can be Mode A, whereas broader scene integration produces Mode B.
- Stability analyses: 1000 resampling repetitions across n = 50, 60, and 70 produce confidence intervals excluding zero and increasingly stable mode proportions.This supports the three-mode partition as a data-driven characterization rather than a sample-size artifact.
- GT versus Top-k divergence: 0.505 versus 0.813 comprehensiveness reveals a gap between human GT regions and model top-k regions, with GT-Comp distributions separating aligned and misaligned cases.Aligned cases have mean GT-Comp 0.963 versus Top-k Comp 0.992; misaligned cases have mean GT-Comp −0.150 versus Top-k Comp 0.945, despite high accuracy in both subsets.
- Document understanding: 43.8% probability reduction follows ablating the top 1% of document tokens, but retaining top-k tokens recovers at most ∼5%, placing Qwen-VL-2.5 in extreme Mode B.ChartQA similarly shows rising comprehensiveness with a sufficiency gap near 1, extending the framework beyond VQA.
5 Conclusion
The paper evaluates visual attention faithfulness in VLMs through token-level causal perturbation and finds that it is heterogeneous rather than binary. It identifies three processing modes with distinct visual-dependence patterns and reports divergence between model-used regions and human-annotated evidence.
- Token-level causal perturbation evaluates visual attention faithfulness in VLMs rather than relying only on attention visualization.
- Visual attention faithfulness manifests in three processing modes with different patterns of visual dependence.
- Human-annotated regions align with model-causally used regions on only some tasks, while models may rely on hidden semantically corresponding regions.
- Experiments on document information extraction show that these faithfulness patterns extend beyond general VQA to structured visual tasks.
Limitations
The study focuses on current dynamic-resolution VLMs and does not directly establish whether its findings extend to fixed-resolution models or reasoning-oriented VLMs.
- The findings may not directly extend to fixed-resolution image-input models or reasoning-oriented VLMs performing multi-step deliberation.
A Details of Experiment Setup
The experiment defines human-annotated visual regions and represents images as spatially ordered visual-token sequences, with additional evaluation on excluded VQA samples.
- Ground-truth regions are manually selected and refined from COCO masks to cover task-relevant objects or areas, then represented as visual-token grid coordinates.
- Qwen2.5-VL produces variable spatial visual-token grids through patch embedding and 2 × 2 spatial merging.
- InternVL2.5 uses dynamic tiling, partitioning images into a variable number of tile tokens while preserving spatially ordered visual-token input.
- Attention-ranked top-k ablation is evaluated on five excluded VQA samples.
B Discussion
The discussion positions token-level causal perturbation as a mechanism-level faithfulness test and examines how perturbation design, task structure, architectures, and human annotations affect interpretation.
- Perturbation design: Visual tokens are perturbed at the first decoder-layer input so probability changes reflect language-model reliance rather than vision-encoder responses to corrupted pixels.
- Perturbation design: Zero-ablation suppresses targeted token content while preserving sequence length and encoded spatial structure.
- Sample selection: VQAv2, VRDU, and document tasks probe different visual regimes, including diverse question types, text-rich layouts, field-level evidence, and visual redundancy.
- Results: The three-mode structure separates in comprehensiveness–sufficiency-gap space, reproduces on InternVL2.5-8B, and remains consistent under a separate ratio-based protocol.
- Broader impact: Different VLM backbones can adopt different visual processing preferences on the same information-extraction task, limiting transferability of attention-aware methods across architectures.
- Perturbation strategies: Overall top-k has the highest comprehensiveness and lowest sufficiency gap among four perturbation strategies on both models.
- Comparison with attention visualization: Causal evaluation identifies faithfulness modes and attention–evidence mismatches that attention maps alone cannot determine.
- Relation to gradient-based saliency: Attention faithfulness is evaluated through causal interventions, unlike gradient-based saliency, which derives importance estimates from gradients.
C Excluded VQA Samples
Five VQA samples were excluded because their ground-truth regions covered nearly the entire image; a ratio-based re-evaluation recovered the faithfulness framework for these cases.
- Five of 90 VQA samples were excluded because their human-annotated regions covered almost the entire image.This left little area for ground-truth-aligned strategies to differ meaningfully from full-image ablation.
- Three re-evaluated samples fell into Mode B, Faithful-Distributed, with comprehensiveness near 1 and sufficiency gaps remaining high across masking ratios.The ratio-based procedure evaluated top-k attention tokens at 10%, 30%, and 50% of visual tokens.
- Absence detection, counting “0”, and holistic scene recognition draw evidence from the whole image rather than a key spatial subset.The ratio-based protocol therefore recovered the three-mode characterization when ground-truth-aligned evaluation was unsuitable.
D InternVL2.5 Cross-Model Replication
InternVL2.5 reproduces the three processing modes while differing from Qwen2.5-VL in how visual necessity is distributed across tokens. Attention-ranked tokens remain the strongest selection on VQA, whereas document and chart tasks are closer to Faithful-Sufficient processing.
- VQA replication: 0.733 mean comprehensiveness and 0.468 mean sufficiency gap make InternVL’s top-k visual tokens its most faithful VQA selection.These were the best mean values among the four evaluated perturbation strategies.
- VQA replication: 50.6% per-sample mode overlap with Qwen labels shows that VQA items can activate different attention regions across models.The three modes nevertheless coexist across architectures.
- Architecture comparison: InternVL’s 0.733 comprehensiveness versus Qwen’s 0.813 suggests more redundantly distributed visual information under its dynamic-tile encoder.InternVL’s threshold τc=0.49 was also below Qwen’s τc=0.59.
- Document and chart tasks: 0.189 sufficiency gap at k=10% follows a steady decline in InternVL’s IE replication, while comprehensiveness rises and saturates near 0.650 by k=3%.The same trend places InternVL’s IE and ChartQA tasks in Mode A, Faithful-Sufficient.
- Architecture comparison: InternVL’s localized tiles and global thumbnail concentrate some local evidence while adding a compact global-context representation.This contrasts with Qwen’s broader contextual aggregation for layout reconstruction.
E Usage of AI Assistant
The paper reports using AI assistants and tools for grammar correction and language polishing.
- AI assistants or tools such as Claude and Grammarly were used during preparation.
- The reported use concerned correcting grammar errors.
- The reported use also concerned polishing the language.