Source-linked AI summary
Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation
Ruoyu Chen, Xiaoqing Guo, Kangwei Liu, Siyuan Liang, Shiming Liu, Qunli Zhang, Laiyuan Wang, Hua Zhang, Xiaochun Cao
TL;DR
MLLMs generate tokens using visual inputs and language context, but how much individual tokens depend on visual evidence remains poorly understood, complicating interpretability and reliability. EAGLE attributes selected tokens to image regions and measures their relative reliance on perceptual evidence and language priors. Across evaluated tasks, it reports stronger faithfulness, localization, and hallucination diagnosis than existing methods, while iterative search limits scalability.
Problem
How much MLLM output tokens depend on visual modalities remains poorly understood, and existing attribution methods face challenges with autoregressive generation and cross-modal interactions.
Source: §1 · p. 2Method
EAGLE uses a black-box framework that attributes selected tokens to image regions with a sufficiency-and-necessity objective, then assesses reliance on perceptual evidence versus language priors.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3Results
Across open-source MLLMs, EAGLE outperforms existing methods in faithfulness, localization, and hallucination diagnosis; for captioning, average gains are 20.0% in insertion and 13.4% in deletion.
Source: §1 · p. 2Takeaways & Limitations
EAGLE pinpoints minimal visual factors associated with hallucinations, providing an interpretation-focused account of their causes.
Source: §5 · p. 8Iterative subset selection and greedy search limit scalability compared with lightweight visualization methods.
Source: §8 · p. 11
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in aligning visual inputs with natural language outputs. Yet, the extent to which generated tokens depend on visual modalities remains poorly understood, limiting interpretability and reliability. In this work, we present EAGLE, a lightweight black-box framework for explaining autoregressive token generation in MLLMs. EAGLE attributes any selected tokens to compact perceptual regions while quantifying the relative influence of language priors and perceptual evidence. The framework introduces an objective function that unifies sufficiency (insight score) and indispensability (necessity score), optimized via greedy search over sparsified image regions for faithful and efficient attribution. Beyond spatial attribution, EAGLE performs modality-aware analysis that disentangles what tokens rely on, providing fine-grained interpretability of model decisions. Extensive experiments across open-source MLLMs show that EAGLE consistently outperforms existing methods in faithfulness, localization, and hallucination diagnosis, while requiring substantially less GPU memory. These results highlight its effectiveness and practicality for advancing the interpretability of MLLMs.
1. Introduction
MLLM token attribution is difficult because autoregressive generation and cross-modal interactions challenge existing methods, while opacity and hallucinations raise transparency and reliability concerns. EAGLE combines region attribution with analysis of whether selected tokens rely more on visual evidence or language priors, and is evaluated across open-source MLLMs and tasks.
1. Introduction
- Autoregressive generation, cross-modal interactions, and textual priors make existing activation- and gradient-based attribution approaches challenging to adapt or interpret reliably.The subset-selection VPS objective also cannot be directly transferred to MLLMs.Source: §1 · p. 2
- EAGLE is a black-box framework that attributes selected output tokens to perceptual regions and quantifies the relative roles of language priors and visual evidence.It combines an insight score for sufficiency with a necessity score for regions whose removal impairs generation.Source: §1 · p. 2
- EAGLE outperforms existing attribution methods by average 20.0% in insertion and 13.4% in deletion for captioning, and 20.6% and 8.1% on those metrics for VQA.On the Pointing Game, EAGLE surpasses TAM by 36.42% and 42.63% under box- and mask-level annotations, respectively; it also localizes hallucination sources for minimal-region removal.Source: §1 · p. 2
- The paper presents EAGLE as lightweight and reports state-of-the-art interpretability across faithfulness, localization, and hallucination diagnosis.Its contributions include greedy search over image regions and token-level modality analysis.Source: §1 · p. 2
2. Related Work
Prior MLLM attribution work includes attention, activation-map, gradient, and search-based approaches, each with stated limitations in faithfulness, scope, or stability. EAGLE is presented as a black-box alternative that localizes outputs to compact input regions and analyzes modality influence and hallucination sources.
2. Related Work
- Attention-based methods face layer sensitivity and limited faithfulness, VPS is restricted to grounding and detection, and IGOS++ remains parameter-sensitive.TAM improves token-level attribution by reducing contextual noise in activation maps, while gradient-based methods remain memory-intensive and unstable.Source: §2 · pp. 2–3
- EAGLE localizes outputs to compact input regions without token selection and quantifies the influence of language priors versus perceptual evidence.The framework also explains causes of object hallucinations in MLLMs.Source: §2 · p. 3
- Prior hallucination studies examine representation correction, incorrect-region attention, and noisy-region attribution; this work focuses on identifying input regions behind incorrect decisions for minimal-region suppression.The cited approaches include,, and VaLSe [6].Source: §2 · p. 3
3. Method
EAGLE frames autoregressive token attribution as ordered selection over image subregions, then evaluates selected regions for sufficiency, necessity, and modality influence. Its scoring and greedy-search procedure supports token- and sentence-level explanations, including analysis of reliance on perceptual evidence versus language priors.
3.1. Task Formulation
- EAGLE decomposes the high-dimensional image into SLICO superpixels and formulates attribution as selecting an ordered subset of regions under a size bound.The ordered formulation accounts for the sequence in which regions contribute to the decision.Source: §3.1 · p. 3
3.2. Explaining Autoregressive Generation
- The insight score evaluates whether selected regions suffice to maximize the probability of the target token, while the necessity score identifies evidence whose removal lowers that probability.The scores address sufficiency and indispensability, respectively.Source: §3.2 · pp. 3–4
- EAGLE combines insight and necessity scores in a unified objective, with larger values indicating more important selected input combinations.The objective encourages diminishing returns as more regions are added, though it may not be strictly submodular for MLLMs.Source: §3.2 · pp. 3–4
- Greedy search ranks regions by their marginal objective increments; increments approaching zero indicate later regions add little information and have weak saliency differentiation.The method uses greedy search to optimize the objective, which is described as NP-hard.Source: §3.2 · p. 4
3.3. Language Prior vs. Perception Evidence
- A full-image versus masked-image probability comparison can misrepresent visual relevance when a token’s probability first rises and then falls as visual inputs are inserted.For a visually irrelevant token, the probability should remain stable as the image is modified.Source: §3.3 · p. 4
- EAGLE tracks token effects as ordered subregions are progressively added; larger influence scores indicate greater perceptual-evidence influence, while smaller scores suggest greater reliance on language priors.The framework remains token-agnostic, including for tokens strongly influenced by language priors.Source: §3.3 · p. 4
- Users can target attribution to specific sentences, words, or tokens, supporting fine-grained and interactive explanation.The framework permits this targeted analysis at arbitrary granularity.Source: §3.3 · p. 4
4. Experiments
EAGLE was evaluated on image captioning, VQA, word-level explanations, and hallucination interpretation across open-source MLLMs. It generally outperformed attribution baselines, while finer image partitions increased attribution time.
4.1. Experimental Setup
- Experiments cover MS COCO image captioning, MMVP VQA, and RePOPE object hallucination, comparing EAGLE with gradient-based methods and TAM across three MLLMs.Evaluation includes faithfulness, localization, and correction-oriented metrics; TAM is limited to attributing one token at a time.Source: §4.1 · pp. 4–5
4.2. Faithfulness on Sentence-level Explanations
- EAGLE achieves state-of-the-art sentence-level faithfulness across models and metrics for both captioning and VQA; VQA margins are generally smaller.The paper attributes the smaller VQA margins to generated content relying more on reasoning and language priors than purely on perceptual evidence.Source: Table 1 · p. 5, §4.2 · pp. 5–6
4.3. Experiments on Word-level Explanations
- EAGLE consistently leads word-level faithfulness and localization evaluations across models, with focused attributions that remain effective on LLaVA-1.5.TAM performs well on stronger models but poorly on LLaVA-1.5; EAGLE's attribution does not depend on model-specific internal activations.Source: §4.3 · pp. 6–7, Table 2 · p. 7
4.4. Interpreting Object Hallucinations
- EAGLE outperforms baselines in hallucination attribution and correction across three models, and removing identified regions can correct outputs.Hallucination attribution is evaluated with faithfulness metrics, while correction tests progressively removing responsible regions using AMCR and CSR@10%.Source: §4.4 · p. 7, Table 3 · p. 8
4.5. Ablation Study
- Using both Insight and Necessity Scores consistently improves faithfulness, while finer image partitions generally improve faithfulness at the cost of attribution time.The ablations use Qwen2.5-VL 7B on MS COCO captioning and point to scalable attribution as a future need.Source: §4.5 · p. 8, Table 4 · p. 8, Table 5 · p. 8
5. Conclusion
EAGLE explains autoregressive MLLM token generation by identifying minimal visual factors and separating where models attend from what they rely on. Experiments report gains in faithfulness, localization, and hallucination diagnosis.
5. Conclusion
- EAGLE uses a submodular-inspired objective that unifies sufficiency and indispensability to explain autoregressive MLLM token generation.It reveals both where models attend and what they rely on.Source: §5 · p. 8
- Experiments across diverse models and datasets show gains in faithfulness, localization, and hallucination diagnosis.Source: §5 · p. 8
- EAGLE pinpoints minimal visual factors associated with hallucinations, providing an interpretation-focused account of their causes.Source: §5 · p. 8
6. EAGLE Algorithm
The section outlines EAGLE’s algorithm and identifies its inputs and outputs. The available algorithm fragments also show loops over target-token positions and vocabulary indices.
6. EAGLE Algorithm
- EAGLE takes a prompt, generated sequence, target-token positions, and vocabulary indices, and returns an ordered subset and saliency map.Source: §6 · p. 11
- The algorithm includes a loop over the target-token positions T.Source: §6 · p. 11
- The algorithm includes a loop over vocabulary indices V.Source: §6 · p. 11
7. Additional Experimental Details
The experimental details specify prompts for image captioning and hallucination detection. The captioning prompt limits responses to one factual sentence, while the hallucination task requests a yes-or-no answer first.
7. Additional Experimental Details
- For MS COCO image captioning, the prompt requests one factual English sentence of no more than 20 words and excludes information not clearly visible.Source: §7 · p. 11
- For RePOPE hallucination detection, the prompt frames the task as visual question answering and requests a strict yes-or-no answer first.Source: §7 · p. 11
- The hallucination-detection prompt ends with the question-and-answer template “Question: {question} Answer:”.Source: §7 · p. 11
8. Limitations and Future Works.
The authors identify scalability and proactive hallucination prevention as limitations. They propose more efficient attribution algorithms and future work on detection, mitigation, and model repair.
8. Limitations and Future Works.
- Iterative subset selection and greedy search limit scalability compared with lightweight visualization methods.Source: §8 · p. 11
- The framework focuses on hallucination explanation and partial mitigation, leaving proactive prevention unexplored.Source: §8 · p. 11
- Future work will pursue more efficient attribution algorithms, automatic hallucination detection and mitigation, and data- or parameter-efficient model repair.Source: §8 · p. 11
9. Additional Qualitative Results
The supplementary qualitative results compare sentence- and word-level explanations across models and datasets, then show examples of regions associated with hallucinations.
9.1. Sentence-level Explanations on MS COCO and MMVP
- For LLaVA-1.5, sentence-level highlights align with caption tokens and VQA queries, while baselines diffuse attention or include irrelevant backgrounds.The examples span MS COCO and MMVP.Source: §9.1 · p. 11, Figure 6 · p. 13, Figure 7 · p. 14
- For Qwen2.5-VL, sentence-level maps selectively highlight caption objects and queried entities, avoiding redundancy while baselines miss targets or add noise.Examples cover MS COCO and MMVP.Source: §9.1 · p. 12, Figure 8 · p. 15, Figure 9 · p. 16
- For InternVL3.5, sentence-level highlights correspond to caption tokens and VQA queries, whereas baselines scatter attention or miss queried objects.The examples include MS COCO and MMVP.Source: §9.1 · p. 12, Figure 10 · p. 17, Figure 11 · p. 18
9.2. Word-level Explanations on MS COCO
- On MS COCO word-level tests, EAGLE produces sparse, accurate object localization, while IGOS++ covers broad regions and LLaVA-CAM and TAM often miss precise objects.The examples are evaluated against ground-truth bounding boxes.Source: §9.2 · p. 12, Figure 12 · p. 19, Figure 13 · p. 20
9.3. Additional Hallucination Attribution Visualizations
- Hallucination examples localize misleading regions: visually similar structures and small background cues for LLaVA-1.5, small or occluded objects for Qwen2.5-VL, and overlaps or occlusions for InternVL3.5.Examples include surfboard-like regions, reflective phone-like regions, and fork–spoon confusion.Source: §9.3 · p. 12, Figure 15 · p. 22, Figure 16 · p. 23