Tracing the Evidence: Faithful Token Attribution Through Vision-Language Reasoning

• Token attribution assigns preceding tokens scores reflecting their contribution to a generated token or target span, with faithful scores reflecting how input evidence contributes to that output.Source: §1 · p. 2
• Joint image-text attribution can rank relevant image patches below less relevant text, while limited path tracing can miss visual evidence propagated through multiple reasoning tokens.In Figure 2, aggregating all paths raises the officer-cap patch to rank 2, compared with rank 51 when tracing the highest-attribution path.Source: §1 · p. 2, Figure 2 · p. 2
• VTRACE combines modality-aware pairwise attribution, closed-form aggregation of direct and indirect reasoning paths, and cross-modal calibration for unified image-text rankings.Source: §1 · p. 2
• Across six visual reasoning benchmarks, VTRACE improves mean RISE insertion AUC by 7.1% and reduces deletion AUC by 18.0% against seven baselines.Source: §1 · p. 3

• Token attribution methods use attention aggregation (Attention Rollout, ALTI) (Abnar & Zuidema, 2020; Ferrando et al., 2022b), relevance propagation (AttnLRP) (Achtibat et al., 2024), Hessian sensitivity (HETA) (Pramanik et al., 2026), or perturbation (ReAGent) (Zhao & Shan, 2024).Source: §2 · p. 3
• Graph-based methods trace contribution paths: IFR builds token-state and component graphs (Ferrando & Voita, 2024), FlashTrace tracks generated tokens (Pan et al., 2026), and FlowTracer models conserved attention-graph flow (Dong et al., 2026).Source: §2 · p. 3
• Multimodal attribution includes relevance propagation through self- and cross-attention (Chefer et al., 2021), LVLM-Interpret, and GLIMPSE (Shen, 2025); VTRACE targets incomplete reasoning-path tracing and cross-modal score differences.Source: §2 · p. 3

3.1 PRELIMINARIES

• Given an image and text prompt, an LVLM generates a response containing intermediate reasoning and a final answer; attribution aims to connect a selected output token or span to its supporting input evidence.Source: §3.1 · p. 3
• Pairwise attribution represents direct source-to-receiver token contributions in a strictly upper-triangular matrix, allowing evidence to pass through intermediate reasoning tokens.Source: §3.1 · p. 3

3.2 VISUAL EVIDENCE IS OBSCURED BY TEXTUAL ATTRIBUTION

• When counting two dogs, IFR and FlashTrace omit image patches from their top 10, whereas VTRACE aligns image and text scores and includes seven image patches.Source: Figure 3 · p. 3, §3.2 · p. 4

3.3 LIMITED REASONING-PATH TRACING MARGINALIZES VISUAL EVIDENCE

• Direct or limited-hop tracing can leave upstream visual evidence weakly attributed when its contribution is integrated into later reasoning tokens; VTRACE aggregates direct and indirect paths.Source: §3.3 · p. 4

4.1 MODALITY-AWARE PAIRWISE TOKEN ATTRIBUTION

• VTRACE subtracts each source token’s own-modality mean before measuring its alignment with a receiver update, yielding strictly upper-triangular direct pairwise weights.Source: §4.1 · pp. 4–5

4.2 COMPOSITIONAL MULTI-HOP ATTRIBUTION: DIRECT AND INDIRECT CONTRIBUTIONS

• CMH attribution uses normalized direct weights in a Katz-based closed-form sum over path lengths, then scores tokens using incoming and outgoing attribution to selected receivers.The token score adds 1 to preserve the contribution of a token with no incoming attribution.Source: §4.2 · pp. 5–6

4.3 ALIGNING IMAGE AND TEXT ATTRIBUTION SCORES

• Final cross-modal scores use masked-token response-likelihood contributions, exact two-player Shapley allocation, and image-only rescaling that preserves each modality’s internal ranking.Source: §4.3 · p. 6

5 EXPERIMENTS

• The experiments use six benchmarks spanning visual understanding, mathematical reasoning, and knowledge-based visual reasoning, with one fixed generated trace per sample across attribution methods.The default model is Qwen3-VL-8B; evaluations use Image-only and Joint token rankings and RISE and MAS insertion/deletion metrics.Source: §5 · p. 6

5.1 QUANTITATIVE ANALYSIS

• VTRACE outperforms all baselines across six benchmarks, improving RISE insertion by 6.8% and reducing deletion by 9.3% in Image, and improving insertion by 7.1% and reducing deletion by 18.0% in Joint.Table 1 reports RISE results on Qwen3-VL-8B; Image perturbs image tokens, while Joint perturbs image and text tokens.Source: Table 1 · p. 7, §5.1 · p. 7
• Removing modality centering, multi-hop aggregation, or cross-modal calibration lowers attribution faithfulness; calibration particularly improves Joint attribution while preserving image-token ranking.Modality centering and multi-hop aggregation improve faithfulness in both variants.Source: §5.1 · p. 7, Table 2 · p. 7
• At around 3,000 tokens, VTRACE uses 31 GB versus AttnLRP’s 87 GB and remains faster than baselines as sequence length grows.Attributing all generated tokens raises medium runtime only from 0.24 s to 0.26 s; Figure 6(b) reports best faithfulness at substantially lower attribution time than baselines.Source: §5.1 · p. 7, Figure 6 · p. 8
• VTRACE achieves the highest RISE insertion and lowest deletion scores on Qwen3-VL-4B and InternVL3.5-8B across diverse benchmarks.The reported results cover different LVLM families and sizes.Source: §5.1 · p. 8, Figure 5 · p. 8
• VTRACE consistently outperforms the strongest baselines across evaluation settings on both correct and incorrect predictions.The comparison uses the strongest baseline separately for each metric.Source: §5.1 · p. 8, Table 3 · p. 8
• Attribution-guided GRPO raises Qwen3-VL-4B average benchmark performance from 63.99 to 65.27.The experiment uses VTRACE token-level attribution for credit assignment during post-training.Source: §5.1 · p. 8, Table 4 · p. 8

5.2 QUALITATIVE ANALYSIS

• In a case study, VTRACE links plant-related reasoning tokens to relevant image patches, while the baseline emphasizes background regions and largely misses those paths.Among recovered top 10% tokens, image tokens make up around a quarter for VTRACE and almost none for baselines; Figure 8 contrasts diverse-hop attribution with IFR.Source: §5.2 · p. 9, Figure 7 · p. 9, Figure 8 · p. 9

6 CONCLUSION

• VTRACE represents token attributions in a pairwise matrix, accounts for direct and indirect paths, and calibrates image and text scores for cross-modal ranking.The paper reports consistent improvements over baselines across six visual-reasoning benchmarks while maintaining efficiency.Source: §6 · p. 9

APPENDIX

• The appendix lists extended related work, a proof, the VTRACE algorithm, benchmark and baseline descriptions, experimental and evaluation details, extended experiments, and case studies.These topics are listed as appendix sections B through I.Source: APPENDIX · p. 13

A LIMITATIONS

• The current evaluation focuses primarily on image-based visual reasoning, leaving multi-image reasoning, long-context multimodal understanding, and longer-horizon agentic interaction as settings for future evaluation.The authors state that extending VTRACE to those settings would further establish its generality.Source: A LIMITATIONS · p. 13

B.1 TOKEN ATTRIBUTION AND INFORMATION FLOW

• Token-attribution methods use internal signals such as attention or relevance propagation, or behavioral changes under input intervention, to estimate token contributions.Examples include Attention Rollout, ALTI, AttnLRP, HETA, and ReAGent.Source: §B.1 · pp. 13–14
• In generative settings, source tokens can influence predictions directly or through generated intermediate tokens; prior approaches trace these routes through prefixes, computation graphs, or limited recursive hops.VTRACE instead aggregates multiple forward paths between source and receiver positions.Source: §B.1 · pp. 13–14

B.2 ATTRIBUTION ACROSS IMAGE AND TEXT

• Multimodal attribution must identify important evidence fairly across heterogeneous visual and textual representations.Prior approaches include relevance propagation through multimodal attention and visualizations or gradient-weighted attention over generated tokens.Source: §B.2 · p. 14

B.3 ATTRIBUTION FOR REASONING LVLM

• Recent LVLM approaches make generated intermediate reasoning an explicit part of inference before the final answer.Examples include Multimodal-CoT and staged LLaVA-CoT reasoning.Source: §B.3 · p. 14
• Prior studies report that models may acknowledge subtle visual cues less often than explicit textual cues, motivating inspection of model-internal contributions beyond generated rationales.The cited work examines reasoning consistency and whether generated reasoning reflects visual and textual input biases.Source: §B.3 · p. 14
• For selected reasoning steps, answer tokens, or output spans, VTRACE traces contributions from earlier image, prompt, and generated tokens through direct and indirect paths.This supports inspection of sources contributing to intermediate steps and the final answer.Source: §B.3 · p. 14

C PROOF OF EQ. 6

• Because the attribution matrix is strictly upper triangular, its Tth power is zero, and the proof uses a finite matrix geometric series through powers up to T−1.The displayed series is I + γcW + γ²cW² + ··· + γ^(T−1)cW^(T−1).Source: C PROOF OF EQ. 6 · p. 15

D VTRACE ALGORITHM

• The algorithm takes cached states from one forward pass, including value vectors, attention weights, and residual streams around each attention block.It also takes receiver positions, image and user-text positions, and γ as inputs.Source: D VTRACE ALGORITHM · pp. 15–16
• VTRACE normalizes the pairwise attribution matrix, computes multi-hop attribution in closed form, then scores source tokens by incoming attribution and outgoing attribution to selected receivers.The algorithm uses R = inv(I − γ * W_hat) − I for the multi-hop matrix.Source: D VTRACE ALGORITHM · p. 16
• Calibration estimates image and text contributions from three padding-replacement passes, then rescales image scores to match the estimated modality share while preserving within-modality rankings.When boundary conditions are unmet, the original scores are retained; otherwise text scores are unchanged.Source: D VTRACE ALGORITHM · p. 16

E.1 BASELINE METHODS

• The compared baselines include perturbation, gradient, attention aggregation, relevance propagation, token-flow, and recursive reasoning-trace attribution methods.The appendix describes seven methods: ReAGent, HETA, Attention Rollout, AttnLRP, IFR, FlowTracer, and FlashTrace.Source: §E.1 · p. 17

E.2 BENCHMARK STATISTICS

• MMStar, MathVista, MathVerse, and MMMU cover visual perception, multimodal knowledge, and mathematical reasoning across varied task formats.MMStar reduces text-only shortcuts; the paper uses the testmini splits for MathVista and MathVerse and the validation split for MMMU.Source: §E.2 · p. 17
• MMMU-Pro raises multimodal reasoning difficulty, while VisualPuzzles isolates reasoning from domain knowledge across five reasoning categories.MMMU-Pro removes text-only-solvable questions, expands answer options, and adds a vision-only setting.Source: §E.2 · p. 18
• HallusionBench, RealWorldQA, and MathVision provide additional benchmarks for post-training experiments, spanning visual hallucination, real-world scenes, and competition mathematics.These three benchmarks are introduced as additional tests for the post-training experiments.Source: §E.2 · p. 18

F EXPERIMENTAL DETAILS

• Experiments evaluate publicly released Qwen3-VL checkpoints at 8B and 4B parameters and InternVL3.5-8B; each model generates one reasoning-and-answer trace per question using greedy decoding with a 2,048-token budget, then freezes it across attribution methods.Traces are not filtered by answer correctness; metrics are reported separately for correct and incorrect predictions.Source: F EXPERIMENTAL DETAILS · p. 18

F.1 ATTRIBUTION-GUIDED LEARNING DETAILS

• Attribution-guided learning fine-tunes Qwen3-VL-4B-Instruct with GRPO using eight sampled rollouts per training example.The training data combine filtered multimodal reasoning examples from MMR1-RL and ReVisual-R1.Source: §F.1 · p. 18

G EVALUATION DETAILS

• Faithfulness is evaluated with RISE and MAS insertion and deletion metrics, which track changes in confidence in the generated trace as input tokens are removed or restored.There are no token-level ground-truth labels for which input tokens support the reasoning trace.Source: G EVALUATION DETAILS · p. 19

G.1 PERTURBATION PROTOCOL

• The perturbation protocol changes image and question tokens while excluding chat-template and generated tokens; image regions are blurred and selected question tokens are replaced with padding.Tokens are grouped by decreasing attribution magnitude into up to 20 steps, requiring one more forward pass than the number of groups.Source: §G.1 · p. 19

G.2 METRIC DEFINITIONS

• Response curves are normalized between fully perturbed and clean endpoints and constrained to [0, 1] before metric calculation.Deletion perturbs groups from highest to lowest attribution, while insertion restores groups in that same order.Source: §G.2 · p. 19, §G.1 · p. 19

G.2.1 RISE

• RISE measures area under the normalized perturbation curve; lower deletion and higher insertion scores indicate more faithful rankings.RISE evaluates ranking alone and does not account for attribution magnitude.Source: §G.2.1 · p. 20

G.2.2 MAS

• MAS evaluates whether attribution magnitude agrees with observed response changes, with lower deletion and higher insertion scores indicating greater faithfulness.MAS incorporates the mismatch between attribution mass and model response into the perturbation curve.Source: §G.2.2 · p. 20

G.3 SYSTEM PROMPT

• A fixed system prompt asks the model to reason step by step over visual evidence before giving a final answer.The prompt requires the final answer on a line formatted as “Final answer: <answer>”.Source: §G.3 · p. 20

H.1 MAIN RESULTS

• MAS insertion improves 9.9% for Image and 18.4% for Joint, while deletion scores fall 7.7% and 29.6%, respectively, on Qwen3-VL-8B.The reported results state that VTRACE consistently outperforms baselines across both variants.Source: §H.1 · p. 20, Table 5 · p. 21, Table 6 · p. 21

H.2 HYPERPARAMETER SENSITIVITY STUDY

• γ=1.0 gives VTRACE its strongest benchmark performance, while IFR performs best overall around γ=0.5.The sensitivity analysis also reports that multi-hop aggregation can enhance existing attribution methods.Source: Figure 9 · p. 22, Figure 10 · p. 22, §H.2 · p. 22

H.3 FULL RESULTS ON GENERALIZABILITY ACROSS LVLM FAMILIES AND SIZES

• VTRACE consistently outperforms baselines across InternVL3.5-2B, Qwen3-VL-4B, and InternVL3.5-8B.The reported evaluations span Image and Joint variants and both RISE and MAS metrics.Source: §H.3 · p. 22, Table 7 · p. 23, Table 8 · p. 23

H.4 FULL RESULTS ON ROBUSTNESS TO PREDICTION CORRECTNESS

• VTRACE achieves the best performance on both correct and incorrect predictions in the Image and Joint variants.The authors report that the attribution gains arise from the attribution method rather than answer correctness.Source: Table 10 · p. 24, §H.4 · pp. 24–25, Table 11 · p. 25

H.5 FINE-GRAINED PERTURBATION CURVES

• VTRACE recovers the response faster under insertion and degrades it more sharply under deletion, especially in early perturbation stages.Restoring a small fraction of top-ranked tokens rapidly recovers the response, while removing them causes substantial degradation.Source: Figure 11 · p. 25, §H.5 · p. 25

H.6 SUMMING PATHS CARRIES CREDIT BACK TO THE INPUTS.

• Figure 12 examines how far generated-token credit reaches back and compares summed-hop image credit with diverse hops, VTRACE, and IFR.Its panels show credit distance, answer image credit as hops are summed, and a performance comparison.Source: §H.6 · p. 25, Figure 12 · p. 26

I CASE STUDIES

• Across joint image-text examples, VTRACE more consistently selects relevant objects and labels and highlights question cues and reasoning, while baselines often emphasize backgrounds, borders, or answer options.The harder ball-selection case shows VTRACE’s image selection is less focused, with some patches on the dark background.Source: I CASE STUDIES · p. 26, Figure 14 · p. 28, Figure 15 · p. 29
• VTRACE scores 0.50 and 0.42 on the first two image examples, versus 0.46 and 0.35 for the strongest baseline; it ties FlowTracer on the third and trails AttnLRP on the fourth.The fourth example shows VTRACE assigning attribution to the player and table edge in addition to important balls.Source: I CASE STUDIES · p. 26, Figure 13 · p. 27