Source-linked AI summary
From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making
Davide Testa, Hugh Mee Wong, Alessandro Lenci, Bernardo Magnini, Albert Gatt
TL;DR
The paper addresses whether VLM decisions are grounded in visual evidence rather than relying on textual regularities. It applies layer-wise Attention Knockout to video-text pathways in caption–foil decisions, finding that candidate options are the primary textual grounding site, while temporal decisions show distinctive fragility that may partly reflect lexical biases.
Problem
Final predictions do not reveal whether VLMs use relevant visual evidence or rely on textual prompt regularities and dataset biases.
Method
The study applies layer-wise Attention Knockout to selectively block video-to-text attention pathways during video-based caption–foil decisions across spatial, causal, and temporal reasoning.
Results
Visual information mainly contributes while models process candidate answer options, which are the primary textual sites of cross-modal grounding for final decisions.
Takeaways & Limitations
Visually enriched textual representations carry visual information toward decisions, whereas temporal reasoning can retain late direct dependence on video evidence.
Takeaways & Limitations
The study evaluates only two relatively small VLMs, restricting generalizability across the rapidly evolving multimodal-model landscape.
Abstract
from arXiv · showhide
Vision-Language Models are commonly evaluated through their final predictions, but understanding whether these decisions are grounded in visual evidence requires tracing how visual information contributes to language-based decisions. With this purpose in mind, we investigate cross-modal information flow in a video-based generative multiple-choice-like setting by applying a layer-wise causal intervention on video-text attention pathways. We target spatial, causal, and temporal visual reasoning. Our results show that visual information is mainly integrated while the model processes the candidate answer options, which serve as the primary textual grounding sites for the final decision. We further show that nouns play an important role as semantic anchors during multimodal enrichment, while verbs are more relevant when temporal relations are processed. Finally, we identify a distinct pattern in temporal reasoning, suggesting that VLMs struggle to reconstruct sequential information across video frames, but we remark that such fragility may also reflect linguistic biases associated with specific temporal expressions used for defining the relation between events within a scene.
1 Introduction
The paper argues that final VLM predictions do not reveal whether decisions are grounded in visual evidence. It therefore studies how video information flows into textual representations during multimodal decision-making.
- Video-based tasks expose how visual information must support spatial, causal, and temporal relations across frames.
- The study investigates where visual information interacts with textual representations and how it contributes across three reasoning dimensions.
- The controlled Italian task asks models to choose between a video-linked caption and a minimally different foil.
- Layer-wise Attention Knockout blocks video-text attention pathways to trace causal information flow and measure effects on candidate-answer plausibility.
2 Related Work
Prior VLM evaluations often reveal whether models choose correctly, but not whether decisions use relevant visual evidence or how information flows internally. This work addresses that gap by applying attention knockout to a caption-foil discrimination problem, combining multiple-choice structure with fine-grained causal analysis.
- Foiling methods test whether models discriminate alternatives differing in a targeted semantic element, creating a fine-grained contrastive evaluation.
- Correct answers can arise from linguistic priors or dataset regularities rather than relevant visual grounding, motivating analyses beyond final accuracy.
- Attention patterns alone do not establish causal explanations, so intervention-based methods target attention connections directly.
- Existing VLM mechanistic studies largely examine image-based tasks, leaving video-specific modality circuits and interactions less understood.
- Attention knockout applied to caption-foil discrimination combines multiple-choice evaluation with fine-grained control for examining internal decision support.
3 Experimental Setting
The study uses controlled Italian video caption–foil tasks across spatial, temporal, and causal reasoning, then applies attention interventions to trace visual information flow. It compares coarse region-level and fine token-level masking, evaluates two VLMs, and measures layer-wise effects under different scoring schemes.
- Task and data: Each item pairs a video with a caption and a minimally altered foil that introduces a subtle semantic inconsistency.The model selects the more plausible description among the two candidates.
- Task and data: The evaluation uses 4,800 Italian caption–foil pairs across SPATIAL, TEMPORAL, and CAUSAL subsets, with 1,600 pairs per subset.Each subset contains 1,600 pairs associated with 100 videos.
- Models: The experiments test Qwen2.5-VL 7B and LLaVA-OneVision 7B, which share a vision-encoder and decoder-only-language-model architecture but differ in visual and spatiotemporal encoding.The shared architecture permits the same intervention framework across both models.
- Intervention framework: Attention Knockout blocks selected pre-softmax attention connections by setting targeted scores to −∞, allowing controlled tracing of visual-to-text information flow.The main direction blocks attention from textual queries to visual tokens, while masking constraints are applied globally across layers.
- Intervention framework: Region-based masking traces video influence on prompt components, whereas token-level masking isolates the roles of nouns, verbs, prepositions, and category-critical tokens within answer options.Category-critical tokens are the lexical elements distinguishing each caption from its foil and defining the targeted reasoning category.
- Scoring and behavioral check: Full-sentence scoring matches A/B-label scoring on Spatial and Causal items but produces substantially lower Temporal accuracy, with performance dropping below chance.The comparison motivates caution when interpreting temporal results under autoregressive sentence scoring.
4 Results
Layer-wise interventions show that visual information is integrated mainly through textual candidate options, with nouns serving as semantic anchors and verbs becoming more involved in temporal reasoning. Temporal decisions follow a distinct, more fragile pattern that may also reflect biases tied to temporal expressions.
- Region-based analysis: The text-only condition consistently underperforms the full-input condition across both models and all three subsets, indicating that linguistic priors alone are insufficient.The comparison uses final-layer accuracy between text-only and the multimodal none condition.
- Region-based analysis: Temporal caption–foil decisions are determined later and are more fragile than Spatial/Causal decisions, with incorrect cases driving the apparent late-layer reversal.The averaged temporal trajectory reflects a late shift toward the foil in the incorrect split.
- Region-based analysis: Visual information is primarily integrated while the model processes candidate options, which act as textual grounding sites for final decisions.Direct vision-to-assistant masking has minor effects, whereas visually enriched textual tokens remain influential into later processing.
- Token-level analysis: Across models and subsets, nouns are consistently among the most affected token categories, linking visual input to grounded entities in candidate sentences.These lexical elements function as semantic anchors between visual input and linguistic representations.
- Token-level analysis: Temporal cases differ from Spatial and Causal cases, showing irregular layer-wise effects and stronger dependence on verbs.This pattern is consistent with temporal items requiring action, event-dynamics, and ordering information across video frames.
- Temporal analysis: Temporal-marker preferences reveal construction-specific linguistic biases, including strong text-only preference for Before over After and below-chance When and While performance with visual input.For Qwen and LLaVA, text-only Before versus After preferences are 0.82 vs. 0.42 and 0.75 vs. 0.36; When and While remain below chance in the none condition.
5 Conclusion
The study traces how VLMs use visual evidence in video-based decisions through layer-wise Attention Knockout across spatial, causal, and temporal reasoning. It examines where visual information enters textual representations.
- Conclusion: Layer-wise Attention Knockout traces video-to-text information flow by selectively blocking visual-to-text attention pathways across spatial, causal, and temporal reasoning.The analysis focuses on where and how visual information enters textual representations.
Limitations
The study is limited by its evaluation scope and scoring design: it uses only two relatively small VLMs and does not systematically compare A/B-label with full-sentence scoring.
- Scope: The evaluation includes only two relatively small VLMs, limiting generalizability across the rapidly evolving multimodal-model landscape.The authors frame this as a controlled diagnostic analysis rather than a leaderboard comparison.
- Scoring: The mechanistic analysis uses sentence-level scoring and does not systematically analyze differences from A/B-label scoring.Sentence-level scoring is required to access candidate token structure and measure changes in full-option plausibility under interventions.
A.1 Text-to-vision Masking Direction
The text-to-vision analysis reverses the usual information-flow direction using an inverted Text-Video prompt, but remains a complementary, weaker diagnostic with effects concentrated in broader textual context.
- Intervention design: An inverted Text-Video prompt makes text-to-vision intervention well-defined because visual tokens can attend to preceding textual tokens under causal attention.The intervention blocks visual-token attention to selected textual regions, including instructions and answer options.
- Model scope: The analysis is reported only for Qwen2.5-VL because LLaVA-OneVision reconstructs inputs into its default Video-Text order.That template prevents reliable application of the inverted prompt required for this intervention.
- Results: Text-to-vision masking produces weaker and less localized effects than the main vision-to-text analysis, with the broad user-text region showing the clearest changes.Masking isolated option A or B regions produces only marginal changes, while both options remain included in the broader user-text span.
- Interpretation: The reverse-direction analysis is treated as a complementary control rather than a fully symmetrical counterpart to the main experiments.Its interpretation may be influenced by the non-standard prompt order used to make the intervention valid under causal attention.
- Temporal pattern: Temporal items show behavior distinct from Spatial and Causal items, supporting a qualitatively different temporal challenge and compatible but weaker reverse-conditioning effects.The main evidence still points to a dominant video-to-textual-representation pathway.
A.2 Cross-lingual analysis
English translations broadly reproduce the Italian information-flow trends, especially the central role of candidate-option regions, while temporal processing shows language-sensitive differences.
- Overall comparison: English experiments on 200-item samples from each subset confirm the main Italian trends overall.The translated analysis tests whether findings depend on the original Italian formulation.
- Region-based analysis: Candidate-option regions remain the strongest sites of cross-modal integration when visual access is blocked.This supports their role as the main textual locations where visual information is integrated.
- Token-level analysis: Nouns remain central for grounding in the English token-level analysis, consistent with the original Italian setting.The broader lexical-category pattern remains comparable across languages.
- Temporal analysis: English Temporal dynamics differ partly from Italian: layer-wise behavior is more stable, models retain opposite final tendencies, and verbs have reduced impact.The authors caution that these observations do not support definitive conclusions and may reflect linguistic formulation.
A.3 Correct Vs. Incorrect Split Analysis
Splitting results by final caption–foil preference clarifies that Causal decisions separate coherently, Spatial errors can resolve late, and the apparent Temporal inversion is driven mainly by incorrect cases.
- Causal items: For Causal items, correct cases increasingly favor the caption in final layers, whereas incorrect cases retain a stable foil preference.This produces a coherent layer-wise separation between correct and incorrect trajectories across both models.
- Spatial items: Correct Spatial cases converge toward the caption in final layers, but incorrect cases show unstable late trajectories and may resolve only near the end.Both models move toward the caption around layers 22–25 before shifting toward the foil at the final layer.
- Temporal items: The apparent Temporal inversion in averaged heatmaps is driven mainly by cases where the model ultimately prefers the foil.Correct Temporal cases become caption-favored by the final layer, while incorrect cases become foil-favored only late in processing.
- Temporal items: Temporal reasoning is more fragile because successful cases stabilize on the caption late, while unsuccessful cases make the foil dominant only in final layers.The pattern is consistent with difficulty preserving and exploiting sequential information across the video until late computation.
B Additional Materials
The appendix provides additional materials, including the original Italian caption–foil pair examples used in the study.
- Table 5 presents the original Italian caption–foil pair examples that Section 3.1 presented in English.
C Experiments
The experiments include implementation details, model descriptions, and an analysis of temporal-item behavior across layers and token regions. The appendix also documents the authors’ use of ChatGPT for grammar, spelling, and coding assistance.
- The appendix reports that Attention Knockout experiments on subsets of MAIA used A100 GPUs with 40GB of memory.
- Figure 14 compares caption–foil ∆log-probability across layers and token regions for correct and incorrect temporal-item predictions.
- LLaVa-OneVision is a 7B-parameter vision-language model built on the LLaVA framework with a Qwen2 language-model backbone.
- Qwen2.5-VL is a 7B-parameter Qwen-family vision-language model designed to improve grounding, longer-video processing, and event capture.
- The authors used ChatGPT for grammar and spelling checks and coding assistance, then reviewed and edited the resulting content.