PaperlayerView exact revision on arXiv

Source-linked AI summary

Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs

Minji Kim, Taekyung Kim, Bohyung Han

arXiv:2510.13251v2cs.CV

TL;DR

VideoLLMs’ internal extraction and propagation of temporal information remain less explored despite their use in video question answering. This paper uses mechanistic interpretability to trace information flow across layers and modalities. It finds recurring pathways from cross-frame interactions to video-language integration and answer generation, with sparse pathways sufficient for VideoQA.

  • Problem

    How VideoLLMs internally extract temporal information from videos and propagate it through text tokens to generate answers remains poorly understood.

    Source: §1 · pp. 1–2

  • Method

    The paper uses mechanistic interpretability to trace VideoLLM information flow and selectively masks attention interactions to assess their contributions.

    Source: §1 · pp. 1–2, §2.2 · p. 3

  • Results

    Across VideoQA tasks, reasoning proceeds from early-to-middle-layer cross-frame interactions to temporal-keyword integration and middle-to-late-layer answer generation, with the identified pathways sufficient for accurate answers.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The analysis offers a blueprint for VideoLLM temporal reasoning and identifies sparse information pathways that suffice for solving VideoQA tasks.

    Source: §5 · p. 10

    The analysis uses a knockout window size of k = 9 because narrower windows can allow information to bypass interventions through residual connections.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Video Large Language Models (VideoLLMs) extend the capabilities of vision-language models to spatiotemporal inputs, enabling tasks such as video question answering (VideoQA). Despite recent advances in VideoLLMs, their internal mechanisms on where and how they extract and propagate video and textual information remain less explored. In this study, we investigate the internal information flow of VideoLLMs using mechanistic interpretability techniques. Our analysis reveals consistent patterns across diverse VideoQA tasks: (1) temporal reasoning in VideoLLMs initiates with active cross-frame interactions in early-to-middle layers, (2) followed by progressive video-language integration in middle layers. This is facilitated by alignment between video representations and linguistic embeddings containing temporal concepts. (3) Upon completion of this integration, the model is ready to generate correct answers in middle-to-late layers. (4) Based on our analysis, we show that VideoLLMs retain their VideoQA performance by selecting these effective information pathways while suppressing a substantial amount of attention edges, e.g., 58% in LLaVA-NeXT-7B-Video-FT. These findings provide a blueprint for how VideoLLMs perform temporal reasoning and offer practical insights for improving model interpretability and downstream generalization. Our project page with the source code is available at https://map-the-flow.github.io

1 INTRODUCTION

The paper addresses the limited understanding of how VideoLLMs extract and propagate temporal information internally. It traces this process across layers and reports recurring information-flow patterns across VideoQA tasks.

1 INTRODUCTION

  • VideoLLMs extend vision-language models to spatiotemporal inputs, and VideoQA requires temporal reasoning over multiple frames.The paper frames its analysis as a blueprint for VideoLLM information flow and reports that code and data are available on its project page.Source: §1 · pp. 1–2
  • Prior VideoLLM work has emphasized external model design, while how models internally extract and propagate temporal information remains poorly understood.The paper also asks whether image-based MLLM findings hold for VideoLLMs and what video-text alignment adds beyond image-text pretraining.Source: §1 · p. 1
  • The study uses mechanistic interpretability to investigate how VideoLLMs encode video information, connect it to temporal concepts in questions, and prepare answers.Its analysis examines where effective information pathways occur and whether they suffice for VideoQA.Source: §1 · pp. 1–2
  • VideoQA reasoning begins with cross-frame interactions in early-to-middle layers, integrates video information with temporal question keywords, then conveys it to the last token for answer generation.The model becomes ready to predict correct answers after integration concludes in the middle layers.Source: §1 · p. 2

2 PRELIMINARY

The preliminary section describes how video and instruction tokens enter VideoLLMs and how causal attention processes them. It also introduces Attention Knockout as an intervention for tracing information flow and measuring prediction changes.

2.1 VIDEO LARGE LANGUAGE MODELS (VIDEOLLMS)

  • Video frames are patchified and encoded into video tokens, instruction text becomes word-embedding tokens, and the two sequences are combined for multimodal processing.The video tokens correspond to frame patches produced by a vision encoder.Source: §2.1 · p. 3
  • Each transformer layer projects its input into query, key, and value representations; causal scaled dot-product attention produces head outputs that are concatenated and projected.The attention mask is causal, and the resulting module output is formed from all heads.Source: §2.1 · p. 3

2.2 ATTENTION KNOCKOUT

  • Attention Knockout blocks selected source-to-target token interactions by masking them, and the analysis measures the resulting relative change in the predicted answer’s probability.The method selectively disables attention connections during inference to trace their contribution.Source: §2.2 · p. 3

3 INFORMATION FLOW DYNAMICS IN VIDEOLLMS

VideoQA reasoning proceeds from early-to-middle cross-frame interactions to middle-layer video-language integration and middle-to-late answer generation. Retaining the identified information pathways preserves performance with only marginal accuracy degradation, unlike random sparsification.

3.1 EXPERIMENTAL SETUP

  • The analysis uses five TVBench tasks and correct-answer instances, focusing on LLaVA-NeXT-7B fine-tuned on video instruction data with eight-frame sampling.The paper also extends analyses to other models and tasks in the appendix.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 TEMPORAL INTERACTION WITHIN VIDEO TOKENS

  • Early-to-middle cross-frame interactions consistently affect the video-fine-tuned model, while the image-only model is mostly unaffected, indicating video tuning fosters cross-frame dependency.The analysis inhibits attention to tokens in preceding frames within a nine-layer window.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
  • Blocking cross-frame attention in the first half of the layers drops accuracy by at least 18% on samples correctly answered under full causal attention.The intervention also yields incorrect or contradictory open-ended answers across tasks.Source: §3.2 · p. 5, Table 2 · p. 5

3.3 VIDEO-LANGUAGE INTEGRATION ON TEMPORAL REASONING KEYWORDS

  • Video concepts emerge sequentially: spatial concepts appear early on salient foreground tokens, while temporal concepts develop later on remaining tokens.The temporal representations do not override already stabilized spatial representations.Source: §3.3 · p. 6, Figure 4 · p. 6, Figure 5 · p. 7
  • Cross-frame interactions align temporal question words with relevant video segments; blocking them produces positional bias instead of semantic alignment.Video information reaches true option tokens through direct or question-mediated routes, with route choice varying by task and prompt structure.Source: §3.3 · pp. 7–8, Figure 6 · p. 7, Figure 7 · p. 8

3.4 ANSWER GENERATION BEHAVIOR AT MIDDLE-TO-LATE LAYERS

  • True-option probability rises abruptly around layer 20 after video-to-question information flow completes, rather than gradually competing among options.The authors suggest correct-answer decisions depend on successful video-to-language propagation in preceding middle layers.Source: §3.4 · p. 9, Figure 9 · p. 9

3.5 DOMINANT CONTRIBUTION OF EFFECTIVE INFORMATION FLOW TO VIDEOQA

  • Retaining only 42% of attention edges in LLaVA-NeXT-7B-Video-FT causes only marginal accuracy degradation across TVBench and TOMATO, while random sparsification of the same proportion causes a severe drop.The analysis evaluates performance after retaining only identified effective pathways.Source: §3.5 · p. 9, Table 3 · p. 9

4 DISCUSSION

VideoQA failures may originate in the initial video representation rather than later integration, and disabling cross-frame interaction reveals distinct error patterns. The findings motivate pathway regularization, earlier temporal concepts, reduced static-scene bias, and early-exit strategies.

4 DISCUSSION

  • Failed predictions can reflect inaccuracies in initial video representations even when later cross-modal routing resembles routing in successful samples.Disabling cross-frame interaction decreases the incorrect-option probability in one case, implicating erroneous edge signals; in another, it increases, indicating fallback to static cues.Source: §4 · p. 10
  • The two ablation outcomes distinguish errors carried by cross-frame edges from reliance on uninformative static cues when temporal context is absent.The incorrect-option probability falls in Case 1 and rises in Case 2 when cross-frame interaction is disabled.Source: §4 · p. 10
  • The authors suggest pathway regularization, earlier formation of temporal concepts, reduced static-scene bias, and early exit as directions for improving efficiency or robustness.Early exit is motivated by findings that interactions beyond effective information-flow ranges contribute marginally to the final decision.Source: §4 · p. 10

5 CONCLUSION

The paper describes a staged account of VideoQA reasoning, from early cross-frame representation formation through temporal concept transfer to answer generation. It also situates this analysis among prior work, identifies the benchmarks examined, and points to reproducibility materials and further analyses.

5 CONCLUSION

  • VideoLLMs form spatiotemporal video representations through early-to-middle cross-frame interaction, transfer visual information to aligned temporal keywords, then convey decision-relevant information to the final token in middle-to-late layers.The resulting sparse pathways are reported to suffice for VideoQA.Source: §5 · p. 10

A RELATED WORK

  • The study focuses on VideoLLMs’ internal temporal-reasoning mechanisms, contrasting with prior work on datasets, frame selection, memory banks, and video-token compression.Related interpretability studies examined multimodal information storage, object identification, or cross-modal transfer; this work extends attention knockout and layer-wise logit probing to VideoLLMs.Source: A RELATED WORK · p. 16

APPENDIX

  • The analysis primarily uses TVBench, TOMATO, VCGBench, LongVideoBench, and Video-MME, benchmarks covering varied video understanding and temporal reasoning tasks.The paper also reports publicly available datasets, model configurations, implementation details, and code and models on its project page.Source: REPRODUCIBILITY STATEMENT · p. 11, A RELATED WORK · p. 16

5 CONCLUSION

  • Ground-truth answer probability in open-ended QA starts rising immediately after the middle layers, consistent with answer generation following integration.The conclusion places conveyance of critical information to the final token in middle-to-late layers.Source: §5 · p. 10, Figure C · p. 17

B ANALYSIS ON OPEN-ENDED VIDEOQA

The open-ended VideoQA analysis tests whether information-flow patterns found in multiple-choice tasks also hold when answers must be generated. It reports continued reliance on cross-frame interaction and evolving checkpoints as temporal words are generated.

B ANALYSIS ON OPEN-ENDED VIDEOQA

  • The open-ended analysis asks whether missing explicit temporal option keywords alter information propagation, examining both single-token and multiple-token generation.The section contrasts generated answer vocabulary with selecting from provided options.Source: B ANALYSIS ON OPEN-ENDED VIDEOQA · p. 17

B.1 SINGLE TOKEN GENERATION

  • Disabling early-to-middle cross-frame interactions lowers answer probability in open-ended questions, supporting their role across question-answering formats.The analysis uses sentence-completion prompts and tasks with relatively constrained first answer subwords.Source: §B.1 · p. 18
  • Without explicit candidate answers, open-ended video and question information converges at the last token, either directly or through non-option question tokens.This places the integration checkpoint at the last token rather than at an explicit option token.Source: §B.1 · p. 18

B.2 MULTIPLE TOKEN GENERATION

  • For multiple-token analysis, verbs are used as candidate temporal anchors, with flow traced as each anchor is generated after progressively longer response prefixes.The study detects verbs in VCGBench Temporal QA responses and measures probability changes after attention knockout.Source: §B.2 · p. 19
  • As open-ended generation proceeds, temporal verbs increasingly serve as checkpoints: response-to-last flow grows while video-to-last and question-to-last contributions decline.The route carrying video evidence shifts from video through the question toward the last position at the first anchor to increasingly passing through the generated response as more anchors appear; the reported monotonic trend covers one to three anchors.Source: §B.2 · p. 19

C ADDITIONAL VISUALIZATION OF VIDEO-TO-QUESTION ATTENTION MAPS

  • Across varied TVBench tasks, baseline query tokens attend to semantically aligned video tokens, while disabling cross-frame interaction makes attention maps less adaptive and less able to reflect temporal context.Examples include attention to action changes, scene transitions, and object motion; without cross-frame interactions, maps become indistinguishable across frames.Source: C ADDITIONAL VISUALIZATION OF… · pp. 19–20

D FULL VISUALIZATION OF LOGIT LENS ANALYSIS

The Logit Lens visualization describes when spatial and temporal concepts emerge in video-token representations and how their positions relate to visual regions.

D FULL VISUALIZATION OF LOGIT LENS ANALYSIS

  • Static concepts emerge in earlier layers and temporal concepts in deeper layers; temporal concepts align with relevant foreground regions and emerge mainly on residual tokens.Spatial concepts tend to stabilize on salient regions early rather than being replaced by temporal concepts.Source: D FULL VISUALIZATION OF LOGIT LENS… · p. 20

E SCALABILITY OF OUR FINDINGS

In LLaVA-NeXT-13B-Video-FT, the information-flow patterns identified in the smaller model persist at scale. Temporal interactions build video representations, video information reaches answer positions through question or option tokens, and generation follows video-language fusion.

E SCALABILITY OF OUR FINDINGS

  • Spatial concepts settle on salient regions early, while temporal concepts emerge mainly on remaining tokens rather than replacing stabilized spatial tokens.The concepts are extracted using Logit Lens.Source: E SCALABILITY OF OUR FINDINGS · p. 21
  • Cross-frame interactions in early-to-middle layers consistently affect performance across tasks, matching the pattern in LLaVA-NeXT-7B-Video-FT.Blocking these interactions degrades performance in the VideoLLM, unlike its ImageLLM baseline.Source: E SCALABILITY OF OUR FINDINGS · p. 21
  • Video information reaches the last token through question tokens, while pathways to true option tokens can be direct or pass through non-option question tokens.The balance between option-token pathways varies by task.Source: E SCALABILITY OF OUR FINDINGS · p. 21
  • Generation occurs after video and language information has been fused, primarily through option tokens.The transition occurs at a different layer range than in the smaller model, while the overall pattern remains consistent.Source: E SCALABILITY OF OUR FINDINGS · p. 22

F GENERALIZATION OF OUR FINDINGS ACROSS VIDEOLLMS

The effective information pathways persist across VideoLLM architectures and scales, and remain useful for long-form VideoQA. The findings also describe how cross-frame interactions, video-language integration, and answer generation unfold across models.

F GENERALIZATION OF OUR FINDINGS ACROSS VIDEOLLMS

G.1 QUANTITATIVE VALIDATION OF EFFECTIVE INFORMATION FLOW PATHWAYS ON LONG-FORM VIDEOS

  • 42% of the original attention edges retain competitive LongVideoBench performance, with only a 0.6%p accuracy drop.The evaluation uses LLaVA-NeXT-7B-Video-FT on long-form video understanding.Source: §G.1 · p. 22

G.2 GENERALIZATION OF OUR ANALYSIS TO VARIOUS BENCHMARKS

Tests across long-video and spatial-understanding benchmarks show that information-flow patterns vary with input and question type. Failure analysis and interventions also locate active cross-frame interaction primarily in early-to-middle layers.

G.2 GENERALIZATION OF OUR ANALYSIS TO VARIOUS BENCHMARKS

  • On LongVideoBench, effective layer ranges remain similar to short-video benchmarks with either 8- or 24-frame inputs, though cross-frame and video-to-question effects are relatively smaller.The authors conjecture that not all frames are equally informative and that descriptive questions encourage reliance on textual cues.Source: §G.2 · pp. 26–27
  • Blocking cross-frame interaction causes a clear, consistent drop in action recognition, but a much smaller and more variable effect in spatial perception.Some spatial questions benefit from temporal information, while many can be answered from static scenes.Source: §G.2 · pp. 27–28

G.3 FAILURE CASE ANALYSIS

  • Failed Moving Direction answers are associated with erroneous cross-frame signals or static bias, while cross-modal integration remains similar to successful samples.The analysis attributes the failures mainly to the earlier spatiotemporal building process.Source: §G.2 · p. 28, §G.3 · p. 29

G.4 SIGNAL LEAKAGE CHECK

  • Progressively expanding or reversing the cross-frame knockout causes no additional significant performance drop unless it overlaps early-to-middle layers.This intervention tests whether bypassing pathways leak residual video signals beyond the identified effective range.Source: §G.4 · p. 29

G.5 IMPACT OF BLOCKING CROSS-FRAME ATTENTION IN THE SECOND HALF LAYERS

  • Blocking cross-frame attention in the second half of layers has only a marginal accuracy impact compared with blocking it in the first half.This supports the finding that active temporal interaction occurs in early-to-middle layers.Source: §G.5 · p. 29, §G.3 · p. 29

G.6 ROBUSTNESS OF OUR ANALYSES ON THE CHOICE OF WINDOW SIZE k

The window size used for Attention Knockout affects how readily information can bypass the intervention. Separately, late question-to-last-token attention can amplify both true and false option probabilities after evidence has stabilized.

G.6 ROBUSTNESS OF OUR ANALYSES ON THE CHOICE OF WINDOW SIZE k

  • k=1 yields only marginal probability drops because narrow attention blocks are easily bypassed; wider windows k=5, 9, and 13 produce significant drops.The authors use k=9 as the default, following Geva et al. (2023).Source: §G.6 · pp. 29–30

G.7 DISCUSSION ON THE PROBABILITY INCREASES IN THE LAST LAYERS

  • Late question-to-last-token attention can increase both true and false option probabilities after the model's belief is stabilized.Blocking this pathway only in final layers preserves earlier evidence for the true option while preventing further amplification of false options.Source: §G.7 · p. 30

H IMPLEMENTATION DETAILS

Appendix H describes model and training configurations alongside interpretability-analysis procedures. Appendix I states that LLMs were used only for manuscript polishing, not research ideation or experimental design.

H IMPLEMENTATION DETAILS

  • Video instruction tuning uses 874k VideoChat2-IT-derived samples; vision encoders are frozen while MLP projectors and LLM backbones are fully fine-tuned.The samples cover VideoQA, captioning, reasoning, classification, and conversation.Source: H IMPLEMENTATION DETAILS · p. 30
  • VideoLLaMA3-7B is used without fine-tuning with 8×12×12 video tokens; fine-tuned LLaVA-NeXT and Mini-InternVL models use sampled frames and model-specific training recipes.LLaVA-NeXT-13B follows the 7B training recipe and video-token sampling strategy; Mini-InternVL uses 8×16×16 video tokens.Source: H IMPLEMENTATION DETAILS · pp. 30–32
  • Attention Knockout measures how disabling attention connections changes answer probability, using a 9-layer window and tracking the first answer subword unless otherwise stated.The reported change is relative to the answer’s baseline probability.Source: H IMPLEMENTATION DETAILS · p. 32
  • Logit Lens traces video-token representations through the language-model head against spatial and temporal keywords parsed from Action Sequence questions.Keywords are lowercased and converted to present tense to reduce interference from later linguistic completion.Source: Table D · p. 31, H IMPLEMENTATION DETAILS · p. 32
  • Effective pathways are selected from Action Antonym intervals with probability drops below −5%; designated cross-frame and video-to-question-to-answer flows are enabled only in those layers.Video-to-answer and answer-to-answer connections are disabled across layers, with late-layer flows to video and question tokens blocked.Source: H IMPLEMENTATION DETAILS · pp. 31–32

I THE USAGE OF LARGE LANGUAGE MODELS

  • The authors report using LLMs only for sentence-level manuscript polishing, with initial writing, research ideation, experimental design, methods, and findings attributed to the authors.Source: I THE USAGE OF LARGE LANGUAGE MODELS · p. 32

Related papers

Loading 2510.13251v2…