PaperlayerView exact revision on arXiv

Source-linked AI summary

Attending to Multimodal Generation One Token at a Time

Varun Gupta, Vineet Gandhi, Makarand Tapaswi

arXiv:2607.03738v1cs.CVcs.AI

TL;DR

The paper addresses the limited understanding of token-level multimodal computation during autoregressive generation. It analyzes evolving attention across semantic context roles and validates the patterns with causal interventions, finding consistent routing dynamics and improved performance from targeted attention boosting.

  • Problem

    Token-level interpretability of how MLLMs route visual and linguistic information during evolving autoregressive generation remains underexplored.

  • Method

    The paper tracks attention to image, text, instruction, and previously generated tokens during decoding, then applies causal blocking and test-time attention boosting interventions.

  • Results

    Across diverse model families, attention follows semantic role, image attention peaks for image-relevant concepts, instruction tokens recur at task transitions, and generated-token attention increases over time.

  • Takeaways & Limitations

    Token-level attention dynamics provide a complementary perspective for interpreting MLLMs and can inform interventions that improve multimodal task performance.

  • Takeaways & Limitations

    Raw attention scores can be high for possibly unimportant tokens, so normalized scores are needed when interpreting changes across timesteps.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context. Prior work on interpretability has focused on individual layers and circuits (where), leaving the token-level dynamics of multimodal computation during generation (when) underexplored. We address this gap and study attention shifts as per semantic role; tracking model attention to image, text, instruction, and previously generated tokens, One Token at a Time (OTaT). We introduce multimodal tasks that require explicit switching between visual and textual context within a single response. Across two mainstream model families and four open-weight MLLMs of varying sizes, we establish consistent patterns: attention to image peaks at tokens requiring image-derived information, instruction tokens are revisited during task transitions, and attention to previously generated tokens increases as the generation progresses. Causal attention blocking interventions validate the functional role of these trends. We profile model behavior under disrupted attention and observe responses falling back to language priors, or exhibiting cross-modal leakage, denial, or recovery. Finally, informed of the attention dynamics through our novel analysis, we propose a simple test-time intervention to boost attention to the relevant modality at the right time, significantly improving multimodal task performance.

1 Introduction

The paper studies how multimodal attention unfolds token by token during autoregressive generation, complementing analyses of individual components. It finds consistent, causally validated attention patterns and uses them to improve multimodal performance.

  • Research gap: Token-level attention dynamics remain underexplored because MLLMs generate responses iteratively using an evolving context.Prior interpretability work has emphasized individual heads, layers, and circuits rather than temporal multimodal computation.
  • Approach: The study tracks how attention patterns evolve during decoding, complementing finer-grained component-level analysis.It examines how computation unfolds as models generate responses one token at a time.
  • Findings: Attention to image tokens peaks for image-relevant concepts, while instruction tokens are revisited when models switch between tasks.These findings are reported as consistent across diverse model families and generalize beyond the base task.
  • Findings: Attention to previously generated tokens increases progressively, showing that models leverage their own outputs during generation.The paper presents this as supporting continued fluency as decoding progresses.
  • Interventions: Causal attention blocking validates the functional role of these attention patterns, while disrupted image access produces language-prior reliance, leakage, denial, or recovery.The study profiles these behaviors across models under blocked information flow.
  • Application: A simple test-time strategy that boosts attention to relevant modalities at critical decoding steps significantly improves multimodal performance.The intervention selectively amplifies modality-specific attention during generation.

2 Method

The method tracks how the currently generating token attends to semantic chunks across autoregressive decoding, then uses blocking and boosting interventions to test and alter their functional roles.

  • Autoregressive multimodal context: MLLMs process image embeddings, text embeddings, and previously generated tokens in an evolving autoregressive context.At each step, the decoder predicts the next token from the input and prior outputs under causal masking.
  • Attention extraction: OTaT quantifies the currently generating token’s attention to static and dynamic semantic chunks at each decoding step.The dynamic context contains previously generated tokens and the currently generating token, while static chunks include image, text, and instruction tokens.
  • Attention extraction: Chunk attention is computed by averaging over layers and heads, then summing attention over the chunk’s token span.The resulting aggregation produces per-token, modality-level attention signals across decoding.
  • Attention normalization: Mean-subtracting chunk attention across decoding steps and samples removes baseline effects and improves trend interpretability.The paper notes that raw attention can make semantically uninformative tokens appear highly important.
  • Attention interventions: Blocking interventions test functional importance, while boosting multiplicatively reweights a selected chunk’s attention and renormalizes the scores.Total blocking prevents information flow from a chunk to any future token, unlike lazy blocking, which only blocks the currently generating token.
  • Attention interventions: The framework blocks attention at output-tag-specific steps to assess how instruction following, grammatical continuation, and task solving depend on different chunks.The analysis compares lazy and total blocking and examines image, text, instruction, and other semantic chunks.

3 Experimental Setup

The experiments evaluate attention dynamics on multimodal tasks designed to combine or contrast visual and textual information, using four open-weight models across two model families and automatic response evaluation.

  • Multimodal tasks: The task suite includes Fruit-Math, Visual Spatial Reasoning, and ChartQA, covering image recognition, conflicting visual-textual relations, and diagram-based questions.The tasks require models to switch between or integrate visual and textual information within responses.
  • Multimodal tasks: Fruit-Math pairs fruit images with unrelated GSM-8K math puzzles to isolate visual and textual contributions.The model must identify the fruit and solve the math puzzle, with non-overlapping modality information reducing redundancy as a confounder.
  • Models and generation: Experiments use LLaVA-OneVision models at 0.5B and 7B parameters and Qwen2.5-VL models at 3B and 7B parameters.The models differ in base language models, parameter counts, input ordering, training strategies, data, and methods.
  • Evaluation and aggregation: Responses are grouped by output-token tags and averaged within tags to produce robust dataset-level attention observations despite varied lengths and wording.A prescribed response format supports aggregation, while per-sample attention remains independently analyzable.
  • Evaluation and aggregation: Gemini 2.5 Pro evaluates model answers, with spot-checks used to assess the reliability of the automatic evaluation procedure.

4 Results and Findings

The results show that MLLMs dynamically shift attention among image, text, instruction, and previously generated tokens as decoding unfolds. Causal blocking disrupts modality-specific behavior, while timestep-aware boosting improves multimodal performance.

  • Attention Orchestrates Autoregression: MLLMs dynamically reallocate attention to the modality relevant to the concept being generated.Image attention peaks for Fruit Concept, whereas text attention peaks for Math Answer; reversing task order reverses these trends.
  • Attention Orchestrates Autoregression: Models revisit instruction tokens at task transitions, especially around the Handoff between visual and arithmetic subtasks.Instruction attention also spikes at the start of generation, supporting multi-answer responses.
  • Attention Orchestrates Autoregression: Attention to previously generated tokens increases consistently as decoding progresses, supporting grammatical structure and format adherence.The same broad trends appear across additional models and in language-only models with two textual contexts.
  • Attention Orchestrates Autoregression: In ChartQA, image attention recurs when answering both questions, while previous-token attention rises and instruction attention spikes near handoff.These findings extend the attention patterns beyond the disentangled Fr-Ma setting.
  • Attention Orchestrates Autoregression: Raw attention can be high for uninformative tokens, but normalization makes timestep changes more interpretable and reduces their apparent impact.The unexplained first-token score becomes close to zero after normalization.
  • Effects of Blocking Information Flow: Blocking image attention at critical visual tokens causes language-prior fallback, cross-modal leakage, denial, or recovery behaviors.For LOV-0.5B, 50-60% of responses contain leaked terms; stronger models can sometimes recover after disruption.
  • Effects of Blocking Information Flow: Targeted timestep-aware boosting produces the best improvements across vision, text, and both accuracy, whereas role-agnostic boosting harms task performance.Reported gains for the targeted intervention are LOV-7B +28.5%, QVL-3B +8.6%, and QVL-7B +2.7%.

5 Related Work

Related work has examined multimodal representations, attention, autoregressive behavior, and test-time interventions, but token-level temporal interpretation remains distinct from these efforts. The paper situates its analysis within causal attention studies and existing intervention strategies.

  • Multimodal interpretability: Prior multimodal interpretability work studies how visual information is distributed, aligned with language, and transformed across layers.Existing findings disagree on whether visual grounding occurs early, progressively, or is attenuated in later layers.
  • Multimodal interpretability: Controlled benchmarks address representation entanglement, while this work uses Fr-Ma and extends analysis to ChartQA and VSR.Fr-Ma is designed so visual and textual modalities contribute independently.
  • Autoregressive dynamics: Existing autoregressive analyses include token-level spatial explainability, textual-importance tracking for pruning, and semantic-chunk attention dynamics.These efforts motivate studying evolving internal attention during multimodal generation.
  • Test-time attention intervention: Test-time interventions modify logits or post-softmax attention, with intervention location in the architecture remaining a central design choice.The paper adopts a causal perspective by validating attention patterns through blocking before applying boosting.

6 Conclusion

Across multiple tasks and models, the paper finds consistent token-level attention dynamics during autoregressive multimodal generation. These patterns also appear in text-only models and motivate attention-based analysis and intervention.

  • 6 Conclusion: Across multiple tasks and open-weight MLLMs, relevant-context attention peaks were disrupted to produce language priors, leakage, denial, or degraded outputs.Previously generated-token attention increased over time, and blocking it caused incomplete or repetitive outputs.
  • 6 Conclusion: Image attention peaks for fruit concepts, text attention peaks for math answers, and instruction attention rises during task handoff across additional models.The same global trends held despite local model variance, while the smallest model showed roughly 4% math accuracy.
  • 6 Conclusion: Layer-stratified analysis finds the strongest task-specific spikes in late and sometimes middle layers, while directional attention trends remain consistent across depth.Early layers show a flatter profile, so the global summary mainly reflects later predictive layers.
  • 6 Conclusion: Text-only Qwen2-7B models reproduce MLLM-like attention patterns across disjoint math and sport tasks.Attention spikes to sport text for sport concepts and math text for math answers, alongside increased previous-token and handoff-instruction attention.

A.5 Qualitative Results on Fruit Prediction: Blocking Information Flow from Image

Blocking image information during fruit prediction causes varied failures, including cross-modal leakage, language-prior responses, denial, and partial recovery. Smaller models may copy nouns from the math puzzle into their fruit predictions.

  • A.5 Qualitative Results on Fruit Prediction: Blocking Information Flow from Image: Image-attention blocking causes accuracy drops across models, with qualitatively different failure modes.The reported examples include language-prior reliance, cross-modal leakage, denial, and recovery.
  • A.5 Qualitative Results on Fruit Prediction: Blocking Information Flow from Image: QVL-3B sometimes recovers the correct fruit after image blocking, but its outputs can contain repetition, placeholders, or unnecessary details.Examples include “not apples; it is a crate of apples” and “not specified, but appears to be plums.”

A.6 Qualitative Results: Other Types of Blocking

Blocking different semantic chunks disrupts generation in distinct ways. Previous-token blocking causes repetition or collapse, while instruction blocking prevents the handoff from fruit identification to math solving.

  • A.6 Qualitative Results: Other Types of Blocking: QVL-3B shows denial, language-prior reliance, and some recovery after image-information blocking.These behaviors complement the failures observed when previously generated or instruction tokens are blocked.
  • A.6 Qualitative Results: Other Types of Blocking: Blocking previously generated tokens causes stuttering, repetition, premature endings, and inability to continue to the next task.Qwen2.5-VL shows greater resilience than the other compared model in quantified results.
  • A.6 Qualitative Results: Other Types of Blocking: Blocking instruction attention at the handoff prevents models from producing the math response after fruit identification.The affected outputs either omit the math answer or end prematurely.

A.7 Boost Factor Analysis and γ Computation

The appendix examines attention boosting across hyperparameters, computes intervention scaling values, and reports qualitative evidence that targeted boosting can improve modality-specific reasoning. It also clarifies how token order and attention normalization affect interpretation.

  • A.7 Boost Factor Analysis and γ Computation: LOV performance peaks at 49.8 for β = 50 and remains within 1.1 points of its maximum for β ∈[20, 100].QVL varies by less than 1 point between β = 30 and β = 400, with a shallow optimum around 150–300.
  • A.7 Boost Factor Analysis and γ Computation: For VSR, the reported intervention calculations produce γ_i = 10.62 for LOV-7B and γ_i = 13.08 for QVL-7B.The values use β = 50 for LOV-7B and β = 200 for QVL-7B.
  • A.7 Boost Factor Analysis and γ Computation: Targeted boosting improves both image-based and text-based spatial reasoning accuracy for LLaVA-OneVision, while Qwen2.5-VL shows mixed gains.Some Qwen2.5-VL examples improve text reasoning, or both image and text reasoning.
  • A.7 Boost Factor Analysis and γ Computation: Answer-bearing ChartQA tokens show substantially stronger image attention than introductory tokens, with weaker text-attention separation; reported differences are statistically significant.The comparisons use mean attention differences, where positive values favor the first group.
  • A.7 Boost Factor Analysis and γ Computation: LLaVA-OneVision places image tokens before text and instruction tokens, whereas Qwen2.5-VL places text and instruction before image tokens.Because decoder attention is causal or masked, token order determines information flow and can affect interpretation.

B.2 Dataset Details

The appendix documents the fruit and sport datasets, token-role tagging, and rule-based procedures for aligning attention blocking with autoregressive generation.

  • Dataset Details: The fruit dataset contains 11 categories, shown with their respective counts in Figure 22.
  • Dataset Details: The sport identification task uses excerpts generated by Gemini 2.5 Pro for nine sport classes, without naming the sport explicitly.
  • Dataset Details: Dataset-level attention trends are compiled by automated POS tagging that groups token roles and aggregates them across tasks.
  • Dataset Details: The Fruit-Math response format combines a fruit description with a numeric math answer in one sequence.The expected format is illustrated using an apple example with math answer 7.
  • Dataset Details: Blocking experiments use the previous generated token to identify when the next token should depend on a targeted semantic chunk.For fruit-concept blocking, attention is blocked when the previous token is “is,” before the fruit name is generated; step constraints can restrict the first matching occurrence.
  • Dataset Details: The rule-based blocking procedure scales from individual examples to dataset-level analysis of attention changes over time.

C Limitations and Future Work

The paper identifies limitations involving evaluation scope, attribution of model differences, intervention settings, computational cost, and modality coverage.

  • Limitations: Complete free-form, in-the-wild generation remains unexplored because generation-centric evaluation and causal intervention are difficult to scale.
  • Limitations: The causes of differing disruption behaviors across models remain difficult to attribute to architecture or training process.The paper contrasts frequent visual-content denial by Qwen2.5-VL with its absence in LLaVA-OneVision models.
  • Future Work: Decoding strategies and model-specific optimal boost factors remain underexplored for role-aware attention boosting.The reported intervention was evaluated globally across all layers and heads.
  • Limitations: Attention analysis requires eager attention mode, which materializes the full attention matrix and is slower and more compute-intensive than optimized implementations such as FlashAttention.
  • Future Work: The paper’s multimodal interpretability scope is limited to vision-language interactions, leaving other modalities for future work.

F Compute Resources

The work is described as a training-free analysis and intervention effort using NVIDIA A6000 GPUs, with supplementary prompts supporting tagging and evaluation across tasks.

  • Compute Resources: The analysis and intervention are training-free and do not require a large-scale compute setup.
  • Compute Resources: All experiments were conducted on NVIDIA A6000 GPUs with 48GB of VRAM.
  • Compute Resources: Supplementary prompts define semantic token tagging and task-specific evaluation procedures for the reported experiments.The materials include prompts for Fruit-Math, ChartQA, VSR, and Math-Sport, plus evaluation prompts for fruit, math, and spatial-relation responses.
Loading 2607.03738v1…