PaperlayerView exact revision on arXiv

Source-linked AI summary

See What You Are Told: Visual Attention Sink in Large Multimodal Models

Seil Kang, Jinyeong Kim, Junhyeok Kim, Seong Jae Hwang

arXiv:2503.03321v1cs.CVcs.AI

TL;DR

LMMs often assign high attention to irrelevant visual tokens, raising questions about their role in image understanding. The paper identifies these tokens as visual attention sinks and introduces VAR to redirect their surplus attention through image-centric heads. Across multiple task types, VAR improves performance without extra training, models, or inference steps, while the authors note limitations of raw attention-map grounding and evaluation on different larger LMMs.

  • Problem

    LMMs consistently attend to irrelevant visual tokens in fixed locations, but the cause and meaning of this behavior remain open questions.

  • Method

    The paper identifies visual sink tokens through extreme hidden-state activations and introduces VAR, which redistributes their surplus attention to visual tokens in image-centric heads.

  • Results

    VAR reliably improves performance across general vision-language, visual hallucination, and vision-centric benchmarks without benchmark-specific hyperparameter tuning.

  • Takeaways & Limitations

    Editing LMM attention maps can intensify focus on relevant image information and improve multimodal capability without additional training, models, or inference steps.

  • Takeaways & Limitations

    The method uses raw visual attention maps, which may not perfectly ground objects, and further evaluation on different and larger LMMs remains to be explored.

Abstract

from arXiv · show

Large multimodal models (LMMs) "see" images by leveraging the attention mechanism between text and visual tokens in the transformer decoder. Ideally, these models should focus on key visual information relevant to the text token. However, recent findings indicate that LMMs have an extraordinary tendency to consistently allocate high attention weights to specific visual tokens, even when these tokens are irrelevant to the corresponding text. In this study, we investigate the property behind the appearance of these irrelevant visual tokens and examine their characteristics. Our findings show that this behavior arises due to the massive activation of certain hidden state dimensions, which resembles the attention sink found in language models. Hence, we refer to this phenomenon as the visual attention sink. In particular, our analysis reveals that removing the irrelevant visual sink tokens does not impact model performance, despite receiving high attention weights. Consequently, we recycle the attention to these tokens as surplus resources, redistributing the attention budget to enhance focus on the image. To achieve this, we introduce Visual Attention Redistribution (VAR), a method that redistributes attention in image-centric heads, which we identify as innately focusing on visual information. VAR can be seamlessly applied across different LMMs to improve performance on a wide range of tasks, including general vision-language tasks, visual hallucination tasks, and vision-centric tasks, all without the need for additional training, models, or inference steps. Experimental results demonstrate that VAR enables LMMs to process visual information more effectively by adjusting their internal attention mechanisms, offering a new direction to enhancing the multimodal capabilities of LMMs.

1 INTRODUCTION

LMMs use transformer-decoder attention between text and visual tokens to incorporate image information, but they also consistently attend to irrelevant visual tokens. The paper identifies these tokens as visual attention sinks and proposes recycling their surplus attention through VAR.

  • LMMs combine a pretrained visual encoder with a language-model transformer decoder to generate text responses for vision-language tasks.
  • Attention weights between text and visual tokens indicate which image information each text token uses.
  • Irrelevant visual tokens receive high attention weights and recur in fixed locations across instructions and generated responses.
  • Specific hidden-state dimensions massively activate in irrelevant tokens, which resemble language-model attention sinks; removing them does not significantly affect response quality.
  • VAR recycles attention assigned to sink tokens and redistributes it toward images in image-centric heads.
  • VAR improves LMM performance across general vision-language, visual hallucination, and vision-centric tasks without additional training, models, or inference steps.

2 RELATED WORK

Prior work characterizes visual attention as the mechanism connecting image information to text responses and identifies unintuitive attention patterns in multimodal and language models. The paper extends the language-model attention-sink concept to irrelevant visual tokens in LMMs.

  • LMM visual attention maps represent how text tokens interact with image tokens while incorporating visual information into responses.
  • Language-model attention sinks are semantically limited tokens that receive disproportionately high attention despite contributing minimally to predictions.
  • Prior analyses associate attention sinks with massive activation in specific hidden-state dimensions before high attention allocation.
  • The paper extends this attention-sink concept to visual tokens and redistributes sink-token attention in selected heads.

3 PRELIMINARIES

LMMs process system instructions, visual features, and text through transformer blocks containing multi-head attention and feed-forward networks. Visual attention maps expose how text tokens attend to visual tokens, including irrelevant and relevant locations.

  • An LMM typically contains a visual encoder, projector, and large language model that processes image features alongside system instructions and text.
  • Each transformer block contains multi-head attention and a feed-forward network operating on token representations.
  • The analysis partitions input-token indices into system, visual, and text-token sets.
  • Visual attention maps distinguish relevant visual tokens associated with a text token from irrelevant visual tokens that receive attention without that association.
  • Attention from visual tokens as keys to text tokens as queries is represented as a visual attention map.

4 VISUAL ATTENTION SINK

LMMs consistently assign high attention to irrelevant visual tokens that occupy fixed locations, and these tokens exhibit attention-sink-like hidden-state activations. Experiments indicate that masking them has little effect because their contribution to the model’s residual stream is low.

  • Irrelevant visual tokens recur in fixed locations regardless of the corresponding text token, distinguishing them from text-relevant visual tokens.
  • Specific hidden-state dimensions show massive activation in irrelevant visual tokens, matching the dimensions activated by the language-model BOS sink token.
  • Sink dimension values separate high-attention visual tokens into low- and high-value groups, enabling identification of irrelevant visual tokens.
  • Visual sink tokens are defined using a threshold on sink-dimension values, with τ = 20 in the subsequent analysis.
  • Masking visual sink tokens has little performance impact, whereas masking the same number of random visual tokens causes a significant performance drop.
  • Visual sink tokens make significantly smaller attention contributions to text-token residual streams than other visual tokens and are often located in less-informative backgrounds.

5 VISUAL ATTENTION REDISTRIBUTION

Visual Attention Redistribution (VAR) selects heads that focus on important visual information and reallocates surplus attention from visual sink tokens to visual non-sink tokens. The method preserves the attention sum while concentrating redistributed attention in selected image-centric heads.

  • 5 VISUAL ATTENTION REDISTRIBUTION: VAR has two steps: selecting image-centric heads using visual attention sink information and redistributing sink-token attention to visual non-sink tokens.
  • 5.1 SELECTING IMAGE-CENTRIC HEADS: Heads with visual attention to image tokens below 0.2 are discarded before selecting image-centric heads using the visual non-sink ratio.
  • 5.1 SELECTING IMAGE-CENTRIC HEADS: Heads with high visual non-sink ratios tend to focus on visual tokens relevant to the text token, whereas low-ratio heads show vague or scattered attention patterns.
  • 5.1 SELECTING IMAGE-CENTRIC HEADS: The threshold ρ controls how many heads are selected as image-centric heads, with heads satisfying rℓ,h_i ≥ ρ chosen.
  • 5.2 REDISTRIBUTING ATTENTION WEIGHTS: VAR accumulates a portion p of sink-token attention into an attention budget Ω and allocates it to visual non-sink tokens in selected heads.
  • 5.2 REDISTRIBUTING ATTENTION WEIGHTS: The redistribution keeps the total attention sum equal to 1 and applies to all text tokens, including instructions and generated responses.

6 EXPERIMENTS

VAR is evaluated across diverse vision-language, hallucination, and vision-centric benchmarks, where it consistently improves base-model performance without benchmark-specific tuning. Ablations and qualitative analyses indicate that selecting image-centric heads and reallocating surplus sink-token attention toward visual tokens are important to its effectiveness.

  • Main results: VAR improves base-model performance across general vision-language, visual hallucination, and vision-centric benchmarks without benchmark-specific hyperparameter tuning.The method is applied to multiple LMMs and task categories using shared settings within each task type.
  • Main results: LLaVA-1.5-7B with VAR outperforms vanilla LLaVA-1.5-13B on GQA, VizWiz, MME, and POPE.This comparison suggests performance gains from increasing image focus without increasing model size.
  • Analyses and discussions: VAR qualitatively intensifies attention on key visual tokens related to each text token, enabling more accurate responses by focusing more effectively on the image.The method can also be incorporated into existing approaches such as visual contrastive decoding.
  • Ablation studies: Redistributing attention across all heads can severely impair model function, whereas selecting image-centric heads is crucial for performance improvement.The all-head variant received 0.0 scores in the reported ablation, while head selection preserves the intended attention-head roles.
  • Ablation studies: Redistribution to visual tokens is more effective than redistribution to text tokens or to both visual and text tokens.Text-only redistribution yields little improvement or decreases performance, while visual-only redistribution best supplements insufficient visual attention.
  • Analyses and discussions: VAR remains robust across reasonable hyperparameter ranges, with consistent τ and p choices across tasks and one ρ value shared within each task type.The reported settings are τ = 20, p = 0.6, and task-specific ρ values of 0.8, 0.5, and 0.9.

7 CONCLUSION

The paper identifies visual attention sinks as irrelevant visual tokens associated with massive activation in specific hidden dimensions, analogous to language-model attention sinks. It proposes VAR to recycle their surplus attention and reports improved multimodal performance without additional training, models, or inference steps.

  • Conclusion: VAR reallocates attention from visual sink tokens to visual information relevant to the corresponding text tokens.The method is presented as recycling surplus attention budget rather than adding model capacity.
  • Conclusion: Experimental results show that VAR improves LMM performance across general vision-language, visual hallucination, and vision-centric tasks.The reported gains are achieved by editing the attention map rather than adding training, models, or inference steps.
  • Conclusion: The sink dimensions in LLaVA-1.5-7B remain D_sink = {1415, 2533} after multimodal fine-tuning, matching its base language model.This supports the reported similarity between visual attention sinks and language-model attention sinks.
  • Conclusion: Visual sink tokens have massive hidden-state activation in the same dimensions as the BOS token across various LMMs.The shared activation pattern appears across models with different architectures, training schemes, and scales.

A.2 FURTHER ANALYSES AND DISCUSSIONS ON VISUAL ATTENTION SINK

Further analyses find that visual sink tokens are semantically less meaningful, tend to occur in image backgrounds, and emerge early before persisting through later layers. The paper leaves their selection and training-time emergence as open questions.

  • Location of visual sink tokens: Visual sink tokens are more likely than randomly selected visual tokens to be located in background regions.Table 6 reports this comparison for LLaVA-1.5-7B.
  • Location of visual sink tokens: 82.9% of visual sink tokens and 90.5% of all visual tokens are reported in the provided location analysis.The surrounding text compares the proportions of sink tokens and all visual tokens located outside the main subject regions.
  • Layer-wise analysis: Massive hidden-state activation emerges in early LMM layers, and visual sink tokens persist until the last layer.In LLaVA-1.5-7B, the activation emerges in layer 2 in the cited analysis.
  • Location of visual sink tokens: Visual sink tokens are described as semantically less meaningful and are argued to be unrelated to the image’s main subject.The analysis evaluates their locations using Pascal-VOC and MS-COCO segmentation data.
  • Further discussions and future works: The mechanisms by which LMMs select visual sink tokens and develop visual attention sinks during training remain open questions.The authors suggest the phenomenon may be inherited from pretrained language models because sink dimensions match those of the base models.
  • Further discussions and future works: The reported sink dimensions in multimodal models are identical to those in their base language models.This is presented as supporting evidence for inheritance from language models during multimodal training.

A.3 ANALYSIS ON IMAGE-CENTRIC HEADS

Image-centric heads are selected to identify attention heads that focus on visual information, and ablations show they are important for integrating visual content. Their prevalence also varies with the text token, increasing when visual information is needed.

  • Image-centric heads are selected to emphasize important visual information in LMMs.
  • Ablating image-centric heads causes larger performance drops than ablating random heads.Ablating complementary heads leaves performance relatively stable across selection thresholds.
  • Image-centric heads selected by the visual non-sink ratio are essential for integrating visual information into text responses.
  • Text tokens related to visual information have more image-centric heads.The analysis is conducted on LLaVA-1.5-7B using LLaVA-Bench (In-the-Wild).
  • VAR is also evaluated as an addition to Visual Contrastive Decoding on MME, POPE, and GQA.The combined method improves VCD on all three benchmarks.

B.2 HYPERPARAMETER ANALYSIS

VAR’s hyperparameters are selected with a 10% pseudo-validation sample and show consistent behavior across benchmarks of the same task type. Performance remains robust across reasonable parameter ranges, although the preferred head-selection threshold varies by task.

  • Hyperparameters are tuned on 10% of benchmark samples and then applied to the full benchmark.A separate benchmark within the same task type is used to assess consistency.
  • The same optimal hyperparameter values can be obtained from different benchmarks within the same task type.The optimal values of τ and p are consistent across tasks, while optimal ρ varies across tasks but remains consistent within task types.
  • VAR robustly improves performance across MME, POPE, and averaged CV-Bench 2D+3D within a reasonable hyperparameter range.
  • The best performance is obtained at τ = 20, consistent with separation between irrelevant and relevant visual tokens at that threshold.
  • The suitable ρ varies by task because tasks require different degrees of head attention, but VAR consistently improves performance when ρ ≥0.5.
  • The best value is p = 0.6 across all benchmarks, although other p values can also improve performance.Here p denotes the portion of sink-token attention weights transferred into the attention budget.
  • Overall, VAR is not highly sensitive to specific hyperparameter choices within plausible ranges.

C LIMITATIONS AND FUTURE WORKS

The authors identify limitations in using raw visual attention maps and note that the method should be studied on different and larger LMMs. They also describe visual sink tokens as a basis for future attention-map and attention-budget applications.

  • The method uses raw visual attention maps without post-processing, which may not perfectly ground objects.Errors may arise when the model recognizes an object incorrectly or the visual encoder misinterprets the image.
  • The methodology remains to be explored on different and larger LMMs despite experiments across various models and benchmarks.
  • Masking visual sink tokens may remove noise and produce more reliable and interpretable visual attention maps.
  • The attention budget freed from sink tokens may be used to emphasize image regions according to user interests.
  • Understanding how LMMs identify visual sink tokens during multimodal training is proposed as a future research direction.

D.3 DETAILS OF ANALYSIS

The analysis evaluates sink dimensions, token masking, random-token knockouts, and attribution contributions using specified LMMs and benchmarks. The authors caution that attention knockout can induce out-of-distribution behavior and layer-wise inconsistencies.

  • Sink dimensions are used to separate irrelevant visual tokens from relevant visual tokens in MME visual question-answering samples.The analysis selects 10 samples from each category, totaling 100 samples, and computes sink-dimension values for visual tokens.
  • Token masking blinds the model to visual sink tokens by setting their attention from visual tokens to text tokens to zero.The experiments use τ = 20 to define the sink-token indices.
  • Attention weights are computed from query and key matrices with a causal mask before being passed to subsequent layers.The causal mask prevents attention to future tokens by using negative-infinity mask values.
  • Random-token knockouts mask the same number of tokens as sink-token knockouts, with random selections made independently at each layer.This comparison is evaluated on LLaVA-1.5-7B using POPE.
  • Attention knockout may induce out-of-distribution behavior and layer-wise inconsistencies, limiting how accurately masking reflects token influence.The authors nevertheless regard the masking experiments as informative about visual sink-token influence.
  • Attribution contributions of visual sink tokens are compared with the average contribution of all visual tokens on LLaVA-1.5-7B and POPE.

E ADDITIONAL QUALITATIVE RESULTS

Additional qualitative results show that visual sink tokens attract high attention across diverse text tokens, while VAR improves the relevance and accuracy of generated responses to image content.

  • Visual Attention Maps: Visual sink tokens consistently receive high attention from text tokens regardless of the text content, including instructions and generated responses.Attention maps in LLaVA-1.5-7B show this fixed-location behavior across the text sequence.
  • VAR Qualitative Comparison: LLaVA-1.5-7B with VAR generates more accurate and image-relevant responses than the model without VAR on LLaVA-Bench (In-the-Wild).The qualitative comparison indicates that VAR incorporates visual information more effectively.
  • GPT-4 Evaluation: GPT-4 is used to evaluate LMM performance in open-ended question-answering tasks, with qualitative evaluation results shown in Fig. 15.The figure identifies Assistant 1 as GPT-4 and Assistant 2 as LLaVA-1.5-7B + VAR.
  • Qualitative Examples: The qualitative examples include answers describing people and unusual image scenes.One example answers that a person is standing on a rock near water, while another describes a man standing on a clothesline attached to a yellow car.
Loading 2503.03321v1…