PaperlayerView exact revision on arXiv

Source-linked AI summary

ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention

Wenjie Liu, Hao Wu, Xin Qiu, Xudong Wang, Yingqi Fan, Yihan Zhang, Anhao Zhao, Yunpu Ma, Xiaoyu Shen

arXiv:2602.07574v2cs.CVcs.CL

TL;DR

Unified self-attention repeatedly processes visual tokens, creating substantial computational overhead, while the necessity of these updates remains in question. ViCA freezes visual tokens and retains sparse text-to-vision cross-attention; it preserves about 98% of baseline accuracy with 4% of visual-side computation. The authors also report substantial inference speedups and compatibility with token pruning.

  • Problem

    Unified self-attention updates visual tokens at every Transformer layer, creating substantial vision-side overhead, while pruning can incur unstable selection and runtime overhead.

    Source: §1 · p. 1

  • Method

    ViCA freezes projected visual tokens and lets text read them as keys and values through sparse cross-attention at selected layers.

    Source: §1 · p. 2

  • Results

    Across three backbones and nine benchmarks, ViCA preserves 98% of baseline accuracy while reducing visual-side computation to 4%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ViCA provides a regular, hardware-friendly inference pipeline with approximately 3.5× single-batch and greater than 10× multi-batch prefill speedups, and can be combined with token dropping.

    Source: §6 · p. 8

    Evaluation is within LLaVA-1.5; extending the training-based method to newer models is challenging when training data or recipes are unavailable or require larger-scale training.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Modern multimodal large language models (MLLMs) adopt a unified self-attention design that processes visual and textual tokens at every Transformer layer, incurring substantial computational overhead. In this work, we revisit the necessity of such dense visual processing and show that projected visual embeddings are already well-aligned with the language space, while effective vision-language interaction occurs in only a small subset of layers. Based on these insights, we propose ViCA (Vision-only Cross-Attention), a minimal MLLM architecture in which visual tokens bypass all self-attention and feed-forward layers, interacting with text solely through sparse cross-attention at selected layers. Extensive evaluations across three MLLM backbones, nine multimodal benchmarks, and 26 pruning-based baselines show that ViCA preserves 98% of baseline accuracy while reducing visual-side computation to 4%, consistently achieving superior performance-efficiency trade-offs. Moreover, ViCA provides a regular, hardware-friendly inference pipeline that yields over 3.5x speedup in single-batch inference and over 10x speedup in multi-batch inference, reducing visual grounding to near-zero overhead compared with text-only LLMs. It is also orthogonal to token pruning methods and can be seamlessly combined for further efficiency gains. Our code is available at https://github.com/EIT-NLP/ViCA.

1 Introduction

Unified self-attention repeatedly updates visual tokens, creating substantial vision-side overhead, while pruning can add runtime costs. ViCA removes visual-token updates and concentrates cross-modal reads in a few layers, preserving accuracy while sharply reducing visual computation.

1 Introduction

  • Unified self-attention processes visual tokens through attention and feed-forward networks at every layer, causing substantial vision-side computation.Source: §1 · p. 1
  • Pruning methods can rely on unstable input-dependent importance estimates and add scoring, selection, and reordering overhead that limits practical speedups.Source: §1 · p. 1
  • Projected visual embeddings are already well aligned with the language space, and effective cross-modal interaction is concentrated in only a few layers.Freezing visual tokens after projection causes only a marginal performance drop.Source: §1 · p. 2
  • ViCA preserves 98% of baseline accuracy while reducing visual-side computation to 4% and outperforming 26 pruning methods on accuracy–efficiency trade-offs.It is evaluated across three backbones and nine multimodal benchmarks.Source: §1 · p. 2

2 Related Work

Prior efficiency work reduces visual computation by dropping tokens, pruning operations, or changing computation across layers or within them. ViCA instead turns redundancy diagnostics into a compact architecture that retains necessary visual-update paths.

2 Related Work

  • Visual-token pruning shortens the sequence, while operation pruning sparsifies attention or feed-forward computations.Token importance may be estimated from text-to-vision attention, vision-to-vision attention, or model-guided signals.Source: §2 · p. 2
  • Depth-wise methods reduce computation across layers, whereas width-wise methods sparsify computation within a layer.Examples include freezing or skipping updates across depth and restricting vision-to-vision attention or pruning feed-forward dimensions within layers.Source: §2 · pp. 2–3
  • Unlike heuristic pruning designs, this work converts attention- and FFN-level redundancy diagnostics into a compact visual-update architecture and experimentally validates retained paths.Source: §2 · p. 3

3 Information-Flow View: Self-Attention vs. Cross-Attention Interaction

The information-flow view distinguishes joint visual-text processing in self-attention from read-only visual memory in cross-attention. This distinction identifies which visual operations write updates and which let text read visual information.

3 Information-Flow View: Self-Attention vs. Cross-Attention Interaction

  • In causal self-attention with vision tokens as a prefix, the active channels are V2V, T2V, and T2T; the mask sets vision-to-text attention to zero.Source: §3 · p. 3
  • Self-attention jointly reads and writes visual and text tokens, updating vision through V2V attention and a visual FFN branch.Source: §3 · p. 3
  • Cross-attention treats visual tokens as read-only memory: text queries attend to visual keys and values while visual attention and FFN writes are bypassed.The paper connects this design to prior evidence of redundant visual-token writes and to early cross-attention MLLMs.Source: §3 · p. 3

4 Methodology

The methodology diagnoses which visual-token updates affect representations and outputs, then builds ViCA around sparse text-to-vision reads. It also keeps attention shapes regular and supports combining the architecture with token dropping.

4.1 Where Visual Information Matters

  • The diagnostics measure representation change with cosine similarity and output impact with KL divergence after disabling an update.Cosine similarity captures embedding change; KL divergence compares the original and modified output distributions.Source: §4.1 · pp. 3–4
  • Output-critical visual updates occupy narrower layer ranges than representation-changing updates: attention layers 6–11 and FFN layers 5–11, versus attention 6–17 and FFN 5–31.Training-based adaptation nearly restores performance even when all visual-token updates are frozen.Source: §4.1 · p. 4
  • Text-to-vision cross-attention is concentrated in a few early-to-mid layers, peaking around layer 10; later visual updates are rarely read by text tokens.Restricting T2V attention and visual updates to essential layers largely preserves performance.Source: §4.1 · p. 4

4.2 ViCA: Vision-Only Cross-Attention

  • ViCA freezes visual representations after projection, removing their self-attention and FFN updates without freezing the vision encoder itself.Source: §4.2 · pp. 4–5
  • ViCA retains text-to-vision cross-attention only at selected key layers, where fixed visual tokens serve as keys and values.Other layers operate purely on text tokens.Source: §4.2 · p. 5

4.3 Practical Acceleration and Compatibility

  • Keeping token counts and attention shapes fixed avoids dynamic shapes and kernel reconfiguration, making reduced computation map more directly to inference speedups.Source: §4.3 · p. 5
  • ViCA retains compatibility with FlashAttention’s text-query, combined text-and-visual key/value pattern under causal masking.Source: §4.3 · p. 5
  • Because ViCA prunes computation paths rather than tokens, it can be combined with token-dropping methods at inference time to further reduce visual computation.Source: §4.3 · p. 5

5 Experiments

ViCA delivers strong accuracy with a small fraction of vision-side computation, and its efficiency gains persist across model scales and inference batch sizes. The experiments also show that combining ViCA with token pruning and restricting cross-attention to selected layers can reduce computation further while retaining much of baseline performance.

5.1 Setup

  • Experiments use three LLaVA-1.5 backbone scales and nine image-based benchmarks, with training-based experiments following the official two-stage protocol.Practical efficiency is benchmarked on 5,000 TextVQA samples using an NVIDIA A6000 GPU.Source: §5.1 · pp. 5–6

5.2 Main Results

  • 97.8% relative accuracy with 4.1% vision-side computation; ViCA+PDrop retains 96.3% accuracy at 2.0% computation on LLaVA-1.5-7B.Across model scales and task types, ViCA maintains a favorable accuracy–efficiency trade-off.Source: §5.2 · pp. 6–7, Table 2 · p. 6
  • Combining ViCA with PyramidDrop reduces vision-related computation to 1.7%–2.0% with a 1.8%–3.7% accuracy drop relative to the original full models.The paper describes architectural compute removal and token-level pruning as complementary.Source: §5.2 · p. 7
  • ViCA reduces vision-related computation to 4.5%/4.1%/3.3% for LLaVA-1.5-3B/7B/13B, equivalent to 27/24/19 visual tokens.Visual KV-cache sizes fall to 28.1%/25.0%/20.0% of the original; with PyramidDrop, computation falls to 2.0%/2.0%/1.7%.Source: §5.2 · p. 7, Table 3 · p. 7
  • 2.4× / 3.5× / 3.6× prefill speedups over original LLaVA-1.5 3B / 7B / 13B models; at batch size 8, speedup is approximately 10×.The speedup trends persist across batch sizes, with larger absolute latency reductions for larger models.Source: §5.2 · p. 7, Table 4 · p. 7, Figure 5 · p. 7

5.3 Ablation Studies

  • Across LLaVA-1.5-3B/7B/13B, freezing visual attention and FFN updates retains 100.3%/98.8%/97.4% of baseline accuracy at about 16% computation; sparse cross-attention then reduces FLOPs to 4% with a 1–3% accuracy drop.The freezing-strategy comparison found that freezing attention updates improved relative average accuracy by 0.2%, while freezing FFN updates reduced it by 0.8%.Source: §5.3 · p. 8, Table 6 · p. 8, Table 5 · p. 8

6 Conclusion

ViCA removes redundant visual computation while retaining sparse cross-modal interaction, yielding faster, more hardware-friendly inference. It remains compatible with token dropping for further efficiency gains.

6 Conclusion

  • ViCA achieves approximately 3.5× prefill speedup in single-batch inference and > 10× speedup in multi-batch inference.The paper attributes the speedups to a more regular, hardware-friendly inference pipeline.Source: §6 · p. 8
  • The diagnosis finds that visual-token self-attention and FFN updates are largely redundant, while effective cross-modal interaction is concentrated in a small subset of layers.The conclusion says visual computation paths are removed on this basis.Source: §6 · p. 8
  • ViCA is compatible with existing token-dropping methods, which can be combined with it for further efficiency gains.Source: §6 · p. 8
  • Future work will explore adaptive cross-modal fusion architectures and a framework unifying pruning paradigms.Source: §6 · p. 8

Limitations

The evaluation is centered on LLaVA-1.5 and the single-image setting, while cross-scale results are reported for 3B and 13B models. The authors also disclose that LLMs were used only for language refinement.

Limitations

  • The evaluation is limited to single-image MLLMs, and multi-image or video settings may require richer vision-to-vision interactions.The authors identify cross-image correspondence, temporal continuity, and fine-grained grounding as challenges for future extensions.Source: Limitations · p. 9
  • Extending controlled evaluation beyond LLaVA-1.5 is challenging because complete training data, data mixtures, or training recipes are not always public, and reproducing them can require substantially larger-scale training.Source: Limitations · p. 9

A The Use of Large Language Models

B.1 Cross-Scale Generalization

  • On LLaVA-1.5-3B, ViCA preserves 98.7% of baseline accuracy at 4.5% vision-side computation; on LLaVA-1.5-13B, it retains 97.0% at 3.3%.With PyramidDrop, computation falls to 2.0% for 3B and 1.7% for 13B.Source: §B.1 · p. 13

B.2 Fine-Grained Tasks

The supplied section reports ViCA comparisons on vision-heavy and spatial-grounding tasks, then examines which cross-attention, vision-attention, and FFN layers matter for performance. The diagnostics and retraining experiments show that the important updates are concentrated in limited layer ranges.

B.2 Fine-Grained Tasks

  • The fine-grained evaluation covers vision-heavy tasks requiring localization, text recognition, and detailed perception, as well as spatial-grounding benchmarks.The named tasks include OCRBench, BLINK, AI2D, InfoVQA, Flickr30k-CIDEr, RefCOCO, RefCOCO+, and RefCOCOg.Source: §B.2 · p. 13

LLaVA-1.5-13B

  • 94.3% vs. 91.3% relative average performance, with 4.1% vs. 11.1% vision FLOPs, for ViCA versus PDrop on vision-heavy benchmarks.For spatial grounding, ViCA raises relative average performance from 57.0% to 74.8% while using 4.1% vision FLOPs; a gap to the original model remains, especially on grounding.Source: LLaVA-1.5-13B · p. 14

C.1 Layerwise Redundancy in Text-to-Vision Cross-Attention

  • Text–vision interaction is concentrated in layers 0–1, 7–11, and 14: retaining cross-attention only there limits the training-free performance drop to 6.5%, and retraining restores 99.5% of baseline performance.Adding layer 16 in retraining yields 100.5% of baseline performance.Source: §C.1 · p. 15

C.2 Layerwise Redundancy in Attention-Based Vision Updates

  • Retraining with all attention-based vision updates removed achieves 100.2% of baseline performance across nine benchmarks.In training-free masking, removing all such updates causes a 9.3% drop; retaining updates in layers 6–17 reduces it to 1.2%, while masking layers 6–11 alone causes a 4.7% drop.Source: §C.2 · pp. 15–16

C.3 Layerwise Redundancy in FFN-Based Vision Updates

  • Skipping all vision-token FFNs after retraining preserves 99.2% of baseline performance; enabling FFNs only in layers 5–11 or layers 5, 7, and 9 reaches 100.5%.The reported training-free tests find that retaining FFNs in layers 5–11 reduces the performance drop to 3.2%, compared with a 28.8% drop when skipping them across all layers.Source: §C.3 · pp. 16–17

C.4 Redundancy Diagnosis Across Model Scales

Layerwise redundancy patterns observed in LLaVA-1.5-7B are examined at 3B and 13B scales, then used to transfer the pruning strategy. The resulting ViCA variants preserve most baseline accuracy with a small fraction of vision computation.

C.4 Redundancy Diagnosis Across Model Scales

  • Training-free masking shows that retaining candidate text-to-vision cross-attention layers preserves 87.1% and 94.2% of original average accuracy for 3B and 13B, respectively.The candidate key-layer sets differ across the two model scales.Source: §C.4 · p. 17
  • Transferred ViCA variants retain 98.7% and 97.0% accuracy on LLaVA-1.5-3B and 13B, respectively, using 4.5% and 3.3% of vision-related computation.FreezeVis variants recover 100.3% and 97.4% of baseline accuracy at the two scales, respectively.Source: §C.4 · pp. 17–18

D.1 Details of Backbones

D.2 Configuration Details of Our Method

  • The preserved cross-attention layers vary by backbone: for example, 7B uses layers 0, 1, 7–11, and 14, while 13B uses 0, 6, 8–10, 13, 14, and 16.PyramidDrop is integrated at inference, dropping 25% of vision tokens at selected layers for each backbone.Source: §D.2 · p. 18

D.3 Details of Calculation Equation of FLOPs

The FLOPs analysis separates visual attention and FFN costs, then examines how visual-token and text-token counts shape computation. In LLaVA-1.5-7B, visual attention becomes more prominent as image count rises, while ViCA leaves text inference as the dominant cost.

Total Computation

  • When n ≪ d, visual computation scales near-linearly with visual-token count, and FFNs dominate its linear component when m ≫ d.The quadratic attention term is then smaller than attention-projection and FFN costs.Source: §D.3 · p. 19

Equivalent Visual Token Count

  • The equivalent visual token count neq represents the number of original-model visual tokens yielding the same vision-side FLOPs.It is reported to enable fair comparison.Source: §D.3 · p. 19

Case Study on LLaVA-1.5-7B

  • For LLaVA-1.5-7B, increasing question length from 0 to 40 tokens leaves the visual QK⊤V share around 2.27%.The analysis fixes the question length at 20 tokens in subsequent calculations.Source: §D.3 · p. 19, Figure 7 · p. 20
  • As image count rises from 1 to 32, visual QK⊤V grows from 2.27% to 42.68%, while visual FFN falls from 59.49% to 38.12% and projector cost from 29.52% to 18.91%.The profile shifts from FFN-dominated to attention-dominated computation; Figure 7 shows the breakdown.Source: §D.3 · p. 19, Figure 7 · p. 20

Cost of visual updating in LLaVA-1.5

  • Visual-token updates account for more than 80% of total visual-path computation in LLaVA-1.5.The update comprises self-attention and FFN computation.Source: §D.3 · p. 20, Table 14 · p. 20

Computation of Our Method

  • With eight text–vision cross-attention layers retained, ViCA’s remaining visual operations are projections and cross-attention, while text inference dominates total computation.The remaining vision-related components are visual-token key–value projections and text–vision cross-attention multiplication.Source: §D.3 · p. 20

D.4 FlashAttention Acceleration Implementation

ViCA’s asymmetric text-query, visual-key/value attention is compatible with FlashAttention and, under saturated CUDA utilization, achieves approximately 3.5× prefill speedup. Its training uses the LLaVA-1.5 data and recipe, with the attention-update strategy and projector pretraining adapted to ViCA.

D.4 FlashAttention Acceleration Implementation

  • ViCA uses text-only queries and text-plus-visual keys and values; FlashAttention v2.1 bottom-right causal-mask alignment supports this short-query, long-key/value pattern.The eager implementation instead freezes visual tokens through explicit masking.Source: §D.4 · pp. 20–21, Figure 8 · p. 21
  • Approximately 3.5× prefill speedup is achieved under saturated CUDA utilization with FlashAttention.The attention difference from eager mode causes only minor accuracy degradation across multiple benchmarks.Source: §D.4 · p. 21

D.5 Training Dataset

  • ViCA uses the LLaVA-1.5 two-stage training data: 558K BLIP-LAION-CC-SBU image-text examples for projector pretraining and a 665K visual-instruction mixture for instruction tuning.The same training data is used as the LLaVA-1.5 7B baseline.Source: §D.5 · p. 21

D.6 Training Configuration

  • Across both stages, ViCA keeps the LLaVA-1.5 hyperparameters and data while freezing visual tokens and enabling selected-layer text-to-vision cross-attention.The projector is pretrained under the ViCA attention configuration before instruction fine-tuning.Source: §D.6 · p. 21

E Details of Comparison Methods

The comparison methods span token pruning, token selection and compression, and operation-level pruning, using different signals and strategies to reduce visual computation.

Token and operation pruning

Token compression and merging

  • ToMe merges similar tokens, VisionZip clusters tokens around dominant ones, TokenPacker forms tokens coarse-to-fine, PruMerge combines pruning and merging, and Delta-LLaVA uses low-rank delta projections.These approaches compress or merge visual representations through different mechanisms.Source: E Details of Comparison Methods · pp. 22–23

Attention- and instruction-guided selection

  • FastV prunes low visual–text-attention tokens after shallow layers, while SparseVLM uses instruction-relevant tokens to guide adaptive progressive pruning and TRIM uses image–text similarity.These methods use cross-modal or instruction-alignment signals to identify tokens for removal.Source: E Details of Comparison Methods · pp. 22–23

Spatial allocation and diversity

  • HiRED allocates tokens across image partitions using CLS attention, HoloV allocates budgets across spatial regions, and D2Pruner combines semantic importance with spatial diversity.Their selection strategies aim to preserve spatial coverage or diversity.Source: E Details of Comparison Methods · p. 22

Aggregation and conditional selection

  • VISA aggregates removed tokens into retained tokens, FiCoCo uses Filter–Correlate–Compress, and CDPruner selects tokens through instruction-conditioned similarity and DPP optimization.These methods combine or summarize information rather than relying only on token removal.Source: E Details of Comparison Methods · pp. 22–23

Stage-adaptive methods

  • VScan adapts token selection across inference stages, Dynamic-LLaVA sparsifies across prefill and decoding, and TwigVLM combines twig-guided pruning with self-speculative decoding.Their mechanisms adapt pruning to relevance or inference stage.Source: E Details of Comparison Methods · pp. 22–23

Spatial allocation and diversity

  • DivPrune maximizes minimum pairwise distance, DART iteratively selects dissimilar tokens, and VisPruner combines attention importance with duplicate removal; HiPrune categorizes tokens by hierarchical attention.These approaches emphasize diversity or token structure during selection.Source: E Details of Comparison Methods · pp. 22–23

Operation and information-flow pruning

  • DOP prunes token-processing operations and reallocates computation, while FlowCut targets tokens with minimal contribution to information propagation.Both methods frame efficiency around operation or information-flow reduction.Source: E Details of Comparison Methods · p. 22

F Details of Benchmarks

The benchmark suite probes multimodal understanding across perception, text-sensitive tasks, structured reasoning, captioning, and language-guided visual grounding. Its benchmarks target complementary visual abilities, from binary perception and diagrams to object localization and hallucination detection.

Perception and diagrams

  • MMEP tests 14 categories of visual perception with binary questions, while BLINK targets fine-grained perception and AI2D tests structured science-diagram understanding.MMEP includes coarse- and fine-grained perception and OCR; BLINK includes depth, correspondence, similarity, forensics, and multi-view reasoning.Source: F Details of Benchmarks · pp. 23–24

Hallucination and spatial reasoning

  • POPE diagnoses object hallucination through binary presence questions and reports accuracy, precision, recall, and F1; SEEDBench’s image-only subset covers spatial understanding and reasoning.SEEDBench includes scene understanding, attributes, location, counting, relations, interactions, reasoning, and text recognition.Source: F Details of Benchmarks · p. 24

Text-centric understanding

  • OCRBench, TextVQA, and InfoVQA assess text-rich visual understanding, including text recognition, document questions, OCR with context, and infographic interpretation.InfoVQA additionally requires interpreting graphical, numerical, and layout-based information.Source: F Details of Benchmarks · pp. 24–25

Visual search and real-world scenes

  • V*Bench evaluates guided search in high-resolution, crowded images, while RealWorldQA tests everyday scenes, object relations, and spatial configurations.Both descriptions emphasize locating or preserving task-relevant visual cues.Source: F Details of Benchmarks · pp. 24–25

Broad multimodal evaluation

  • MMStar reduces language-only shortcuts to test vision-indispensable reasoning, while MMBench and MMBench-CN cover broad abilities through hierarchical multiple-choice evaluation.MMBench-CN retains MMBench’s ability structure in Chinese-language settings.Source: F Details of Benchmarks · pp. 23, 25

Captioning and question answering

  • Flickr30k-CIDEr measures caption agreement with human references, while VQAv2, GQA, and ScienceQA image questions assess visual question answering and grounded reasoning.GQA emphasizes compositional reasoning over scene-graph structures; ScienceQA’s image subset requires integrating visual information with domain knowledge.Source: F Details of Benchmarks · pp. 23–25

Visual grounding

  • RefCOCO, RefCOCO+, and RefCOCOg test language-guided object localization, with increasingly distinct emphasis on spatial terms, appearance attributes, and longer descriptions.RefCOCO+ relies less on absolute location words, while RefCOCOg uses longer, more descriptive expressions.Source: F Details of Benchmarks · p. 25

Related papers

Loading 2602.07574v2…