PaperlayerView exact revision on arXiv

Source-linked AI summary

VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions

Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas, Yassine Ouali, Georgios Tzimiropoulos

arXiv:2603.23495v1cs.CVcs.AIcs.LG

TL;DR

Existing token-reduction methods improve LVLM efficiency but can lose information on fine-grained, high-resolution tasks. VISOR instead sparsifies image-text and image-image interactions while retaining full visual tokens, using cross-attention for context, self-attention for refinement, and adaptive computation. Across vision-language benchmarks, VISOR reduces computational cost and outperforms prior methods, especially on challenging tasks.

  • Problem

    Token reduction can create an information bottleneck that impairs LVLM performance on complex tasks requiring fine-grained visual understanding.

  • Method

    VISOR retains full visual tokens while selectively executing cross-attention and self-attention layers, with a universal multi-budget model and lightweight adaptive inference policy.

  • Results

    VISOR consistently outperforms prior methods across vision-language benchmarks, particularly on challenging tasks requiring detailed visual reasoning.

  • Takeaways & Limitations

    Sparse visual interactions can improve LVLM efficiency without discarding visual information, while allocating more refinement to complex tasks.

Abstract

from arXiv · show

Existing approaches for improving the efficiency of Large Vision-Language Models (LVLMs) are largely based on the concept of visual token reduction. This approach, however, creates an information bottleneck that impairs performance, especially on challenging tasks that require fine-grained understanding and reasoning. In this work, we challenge this paradigm by introducing VISion On Request (VISOR), a method that reduces inference cost without discarding visual information. Instead of compressing the image, VISOR improves efficiency by sparsifying the interaction between image and text tokens. Specifically, the language model attends to the full set of high-resolution visual tokens through a small, strategically placed set of attention layers: general visual context is provided by efficient cross-attention between text-image, while a few well-placed and dynamically selected self-attention layers refine the visual representations themselves, enabling complex, high-resolution reasoning when needed. Based on this principle, we first train a single universal network on a range of computational budgets by varying the number of self-attention layers, and then introduce a lightweight policy mechanism that dynamically allocates visual computation based on per-sample complexity. Extensive experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results across a diverse suite of benchmarks, and excels in challenging tasks that require detailed visual understanding.

1. Introduction

VISOR replaces visual-token compression with sparse execution of cross-attention and self-attention layers, preserving access to the full visual-token set while reducing computation. It adapts visual processing to task complexity through selective layer execution and per-sample allocation.

  • Motivation: Prior token reduction methods lower LVLM cost by pruning or compressing visual tokens, but can lose information on complex high-resolution tasks.They perform well for coarse understanding yet incur substantial information loss when fine-grained visual understanding is required.
  • VISOR: VISOR sparsifies the computational layers that interact with visual tokens instead of reducing the number of visual tokens processed.It executes selected cross-attention and self-attention layers while retaining the full visual-token set throughout the forward pass.
  • VISOR: A small number of cross-attention layers provides coarse visual context, while self-attention layers update visual tokens for fine-grained reasoning.Cross-attention alone suffices for coarse understanding, whereas self-attention enables gradual refinement of visual features on complex tasks.
  • Adaptive computation: VISOR trains one universal network across computational budgets and dynamically selects self-attention layers per sample using a lightweight policy.The universal model varies the number of self-attention layers, and the policy selects the executed configuration during inference.

2. Closely related work

Existing efficient LVLM methods primarily reduce visual tokens through pruning or compression, but their benefits weaken on challenging high-resolution tasks. VISOR instead limits when visual information is processed, preserving full visual context while remaining compatible with token-reduction methods.

  • Token reduction: Efficient LVLM research mainly uses dynamic token pruning, token merging, or learned token compression to reduce the visual-token load.These approaches target redundancy before or within the language model.
  • Limitations: Prior token-reduction methods often focus on coarse or lower-resolution tasks, while fine-grained settings can expose accuracy drops or limited speedups.The related-work discussion identifies this as a general trend under challenging, higher-resolution evaluation.
  • Comparison: Figure 1 compares FLOPs reduction against accuracy, with the caption highlighting efficiency gains while retaining performance on harder datasets.The figure uses the paper’s easy-versus-hard task distinction for interpreting the comparison.
  • VISOR: VISOR avoids discarding tokens by limiting the layers where the language model interacts with and updates visual information.The full high-resolution visual context remains available when needed for fine-grained reasoning.

3. Motivation: Image processing within LVLMs

The motivation analysis finds that visual interaction and visual-feature refinement vary with task difficulty. Easy tasks use sparse, mostly textual processing, whereas hard tasks require sustained visual interaction and progressive refinement, motivating adaptive computation.

  • Attention patterns: Cross-modality attention is task-dependent: coarse tasks rely mainly on text-to-text attention, while fine-grained tasks attend to images across the network.The analyzed interaction types are Query-to-Image, Answer-to-Image, and Answer-to-Query.
  • Visual refinement: Visual features remain largely static on easy tasks but are progressively refined across layers on harder tasks.Pairwise CKA similarity is high for ScienceQA-like tasks (CKA > 0.9) and drops to 0.6 for DocVQA-like tasks.
  • Visual refinement: The CKA analysis measures pairwise similarity between vision features from transformer layers to characterize their evolution.The method uses Centered Kernel Alignment similarity across layer pairs.
  • Sensitivity analysis: Dropping vision tokens affects tasks differently: easy tasks remain robust, whereas DocVQA, ChartQA, and InfoVQA are highly sensitive.The sensitivity analysis yields coarse-vision and vision-sensitive task clusters.
  • Design implication: The findings motivate adapting the visual-processing budget to each sample or task rather than using one fixed configuration.The paper identifies varying interaction sparsity, feature refinement, and dropout sensitivity across tasks.

4. Method

VISOR reduces LVLM inference cost by sparsifying visual-token interactions rather than discarding visual tokens. It combines efficient cross-attention with selective self-attention, then uses universal training and routing to adapt computation to input complexity.

  • Efficient visual processing: A standard LVLM’s concatenated self-attention captures image-image, image-text, and text-text interactions at O((N_v + N_t)^2 · d) cost.Because N_v is typically at least N_t for high-resolution images, image-image interactions dominate inference cost.
  • Efficient visual processing: VISOR decouples text and vision processing, executing cross-attention and self-attention only at selected layers while retaining the full visual-token set.Most layers operate only on text tokens; selected layers integrate text-image and image-image interactions based on sample or task complexity.
  • Cross-attention: Cross-attention uses text tokens as queries and visual tokens as keys and values, updating the text stream while leaving visual tokens unchanged.This provides visual context efficiently when tasks do not require visual-feature refinement.
  • Selective self-attention: Selective self-attention updates visual tokens and enables later cross-attention layers to use refined, higher-level visual representations.Uniformly distributing a few cross-attention and self-attention layers yields strong performance.
  • Adaptive inference: VISOR trains one universal model across viable self-attention configurations, then uses a lightweight policy network to choose the computation budget per input.Offline pseudo-labeling selects the most efficient configuration meeting at least 99% of full-model accuracy, while routing predicts configurations from a special routing token.
  • Adaptive inference: VISOR remains orthogonal to token reduction and can be combined with pruning or token packing through VISOR-TR.The supplied method description presents this combination as a route to further efficiency gains.

5. Experiments

VISOR is evaluated against state-of-the-art methods across diverse vision-language benchmarks under a shared LLaVA-OV backbone. It matches or exceeds prior methods on coarse tasks and performs especially strongly on challenging fine-grained reasoning tasks while reducing computation.

  • Evaluation setup: VISOR is evaluated on benchmarks spanning coarse and fine-grained visual understanding, including reasoning, OCR, chart, document, and general vision-language tasks.The benchmark suite includes RealWorldQA, ScienceQA, GQA, MME, MMSTAR, MMBench, POPE, AI2D, ChartQA, TextVQA, InfoVQA, OCRBench, and DocVQA.
  • Implementation: The implementation builds on LLaVA-OV with a SigLIP-400M vision encoder, Qwen2 language model, and 2-layer MLP connector.The model uses 384 × 384 image patches producing 729 visual tokens and inserts attention layers across the language model.
  • Implementation: Training uses a two-stage procedure that first tunes the new attention layers and then fine-tunes the entire model on a high-quality dataset.The reported training spans three epochs across both stages.
  • Evaluation setup: Comparisons use a shared LLaVA-OV (0.5B) backbone, with accuracy and average FLOP savings reported relative to baseline LLaVA-OV.Vision-encoder FLOPs are excluded because they are common to all methods, and VISOR uses one routed universal model.
  • Main results: VISOR matches or exceeds prior methods on coarse visual-context tasks while achieving up to 8.6× FLOP savings.For fine-grained visual reasoning, it outperforms the listed token-reduction baselines, including VisionZip, HiRED, and M3.
  • Main results: VISOR-TR reaches up to 18× FLOP savings when combined with token reduction while maintaining state-of-the-art accuracy.This result demonstrates the reported efficiency benefit of combining VISOR with token-reduction techniques.

6. Ablation studies and analysis

Ablations show that cross-attention is sufficient for coarse tasks, whereas fine-grained reasoning requires visual-token refinement through self-attention. VISOR also offers favorable efficiency trade-offs, including real hardware speedups and compatibility with token reduction.

  • Attention-layer ablations: Cross-attention performance saturates around 8 layers on coarse tasks but is insufficient for fine-grained reasoning.Adding self-attention substantially improves fine-grained performance, with a 7-layer configuration nearly matching the full model.
  • Token-reduction combinations: Combining VISOR with token reduction achieves up to 35× FLOPs savings with only a minor accuracy drop.More aggressive reduction, such as greater than 4× token reduction, causes larger performance drops on hard tasks because the information bottleneck becomes more pronounced.
  • Efficiency trade-offs: VISOR provides a better accuracy-efficiency trade-off than prior methods while substantially reducing transformer-layer FLOPs relative to LLaVA-OV.The FLOP accounting excludes the vision encoder, which is common to all methods.
  • Real inference speed: On MI300X GPUs, full LLaVA-OV takes 0.0738 sec/sample, while VISOR takes 0.0384 sec/sample with 8CA-7SA and 0.0261 sec/sample with 8CA-2SA.The reported measurements use the maximum batch size that fits all methods in memory.
  • Model overhead: VISOR’s cross-attention layers add less than 7.5% model-size overhead compared with baseline LLaVA-OV.The overhead is concentrated in the linear projection layers.

7. Conclusion

VISOR reduces LVLM inference cost by sparsifying image-text and image-image interactions without discarding visual information. It combines efficient cross-attention, selective self-attention, universal multi-budget training, and dynamic computation allocation.

  • VISOR sparsifies image-text and image-image interactions while preserving the full visual information available to the LVLM.
  • Efficient cross-attention models text-image interactions, while selective self-attention refines visual features for fine-grained understanding and reasoning.
  • A single universal network supports multiple computational budgets, with lightweight policies allocating visual computation according to per-task or per-sample complexity.
  • VISOR improves efficiency and outperforms state-of-the-art token compression methods across diverse benchmarks, especially on detailed visual-understanding tasks.

A. Identifying promising configurations for adaptive training and inference

The configuration space grows exponentially with the number of self-attention layers, while some layer subsets can severely damage performance. The selected configurations therefore preserve critical layers and span increasing computational budgets for task-dependent visual processing.

  • 2^LSA configurations arise by independently executing or skipping each self-attention layer, making exhaustive evaluation impractical.
  • Dropping the first self-attention layer causes significant performance degradation across all evaluated datasets.
  • Configurations with only 1 or 2 self-attention layers perform poorly on complex tasks, whereas additional layers generally improve results.
  • Less vision-intensive tasks favor configurations near early exit, while complex tasks benefit from a more uniform distribution of self-attention layers.
  • The 0.5B model uses selected configurations ranging from two self-attention layers, [1] [4] and [1], to seven layers, [1] [4] [10] [16] [19] [22].

B. Per-dataset saving rate

Per-dataset results show that visual-computation needs vary by task: easy tasks can use few self-attention layers, while harder tasks require more layers for optimal performance. Routing allocates similar layer counts within datasets and adapts budgets across datasets.

  • Easy tasks can be solved with very few self-attention layers, while hard tasks require more layers for optimal performance.
  • Across datasets, the results preserve the pattern that computationally easier tasks need fewer layers than more demanding tasks.
  • Router-selected configurations tend to allocate a similar number of layers within each dataset, indicating dataset-consistent computational budgets.
  • Performance heatmaps compare configuration rows against dataset columns, with color intensity representing relative accuracy.
  • Table 5 reports accuracy for most datasets and an MME score, with corresponding FLOPs savings relative to the LLaVA-OV baseline.

D. Oracle performance analysis

VISOR’s adaptive-computation analyses show that most samples can use few self-attention layers, while difficult visual tasks benefit from more processing. The method also reports efficiency gains across architectures, token-packing settings, and multi-image benchmarks.

  • Challenging tasks show nearly linear performance improvement as the computational budget increases, highlighting the value of additional self-attention layers for fine-grained reasoning.
  • The router’s layer configurations are visualized per test dataset, while configuration performance is compared across tasks and computational budgets.
  • Oracle selection finds that most samples are accurately processed with few self-attention layers, while hard tasks require more layers for optimal performance.
  • VISOR’s FLOPs decrease relative to LLaVA-OV as visual sequence length grows, excluding the shared vision-encoder cost.
  • At approximately 16× FLOPs reduction, VISOR-TR is evaluated head-to-head against M3 and VisPruner under matched end-to-end finetuning.
  • On QwenVL2 (2B), VISOR largely matches the full model’s performance using significantly fewer FLOPs, though unavailable full training data disadvantages the method and causes swings on some datasets.
  • Token packing halves visual-token count through spatial interpolation and pixel shuffle without added parameters, complementing sparse-attention savings.
  • On MUIR, Blink, and MMIU, finetuned VISOR matches or outperforms LLaVA-OV while running over 3× faster.

J. Scaling VISOR with Training Data Size

VISOR benefits from more training data, with performance losses from halving the dataset concentrated on harder vision-intensive tasks. With FastVLM, VISOR further reduces LLM computation while improving benchmark performance over the corresponding baseline.

  • Scaling with training data: Halving the training data generally reduces performance, with the largest degradation on vision-intensive tasks such as DocVQA and InfoVQA.Performance on simpler datasets, including MME, POPE, and RWQA, remains similar.
  • Using VISOR with FastVLM: FastVLM produces 36 visual tokens per 384 × 384 image patch, compared with 729 from standard LLaVA-OV with SigLIP-400M.This comparison motivates testing whether VISOR remains effective with an encoder that already outputs fewer visual tokens.
  • Using VISOR with FastVLM: VISOR improves over LLaVA-OV with FastVLM on both Easy and Hard benchmarks while achieving 60× versus 20× LLM FLOPs savings.The reported FLOPs savings concern LLM computation and exclude vision encoding.
  • Using VISOR with FastVLM: Compared with SigLIP-based LLaVA-OV, the FastVLM version achieves comparable Easy-benchmark results but has a larger gap on Hard benchmarks.The passage attributes this pattern to aggressive token reduction mainly hurting fine-grained benchmarks.

L. Re-implementation of baselines

The baseline re-implementation adapts token-selection and token-reduction methods to the LLaVA-OV setting with specified token budgets and selection rules. The section also details the CKA computation used to compare visual features across layers.

  • Baseline adaptations: Methods relying on CLS-token attention instead use each token’s average attention received from all other sequence tokens because LLaVA-OV’s SigLIP-400M encoder lacks a CLS token.This substitution follows Yang et al.’s proposed approach.
  • Token reduction baselines: VisionZip selects dominant visual tokens using encoder attention scores and merges remaining tokens into contextual tokens based on semantic similarity.The LLaVA-Next implementation is adapted to the LLaVA-OV backbone.
  • Token budgets: The re-implemented baselines use 128 tokens per patch for VisionZip, VisionZip†, and VisPruner, while HiRed uses a 20% token budget and M3 reports an 8× reduction.VisPruner divides its retained tokens into important and diverse subsets; VisionZip divides them into dominant and contextual subsets.
  • CKA analysis: CKA compares centered Gram matrices formed from vision features X and Y extracted at different layers using a normalized Frobenius inner product.The feature matrices have n tokens and d feature dimensions, and centering uses H = I_n − 1_n.
Loading 2603.23495v1…