PaperlayerView exact revision on arXiv

Source-linked AI summary

A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs

Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yibing Song, Kai Wang, Zhangyang Wang, Yang You

arXiv:2412.03324v2cs.CV

TL;DR

Large VLMs face inference overhead from numerous visual tokens, and single-layer attention can be inadequate for pruning at low retention. SGL uses a small VLM’s aggregated attention to guide large-model pruning and uses early exiting; across 11 benchmarks, it reports up to 91% visual-token pruning with competitive performance.

  • Problem

    Large VLM inference is burdened by numerous visual tokens, while single-layer attention may not identify essential tokens accurately at low retention and all-layer attention requires a costly full pass.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    SGL uses a small VLM’s all-layer aggregated attention to guide visual-token pruning in a large VLM and uses the small model’s predictions for early exiting.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Across 11 benchmarks, SGL prunes up to 91% of visual tokens while maintaining competitive performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Easy questions can bypass large-model inference, while cases that invoke the large model can still reduce computation by pruning visual tokens.

    Source: §1 · p. 2

    The method is primarily validated on multimodal understanding tasks; applying it to VLMs that unify understanding and generation remains future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) have shown remarkable success across various multi-modal tasks, yet large VLMs encounter significant efficiency challenges due to processing numerous visual tokens. A promising approach to accelerating large VLM inference is using partial information, such as attention maps from specific layers, to assess token importance and prune less essential tokens. However, our study reveals three key insights: (i) Partial attention information is insufficient for accurately identifying critical visual tokens, resulting in suboptimal performance, especially at low token retention ratios; (ii) Global attention information, such as the attention map aggregated across all layers, more effectively preserves essential tokens and maintains comparable performance under aggressive pruning. However, the attention maps from all layers requires a full inference pass, which increases computational load and is therefore impractical in existing methods; and (iii) The global attention map aggregated from a small VLM closely resembles that of a large VLM, suggesting an efficient alternative. Based on these findings, we introduce a \textbf{training-free} method, \underline{\textbf{S}}mall VLM \underline{\textbf{G}}uidance for accelerating \underline{\textbf{L}}arge VLMs (\textbf{SGL}). Specifically, we employ the attention map aggregated from a small VLM to guide visual token pruning in a large VLM. Additionally, an early exiting mechanism is developed to fully use the small VLM's predictions, dynamically invoking the larger VLM only when necessary, yielding a superior trade-off between accuracy and computation. Extensive evaluations across 11 benchmarks demonstrate the effectiveness and generalizability of SGL, achieving up to 91\% pruning ratio for visual tokens while retaining competitive performance.

1. Introduction

Large VLMs incur inference overhead from visual tokens, and global attention better preserves essential tokens than single-layer attention but requires a full inference pass. SGL uses a small VLM to guide large-model token pruning and early exiting.

1. Introduction

  • Visual-token volume creates substantial VLM inference overhead; existing compression approaches may require training, while attention-based pruning uses partial LM information.Attention-based pruning is presented as an approach that can integrate without fine-tuning.Source: §1 · p. 1
  • FastV struggles below 35% retention, whereas all-layer oracle attention remains competitive at 9%, showing global attention better identifies essential tokens.Obtaining the oracle map requires a full inference pass, making it costly before large-model inference.Source: §1 · p. 1
  • The small VLM’s aggregated attention pattern closely mirrors the large VLM’s and preserves answer-relevant tokens regardless of answer correctness.This makes the small model’s map a proxy for guiding pruning in the large model.Source: §1 · p. 2
  • SGP runs a small VLM first, aggregates its all-layer attention over prompt and generated tokens, then ranks visual tokens to guide pruning in the large VLM.The stated goal is reducing computation while preserving information needed for a correct answer.Source: §1 · p. 2
  • SEE evaluates the small VLM’s decision score and skips the large VLM when that score exceeds a threshold.When the large model is invoked, SGP prunes less important visual tokens using the small model’s aggregated attention map.Source: §1 · p. 2
  • Across 11 benchmarks, SGL prunes up to 91% of visual tokens in InternVL2 models from 26B to 76B while maintaining competitive performance.The method is also reported to integrate with Qwen-VL and LLaVa-OV.Source: §1 · p. 2

2. Related Works

Related work addresses visual-token costs through compression and attention-based pruning, while SGL uses a small VLM’s aggregated attention and confidence to guide pruning and early exiting.

2. Related Works

  • VLMs connect visual-encoder tokens to a language model, increasing sequence length and computational and memory demands.The section gives examples ranging from 576 tokens for LLaVa at 336×336 resolution to 10,496 tokens in InternVL2.Source: §2 · p. 2
  • Prior visual-token compression includes pruning, merging, skipping, Q-Former, distillation, and parameter alignment; several approaches require additional training.Training-free methods use LM attention, but visual-encoder token merging may miss vision-language interactions and specific-layer pruning may miss essential tokens at low retention.Source: §2 · p. 3
  • SGL instead ranks tokens using attention aggregated across all layers of a small VLM to guide pruning in a larger VLM.Its early-exit decision uses small-model confidence, with a consistency criterion also proposed.Source: §2 · p. 3

3. Small Guides Large (SGL) in VLMs

SGL combines small-VLM-guided visual token pruning with small-VLM early exiting. Its design uses a small model both to identify tokens for the large VLM and to assess whether that larger model is needed.

3.1. Preliminary of Vision-language Models

  • VLMs encode images as visual tokens, combine them with prompt tokens, and feed them to a language model that generates responses autoregressively.The passage notes that image-token counts can greatly exceed prompt-token counts, especially for high-resolution images.Source: §3.1 · p. 3

3.2. Small VLM-Guided Visual Token Pruning

  • Single-layer attention pruning falls short of an all-layer oracle, but obtaining the oracle requires a full, computationally costly inference pass.This motivates seeking a precise attention map before running the large model.Source: §3.2 · p. 3
  • SGP aggregates the small VLM’s attention to visual tokens from prompt and generated tokens, then uses the resulting importance ranking to prune tokens in the large VLM.The aggregated scores include attention received from prompt tokens and generated tokens across heads and layers; the large VLM retains the highest-ranked tokens.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Small VLM Early Exiting

  • SEE estimates answer confidence from sequence probability and consistency after visual-token pruning, then exits at the small VLM when its decision score exceeds a threshold.With 95% of visual tokens removed, computing the consistency score requires less than 10% of the small VLM’s initial inference time.Source: §3.3 · pp. 4–5

4. Experiment

Experiments test SGL’s pruning and early-exiting designs across 11 benchmarks, finding competitive performance under aggressive token pruning and a favorable efficiency–performance trade-off. Ablations and cross-model evaluations examine its design choices and generalizability.

4.1. Experimental Setup

  • Experiments cover four VQA benchmarks, three visual-grounding benchmarks, and four multimodal-understanding benchmarks; InternVL2-2B and InternV2L-26B are default models, while Qwen-VL and LLaVA-OV assess method generalizability.SGP is compared with ToMe and FastV at average token retention ratios of 64%, 35%, and 9%.Source: §4.1 · p. 5, §4.2 · pp. 5–6

4.2. Comparing SGP with Previous Methods

  • At 9% retention, SGP retains over 89% of original InternVL2-26B performance while FastV and ToMe collapse; at 35%, SGP stays competitive where they drop, especially on OCR and grounding tasks.At 64% retention, all methods show competitive performance across tasks.Source: §4.2 · p. 6

4.3. SGP with SEE Towards Improved Efficiency

  • At 40B/76B, SGP has competitive speed and superior performance to FastV and ToMe, especially at low retention; SEE preserves competitive performance at 20%/40% exits while reducing average inference time across sizes.At 26B, SGP without SEE is slower than FastV and ToMe because of the 2B small VLM’s overhead.Source: §4.3 · p. 6, Figure 3 · p. 6

4.4. Ablation Study of Key Designs

  • At 9% retention, aggregating more small-model layers improves average performance over single-layer large-model attention and reaches oracle-comparable results; using prompt and generated tokens avoids the instability of either alone.The small-model aggregation slightly outperforms the oracle on SEED.Source: Table 2 · p. 6, §4.4 · p. 7, Table 3 · p. 7
  • Sconfidence beats the other early-exiting baselines except Sconsistency; combining them into S achieves outstanding performance, while Sconsistency costs less than 1/10 of initial small-VLM inference time.Figure 4 compares decision strategies by the area between each strategy’s curve and the 2B model score.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5. Visualization of Token Pruning and Answers

  • SGP retains answer-relevant tokens across retention levels, including when the small VLM answers incorrectly; FastV preserves only partial relevant thumbnail tokens and misses them in the main image.The visualization includes pruning of 60%, 80%, and 95%, corresponding to average retention ratios of 64%, 35%, and 9%.Source: §4.5 · p. 7, Figure 5 · p. 8

4.6. Generalization on Different Size Models

  • The method is robust to small-model choice and compatible with different language models; InternVL2-1B slightly outperforms InternVL2-2B on three tasks, and 2B guidance extends to 40B and 76B VLMs.The large-model evaluation fixes the small model as InternVL2-2B.Source: §4.6 · p. 8, Table 4 · p. 8, Table 5 · p. 8

4.7. Generalization on Various Architectures

  • With Qwen2-VL-2B or LLaVa-OV-0.5B guidance, the corresponding 72B models retain approximately 96% of original performance at 9% visual-token retention.The result supports applicability across the evaluated architectures.Source: §4.7 · p. 8, Table 6 · p. 8

5. Conclusion

SGL combines small-VLM-guided pruning with early exiting, and both techniques are training-free. Experiments across 11 benchmarks show effectiveness, particularly at low visual-token retention; applicability to VLMs unifying understanding and generation remains future work.

5. Conclusion

  • A small VLM’s global attention resembles a larger VLM’s, guiding SGP pruning in large VLMs, while SEE uses the small VLM’s predictions; both techniques are training-free.Source: §5 · pp. 8–9
  • Experiments across 11 benchmarks demonstrate SGL’s effectiveness, particularly at low visual-token retention ratios.Source: §5 · p. 9
  • The method is primarily validated on multimodal-understanding tasks; applying it to VLMs that unify understanding and generation remains future work.Source: §5 · p. 9

A. SGP with SEE Towards Improved Efficiency

With both SGP and SEE, SGL has competitive inference speeds and superior performance to FastV and ToMe at 40B and 76B, particularly at low token retention. At 26B, SGP without SEE is slower than those baselines, in part because of the small VLM’s computation and autoregressive generation on RefCOCO.

A. SGP with SEE Towards Improved Efficiency

B. Memory Efficiency

Although SGL adds a small VLM alongside the large VLM, the small model uses only a minimal portion of memory, and SGL retains memory efficiency.

B. Memory Efficiency

  • The small VLM adds only a minimal portion of memory relative to the large model, and SGL retains memory efficiency.The authors report that Table 7 verifies this finding.Source: B. Memory Efficiency · p. 12

C. Visualization

The visualizations show cases where a large VLM with SGP-pruned tokens answers correctly despite an incorrect small-VLM prediction, while the large VLM with FastV also fails. The memory analysis reports peak-memory measures for SGL at 9% average retention.

C. Visualization

  • In examples where the 2B small VLM is incorrect, the 26B large VLM with SGP-pruned tokens answers correctly, while large-VLM FastV also fails.Source: C. Visualization · p. 12
  • Table 7 compares small-VLM memory, large-VLM peak memory, and SGL peak memory at 9% average retention; it defines Δ as the SGL-minus-large-model peak-memory difference.Source: Table 7 · p. 13, C. Visualization · p. 13

D. Model Descriptions

The appendix identifies where model configurations are documented and visualizes SGP pruning at several layers and token-retention levels.

D. Model Descriptions

  • InternVL, QWen2-VL, and LLaVa-OV configurations are detailed in Tables 8, 9, and 10, respectively.The tables describe InternVL [7], QWen2-VL [54], and LLaVa-OV [27].Source: D. Model Descriptions · p. 13, Table 8 · p. 13, Table 9 · p. 13
  • At 60%, 80%, and 95% pruning in the 48-layer, 26B large VLM, SGP prunes at layers 19, 9, and 2, yielding average retention ratios of 64%, 35%, and 9%.Figure 8 presents additional visualizations across these retention ratios and answers.Source: Figure 8 · p. 14
  • Figure 8 highlights retained tokens and places InternVL thumbnails in the left corner.Source: Figure 8 · p. 14
Loading 2412.03324v2…