Source-linked AI summary
[CLS] Token Tells Everything Needed for Training-free Efficient MLLMs
Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding
TL;DR
Large visual-token sequences make efficient MLLM deployment difficult, and prompt-based pruning can overlook response-relevant image content. VTC-CLS uses visual-encoder [CLS] attention, ensembling importance scores across layers, to compress tokens without training. The authors report state-of-the-art performance across compression levels and inference acceleration.
Problem
Large numbers of visual tokens increase MLLM computation and memory costs, while FastV's prompt-attention importance signal can overlook response-relevant visual context.
Source: §1 · p. 1Method
VTC-CLS prunes visual tokens using [CLS]-to-token attention from the visual encoder and ensembles importance scores across layers.
Source: §3.3 · p. 6Results
At 256 visual tokens, VTC-CLS improves overall performance by 1.2% over FastV and 1.1% over PruMerge+, and achieves state-of-the-art performance across compression levels.
Source: §4.2 · p. 6Takeaways & Limitations
VTC-CLS is a plug-and-play, training-free method that accelerates MLLM inference while maintaining competitive performance.
Source: §1 · p. 2, §4.3 · p. 7
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across a wide range of vision-language tasks, garnering significant attention in the computer vision. However, their efficient deployment remains a substantial challenge due to high computational costs and memory requirements. Recognizing the redundancy of information within the vision modality, recent studies have explored methods for compressing visual tokens in MLLMs to enhance efficiency in a training-free manner. Despite their effectiveness, existing methods like Fast rely on the attention between visual tokens and prompt text tokens as the importance indicator, overlooking the relevance to response text and thus introducing perception bias. In this paper, we demonstrate that in MLLMs, the [CLS] token in the visual encoder inherently knows which visual tokens are important for MLLMs. Building on this prior, we introduce a simple yet effective method for train-free visual token compression, called VTC-CLS. Firstly, it leverages the attention score of the [CLS] token on visual tokens as an importance indicator for pruning visual tokens. Besides, we also explore ensembling the importance scores derived by the [CLS] token from different layers to capture the key visual information more comprehensively. Extensive experiments demonstrate that our VTC-CLS achieves the state-of-the-art performance across various tasks compared with baseline methods. It also brings notably less computational costs in a training-free manner, highlighting its effectiveness and superiority. Code and models are available at \url{https://github.com/THU-MIG/VTC-CLS}.
1. Introduction
Visual-token redundancy raises MLLM inference costs, while prompt-attention pruning can miss response-relevant context. VTC-CLS uses visual-encoder [CLS] attention to select important tokens, including through an ensemble across layers.
1. Introduction
- Large visual-token sequences increase inference computation and memory use, while training compact connectors requires substantial resources.LLaVA-1.5 maps 336×336 and 672×672 images to 576 and 2304 visual tokens, respectively.Source: §1 · p. 1
- FastV uses attention between visual tokens and prompt text as its importance signal, which can discard visual context relevant to the generated response.The paper calls this perception bias.Source: §1 · p. 2
- The visual encoder's [CLS] token carries prior information about important visual tokens, and the paper reports consistency between those tokens and important tokens in the LLM.The authors use this consistency to motivate pruning visual tokens from the encoder perspective.Source: §1 · p. 2
- VTC-CLS uses [CLS]-to-visual-token attention to select important tokens and ensembles importance scores across layers for comprehensive compression.The authors describe it as a plug-and-play, training-free method.Source: §1 · p. 2
- The authors report state-of-the-art performance across visual-language tasks compared with existing methods, alongside notable inference acceleration.They describe extensive experiments across various tasks.Source: §1 · p. 2
2. Related Work
Related work spans MLLM architectures and efficiency strategies, including visual-token compression. The paper distinguishes VTC-CLS from a prior [CLS]-based method by its training-free, eviction-based design and cross-layer importance ensemble.
Multimodal Large Language Models
- MLLMs combine visual encoders with LLMs, with works such as Flamingo and BLIP-2 bridging the modality gap and MiniGPT-4 and LLaVA using instruction tuning.The cited examples are described as supporting zero-shot visual-language tasks or multimodal generation for complex tasks.Source: §2 · p. 2
Visual Token Compression
- Efficiency work spans data selection, compact backbones, mixture-of-experts, connector designs, and visual-token compression.Visual-token approaches include special-token compression, adaptive pooling, token merging, and stacking tokens into groups.Source: §2 · pp. 2–3
- Prior compression methods may require resources to train new architectures or incur performance degradation from suboptimal pruning, while LLaVA-PruMerge uses [CLS] with a finetuned merging framework.VTC-CLS instead uses training-free, eviction-based compression and ensembles importance estimates across layers; the authors report it outperforms the training-free PruMerge variant.Source: §2 · p. 3
3. Methodology
The method estimates visual-token importance from [CLS] attention in the visual encoder, using its observed consistency with LLM importance to guide training-free token pruning. It also ensembles scores across encoder layers to select tokens for the LLM.
3.1. Preliminary
- Output-token attention to visual tokens reflects their significance in the LLM, while [CLS]-to-patch attention scores estimate visual-token importance in the encoder.The method averages attention across heads to form the visual-encoder score; response tokens are unknown in advance, motivating estimation from the encoder.Source: §3.1 · pp. 3–4, §3.2 · p. 4
3.2. Consistency of Visual Token Importance
- [CLS]-based encoder importance shows consistently high overlap and rank correlation with LLM importance, unlike random scores, supporting encoder-side approximation.Figure 2 reports that ensembling scores across layers further strengthens consistency, with K=3 the best setting.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
3.3. Visual Token Compression by [CLS] Token
- VTC-CLS retains the top U visual tokens by [CLS]-based importance, concatenates them with text tokens, and feeds them to the LLM for inference.The retained tokens use averaged [CLS] attention across heads, with the penultimate layer as the single-layer score source.Source: §3.3 · p. 5, Figure 3 · p. 5
- Ensembling [CLS] importance scores from multiple visual-encoder layers improves overlap and rank correlation with LLM importance; K=3 gives the strongest correlation.The method averages scores from K layers starting at the penultimate layer and moving backward.Source: Figure 2 · p. 4, §3.3 · pp. 5–6
4. Experiments
Experiments evaluate VTC-CLS training-free on LLaVA-1.5 across eight benchmarks, reporting compressed performance, inference efficiency, model analyses, and visualizations of retained tokens.
4.1. Implementation Details
- The evaluation uses LLaVA-1.5 with ViT-L and Vicuna v1.5 across eight benchmarks, with all methods evaluated in a training-free manner.The image subset of ScienceQA assesses zero-shot scientific question answering, while the benchmarks cover tasks including text-rich visual question answering, hallucination, visual reasoning, and conversation.Source: §4.1 · p. 6
4.2. Main Results
- VTC-CLS achieves state-of-the-art compressed performance: at 256 tokens it improves overall by 1.2% over FastV and 1.1% over PruMerge+, and at 64 tokens by 2.1% over PruMerge+ on average.At 128 tokens, it exceeds PruMerge+ by 2.5% on POPE and FastV by 4.1% on GQA; with more than half the tokens compressed, average performance degradation is 0.4%.Source: §4.2 · pp. 6–7
4.3. Model analyses
- With 128 visual tokens retained, VTC-CLS reduces evaluation time by 32.1% on GQA and 29.8% on POPE; with 64 tokens, it yields 1.7× and 1.6× speedups, respectively.The analyses measure evaluation time on a single NVIDIA 3090 GPU and report competitive performance alongside the speedups.Source: §4.3 · p. 7, Table 2 · p. 7
- At 128 retained tokens, VTC-CLS outperforms Random, ToMe, and KMeans overall, although token merging helps those baselines on POPE.The authors attribute the difference to VTC-CLS selecting important tokens rather than potentially merging features with discrepancies.Source: K GQA POPE MMVet SEED Avg · p. 7, Table 3 · p. 7
- Performance with different K values first rises and then falls, peaking at K=3; using K>1 also consistently beats K=1.The authors relate this pattern to pursuing consistency with LLM token-importance scores and aggregating information across layers.Source: K GQA POPE MMVet SEED Avg · pp. 7–8, Table 4 · p. 7
- Average, maximum, and minimum score ensembling each outperform no ensembling across benchmarks, while average gives the best overall performance.This supports the benefit of combining importance scores across layers beyond any one ensemble function.Source: Table 5 · p. 7, K GQA POPE MMVet SEED Avg · p. 8
- When compression is applied after the second LLM layer, VTC-CLS outperforms FastV across benchmarks, indicating its efficacy is not limited to the compression location.The experiments use importance scores derived from the visual encoder for this LLM-layer compression setting.Source: K GQA POPE MMVet SEED Avg · p. 8, Table 6 · p. 8
4.4. Visualization
- Visualizations show VTC-CLS discarding redundant patches while retaining salient regions associated with both prompt and response text.In one example, retained patches represent “stop” in the prompt and “war” in the response, supporting reasoning about “below.”Source: §4.4 · p. 8, Figure 4 · p. 8
5. Conclusion
The conclusion presents visual-encoder token importance as a useful prior for compressing MLLMs’ visual context. VTC-CLS uses [CLS] attention, ensembles scores across layers, and reports state-of-the-art benchmark performance with inference acceleration.
5. Conclusion
- The visual encoder provides a prior for identifying important visual tokens, which VTC-CLS uses to compress MLLM visual context.The method ranks tokens with [CLS] attention scores and ensembles scores from different layers to capture key visual information more comprehensively.Source: §5 · pp. 8–9
- Across various benchmarks, VTC-CLS achieves state-of-the-art performance alongside notable inference acceleration.Source: §5 · p. 9