Source-linked AI summary
HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models
Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S. Nikolopoulos, Hans Vandierendonck, Deepu John, Bo Ji
TL;DR
High-resolution VLMs improve visual detail but create excessive token counts that burden inference on resource-constrained GPUs. HiRED uses ViT CLS attention to allocate a fixed token budget across image partitions and retain informative tokens before LLM generation. On LLaVA-Next-7B, HiRED-20% improves throughput, latency, and memory use while achieving higher accuracy than prior early-dropping methods.
Problem
High-resolution VLMs generate excessive visual tokens from multiple image partitions, increasing inference cost and risking resource failures on commodity GPUs.
Method
HiRED uses ViT CLS attention to allocate a fixed token budget across image partitions and select informative visual tokens before LLM generation.
Results
HiRED-20% on LLaVA-Next-7B delivers higher accuracy than prior early-dropping methods while improving throughput, latency, and GPU-memory use.
Takeaways & Limitations
HiRED provides a plug-and-play approach for more efficient high-resolution VLM inference under resource constraints.
Takeaways & Limitations
HiRED may lose spatial information on tasks requiring crucial spatial relationships, such as ChartQA.
Abstract
from arXiv · showhide
High-resolution Vision-Language Models (VLMs) are widely used in multimodal tasks to enhance accuracy by preserving detailed image information. However, these models often generate an excessive number of visual tokens due to the need to encode multiple partitions of a high-resolution image input. Processing such a large number of visual tokens through multiple transformer networks poses significant computational challenges, particularly for resource-constrained commodity GPUs. To address this challenge, we propose High-Resolution Early Dropping (HiRED), a plug-and-play token-dropping method designed to operate within a fixed token budget. HiRED leverages the attention of CLS token in the vision transformer (ViT) to assess the visual content of the image partitions and allocate an optimal token budget for each partition accordingly. The most informative visual tokens from each partition within the allocated budget are then selected and passed to the subsequent Large Language Model (LLM). We showed that HiRED achieves superior accuracy and performance, compared to existing token-dropping methods. Empirically, HiRED-20% (i.e., a 20% token budget) on LLaVA-Next-7B achieves a 4.7x increase in token generation throughput, reduces response latency by 78%, and saves 14% of GPU memory for single inference on an NVIDIA TESLA P40 (24 GB). For larger batch sizes (e.g., 4), HiRED-20% prevents out-of-memory errors by cutting memory usage by 30%, while preserving throughput and latency benefits. Code - https://github.com/hasanar1f/HiRED
1 Introduction
High-resolution VLMs preserve detail by encoding multiple image partitions, but the resulting visual-token volume strains inference on commodity GPUs. HiRED addresses this with attention-guided early dropping under a fixed budget and reports substantial efficiency gains with competitive accuracy.
- Motivation: High-resolution VLMs encode multiple partitions plus a full-image representation, producing 3–10× more visual tokens than low-resolution counterparts.These tokens are projected into the text-embedding space, concatenated with text and system tokens, and processed by the LLM.
- Motivation: Resource constraints make visual-token control essential because transformer complexity, KV-cache storage, and limited GPU memory can reduce throughput or cause out-of-memory failures.The paper specifically highlights commodity GPUs such as Jetson Orin NX and NVIDIA Tesla T4.
- Method: HiRED is a plug-and-play framework that allocates a fixed token budget across partitions, drops tokens before LLM generation, and targets broad multimodal-task coverage without architectural changes.Its stated design goals include high-resolution support, controllable budgets, early dropping, and wide task coverage.
- Method: HiRED uses early-layer CLS-to-patch attention to estimate partition content and final-layer CLS attention to select informative tokens for the LLM.The framework combines budget allocation and token dropping based on attention patterns in the ViT.
- Results: 4.7× higher token-generation throughput, 78% lower response latency, and 14% lower GPU memory use are achieved by HiRED-20% on LLaVA-Next-7B for single inference.On an NVIDIA TESLA P40, throughput is 2.30 versus 0.49 tokens/sec, latency is 4.21 versus 19.49 seconds, and memory is 13.76 versus 16.04 GB.
- Results: For batch size 4, HiRED-20% cuts memory usage by 30% and avoids out-of-memory errors while maintaining its throughput and latency improvements.The full token budget exceeds the 24 GB GPU capacity in this setting.
2 Related Work
Related work reduces VLM cost through smaller architectures, sparse attention, or early visual-token dropping, but these approaches differ in training requirements, resolution support, task scope, and budget control.
- Lightweight Architectures: Lightweight architectures reduce model size but can compromise reasoning, while quantization and pruning do not directly solve excessive visual-token counts.Some token-ensemble methods target cross-modal transformer VLMs, and others require expensive training or fine-tuning.
- Sparse Attention Computation in LLM and ViT: Sparse-attention methods reduce transformer computation, but FastV is not designed for high-resolution VLMs and FlexAttention does not control visual-token counts by resource constraints.
- Early Dropping of Visual Tokens: Early-dropping methods remove visual tokens before LLM processing, but TokenCorrCompressor is limited to document understanding and PruMerge targets low-resolution LLaVA.The paper applies PruMerge strategies separately to image partitions for comparison with HiRED.
3 Key Insights
The paper identifies sparse visual-token usage and unequal partition importance as opportunities for selective dropping. A small subset of tokens carries most image context, while partitions differ in how many highly attended tokens they contribute.
- Visual Token Sparsity: Visual tokens comprise 80–90% of tokens in the examined VLM but receive significantly less attention than system and text tokens.
- Visual Token Sparsity: The top 20% and 40% of visual tokens account for 60% and 80% of total attention, respectively.
- Visual Token Sparsity: A small subset of visual tokens brings most image context to the LLM, motivating removal of less important tokens without sacrificing accuracy.
- Partition Importance: The number of highly attended visual tokens varies across full-image and sub-image partitions.This variation indicates that some partitions may tolerate more token dropping than others.
4 Our Design: HiRED
HiRED uses ViT CLS-attention at different layers to allocate a fixed token budget across image partitions and retain the most informative tokens before LLM processing.
- CLS-attention Pattern in ViT: Across layers, initial ViT CLS-attention highlights image content, whereas final-layer attention emphasizes patches containing more image features.Figure 3 contrasts subject-focused early attention with informative regions distributed across content and background in later layers.
- HiRED Design: HiRED allocates a fixed token budget across the full image and sub-images, then selects high-importance tokens within each partition's allocation.The two phases are token budget allocation and visual token dropping.
- Evaluation: Table 3 compares HiRED and token-dropping baselines across accuracy metrics, with higher values indicating better performance.The table reports results for HiRED and baselines including PruMerge and PruMerge+.
- Token Budget Allocation: Initial-layer CLS-attention identifies visual content across sub-image regions, assigning larger budgets to partitions with more content.The full-image attention aggregated over corresponding tokens and heads determines each sub-image's budget fraction.
- Visual Token Dropping: Final-layer CLS-attention scores token feature importance, allowing HiRED to retain informative subject and background patches while dropping the rest.Different heads capture different features, so their final-layer attention is aggregated for ranking.
- Visual Token Dropping: The selected visual tokens are concatenated with text and system tokens before being passed to the LLM.This early reduction shortens the input sequence before LLM computation.
5 Evaluation
HiRED is evaluated across high- and low-resolution VLMs, multimodal tasks, accuracy settings, and inference-efficiency conditions. It preserves accuracy under reduced token budgets while improving throughput, latency, memory use, and budget control.
- Accuracy: HiRED-20% maintains nearly full-execution accuracy for VQA with a maximum of 576 tokens, while HiRED-40% remains comparable for fine-grained transcription with 1,152 tokens.Accuracy increases with fewer tokens for ScienceQA and POPE in some settings.
- Accuracy: HiRED outperforms PruMerge and PruMerge+ in transcription accuracy: PruMerge+ is 11% lower on TextVQA and 26% lower on DocVQA despite using 55% of tokens.PruMerge is 13% and 37% lower on TextVQA and DocVQA, respectively, while using 10% of tokens.
- Inference Efficiency: 4.7× higher token-generation throughput, 78% lower TTFT latency, and 14% lower GPU memory usage result from HiRED-20% versus full execution on LLaVA-Next-7B.At batch size 4, full execution runs out of memory on a 24 GB GPU, whereas HiRED-20% uses 16.99 GB, a 30% reduction, while maintaining throughput and latency improvements.
- Inference Efficiency: HiRED enforces a strict token budget across TextVQA samples, unlike Full, PruMerge, and PruMerge+, whose token counts vary with image structure or adaptive allocation.This makes computational cost predictable under resource constraints.
- Ablation Study: A balanced α = 0.5 allocation between full-image and sub-images generally provides the best accuracy, while initial-layer attention allocates budgets and final-layer attention selects tokens.Summing attention across heads improves accuracy over using no aggregation.
- Ablation Study: On low-resolution single-partition VLMs, HiRED stays close to full-execution accuracy at 40% and 20% token budgets across LLaVA-1.5-7B and ShareGPT4V-7B.This evaluation isolates token dropping from budget allocation and supports robustness across model types.
6 Conclusion
HiRED addresses the inference cost of high-resolution VLMs by allocating a fixed token budget and retaining informative visual tokens. The authors report improved efficiency with competitive accuracy, while noting a spatial-information limitation for tasks requiring spatial relationships.
- Conclusion: HiRED allocates a fixed token budget across image partitions, retains informative visual tokens, and drops the rest before LLM generation.The framework is presented as a practical solution for resource-constrained high-resolution VLM deployment.
- Conclusion: HiRED may lose spatial information, which can affect tasks requiring spatial relationships such as ChartQA task understanding.The authors identify 2D positional encodings after token dropping as a possible future solution.