Source-linked AI summary
Adaptive Visual Token Reduction for Accelerated Image Understanding
Seyoung Jeong, Jong Pil Yun, Sang Jun Lee
TL;DR
High-resolution, information-rich images make LVLM inference costly, while token pruning and fixed-size regions can limit preservation of spatially structured information. ReFIT adapts regions to instruction relevance and refines tokens; across four VQA benchmarks, it improves accuracy while reducing computation.
Problem
High-resolution processing of information-rich images increases inference cost, while token pruning and fixed-size region selection have limits in preserving spatially structured information.
Source: §1 · p. 1Method
ReFIT uses RWR to adaptively select instruction-relevant regions and ITR to remove low-relevance visual tokens within them.
Source: §1 · p. 1Results
Across four VQA benchmarks, ReFIT improves accuracy and reduces computational cost; MPDocVQA ANLS is 0.4125, improving over vanilla by 0.0367.
Source: §3.2 · p. 3, §4 · p. 4Takeaways & Limitations
Experiments validate RWR and ITR for efficient LVLM inference, with ReFIT surpassing the vanilla LVLM at substantially lower computation.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Large Vision-Language Models achieve strong VQA performance, but processing high-resolution, information-rich images requires substantial computation, motivating visual token reduction. However, existing methods often prune individual tokens or rely on fixed-size cropping, limiting their ability to preserve spatially structured information such as horizontally or vertically elongated text. To address this limitation, we propose ReFIT, an instruction-guided visual token reduction framework for efficient LVLM inference. ReFIT consists of Relevance-Guided Window Reshaping (RWR) and Instruction-Guided Token Refinement (ITR), where RWR captures instruction-relevant regions by adapting to their spatial characteristics, while ITR further removes unnecessary visual tokens. Experiments on four VQA benchmarks demonstrate that ReFIT improves answer accuracy while reducing computational cost, and qualitative results demonstrate its effectiveness in localizing relevant regions and removing unnecessary visual information.
1. INTRODUCTION
Information-rich images often require high-resolution inputs and long visual token sequences, yet relevant evidence may occupy only a small portion of the image. ReFIT addresses limits in token pruning and fixed-region selection with instruction-guided region reshaping and token refinement.
1. INTRODUCTION
- High-resolution inputs for fine-grained image details produce longer visual token sequences and increase inference cost, although relevant evidence may occupy only a limited image area.The introduction identifies small text, tables, and densely arranged elements as examples of fine-grained evidence.Source: §1 · p. 1
- Token pruning can fail to preserve spatially structured content, while fixed-size region selection may miss text or tables extending beyond window boundaries.Even correctly selected regions may retain irrelevant information that degrades answer accuracy.Source: §1 · p. 1
- RWR adapts window configurations to capture instruction-relevant regions across diverse spatial layouts rather than relying on fixed region structures.ITR uses an instruction-guided relevance map to remove low-relevance tokens while preserving relevant information; cropped regions are re-encoded before token refinement.Source: §1 · p. 1, Figure 1 · p. 2
- ReFIT improves answer accuracy and computational efficiency on InfoVQA, SPDocVQA, MPDocVQA, and GQA.The framework uses RWR to select instruction-relevant regions and ITR to remove low-relevance tokens.Source: §1 · p. 1
2. METHODOLOGY
ReFIT encodes image features and instruction tokens, then uses their relevance to select spatially adapted regions and refine tokens before LLM inference.
2. METHODOLOGY
- ReFIT combines multi-resolution visual features and instruction-aligned token features to estimate relevance, reshape windows, crop and re-encode selected regions, and refine tokens before forwarding them to the LLM.The remaining visual tokens are passed with the instruction for answer generation.Source: §2 · p. 2
2.1. Feature Extraction
- Visual features are projected into T tokens of dimension d, arranged as a spatial grid and grouped into local windows; instruction tokens are embedded and projected into a shared feature space.The passage defines T as the number of visual tokens and d as the feature dimension.Source: §2.1 · p. 2
2.2. Relevance-Guided Window Reshaping
- RWR ranks windows by instruction relevance and adapts neighboring-window aggregation horizontally or vertically to match relevant regions’ spatial extent.The method adjusts the number of neighboring windows from 2 to 4 to reduce fragmentation from fixed boundaries.Source: §2.2 · pp. 2–3
2.3. Instruction-Guided Token Refinement
- ITR builds an instruction-guided relevance map and assigns tokens to Protected, Retained, or Removed states; tokens in the top-2 RWR-ranked regions are always preserved.Protected and Retained tokens form the final set forwarded to the LLM.Source: §2.3 · p. 3
3. EXPERIMENTS
Across four VQA benchmarks, ReFIT improves answer performance while reducing computation. The experiments evaluate its benchmark performance, qualitative localization, and the contributions of its components.
3.1. Implementation and Evaluation Details
- Evaluation uses LLaVA-NeXT-Vicuna-7B across four VQA benchmarks, reports ANLS for three document QA datasets and accuracy for GQA, and measures computation with FLOPs.The vision encoder, projection layer, and LLM backbone remain frozen during training; only the proposed modules are optimized.Source: §3.1 · p. 3
3.2. Quantitative Analysis
- ReFIT achieves the best performance across four VQA benchmarks; MPDocVQA ANLS reaches 0.4125, 0.0367 above vanilla, with FLOPs falling from 50.01 to 27.91 TFLOPs.On GQA, accuracy is 0.7669 at 14.94 TFLOPs, a 50.8% computational-cost reduction versus vanilla.Source: §3.2 · pp. 3–4
3.3. Qualitative Analysis
- Compared with PinPoint, ReFIT better crops horizontally extended text and removes unnecessary tokens within the selected regions.The qualitative comparison marks ground-truth regions in red and model-cropped regions in green; PinPoint's fixed-size windows can truncate extended text and yield incorrect answers.Source: §3.3 · p. 4, Figure 2 · p. 4
3.4. Ablation study
- RWR and ITR each improve performance, their combination performs best across benchmarks, and each outperforms alternative region or token selection strategies.RWR is compared with fixed or random region selection; ITR is compared with random or spatially connected token removal.Source: §3.4 · p. 4, Table 2 · p. 4, Table 3 · p. 4
4. CONCLUSION
ReFIT accelerates LVLM inference for information-rich image understanding by selecting instruction-relevant regions and refining their visual tokens. On four VQA benchmarks, it improves answer accuracy and computational efficiency, including compared with the vanilla LVLM.
4. CONCLUSION
- ReFIT combines adaptive window reshaping to identify instruction-relevant regions with token refinement that removes irrelevant visual tokens within those regions.Together, these stages yield a compact set of instruction-relevant visual evidence.Source: §4 · p. 4
- On InfoVQA, SPDocVQA, MPDocVQA, and GQA, ReFIT improves answer accuracy while reducing computational cost.The paper reports that ReFIT surpasses the vanilla LVLM with substantially lower computation.Source: §4 · p. 4
- The results support RWR and ITR as effective components for efficient LVLM inference on the evaluated VQA benchmarks.Source: §4 · p. 4