When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model

1 INTRODUCTION

• Image-based selection can discard requested details, while text-guided selection can preserve related support yet lose the corresponding value.At 80% pruning on Qwen3-VL-8B, ZOO-Prune loses 28.8% and 34.7% relative to Dense on TextVQA and TextCaps; SparseVLM loses more than 50% on ChartQA and InfoVQA.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Text-to-visual attention identifies answer-relevant regions more strongly at intermediate decoder layers than near the input, making guidance timing critical.Source: §1 · p. 2
• The method exceeds the strongest baseline by 11.10 and 16.84 percentage points at 80% and 90% pruning, respectively.It achieves the highest average Dense-relative performance retention across evaluated pruning ratios, benchmarks, and models.Source: §1 · p. 2
• The method prunes early with vision-encoder attention, retains extra candidates, then uses midpoint text-to-visual attention to select the final tokens.Source: §1 · p. 2

2.1 LARGE VISION-LANGUAGE MODELS

• VLM research extends language models with visual encoders and multimodal alignment, while architecture, training, and high-resolution processing affect visual understanding and token costs.The section discusses CogVLM, Idefics2, LLaVA variants, and InternVL.Source: §2.1 · p. 2

2.2 VISUAL TOKEN PRUNING

• Figure 2 tracks text-guided evidence identification across decoder depth, while Table 1 describes a selection-depth ablation at 80% pruning under matched processing budgets.Figure 2 reports four QA benchmarks; Table 1 reports eight-task Dense-relative recovery.Source: Figure 2 · p. 3, Table 1 · p. 3
• Visual token reduction includes merging, trained pruning, and training-free selection using visual importance, text–visual attention, or combinations of relevance and diversity.Other approaches preserve information through merging or recycling and correct token-reduction distortions; this paper examines when textual guidance becomes informative.Source: §2.2 · pp. 2–3

3.1 MOTIVATION AND DESIGN RATIONALE

• The depth analysis samples 300 images per benchmark, uses two questions per image, and measures answer effects when each of eight spatial regions is masked.The evaluation covers four multimodal QA benchmarks and uses the unpruned model to measure token-averaged negative log-likelihood of the correct answer.Source: §3.1 · p. 3
• Text-to-visual attention alignment with question-relevant regions is weak early but becomes substantially stronger at intermediate decoder depths.The analysis also compares attention from the matching question with attention from another question about the same image.Source: §3.1 · p. 4
• The decoder midpoint achieves the highest recovery across all three models under matched budgets, while retaining enough candidates at practical cost.The depth-wise analysis and budget-controlled ablation motivate setting L = D/2 for text-guided reselection.Source: §3.1 · p. 4

3.2 PROPOSED METHOD

• Vision-encoder attention drives pre-LLM pruning, avoiding decoder computation and reserving textual guidance for later.The early pruning stage remains purely vision-based because textual guidance is less informative in early decoder layers.Source: §3.2 · p. 5
• The method preserves an expanded image-ranked candidate set, then uses midpoint text-to-visual attention alone to form the final visual-token set.Candidates outside the initial vision-guided top-K can enter the final set when they become more relevant to the input.Source: §3.2 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation covers three VLMs, eight benchmarks, and 70%, 80%, and 90% pruning, reporting each benchmark’s native metric and Dense-normalized average relative performance.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

• 11.10 and 16.84 points over the strongest baseline at 80% and 90% pruning, respectively; at 90% pruning, Qwen3-VL-8B retains 90.65% of Dense performance while all competitors remain below 70%.Source: Table 2 · p. 6, Table 3 · p. 6, §4.2 · pp. 6–7
• Gains are especially pronounced on TextVQA, ChartQA, and InfoVQA, where performance depends on preserving localized, prompt-relevant evidence.Source: §4.2 · p. 7

4.3 ABLATION STUDIES

• With the same token-count trajectory, midpoint text-to-visual reselection improves on candidate preservation alone; ablations favor vision-encoder attention for early pruning and midpoint text-to-visual attention for reselection.Source: Figure 5 · p. 8, §4.3 · p. 8, Figure 6 · p. 8

4.4 ANALYSIS

• Increasing reserve fraction improves recovery but raises LLM-prefill latency; the trade-off is approximately linear for α ∈ [0.1, 0.3], and α = 0.2 is the default.The paper reports practical latency with only modest additional inference cost despite retaining extra candidates.Source: Figure 7 · p. 9, §4.4 · p. 9, Table 5 · p. 9

• The method provides substantial performance gains with practical inference efficiency and may serve as a strong baseline for future visual-token-pruning work.Source: §5 · p. 9

A ADDITIONAL PROTOCOL DETAILS

• The evaluation covers eight benchmarks, with task-specific scoring and CIDEr and ANLS displayed after multiplication by 100.The sets range from 900 MMMU examples to 5,000 TextVQA examples.Source: A ADDITIONAL PROTOCOL DETAILS · p. 12
• Avg. Rel. equally averages task scores normalized by Dense scores, while four-task pruning ablations and the depth diagnostic use different benchmark subsets.Avg. Rel. can exceed 100%. The ablations use InfoVQA, ChartQA, TextCaps, and MMStar; the depth diagnostic uses TextVQA, ChartQA, InfoVQA, and AI2D.Source: A ADDITIONAL PROTOCOL DETAILS · p. 12
• Within each backbone, methods share inputs, preprocessing, generation settings, and scorers, with selected baseline implementations adapted for comparison.FastV retains two initial decoder blocks; SparseVLM is matched by average decoder visual-token usage, and RESTORE also applies position and attention correction inside the language model.Source: A ADDITIONAL PROTOCOL DETAILS · p. 13

A.1 SELECTION IMPLEMENTATION

• Qwen aggregates vision-encoder attention received by visual keys across visual queries and heads, while LLaVA-OneVision uses head-averaged CLS-to-patch attention.LLaVA-OneVision scores come from the penultimate vision-encoder block, and scores are aligned to each backbone’s visual-token geometry before top-M selection.Source: §A.1 · p. 13
• Midpoint scores use prompt tokens as decoder queries, without using correct answers or generated answer tokens for selection.When attention probabilities are not exposed by the kernel, scores are recomputed from the corresponding queries and keys.Source: §A.1 · p. 13
• A token–block count measures visual-token processing for matching decoder workloads across selection depths and progressive-pruning baselines.Analytical decoder FLOPs are computed from sequence lengths before and after reselection and averaged over individual inputs.Source: §A.1 · p. 13

A.2 RESERVE-SIZE TRADE-OFF

• At 80% pruning, Figure 8 compares analytical decoder-prefill FLOPs and GPU-input-to-first-token latency across five reserve fractions.Recovery is averaged equally over all eight benchmarks.Source: §A.2 · p. 13
• Across all three backbones, larger reserves improve recovery while increasing computation, with diminishing gains at larger reserve fractions.The same trend appears in analytical FLOPs, so it is not specific to measured latency.Source: §A.2 · p. 13
• The relative increase in first-token latency is smaller because it includes vision processing and other reserve-independent costs.The reserve fraction α controls the trade-off between candidate preservation and inference cost.Source: §A.2 · p. 13

B ADDITIONAL ABLATION ANALYSIS

• Approximately 39–45% of final visual tokens come from candidates outside the initial vision-guided top-K.This indicates that midpoint text-to-visual attention substantially revises the initial visual ranking.Source: B ADDITIONAL ABLATION ANALYSIS · p. 13
• The largest recovery gains from candidate preservation to text-guided reselection appear on InfoVQA and ChartQA.These benchmarks require complex reasoning over relationships among textual and visual elements.Source: B ADDITIONAL ABLATION ANALYSIS · p. 13
• Table 6 reports both the fraction of final tokens selected from outside the initial top-K and the recovery gain from preservation to reselection.Source: B ADDITIONAL ABLATION ANALYSIS · p. 13

C COMPARISON WITH PROGRESSIVE PRUNING

• The progressive-method comparison matches each baseline to Ours’ per-input visual token–block budget at α = 0.2, up to integer rounding.The 80% and 90% settings refer to Ours’ final pruning ratios; baseline final token counts may differ under matched workloads.Source: C COMPARISON WITH PROGRESSIVE PRUNING · pp. 13–14
• Figure 8 compares reserve-size trade-offs across three backbones, showing analytical decoder-prefill FLOPs above and measured GPU-input-to-first-token latency below.Labels vary α from 0.10 to 0.30, the fraction of otherwise pruned tokens retained as candidates until midpoint reselection.Source: Figure 8 · p. 14
• Table 6 reports the share of final tokens selected outside the initial vision-guided top-K and the recovery gain from candidate preservation to midpoint reselection.The recovery gain is measured relative to Dense under the same token-count trajectory.Source: Table 6 · p. 14
• At 80% pruning, Ours exceeds FlowCut and PyramidDrop by 9.51 and 19.26 percentage points; at 90%, the margins are 9.47 and 27.82 points.The comparison matches per-input visual token–block budgets up to integer rounding; progressive baselines may therefore use different final token counts.Source: C COMPARISON WITH PROGRESSIVE PRUNING · p. 14

D QUALITATIVE RESULTS

• Existing pruning strategies can discard visually small but task-relevant evidence; this is especially apparent when answers depend on localized text, numbers, or chart elements.ZOO-Prune may retain salient regions while missing details, whereas SparseVLM may retain related support but lose the precise evidence or relationships needed.Source: D QUALITATIVE RESULTS · p. 14
• Table 7 compares two progressive pruning methods with Ours on Qwen3-VL-8B under matched per-input visual token–block budgets.It reports eight-task mean performance relative to Dense; the 80% and 90% settings denote Ours’ final pruning ratios, and baseline final token counts may differ.Source: Table 7 · p. 15
• Deferred reselection can preserve initially lower-ranked candidates until text-to-visual attention becomes more informative, helping retain question-relevant evidence.Examples extend beyond question answering to diagrams and multimodal reasoning, as well as the visual information needed for open-ended descriptions.Source: D QUALITATIVE RESULTS · p. 15