PaperlayerView exact revision on arXiv

Source-linked AI summary

When 1+1 Backfires: Rethinking Relevance-Diversity Balancing for Sample-Aware Visual Token Reduction

Yihang Huang, Peisong Wen, Chenhao Zhou, Difu Feng, Qianqian Xu

arXiv:2610.16615v1cs.AI

TL;DR

Visual token reduction lowers the inference cost of VLMs, but fixed relevance–diversity balances do not account for variation across tasks and samples. SAVER infers a sample-specific weight from text-relevant token geometry and applies it to relevance and diversity selection. It consistently outperforms competitive baselines across compression budgets, retaining 96.5% of full-model performance with only 11% of visual tokens.

  • Problem

    Existing token-reduction methods use relevance, diversity, or fixed combinations, while the optimal balance varies substantially across benchmarks and inputs.

    Source: §1 · p. 2

  • Method

    SAVER uses the Geometric Intent Prior to infer a sample-specific balance from text-relevant token geometry, then applies it to relevance scoring and diversity-aware selection.

    Source: §1 · p. 2

  • Results

    96.5% of full-model performance is retained using only 11% of visual tokens, and SAVER consistently outperforms existing approaches under equivalent compression budgets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Effective visual token reduction requires sample-aware relevance–diversity balancing rather than a fixed strategy.

    Source: §1 · p. 2, §6 · p. 8

    GIP uses the top 2% of visual tokens by relevance score as indicators of query visual grounding.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Visual token compression has emerged as a critical technique for reducing the inference cost of Vision-Language Models. Existing methods select tokens based on text relevance, visual diversity, or a fixed combination of both---treating the relevance-diversity balance as a universal constant independent of the input. We conduct a systematic analysis to challenge this assumption. At the dataset level, grid-searching the optimal relevance-diversity weight per benchmark reveals substantial cross-task variation. More critically, we further maximize per-sample answer likelihood to recover sample-level oracle weights, finding that their distributions not only differ markedly across benchmarks but also exhibit high within-benchmark variance. Based on these observations, we propose SAVER, a framework for Sample-Aware relevance-diversity balanced Visual tokEn Reduction. We first derive a Geometric Intent Prior (GIP) from the spatial concentration of text-relevant tokens to infer a sample-specific relevance-diversity weight, and then apply this weight to jointly select tokens via a Weighted Text Centroid (WTC) for relevance scoring and a Visual Marginal Coverage Gain (VMCG) for diversity. SAVER requires no additional training and achieves sample-aware balancing entirely at inference time. Extensive experiments demonstrate that SAVER consistently outperforms state-of-the-art baselines across diverse benchmarks, retaining 96.5% of full-model performance on LLaVA-1.5-7B with only 11.1% of the original visual tokens, and further generalizing to LLaVA-NeXT and Mini-Gemini.

1 Introduction

Fixed relevance–diversity combinations miss the variation in optimal balance across benchmarks and individual inputs. SAVER estimates a sample-specific balance and uses it to guide visual token selection.

1 Introduction

  • Existing relevance-only methods can discard peripheral context, diversity-only methods can retain query-irrelevant regions, and fixed combinations do not model sample-dependent trade-offs.Prior input-agnostic balancing does not consistently outperform single-objective baselines, a phenomenon termed “1 + 1 < 1.”Source: §1 · p. 2
  • Optimal relevance–diversity weights vary substantially across benchmarks and individual inputs, so effective token reduction requires sample-aware balancing.The analysis estimates benchmark optima and uses per-sample answer likelihood to recover oracle weights.Source: §1 · p. 2
  • A practical sample-aware prior must use available input knowledge and shallow early-prefill semantics without dataset-specific context or extra inference.The text identifies per-dataset configuration and auxiliary prediction or deeper-layer semantics as barriers to open-world, efficient deployment.Source: §1 · p. 2
  • GIP infers each sample’s balance from the spatial distribution of text-relevant tokens: concentration favors relevance, while dispersion favors broader visual coverage.SAVER pairs GIP with a Weighted Text Centroid for relevance scoring and Visual Marginal Coverage Gain for diversity-aware selection.Source: §1 · p. 2
  • SAVER retains 96.5% of full-model performance using only 11% of visual tokens, while outperforming existing approaches under equivalent compression budgets.It combines a sample-specific weight from GIP with a norm-weighted text centroid for relevance and facility-location marginal gain for coverage.Source: §1 · p. 2

2 Related Work

LVLMs use many visual tokens to represent image semantics, increasing sequence length and computational overhead. Token-reduction methods address this cost through text relevance, visual coverage, or combined objectives, but fixed mixtures do not adapt to input-specific trade-offs.

2.1 Large Vision-Language Models

  • Many visual tokens increase LVLM input length and computational overhead, bottlenecking inference efficiency.The figure compares dataset-level performance across θ with instance-level oracle-weight distributions; POPE favors high θ, MMBench low θ, and GQA intermediate values.Source: §2.1 · p. 3, Figure 1 · p. 3

2.2 Token Reduction Methods

  • Text-relevance methods can discard peripheral context, while visual-coverage methods preserve broader context but may spend tokens on regions irrelevant to the instruction.Recent combined methods often use fixed weighting, although the optimal balance varies across questions; MoB requires knowing the target dataset to set parameters.Source: §2.2 · p. 3

3 Motivation

The section examines whether relevance-diversity balance should be fixed globally by studying benchmark-level optima and per-sample answer-likelihood optima. It finds substantial variation across benchmarks and within each benchmark.

  • A weighted score combines text relevance and visual diversity; θ ∈ [0, 1] controls the balance, with larger θ giving greater weight to diversity under fixed K.Source: §3 · p. 3
  • The analysis defines benchmark-level optimal weights using task evaluation metrics and sample-level oracle weights by maximizing correct-answer likelihood.Per-sample answer likelihood serves as a surrogate because no continuous metric directly reflects sample-level token selection quality.Source: §3 · p. 3
  • SAVER's Geometric Intent Prior uses the spatial distribution of high-relevance tokens to dynamically weight WTC relevance and VMCG diversity in greedy selection.Source: Figure 2 · p. 4
  • Optimal relevance-diversity weights vary substantially across benchmarks and within benchmarks, showing that the balance is not determined by task alone.Source: §3 · p. 4
  • POPE favors high θ for surveying diverse regions, MMBench favors low θ for fine local details, and GQA peaks at an intermediate value.Source: §3 · p. 4

4 Methodology

SAVER compresses visual tokens by combining text relevance and visual diversity, with a sample-specific balance inferred from token geometry. It pairs this prior with attention-free relevance scoring and coverage-based diversity selection.

4 Methodology

  • SAVER combines a sample-specific relevance-diversity balance with Weighted Text Centroid relevance and Visual Marginal Coverage Gain diversity objectives.The framework derives the balance from text-relevant token geometry and selects a fixed-budget token subset for decoding.Source: §4 · p. 4, §4.1 · p. 4

4.1 Preliminaries

  • Visual token reduction keeps a subset of K≪N encoder tokens so the decoder processes a shorter visual-text sequence, reducing prefilling cost.The method selects tokens at an early LLM layer.Source: §4.1 · p. 4

4.2 Geometric Intent Prior

  • The Geometric Intent Prior measures the spatial spread of highly text-relevant tokens, using concentrated tokens to favor relevance and dispersed tokens to favor diversity.The prior uses the determinant of the spatial covariance matrix; its magnitude reflects the area spanned by the anchor tokens.Source: §4.2 · pp. 5–6

4.3 Text Relevance via Weighted Text Centroid

  • Weighted Text Centroid estimates relevance from a norm-weighted centroid of text embeddings, avoiding attention maps and reducing function-word interference.Each visual token is scored by cosine similarity to the centroid; the approach requires one inner product per visual token.Source: §4.3 · p. 6

4.4 Visual Diversity via Visual Marginal Coverage Gain

  • Visual Marginal Coverage Gain scores a candidate by both its novelty and its improvement in coverage of tokens that remain poorly represented.The facility-location criterion suppresses redundant selections and outliers, addressing failure modes of max-min and coverage-based alternatives.Source: §4.4 · p. 6

4.5 Sample-Aware Token Selection

  • The framework scales the GIP mapping using intrinsic image diversity, then greedily adds the highest-scoring token and updates diversity scores until the budget is met.The image-diversity statistic is normalized and used only to scale the GIP mapping.Source: §4 · p. 4, §4.5 · p. 7

5 Experiments

Across three model families, SAVER leads the reported token-retention comparisons, while efficiency results and ablations support sample-aware balancing without added computational overhead. The experiments cover general visual understanding, OCR and document understanding, efficiency, and component choices.

5.1 Experimental Setup

  • The experiments cover eight standard-protocol benchmarks and evaluate SAVER primarily on LLaVA-1.5-7B, with LLaVA-NeXT-7B and Mini-Gemini testing generalizability.The benchmarks include GQA, MMBench, MME, POPE, ScienceQA, SEEDBench, TextVQA, and OCRBench; models use their original inference configurations without additional training.Source: §5.1 · p. 7

5.2 Main Results

  • At K=64, SAVER retains 96.5% of full-model performance on LLaVA-1.5-7B, and it leads the reported comparisons on LLaVA-NeXT-7B and Mini-Gemini.The reported SAVER results are 98.4% at K=128 and 99.6% at K=192 on LLaVA-1.5-7B; 98.5% at K=640 and 96.2% at K=320 on LLaVA-NeXT; and 98.1% at K=128 and 94.7% at K=64 on Mini-Gemini.Source: §5.2 · p. 8

5.3 Efficiency Analysis

  • SAVER matches DART in peak memory and throughput while scoring 83.8 on POPE, versus DART’s 73.9 and SparseVLM’s 75.1.SparseVLM consumes more GPU memory than the uncompressed model; the comparison is at K=64 on LLaVA-1.5-7B.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Ablation Study

  • Figure 3 reports POPE F1 across token budgets; the effect of γ is amplified at higher compression ratios.The text says overly small γ under-emphasizes novelty and favors redundant regions, while overly large γ prioritizes outliers at the expense of global coverage; moderate values work best, and the optimum may vary with budget.Source: Figure 3 · p. 8, §5.4 · p. 8
  • SAVER’s sample-aware variant achieves a 96.9% average, above the static combination’s 95.9%, diversity alone’s 95.0%, and relevance alone’s 91.0%.Relevance alone scores 73.7 on POPE, while diversity alone scores 83.4 on POPE and 59.9 on MMBench; the text attributes SAVER’s gains to dynamically inferring a sample-specific balance via GIP.Source: §5.4 · p. 8

6 Conclusion

The paper finds that relevance-diversity balance varies across tasks and inputs, and presents SAVER as a training-free framework for inferring sample-specific weights. Experiments report consistent gains across compression budgets while matching baseline inference efficiency.

6 Conclusion

  • SAVER combines a sample-specific Geometric Intent Prior with a Weighted Text Centroid for relevance and a Visual Marginal Coverage Gain for diversity.The paper reports that the prior is inferred from the spatial distribution of high-relevance tokens.Source: §6 · p. 8

Related papers

Loading 2610.16615v1…