PaperlayerView exact revision on arXiv

Source-linked AI summary

From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankers

Siyi Liu, Hanjun Yang, Chenchen Zhang, Xiaorong Zhu, Xinyu Zuo, Lisheng Duan, Haijin Liang, Jin Ma, Junfu Pu, Yongqi Zhang

arXiv:2609.00667v1cs.IRcs.CV

TL;DR

Listwise VLM rerankers need to prune many visual tokens without disrupting query-conditioned candidate order, yet attention saliency can emphasize common, order-neutral regions and varies in reliability by layer. RaDiCal combines an attention-free rank-discriminative prior with entropy-calibrated saliency and entropy-based pruning-layer selection. Across retrieval benchmarks and VLM architectures, it preserves strong ranking performance while reducing computation and transferring without dataset-specific retuning.

  • Problem

    Visual-token pruning for listwise VLM reranking must preserve candidate order, but saliency can emphasize visually prominent regions shared across candidates and misaligned with ranking contribution.

  • Method

    RaDiCal is a training-free framework that fuses query-relevant, cross-candidate Discriminative Token Importance with layer-specific saliency using normalized attention entropy for trust calibration and pruning-layer selection.

  • Results

    Across three benchmarks and two VLM architectures, RaDiCal matches or exceeds Dense at a 20% token budget, ranks first among pruning methods on FashionIQ, and delivers up to 1.45× measured speedups.

  • Takeaways & Limitations

    Entropy provides a shared trust landscape for deciding when saliency is informative and where pruning should occur, supporting rank-preserving pruning without dataset-specific retuning.

  • Takeaways & Limitations

    RaDiCal is designed for listwise reranking, has not been evaluated on other multi-image tasks, and assumes softmax-based attention.

Abstract

from arXiv · show

Large vision-language models used as listwise rerankers must jointly process visual tokens from tens of candidates per query, making token pruning essential for practical deployment. Existing pruning methods retain tokens by attention saliency, yet we show that saliency is systematically misaligned with ranking contribution: visually prominent tokens often capture order-neutral patterns shared across candidates. This mismatch is layer-dependent: saliency becomes informative only where attention is concentrated, and normalized attention entropy diagnoses the reliability shift (Pearson r=0.87). We propose RaDiCal (Rank-Discriminative Calibration), a training-free framework that uses normalized attention entropy to decide when saliency can be trusted, fusing it with an attention-free rank-discriminative prior and selecting pruning layers from the same trust landscape. Across three retrieval benchmarks and multiple VLM architectures, RaDiCal matches Dense MRR@10 on Flickr30K and surpasses it on MSCOCO at a 20% token budget, ranks first among all pruning methods on FashionIQ, and holds within 1.2 pp on Flickr30K and MSCOCO at 10% retention. It cuts FLOPs by 39--45% and delivers 1.28--1.45$\times$ measured speedups across two VLM architectures without dataset-specific retuning.

1 Introduction

Listwise VLM reranking makes visual-token pruning necessary, but preserving visual saliency alone can discard evidence that determines candidate order. RaDiCal addresses this mismatch by combining rank-discriminative token scoring with entropy-calibrated saliency and layer selection.

  • Tens of candidates and hundreds or thousands of visual tokens per image make visual-token pruning essential for practical listwise VLM reranking.
  • Saliency can favor visually prominent regions shared across candidates, whereas ranking depends on query-relevant evidence that distinguishes candidates.For the dog-and-frisbee example, the dog may be common, while the red frisbee and catching interaction are discriminative.
  • Saliency–ranking alignment is indistinguishable from a random-ordering null at multiple layers, but becomes informative where attention is concentrated.The layer dependence motivates calibration rather than unconditional use or wholesale removal of attention.
  • RaDiCal combines attention-free Discriminative Token Importance, based on query relevance and cross-candidate distinctiveness, with entropy-calibrated layer-specific saliency.The framework also uses the entropy-derived trust signal to select pruning layers without dataset-specific sweeps.
  • Across three benchmarks and two VLM architectures, RaDiCal matches or exceeds Dense on Flickr30K and MSCOCO at a 20% token budget, ranks first among pruning methods on FashionIQ, and reaches 1.45× measured speedups.The reported results transfer across datasets, model scales, and architectures without retuning.

2 Related Work

Prior visual-token pruning work largely targets single-image preservation, while listwise reranking requires preserving candidate order under shared-context comparison. Existing evidence also shows that attention reliability depends on context and layer, motivating a rank-aware alternative.

  • Existing attention-based, redundancy-aware, and upstream compression methods primarily optimize single-image preservation, often with task-specific training or learned policies.
  • Listwise reranking evaluates candidates relationally within a shared context, so token value must account for cross-candidate comparison rather than one input alone.
  • Attention does not consistently equal contribution: low-attention tokens can affect outputs, high-attention tokens can act as probability dumps, and random pruning can match designed methods.
  • Saliency reliability varies across layers and tasks, while attention entropy can indicate when saliency is informative but remains a weak correctness predictor in isolation.
  • Prior multi-image methods exploit redundancy, diversity, layouts, or temporal continuity, but these criteria do not target discrete candidate-order preservation in listwise reranking.

3 Problem Formulation and Diagnostics

The paper formulates pruning as preserving relative candidate order while diagnosing when attention saliency is reliable across decoder layers. Normalized attention entropy provides the layer-wise trust signal used to motivate calibrated pruning decisions.

  • 3.1 Saliency Reliability Is Layer-Conditioned: A query and candidate set are processed in shared multimodal context, with pruning defined as removing visual tokens while maintaining relative candidate order.
  • 3.1 Saliency Reliability Is Layer-Conditioned: Each layer’s saliency reliability is measured by a permutation-corrected z-score against a random-ordering null.
  • 3.1 Saliency Reliability Is Layer-Conditioned: Reliability is strongly non-monotonic: shallow layers can have z_l ≈ 0, while selected middle and deep layers recover stronger alignment.
  • 3.1 Saliency Reliability Is Layer-Conditioned: A fixed pruning layer can prune where saliency is unreliable or miss the layer window where saliency becomes informative.
  • 3.2 Entropy as a Trust Signal: Hnorm is computed from attention restricted to visual-token columns and renormalized within each text row, measuring how discriminative layer attention is.
  • 3.2 Entropy as a Trust Signal: Hnorm ≈ 1 indicates near-uniform attention, whereas Hnorm ≈ 0 indicates concentrated attention where saliency is informative; 1 − Hnorm(l) measures normalized information gain.
  • 3.2 Entropy as a Trust Signal: The entropy profile is non-monotonic and model-intrinsic, with Flickr30K and MSCOCO yielding Pearson r = 0.9993 and requiring no per-dataset recalibration.
  • 3.2 Entropy as a Trust Signal: Reliability correlates positively with information density I(l) = 1 − Hnorm(l), with Pearson r = 0.87 across 36 decoder layers.This supports Hnorm as an inference-time proxy for how much to trust saliency and motivates an attention-free prior, calibrated fusion, and multi-regime layer scheduling.

4 Method: RaDiCal

RaDiCal combines a layer-invariant rank-discriminative prior with layer-specific attention saliency, calibrating their relative trust from normalized attention entropy. The same trust landscape selects pruning layers and governs token retention without dataset-specific tuning.

  • Overview: RaDiCal uses Hnorm as a shared coordinate for token-score fusion and pruning-layer selection.The framework combines DTI and AttentionInfo through α(l), then applies α-Maximin to choose layers spanning diverse trust regimes.
  • Token signals: DTI scores tokens using query-conditioned cross-candidate discriminativeness, while AttentionInfo captures layer-specific information content.DTI is computed once from ViT outputs before the LLM forward pass and reused across selected layers.
  • Trust calibration: α(l) shifts trust from saliency to DTI as normalized attention entropy increases.At the sharpest layer, saliency fully governs selection; under uniform attention, DTI governs entirely.
  • Trust-calibrated fusion: The log-linear fusion is scale-invariant and softly suppresses tokens receiving near-zero values from either signal.At α=0 the method uses pure saliency, while at α=1 it uses pure DTI.
  • Layer scheduling: α-Maximin selects K pruning layers by spanning diverse trust regimes with a minimum layer-gap guard.Selected layers receive a uniform per-layer keep ratio whose sequential product realizes the global retention budget R.
  • Deployment: RaDiCal separates an offline model-and-budget schedule from an online query-and-candidate-set pass, with K and the layer-gap guard fixed across datasets.The external keep ratio R controls the evaluation budget, while the schedule is fixed for a given model and budget.

5 Experimental Results

Across three retrieval benchmarks and multiple VLM settings, RaDiCal preserves or improves reranking quality under aggressive visual-token pruning while offering favorable measured efficiency. Its calibrated token scoring and layer selection remain robust across budgets, candidate similarity, candidate-set sizes, model architectures, and retriever pools.

  • Efficiency Analysis: RaDiCal lies on the quality–efficiency Pareto frontier, saves over 40% of Dense FLOPs at R=20%, and delivers 1.28–1.45× measured end-to-end speedups across two backbones.At R=10%, analytical FLOP savings exceed 50% with minimal additional degradation; analytical compression does not predict wall-clock speedup.
  • Ablation Analysis: Ablations show that token scoring and pruning-layer selection have roughly equal importance, while fixed-weight mixing underperforms disabling DTI and both individual channels trail calibrated fusion.These results support adaptive trust calibration and complementary DTI and AttentionInfo signals.
  • Robustness Analysis: RaDiCal preserves the gap to Dense across candidate-similarity quintiles and remains nearly flat around Dense as candidate-set size N increases from 5 to 50.Its relative MRR@10 varies by barely 1 pp around Dense, while SparseVLM and FastV degrade by an order of magnitude more.
  • Model Generalization: The method transfers across Qwen3-VL-8B and InternVL2.5-8B, with α-Maximin selecting distinct schedules automatically and RaDiCal leading on MRR@10 across architectures and retriever pools.It matches or exceeds Dense across Qwen, Jina, and SigLIP2 candidate pools over five seeds despite pairwise pool overlap below 40%.

6 Conclusion

The paper shows that attention saliency is misaligned with ranking contribution, while normalized attention entropy identifies when saliency becomes trustworthy. RaDiCal unifies discriminative scoring, trust calibration, and pruning-layer scheduling, achieving strong ranking quality and efficient inference without retuning.

  • Normalized attention entropy diagnoses where attention saliency becomes trustworthy for ranking contribution.
  • RaDiCal unifies token scoring, saliency calibration, and pruning-layer scheduling through a single entropy-derived coordinate.
  • RaDiCal matches Dense on Flickr30K and exceeds it on MSCOCO at a 20% token budget.
  • RaDiCal delivers measured speedups of up to 1.45× and transfers across datasets, model scales, and architectures without retuning.

Limitations

The method’s scope is limited to settings where cross-candidate contrast and softmax-based attention are available, and its training-free calibration choices remain unexplored. Evaluation also does not cover a wider range of retrieval domains or modalities.

  • RaDiCal is designed for listwise reranking, while applicability to other multi-image tasks has not been evaluated.
  • The study does not investigate learning the calibration weight α or DTI scoring weights for task-specific gains.
  • The method assumes softmax-based attention and may not generalize to linear-attention or state-space architectures.
  • Evaluation spans several benchmarks and VLM families but not a wider range of retrieval domains or modalities.

Ethical Considerations

The paper reports no new data or human-subject annotations and relies on public benchmarks and pretrained models. It also identifies deployment concerns involving energy, inherited risks, subgroup effects, and human oversight.

  • The work collects no new data or human-subject annotations, using public benchmarks and pretrained models under their original licenses.
  • The authors do not measure energy use or carbon emissions, or mitigate inherited biases, privacy risks, and hallucinations.
  • Because pruning may alter candidate exposure and affect underrepresented groups, subgroup evaluation and human oversight are recommended before high-stakes deployment.
  • Experiments use public retrieval benchmarks, pretrained models, and fixed training-free baseline configurations under a shared global retention budget.

F.2 Cross-Method Retention Comparison

The comparison examines retention patterns on a difficult, high-similarity query where shared visual content weakens saliency’s discriminative value. RaDiCal concentrates tokens on query-specific evidence, unlike the more uniform or shallow-layer-biased alternatives.

  • Query difficulty: The query requires identifying a woman, three dogs, and a field among candidates sharing the visual pattern “person + dog + outdoor.”
  • Saliency-only: Saliency-only retains prominent storefronts, murals, and high-contrast textures despite their irrelevance to the query.
  • RaDiCal: RaDiCal retains the three dogs and woman’s head in the ground-truth candidate while allocating little budget to clearly mismatched candidates.
  • PyramidDrop: PyramidDrop spreads an approximately equal 19–20% budget per candidate in a near-uniform grid, limiting exploitation of cross-candidate differences.
  • SparseVLM: SparseVLM prunes at shallow layers with Hnorm≈1, removing critical woman-head tokens and producing position-biased retention.

G Layer Selection Strategy Comparison

RaDiCal selects pruning layers by spreading stages across distinct regions of the α trust landscape, with four stages providing the strongest empirical balance. The resulting schedule preserves ranking quality while reducing computational cost without measurable online scoring overhead.

  • Layer Selection Strategy: α-Maximin selects layers [7, 22, 24, 29], spanning DTI-dominant, mixed, saliency-leaning, and saliency-only regimes.The α values are [0.84, 0.43, 0.22, 0.00], providing complementary rather than redundant pruning decisions.
  • Layer Selection Strategy: 1.38 pp: removing shallow layer L7 lowers MRR@10 to 82.60, showing that deep-only selection is inferior.The observed drop supports retaining a high-α shallow stage for full α-space coverage.
  • Stage-Count Sensitivity: K=4 is the empirical optimum: its MRR@10 nearly closes the gap to Dense, whereas K=5 reverses quality after adding insufficiently specialized layer L4.Most of the gain appears by K=3, with K=4 supplying further refinement.
  • Layer-Gap Sensitivity: g ≤2 yields the highest mean MRR@10 on both datasets; over-spacing removes L24 and degrades performance by up to 2.30 pp.The loss follows from breaking dense coverage of the steep Hnorm descent between L22 and L29.
  • Efficiency: 39.2% FLOPs↓: RaDiCal matches random pruning’s InternVL2.5-8B latency closely, at 3.81 versus 3.89 s/q, while achieving higher ranking quality.Online DTI and entropy scoring add no measurable runtime cost; offline profiles are amortized across queries.

J Retriever Robustness

RaDiCal remains robust across substantially different first-stage retriever candidate pools. Its pool-conditioned scoring preserves performance near Dense and outperforms resolution-level compression under matched visual-token budgets.

  • Retriever Robustness: 99.93–100.57% of Dense MRR@10: RaDiCal matches or exceeds Dense across all three retrievers.The candidate pools come from Qwen3-VL-Embedding-2B, Jina Embeddings v4, and SigLIP2base-patch16-512.
  • Retriever Robustness: 4.26–4.72 pp: RaDiCal consistently leads PyramidDrop across the three retriever-generated candidate pools.The result is reported on Flickr30K at R=20% over five seeds.
  • Resolution Baselines: 2.4–3.7 pp: token-level RaDiCal leads the best DynRes variant across both datasets at the same total visual-token budget.Candidate-level resolution allocation cannot substitute for within-image token selection, even when guided by DTI.

L FashionIQ: Conditional and Unconditional Metrics

FashionIQ metrics are reported under both conditional and unconditional evaluation protocols, with identical method ordering. RaDiCal’s reranking-specific design also has measurable transfer boundaries on single-image tasks, especially OCR-heavy benchmarks.

  • Evaluation Protocols: Method ordering is identical under conditional evaluation over 1,599 queries and unconditional evaluation over all 6,016 queries.The conditional protocol requires the ground-truth target to appear in the first-stage top-20 candidate set.
  • Scope: CCU is reranking-specific because cross-candidate uniqueness is undefined without multiple candidates.AttentionInfo, α calibration, and α-Maximin scheduling remain candidate-independent and can transfer to single-image tasks.
  • Single-Image Transfer: On six single-image benchmarks, the −CCU variant leads on POPE-adv. and COCO-Caption and remains close to PyramidDrop on GQA and MME.It is within 0.02 pp on GQA and 0.50 pp on MME, while OCR-heavy DocVQA and TextVQA remain outside its listwise target regime.
Loading 2609.00667v1…