PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond One-Size-Fits-All: Sample-Adaptive Strategy Routing for Vision Token Pruning in MLLMs

Haiji Liang, Pengfei Zhou, Zhenglin Wan, Wei Wang, Yang You, Wangbo Zhao

arXiv:2609.10346v1cs.CVcs.AI

TL;DR

Fixed vision-token pruning strategies can be suboptimal for individual inputs despite strong average performance, limiting efficient MLLM inference. VIP-Router predicts the best strategy or Full from low-cost features at each pruning level. On VTC-Bench Group A, it improves average accuracy by 26.9% and average utility by 22.0% over Best Fixed while remaining lightweight and plug-and-play.

  • Problem

    Existing pruning methods use fixed strategies, while average benchmark rankings conceal substantial sample-level complementarity among strategies.

  • Method

    VIP-Router predicts per-input strategy utility from low-cost visual, textual, and query-conditioned features, retaining Full as an option when pruning is unfavorable.

  • Results

    VIP-Router consistently outperforms Best Fixed across five reduction ratios, improving average accuracy by 26.9% and average utility by 22.0% on VTC-Bench Group A.

  • Takeaways & Limitations

    The results support sample-adaptive pruning-strategy selection as an additional axis for adaptive MLLM inference, with gains across ratios, backbones, and unseen benchmarks.

  • Takeaways & Limitations

    Empirical results are restricted to VTC-Bench Group A, and direct zero-shot router transfer across unseen backbones generally fails to preserve backbone-specific advantages.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) process hundreds or thousands of visual tokens per image, incurring prohibitive inference costs. While existing vision token pruning methods mitigate this overhead, they implicitly assume that a single fixed pruning strategy can be applied uniformly across all inputs. Our analysis further reveals that ranking pruning methods by average benchmark accuracy conceals substantial sample-wise complementarity: although the average-best strategy excels overall, alternative strategies prove superior on a significant fraction of individual samples. To harness this diversity, we propose VIP-Router, a lightweight VIsion Pruning Router that adaptively selects the pruning strategy predicted to be best suited to each input at a specified pruning level. Conditioned on low-cost visual and textual features, VIP-Router identifies the most suitable candidate strategy while retaining full-token inference as an option when pruning is predicted to be unfavorable. Evaluated on a curated suite of pruning-sensitive visual perception benchmarks, VTC-Bench Group A, VIP-Router consistently outperforms the best fixed strategy baseline across all reduction ratios, achieving a 26.9% relative improvement in average accuracy, and a 22.0% relative increase in average utility after accounting for realized token cost. Crucially, VIP-Router operates in a plug-and-play manner without modifying underlying pruning algorithms or model weights, introducing trainable parameters equivalent to merely 0.017\% of the backbone. Furthermore, VIP-Router proves effective across various MLLM backbones and yields consistent gains on unseen benchmarks, highlighting the potential of sample adaptive routing for visual token pruning.

1. Introduction

Vision token pruning reduces MLLM costs, but fixed strategies are not uniformly effective: benchmark- and sample-level complementarity leaves substantial room for adaptive selection. VIP-Router exploits low-cost input features to choose among pruning strategies or full-token inference, improving accuracy and utility without changing the underlying MLLM.

  • Motivation: Hundreds or thousands of visual tokens per image make MLLM inference computationally expensive, motivating vision token pruning.Pruning methods remove or consolidate visual tokens to reduce memory and latency.
  • Complementarity: Different pruning strategies excel on different VTC-Bench Group A tasks, so no single strategy dominates across benchmarks.Group A contains samples that are correct with full tokens but fail after equivalent-ratio downsampling.
  • Complementarity: More than one-third of samples solvable by at least one strategy are missed by Best Fixed, showing that average-best does not imply sample-best.This sample-level complementarity is distinct from ranking methods only by average benchmark accuracy.
  • Approach: VIP-Router uses low-cost visual and textual features to predict the best candidate strategy for each input at a specified pruning level.It performs sample-adaptive selection rather than executing multiple candidate strategies to discover the best outcome after the fact.
  • Results: 26.9% relative accuracy improvement and 22.0% relative utility improvement over Best Fixed are achieved on VTC-Bench Group A.VIP-Router consistently outperforms fixed strategies across all five reduction ratios, adds about 1.4M trainable parameters, and does not modify candidate pruning methods or the MLLM pipeline.
  • Contributions: The work characterizes sample-level complementarity and establishes cost-aware gains across pruning levels, backbones, and unseen benchmarks.Full-token inference remains available when pruning is predicted to be unfavorable.

2. Related Work

Prior work reduces visual-token computation using diverse fixed pruning criteria and adapts computation through token quantity, location, granularity, or compression pathways. VIP-Router instead adapts which pruning criterion is applied to each sample at a specified pruning level.

  • Vision Token Pruning: Vision token pruning removes or consolidates redundant visual tokens before or within the language model to improve MLLM efficiency.Existing criteria include attention, attention-statistic matching, token dominance, diversity, duplication, and decoding-time task relevance.
  • Vision Token Pruning: Existing pruning approaches fundamentally apply a single fixed pruning criterion to all inputs.VIP-Router treats these methods as complementary candidates and selects among them per sample.
  • Adaptive Inference: Adaptive inference methods vary the amount or location of computation, including input- or layer-dependent visual-token retention.Examples include dynamic context sparsification, adaptive retention ratios, and progressive in-model pruning.
  • Positioning: VIP-Router adapts a different axis: which pruning criterion to apply to each sample at a specified pruning level.This distinguishes strategy selection from adapting computation amount, location, representation granularity, or compression pathway.

3. Vision Pruning Router

VIP-Router formulates pruning as cost-aware selection among existing strategies and Full, then predicts candidate utilities from frozen previews, cross-modal features, and the retained-token ratio. A shared router selects one option for execution in the unchanged MLLM.

  • Architecture: VIP-Router uses frozen visual and textual preview encoders, text-to-vision cross-attention, and a ratio-conditioned MLP to estimate candidate utilities.The representation combines global and token-level features with query-conditioned visual information.
  • Candidate Set: The candidate set contains Full, FastV, VisionZip, PruMerge+, and DART, with Full retaining all visual tokens.Compressed candidates apply existing pruning rules within the Group A formulation.
  • Problem Formulation: The retained-token ratio r specifies the compressed candidates’ operating point, while VIP-Router may select Full when pruning is unfavorable.The corresponding reduction ratio is 1 − r, but r is not a hard per-sample token constraint.
  • Objective: The utility objective ranks a correctness-preserving compressed strategy above Full and Full above an incorrect compressed strategy.This provides a cost-aware no-pruning option under Group A.
  • Routing: At test time, a learned router predicts all candidate utilities and selects the highest-utility option at the specified retained-token ratio.A shared router is conditioned on r instead of training a separate router for each ratio.
  • Training and Inference: Only feature projections, cross-attention, and the utility predictor are updated; preview encoders and the MLLM remain frozen during single-pass inference.The selected strategy runs within the unchanged MLLM after utility prediction.

4. Experiments

VIP-Router is evaluated against fixed pruning baselines using cost-aware accuracy, utility, efficiency, parameter overhead, backbone generalization, transfer, and staged ablations. Across these evaluations, it improves performance while retaining a lightweight, adaptable design.

  • Experimental setup: VIP-Router is compared with uniform fixed strategies, Best Fixed, and the Per-Sample Oracle across pruning ratios on VTC-Bench Group A.Best Fixed selects one strategy using training-set average utility, while the oracle uses ground-truth test labels.
  • Cost-aware performance: 22.0%: average utility rises from 30.88 to 37.68 over five reduction ratios, exceeding the fixed-method envelope at every ratio.The gain ranges from 5.07 to 8.91 utility points, while the macro-average oracle headroom recovery is 26.5%.
  • Accuracy across reduction ratios and benchmarks: 26.9%: average accuracy increases from 40.35% to 51.19%, with VIP-Router achieving the highest average accuracy at every reduction ratio.It obtains the best pruning result in 30 of 40 benchmark–ratio combinations and reaches a 45.2% relative gain at 99% token reduction.
  • Accuracy–efficiency trade-off: VIP-Router remains above fixed-strategy curves over most evaluated compute ranges beyond approximately 4.5 TFLOPs after routing and pruning FLOPs are included.At 99% reduction, more frequent full-token selection raises realized cost and makes the matched-compute point less competitive, although same-ratio accuracy remains substantially higher.
  • Generalization and ablations: VIP-Router transfers positive utility gains to all four unseen benchmarks and works across multiple MLLM backbones, although transfer strength varies with domain shift.Ablations show that query-conditioned spatial cross-modal interaction recovers the performance loss from sharing one ratio-conditioned router, reaching parity with five ratio-specific routers at 38.09 versus 38.07 utility.

5. Conclusion

VIP-Router reframes vision token pruning as sample-adaptive strategy selection rather than uniform application of a fixed criterion. It improves accuracy and utility across VTC-Bench Group A while remaining effective across backbones and unseen benchmarks, but the reported empirical scope has important boundaries.

  • VIP-Router consistently improves accuracy and utility over Best Fixed across five reduction ratios on VTC-Bench Group A.The framework selects a suitable pruning strategy per sample instead of applying one fixed strategy uniformly.
  • The router remains effective across different MLLM backbones and transfers zero-shot to unseen benchmarks.These results support pruning-strategy choice as an axis of adaptive inference within the evaluated settings.
  • All empirical results are restricted to Group A, leaving scaling to unfiltered benchmarks insufficiently evaluated.The paper also reports that direct zero-shot transfer across unseen target architectures often fails to preserve backbone-specific gains.
  • Direct zero-shot policy transfer across target backbones often falls behind static baseline policies, suggesting pruning preferences remain backbone-dependent.This limitation concerns transfer without target-specific adaptation, not the router’s backbone-specific evaluation.

A.1. Theoretical Motivation: Utility Prediction from a Compact Representation

The theoretical motivation formalizes routing as utility prediction from a compact preview representation. It characterizes the oracle, decision margins, prediction-regret bounds, and the limits of what representation enrichment can resolve.

  • VIP-Router compares candidate utilities in S = {Full} ∪ Sp, where Sp contains pruning strategies.Under Group A, Full is correct by construction, while pruning utility depends on correctness and retained-token cost.
  • For pruning strategy s, the Bayes-optimal utility predictor equals conditional success probability minus retained-token ratio, while Full has utility 0.Thus routing can estimate conditional pruning success and compare it against the known cost threshold.
  • The representation gap is uncertainty surviving compression of (x,r) into z, so reducing it requires enriching the preview representation rather than enlarging the predictor.The three-branch design uses global text, global vision, and query-conditioned spatial evidence to address this gap.
  • If utility prediction error is at most ε, selected-strategy regret is at most 2ε; preserving the optimal decision requires ε < Δ(x,r)/2.For r ∈ {0.04, 0.0625, 0.1111, 0.25}, ε < 0.02 suffices for all samples, while r = 0.01 requires ε < 0.005.
  • The analysis does not establish preview sufficiency a priori; feature ablations and recovered utility headroom provide empirical evidence instead.This explicitly limits the theoretical claim to accurate enough conditional pruning-success estimation for preserving utility ordering.
  • Full-token fallback contributes at most 0.41 average utility points under the reported worst-case counterfactual, or 6.1% of VIP-Router’s observed gain over Best Fixed.The bound is based on replacing Full selections with compressed strategies that are all assumed incorrect.

A.2. Algorithm

VIP-Router trains a utility predictor from frozen preview features and uses it to select one candidate strategy at inference time. The training and inference procedures standardize candidate utilities, optimize the router, and execute the selected option once.

  • Training: Training constructs utility targets for every sample and candidate, computes training-set normalization statistics, and standardizes each candidate target.The router then forms a standardized utility vector for optimization.
  • Training: Frozen visual and textual preview features are combined through query-conditioned cross-modal features before standardized candidate utilities are predicted.Only the router components are optimized; preview encoders and the MLLM remain frozen.
  • Training: VIP-Router selects the checkpoint with highest validation utility after recovering predictions to the original utility scale.Training uses validation utility for checkpoint selection.
  • Inference: At inference, the router extracts frozen preview features, predicts candidate utilities at the requested retained-token ratio, and selects the highest-utility candidate.The selected candidate is then applied and the frozen MLLM runs once.
  • Inference: The inference procedure executes the chosen strategy once rather than evaluating multiple candidate outputs.This preserves the computational purpose of adaptive pruning selection.

A.3.1. Dataset Construction and Split Details

The routing dataset reconstructs ratio-specific pruning-sensitive Group A examples from released VTC-Bench inference results. Records are split at the image–question level using a deterministic 70/15/15 partition, while eligible-example counts vary by reduction ratio.

  • Routing examples use released per-sample inference results for full-token inference, image downsampling, and four candidate pruning methods.Binary correctness is derived from task-specific evaluation fields, including multiple-choice matching for MMBench and MMBench-CN.
  • Group A includes an image–question record at a ratio when full-token inference is correct but equivalent-ratio image downsampling is incorrect.Correctness labels for FastV, VisionZip, PruMerge+, and DART are associated with each eligible record–ratio pair.
  • The dataset contains 33,091 routing examples from 12,919 unique image–question records.Group A is reconstructed independently at each reduction ratio.
  • The five retained-token ratios correspond to visual-token reduction ratios of 75.00%, 88.89%, 93.75%, 96.00%, and 99.00%.The router receives r and ln(r + 10^-8) as ratio inputs.
  • Splitting uses unique benchmark:doc_id records and a deterministic hash-based 70/15/15 partition, producing 8,988 training, 1,956 validation, and 1,975 test records.No record is shared across splits, although the same image may occur in distinct image–question records.
  • Eligible-example counts vary across reduction ratios because Group A membership is ratio-dependent.This property is documented in the dataset-construction table caption.

A.3.2. Router Architecture and Training Configuration

VIP-Router combines frozen visual-textual preview features, cross-attention, ratio conditioning, and utility prediction to select among pruning strategies or full-token inference. Training standardizes features and candidate utilities using only the training split, then optimizes a Huber objective with AdamW.

  • Router Architecture: VIP-Router uses frozen CLIP visual and textual representations, retaining 49 visual patch tokens and a global visual feature.Questions are capped at 77 tokens, and CLIP parameters remain frozen during router training.
  • Router Architecture: A four-head text-to-vision cross-attention layer produces a query-conditioned visual representation for utility prediction.Projected text tokens serve as queries, while projected visual tokens serve as keys and values; the EOS output is retained.
  • Utility Prediction: The utility predictor scores FastV, VisionZip, PruMerge+, DART, and Full, with the highest predicted utility determining the selected candidate.Full-token inference is treated as an ordinary predicted candidate rather than being analytically fixed.
  • Training Configuration: Training standardizes preview features and candidate utilities using training-set statistics only, while validation and test samples are excluded from statistic estimation.Candidate utility targets are standardized separately, and constant dimensions such as Full utility use scale 1.
  • Training Configuration: The router is optimized with AdamW using learning rate 10^-3, weight decay 10^-4, batch size 256, warm-up, cosine decay, gradient clipping, and early stopping.The objective is an element-wise Huber loss with δ=1 averaged over five candidate utilities; frozen preview features are cached and reused.

A.3.3. Hardware and Software Environment

The evaluation environment uses a single NVIDIA H200 GPU and standard Python, PyTorch, CUDA, and Transformers software. Encoder robustness experiments retain the routing architecture while replacing frozen preview encoders, and detailed results show broad gains across benchmarks and reduction ratios.

  • Hardware and Software Environment: All experiments run on a single NVIDIA H200 GPU with Python 3.10.12, PyTorch 2.3.0, CUDA 12.4, and Transformers 4.49.0.Router training is performed in FP32.
  • Encoder Analysis: Alternative frozen preview encoders preserve the routing objective, candidate set, data split, and architecture while retaining cross-attention, pooling, ratio conditioning, and utility prediction.The default configuration uses CLIP ViT-B/32 and its paired text encoder.
  • Encoder Analysis: VIP-Router remains effective across multiple preview encoder families, with EUPE–CLIP slightly exceeding default utility and SigLIP2-B/32 showing a larger decrease.These results support operation with multiple frozen vision-language representations rather than dependence on one feature family.
  • Per-Benchmark Results: VIP-Router achieves the best pruning result in 30 of 40 benchmark–ratio combinations, or 75.0%, and exceeds the strongest fixed strategy at every evaluated reduction ratio.VisionZip is strongest among fixed strategies at 75.00%, 88.89%, and 99.00% reduction, while PruMerge+ and DART lead at 93.75% and 96.00%.
  • Selection Behavior: Routing remains distributed across pruning methods, with DART selected increasingly under aggressive reduction and VisionZip remaining substantial throughout the evaluated range.Full-token inference is selected for 4.92%, 3.72%, 3.51%, 3.48%, and 6.66% of samples across the five ratios, respectively.
  • Routing Error Diagnosis: High-margin mismatches constitute 40.9% of strategy disagreements but contribute 63.5% of total regret, while 28.9% of argmax mismatches incur zero regret.Near-tie mismatches still contribute 36.5% of total regret, so exact oracle-argmax agreement is insufficient to assess routing quality.
Loading 2609.10346v1…