PaperlayerView exact revision on arXiv

Source-linked AI summary

Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu

arXiv:2606.12412v1cs.CVcs.AI

TL;DR

Visual-token pruning can discard regions that become important deeper in the decoder, harming grounding. Reroute defers rather than deletes tokens, improving grounding under aggressive reduction while preserving general VQA performance and the pruning baseline’s theoretical efficiency class.

  • Problem

    Visual-token importance changes across decoder depth, challenging irreversible pruning of visual tokens for efficient VLM inference and grounding.

  • Method

    Reroute is a training-free plug-in that defers unselected vision tokens for possible re-entry at later routing stages while preserving pruning-baseline efficiency budgets.

  • Results

    Across FastV, PDrop, and Nüwa variants on LLaVA-1.5 and Qwen backbones, Reroute improves grounding under aggressive reduction while preserving general VQA performance.

  • Takeaways & Limitations

    Visual-token reduction can be framed as recoverable routing rather than only irreversible pruning, particularly for grounding-oriented evaluation under aggressive reduction.

  • Takeaways & Limitations

    Reroute inherits attention-router ranking errors, while practical latency depends on optimized gather/scatter kernels and KV-cache management.

Abstract

from arXiv · show

Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory. Existing visual-token reduction methods largely follow a rank-and-remove paradigm: they score visual tokens, keep a compact subset, and permanently discard the rest. We show that this irreversible action is fragile because visual-token importance changes across decoder depth; tokens ranked low at one stage may become relevant in later layers, especially for grounding-sensitive queries. We propose Reroute, a training-free plug-in that replaces removal with recoverable routing. At each routing stage, selected vision tokens pass through decoder blocks, while deferred tokens bypass the stage and re-enter the candidate pool at the next routing decision. Reroute reuses existing attention-score ranking rules and stage-wise schedules, preserving the theoretical TFLOPs and KV-cache budget class of the pruning method it augments. Across FastV, PDrop, and Nüwa variants on LLaVA-1.5 and Qwen backbones, reroute improves grounding under aggressive token reduction while maintaining general VQA performance. These results suggest that VLM token reduction should not be viewed only as irreversible pruning, but also as recoverable routing. The code can be found here: https://github.com/elmma/mllm-reroute/

1 Introduction

VLM visual-token pruning is fragile because token importance changes across decoder depth, causing irreversible removal to discard evidence that becomes relevant later. Reroute replaces removal with training-free, recoverable routing that preserves matched efficiency while improving grounding under aggressive reduction.

  • Motivation: VLM inference is costly because hundreds to thousands of visual tokens cause quadratic attention and linear KV-cache growth.Reducing decoder-participating visual tokens is therefore a key deployment lever.
  • Problem: Token importance shifts across decoder depth, with target-relevant regions emerging in middle and deep layers after diffuse shallow attention.A ground-truth token ranked at percentile 0.11 at layer 3 and 0.25 at layer 8 later rises to 0.97 at layer 25.
  • Problem: At 88.9% reduction, conventional pruning causes grounding collapse, with IoU falling below 0.4 across three backbones.For LLaVA-1.5-7B, the average visual-token count drops from 576 to 64.
  • Method: Reroute selects top-r fraction tokens for Attn+FFN while deferred tokens bypass each stage and re-enter the candidate pool at the next routing layer.It reuses text-to-vision attention as the ranking signal and does not delete deferred tokens.
  • Efficiency: Reroute reuses existing scorers and schedules without extra trainable parameters, preserving the FLOPs and KV-cache budget of the augmented pruning method.Irreversible pruning is the degenerate case where deferred-token re-entry is forbidden.
  • Results: Across FastV, PDrop, and Nüwa on three backbones, Reroute improves matched-budget grounding and preserves general VQA performance, with largest gains under aggressive reduction.The reported backbones include LLaVA-1.5-7B, Qwen2.5-VL-7B, and a hybrid Mamba-Transformer model.

2 Related Work

Prior work reduces VLM visual-token cost through encoder-side compression, decoder-side score-and-remove pruning, and conditional computation. This work is positioned as recoverable routing that addresses depth-varying token importance while complementing KV-cache and system optimizations.

  • Vision-encoder-side visual token reduction: Encoder-side methods compress visual information before decoding through score-and-drop, similarity-based merging, projector or [CLS]-head operations, and budget-aware sub-image partitioning.Examples include LLaVA-PruMerge, VisionZip, FasterVLM, and HiRED.
  • Decoder-side visual token pruning: Decoder-side methods largely retain a score-then-remove design, evolving from FastV’s single rule to progressive schedules, recycling, spatial safeguards, and diversity-aware selection.The cited approaches include PyramidDrop, SparseVLM, FEATHER, Nüwa, and several calibration, budgeting, protected-key, withdrawal, and selection refinements.
  • Mixture-of-depth and conditional computation: Conditional-computation methods route only selected tokens or inputs through blocks, but early exits are terminal whereas Mixture-of-Depths uses residual bypasses for non-selected tokens.Adaptive Computation Time, early-exit networks, layer-skipping policies, Mixture-of-Depths, γ-MoD, and p-MoD form the cited progression.
  • Layer-wise attention dynamics in VLM decoders: Visual-token importance varies across decoder depth, with prior work documenting post-layer-2 attention drops, attention shift, attention dispersion, and attention sinks.These dynamics motivate treating token importance as unstable rather than fixed at one pruning stage.
  • Efficient inference and KV-cache compression: Reroute is complementary to KV-cache eviction and system-level optimizations because it selects visual tokens for full computation while those methods decide which past states remain cached.The passage states that Reroute can in principle be combined with these methods.

3 Preliminaries

Visual-token reduction is formulated as stage-wise selection over vision tokens while text tokens remain active throughout decoding. The framework distinguishes irreversible rank-and-remove pruning from Reroute’s recoverable treatment of deferred vision tokens.

  • Stage-wise visual-token selection: Reduction targets only vision tokens, while text tokens remain active throughout decoding.This isolates the post-ranking action to low-ranked vision tokens.
  • Stage-wise visual-token selection: The decoder is partitioned into S stages, where each stage ranks its candidate vision-token set and selects a top-K subset using keep ratio r_i.Stage i begins at routing layer ℓ_i, with K_i = ⌊r_i|V|⌋ and C_1 = V.
  • Stage-wise visual-token selection: Each stage routes selected vision tokens together with all text tokens through the decoder, while deferred vision tokens determine the reduction method’s behavior.The active set is A_i = T ∪ V_sel_i.
  • Reroute: Reroute keeps deferred vision tokens recoverable instead of removing them.Deferred tokens can therefore remain available for later routing decisions.
  • Rank-and-remove as irreversible pruning: Conventional pruning removes deferred vision tokens and contracts the candidate pool, preventing their re-scoring or re-selection in later stages.This irreversibility defines rank-and-remove pruning.

4 Method

Reroute preserves pruning’s stage-wise ranking and budget schedule while replacing irreversible deletion with recoverable deferral. Deferred visual tokens bypass the current decoder stage, remain eligible for later selection, and are restored to the candidate pool at subsequent routing decisions.

  • Routing mechanism: Reroute uses pruning’s scorer and keep-ratio schedule, routing selected tokens through Attn+FFN while deferring the remainder.At each routing layer, selected tokens traverse the decoder path and deferred tokens bypass the current stage.
  • Routing mechanism: Deferred tokens re-enter the full candidate set at the next routing layer, allowing later re-selection when their scores rise.This converts irreversible deletion into recoverable deferral.
  • Budget matching: Reroute and pruning activate the same number of vision tokens per stage under shared schedules, targeting the same theoretical TFLOPs and KV-cache class.Token identities may diverge after re-entry; practical latency still depends on gather/scatter kernels and KV-cache management.
  • Conditional computation: Reroute is a training-free MoD-style conditional-computation instantiation that reuses pruning’s text-to-vision attention scores instead of learning a router.Excluded tokens form a bypass branch: they skip the current stage but remain eligible for later selection.

5 Experiments

Experiments across LLaVA-1.5 and Qwen backbones show that Reroute improves grounding over matched pruning baselines, especially under tighter token budgets, while preserving general VQA performance and most efficiency savings. Its benefits depend on repeated, appropriately placed rerouting stages.

  • Visual grounding on LLaVA-1.5: At 64 tokens, PDrop+Reroute raises LLaVA HuggingFace grounding from 22.2% to 34.0%, while FastV+Reroute has limited recovery.At 192 tokens, FastV improves from 45.5% to 59.2% and PDrop from 71.3% to 82.3%; at 128 tokens, gains are 12.0 and 7.8 points.
  • Visual grounding on LLaVA-1.5: At 64 tokens, Nüwa+Reroute improves the original-format average ratio from 54.6% to 55.8%, showing benefits atop a stronger grounding pipeline.Reroute matches Nüwa at moderate budgets and replaces irreversible deletion with recoverable deferral rather than introducing a new scoring heuristic.
  • General VQA results on LLaVA-1.5: Across 192-, 128-, and 64-token budgets, Reroute remains comparable to or better than matched FastV and PDrop counterparts on general VQA.The results indicate grounding gains do not require sacrificing broad multimodal behavior.
  • Qwen-series visual grounding results: At 88.9% reduction on Qwen2.5-VL, PDrop+Reroute improves the average ratio from 11.1% to 18.6%, confirming transfer beyond LLaVA-1.5.At 66.7% reduction, FastV improves from 53.7% to 60.3% and PDrop from 62.9% to 65.1%; at 77.8%, PDrop improves from 29.3% to 37.4%.
  • Qwen-series visual grounding results: At 66.7% reduction on Qwen3.5-9B-Hybrid, FastV+Reroute improves from 31.1% to 77.5% and PDrop+Reroute from 40.0% to 77.7%.PDrop further improves from 25.8% to 59.7% at 77.8% reduction and from 19.3% to 37.4% at 88.9%.
  • Efficiency: At a 64-token budget, FastV+Reroute reduces main TFLOPs/KV cache by 80%/86%, while PDrop+Reroute reduces them by 77%/83%.Other budgets remain close to matched pruning baselines; overhead mainly comes from additional reroute stages or budget-matched scheduling.

6 Conclusion · Appendix Overview

The conclusion frames Reroute as recoverable visual-token routing: selected tokens receive computation, while deferred tokens remain eligible for later re-entry rather than being permanently removed. It also notes router, efficiency, scope, and evaluation limitations, while the appendices provide reproducibility details and routing analyses.

  • 6 Conclusion: Reroute defers low-ranked vision tokens instead of permanently removing them, allowing deferred tokens to re-enter at later routing decisions.Selected tokens receive full Attn+FFN computation, while deferred tokens bypass the current stage.
  • 6 Conclusion: Conventional pruning is the special case in which deferred tokens never return, yielding a training-free Mixture-of-Depth-style formulation for VLM visual tokens.
  • 6 Conclusion: Reroute inherits attention-score router weaknesses: poor rankings or overly aggressive budgets can keep useful evidence inactive for multiple stages.
  • 6 Conclusion: Practical latency gains depend on optimized gather/scatter kernels and KV-cache management, making the stated efficiency benefits primarily theoretical.
  • 6 Conclusion: Reroute targets decoder-side reduction and is complementary to encoder-side compression, token merging, and KV-cache eviction, whose combinations remain unevaluated.
  • Appendix Overview: The appendix provides reproducibility details covering datasets, splits, model-specific inference settings, routing configurations, and position-index handling.
  • Appendix Overview: Appendix B analyzes schedule choices, including the early-stage PDrop ablation and frequency/location sweeps for reroute scheduling.

A Experimental Protocol and Configurations

The section specifies the implementation and evaluation details required to reproduce the main results. It covers datasets, evaluation splits, model settings, grounding post-processing, routing configurations, schedule normalization, and position-index handling.

  • Evaluation setup: The protocol defines the datasets, evaluation splits, model settings, and grounding post-processing conventions used for evaluation.
  • Routing configuration: The experiments specify routing configurations and normalize schedules across models.
  • Implementation details: The protocol states how position indices are handled throughout the experiments.

A.1 Evaluation datasets and splits … A.4 Position-index handling for physical pruning

The evaluation covers visual grounding and general multimodal understanding across multiple model families, with explicit routing-schedule conventions and position handling. Physical pruning preserves original visual indices because re-indexing harms both VQA and grounding, whereas Reroute preserves layout by construction.

  • A.1 Evaluation datasets and splits: Grounding is evaluated on RefCOCO, RefCOCO+, and RefCOCOg splits using parsed bounding boxes and standard IoU-based localization.The suite tests whether models localize objects referred to by natural-language expressions.
  • A.1 Evaluation datasets and splits: General multimodal understanding is assessed with MME, GQA, POPE, TextVQA, ScienceQA, and MMBench, plus additional datasets when available.These tasks span perception, recognition, hallucination, OCR, scientific reasoning, and broader multimodal reasoning.
  • A.2 Model and inference settings: Experiments evaluate Reroute and pruning baselines on LLaVA-1.5-7B, Qwen2.5-VL-7B, and Qwen3.5-9B, with Qwen3-VL-8B used for supplementary architecture comparisons.Unless otherwise noted, benchmarks use lmms-eval defaults and run on one NVIDIA RTX 4090 GPU.
  • A.2 Model and inference settings: Grounding prompts and post-processing follow each model family, including pixel-coordinate normalization for Qwen2.5-VL and [0, 1000] normalization for Qwen3-VL and Qwen3.5.LLaVA-1.5 uses the default lmms-eval pipeline, while Qwen models follow official/Nuwa and community conventions.
  • A.3 Routing configurations: LLaVA-1.5 uses 32 decoder layers and 576 visual tokens under standard evaluation, with routing configurations summarized for the Avg. = 192 comparison.The reported routing-layer settings are collected in Table 5.
  • A.3 Routing configurations: PDrop schedules are treated under standard approximately even routing, such as {8, 16, 24}, and early routing, such as {2, 7, 15, 23}, denoted PDrop (L8) and PDrop (L2).Because different schedules can target the same average-token budget, both conventions are considered.
  • A.3 Routing configurations: PDrop (L2) gives stronger grounding results, so the early-routing convention is used as the main PDrop-aligned schedule for Reroute.Cross-model experiments preserve this scheduling principle while normalizing layers to model depth and respecting architecture-specific constraints.
  • A.4 Position-index handling for physical pruning: Re-indexing physically pruned visual tokens substantially degrades general VQA and RefCOCO grounding, so baselines preserve original indices and Reroute retains layout by construction.Table 7 compares contiguous re-indexing with original-index preservation under FastV on LLaVA-1.5-7B.

B Routing Schedule Analysis … C Supplementary Results and Visualizations

The paper analyzes how routing schedules affect Reroute, focusing on routing-layer placement and per-stage retention while reusing pruning baselines’ attention-score ranking interface. Supplementary sections broaden evaluation to general VQA, Qwen3-VL, and qualitative grounding visualizations.

  • B Routing Schedule Analysis: Reroute’s main schedule choices are routing-layer placement and per-stage retention ratios because the method is training-free and reuses pruning baselines’ attention-score ranking interface.The analysis examines early versus standard PDrop schedules, the number of reroute stages, and their decoder-layer placement.
  • B.1 Early-stage routing for PDrop: At Avg. = 192, PDrop (L2) consistently outperforms PDrop (L8) on ACC@0.5 across both evaluated backbones.This supports early-stage routing as the main PDrop-aligned schedule for pruning-versus-Reroute comparisons.
  • B.2 Frequency and location sweeps for Reroute: The frequency sweep varies reroute-stage count with the first routing layer fixed, while the location sweep varies decoder-layer positions with stage count fixed.Both sweeps use the same per-stage retention ratio, r_i = 0.5.
  • B.2 Frequency and location sweeps for Reroute: Routing-stage placement affects the realized average token ratio and grounding behavior, with later stages retaining more tokens and performing strongly on RefCOCO.Early and clustered-early schedules provide stricter-budget comparisons.
  • B.2 Frequency and location sweeps for Reroute: Increasing the number of reroute decisions generally improves both GQA and RefCOCO grounding while keeping the average token ratio fixed.The frequency sweep therefore favors repeated rerouting under a controlled average-token budget.
  • C Supplementary Results and Visualizations: Supplementary evaluation tests whether Reroute preserves broad multimodal capability through general VQA results.The supplementary discussion also covers Qwen3-VL as an architecture-sensitive case study.
  • C Supplementary Results and Visualizations: The supplementary material further presents qualitative grounding and token-selection visualizations to complement the quantitative grounding tables.These visualizations are discussed alongside the Qwen3-VL case study.

C.1 General VQA preservation · C.2 Qwen3-VL grounding case study

Reroute generally preserves broad VQA performance at matched token budgets, with the clearest benefits from multi-stage PDrop routing, while FastV is less stable under extreme reduction. On Qwen3-VL, Reroute helps at the most aggressive compression but remains sensitive to hand-specified, architecture-dependent schedules.

  • C.1 General VQA preservation: Reroute is generally comparable to, and often improves over, corresponding pruning baselines across LLaVA-1.5-7B and Qwen-family backbones.These results test preservation of broad multimodal capability under the same average-token budget.
  • C.1 General VQA preservation: PDrop + Reroute shows the clearest preservation trend because deferred tokens can re-enter computation under multi-stage routing.On Qwen2.5-VL-7B, its average-ratio score improves over PDrop at all three reduction levels.
  • C.1 General VQA preservation: FastV + Reroute remains broadly comparable under moderate budgets but is less stable under extreme reduction because FastV relies on a single early scoring layer.The contrast reflects differences between single-stage and multi-stage routing schedules.
  • C.1 General VQA preservation: Frequency-sweep configurations fix the first routing layer at layer 2 and per-stage retention ratio ri = 0.5 while varying the number of reroute stages.The configurations are reported for LLaVA-1.5-7B; RefCOCO A/B denote testA/testB Acc@0.5.
  • C.1 General VQA preservation: Location-sweep configurations use four reroute stages with per-stage retention ratio ri = 0.5 while varying stage placement across decoder depth.RefCOCO A/B denote testA/testB Acc@0.5.
  • C.2 Qwen3-VL grounding case study: At 66.7% token reduction, PDrop (L2) reaches a 69.5% average ratio, while PDrop + Reroute reaches 57.2%; at 77.8% reduction, the corresponding values are 38.9% and 36.1%.The early PDrop schedule remains the strongest grounding configuration in these moderate-reduction regimes.
  • C.2 Qwen3-VL grounding case study: At 88.9% reduction, PDrop + Reroute improves over PDrop from 12.9% to 19.1%, indicating that recoverable deferral becomes useful under the most aggressive compression.The experiment is interpreted as an architecture-sensitive diagnostic rather than a primary success case.
  • C.2 Qwen3-VL grounding case study: Qwen3-VL demonstrates that Reroute can help under extreme compression, but training-free schedules should account for model-specific architectural constraints and visual-feature injection mechanisms.Reroute reuses the pruning baseline’s attention-score ranking signal and does not learn a router, making routed layers and keep-ratio budgets key design variables.

C.3 Additional qualitative results

Additional qualitative visualizations compare pruning with Reroute across RefCOCO-family grounding tasks and multiple VLM backbones under matched token budgets. The figures also show selected token masks, while Qwen2.5-VL examples illustrate that Reroute preserves deferred visual evidence and improves grounding over pruning.

  • RefCOCO-family grounding: Figures 8–10 compare pruning baselines with corresponding Reroute variants on RefCOCO-family grounding examples for LLaVA-1.5, Qwen2.5-VL, and Qwen3.5 under matched token budgets.The Qwen3.5 comparison uses matched gated-attention token budgets.
  • Selected visual-token masks: Figures 6 and 7 visualize selected visual-token masks for LLaVA-1.5 and Qwen2.5-VL under corresponding token budgets.Highlighted regions indicate the visual tokens selected by the routing policy.
  • RefCOCO-family grounding: The qualitative grounding visualizations cover LLaVA-1.5, Qwen2.5-VL, and Qwen3.5.The examples are provided in Figures 8, 9, and 10.
  • RefCOCO-family grounding: Qwen2.5-VL examples show Reroute preserves deferred visual evidence and improves grounding over matched pruning baselines.This comparison is presented in Figure 9.
Loading 2606.12412v1…