PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference

Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang

arXiv:2609.15131v1cs.CVcs.AI

TL;DR

MLLMs incur substantial computation processing numerous visual tokens, and fixed or prefix-based skipping may not fit input-varying, module-specific needs. AdaVSkip routes visual tokens independently through each layer’s self-attention and MLP modules, using two-stage router training. Across backbones and benchmarks it improves performance–efficiency trade-offs; with compression it reduces FLOPs by 91.2% while retaining 97.2% of original average performance on LLaVA-NeXT-7B.

  • Problem

    MLLM inference is costly for numerous visual tokens, while fixed or prefix-based skipping may not match input-varying, module-specific computation needs.

    Source: §1 · pp. 1–2

  • Method

    AdaVSkip uses per-layer self-attention and MLP routers, trained with necessity-guided supervision followed by answer-feedback reinforcement learning while the backbone stays frozen.

    Source: §1 · p. 2

  • Results

    Across three backbones and multiple benchmarks, AdaVSkip improves performance–efficiency trade-offs; with CDPruner on LLaVA-NeXT-7B, it cuts FLOPs by 91.2% while retaining 97.2% of original average performance.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Combining adaptive skipping with visual-token compression exploits vertical and horizontal redundancy, achieving better performance at lower computational cost than compression alone.

    Source: §5 · p. 10

Abstract

from arXiv · show

Multimodal large language models (MLLMs) require substantial computation to process numerous visual tokens across all transformer layers. Most methods for efficient MLLM inference exploit horizontal redundancy by compressing visual tokens. Beyond token reduction, recent studies exploit vertical redundancy through early exit or fixed-layer skipping. However, we find that the extent and distribution of this redundancy vary across inputs and differ between self-attention and MLP modules. Motivated by these observations, we propose AdaVSkip, which equips each layer with two lightweight routers that independently determine whether visual tokens pass through by or skip the self-attention and MLP modules. These decisions collectively define an input-specific visual-computation path, but their discrete and non-differentiable nature makes learning effective paths challenging. To address this challenge, we develop a progressive two-stage training framework that updates only the routers while keeping the backbone frozen. Stage I establishes an initial routing policy through supervised training with input-specific targets derived from module-wise necessity scores. To further align the routing policy with task performance, Stage II uses reinforcement learning to optimize routing decisions with direct feedback from generated answers. It combines an answer correctness reward with a skip-consistency reward that discourages excessive retention of visual-token computation. Across three MLLM backbones, AdaVSkip maintains strong task performance with substantially less computation. On LLaVA-NeXT-7B, AdaVSkip reduces FLOPs by 53.2\% while preserving the original model's average performance. Combining it with visual token compression increases this reduction to 91.2\%, while retaining 97.2\% of the original performance on average.

1 INTRODUCTION

Visual-token computation is costly, and its redundancy varies by input and module, making fixed skipping patterns a poor fit. AdaVSkip routes visual tokens independently through self-attention and MLP modules and improves performance–efficiency trade-offs across MLLMs and benchmarks.

1 INTRODUCTION

  • Existing token compression reduces visual-token counts, while early-exit and fixed-layer methods exploit vertical redundancy but may not match questions’ varying computation needs.MLLMs process many visual tokens through self-attention and MLP at every transformer layer, creating substantial inference costs.Source: §1 · p. 1
  • Vertical redundancy varies across inputs, and self-attention and MLP can differ in their need for visual-token computation within the same layer.These differences limit fixed skipping patterns and motivate module-wise decisions.Source: §1 · p. 2
  • AdaVSkip uses two lightweight routers per layer to independently decide whether visual tokens undergo self-attention and MLP computation.Skipped visual tokens bypass a module with unchanged hidden states, remain in the sequence, and can resume computation later; text tokens are always processed.Source: §1 · p. 2
  • A progressive two-stage strategy trains only the routers with the backbone frozen, first using necessity-guided supervision and then reinforcement learning with correctness and skip-consistency rewards.Stage I derives input-specific targets from module-wise necessity scores; Stage II refines routing using generated-answer feedback while discouraging excessive visual computation.Source: §1 · p. 2
  • Across three MLLM backbones and multiple benchmarks, AdaVSkip improves performance–efficiency trade-offs over fixed-layer and prefix-based methods, particularly under aggressive skipping.With visual-token compression, it reduces FLOPs by 91.2% while retaining 97.2% of original average performance on LLaVA-NeXT-7B.Source: §1 · pp. 2–3

2 RELATED WORK

Related work addresses visual-token costs through horizontal compression and vertical skipping. AdaVSkip builds on vertical-redundancy work but uses input-specific decisions across layers and separately for self-attention and MLP.

2.1 MULTIMODAL LARGE LANGUAGE MODELS

  • Projector-based MLLMs map images into visual tokens that are processed alongside text tokens, while higher resolution and finer visual representations increase computational costs.The projector-based paradigm connects a pretrained vision encoder to an LLM through a lightweight projector.Source: §2.1 · p. 3

2.2 HORIZONTAL REDUNDANCY IN VISUAL TOKENS

  • Horizontal-redundancy methods either learn compact visual representations before the LLM or prune or merge tokens at inference, but remaining tokens still pass through all LLM layers.Learned compact representations often require architectural modifications and training, whereas inference-time token reduction can be training-free.Source: §2.2 · p. 3

2.3 VERTICAL REDUNDANCY ACROSS LAYERS

  • Prior vertical-redundancy methods use early exits or selected layers, while AdaVSkip chooses input-specific skipping amounts and locations with independent self-attention and MLP decisions.Early-exit methods restrict visual-token computation to a layer prefix; ShortV instead preserves computation in selected layers.Source: §2.3 · p. 3

3 METHOD

AdaVSkip is motivated by input-, depth-, and module-dependent variation in visual-token necessity, and uses independent layerwise routers to adapt visual-token computation. It trains the routers progressively, first from necessity-derived targets and then with answer-feedback reinforcement learning.

3.1 PATTERNS OF VERTICAL REDUNDANCY

  • Each of 2L necessity scores compares original output distributions with a forward pass skipping visual computation in one attention or MLP module.Source: §3.1 · pp. 3–4
  • Necessity varies across benchmarks, layers, modules, and samples; same-source LLaVA-1.5 SFT samples cluster in 64-dimensional profiles projected with t-SNE.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 ADAPTIVE VISUAL-TOKEN SKIPPING

  • Two independent lightweight routers per layer gate visual participation in attention and MLP; text tokens always proceed, and routing occurs only during prefill.Source: §3.2 · p. 4

3.3 STAGE I: NECESSITY-GUIDED SUPERVISED TRAINING

  • Stage I keeps the smallest highest-necessity module subset reaching each type’s threshold, then trains routers against targets and answer-token language-modeling loss.Source: §3.3 · p. 5

3.4 STAGE II: ROUTING POLICY REFINEMENT

  • Stage II samples routing trajectories, evaluates generated answers, and uses normalized exact-match correctness plus a keep-ratio consistency reward before group-relative policy optimization.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate three MLLM backbones across multiple benchmarks against vertical-skipping and visual-token-compression methods. Results cover main comparisons, training and routing ablations, inference efficiency, and input-specific routing visualizations.

4.1 MAIN RESULTS

  • At K = 16, AdaVSkip retains 100.0% average relative performance using 52.0% of original FLOPs on LLaVA-1.5-7B, and 100.4% using 46.8% on LLaVA-NeXT-7B.Source: §4.1 · p. 7
  • At a 5.6% equivalent visual token budget on LLaVA-NeXT-7B, AdaVSkip with CDPruner achieves 97.2% average relative performance at 8.8% of original FLOPs, versus 94.5% at 9.7% for CDPruner alone.On LLaVA-1.5-7B, the combined configurations improve average relative performance and require fewer FLOPs than CDPruner alone at each reported budget.Source: §4.1 · p. 7, Table 2 · p. 8
  • For Qwen3-VL-8B-Instruct, VisionZip alone achieves 71.7% at 13.9% FLOPs; jointly exploiting vertical and horizontal redundancy mitigates losses on fine-grained visual-perception tasks.The combination outperforms MMTok, the strongest standalone compression baseline at this budget, by 18.4, 13.8, and 7.7 points on ChartQA, DocVQA, and InfoVQA, respectively.Source: Table 3 · p. 8, §4.1 · p. 8

4.2 ABLATION STUDY OF ADAVSKIP TRAINING COMPONENTS

  • Module-wise routing raises average performance from 98.9% to 99.3% while reducing retained FLOPs from 52.4% to 51.8%; adding SCR holds performance at 100.0% while reducing retained FLOPs from 53.5% to 52.0%.Input-specific targets further raise average performance to 99.3%, while the downstream CE loss improves it from 98.2% to 98.5%.Source: §4.2 · p. 8, Table 4 · p. 9

4.3 ANALYSIS OF ADAPTIVE ROUTING

  • With skip counts matched, AdaVSkip achieves 100.2% average relative performance versus 98.7% for Count-Matched Prior; its average skip ratio also varies by task, from 36.1% on TextVQA to 62.2% on POPE.Against Fixed-Budget AdaVSkip, it achieves 100.2% versus 98.6% at comparable FLOPs.Source: §4.3 · p. 8, Table 5 · p. 9

4.4 INFERENCE EFFICIENCY ANALYSIS

  • With LLaVA-NeXT-7B, AdaVSkip alone provides 2.09× prefill and 1.78× end-to-end speedups, retains 44.5% of original FLOPs, and reduces KV cache by 52.4%.With CDPruner, speedups rise to 6.10× prefill and 3.00× end-to-end, and KV cache falls by 90.5%.Source: Table 6 · p. 9, §4.4 · p. 9

4.5 VISUALIZATION AND CASE STUDY

  • At K = 24, routing preserves different amounts and locations of visual computation by input: it skips many modules for a coarse scene question but preserves more for OCR and spatial-relation questions.The OCR and spatial-relation samples retain different module subsets.Source: Figure 4 · p. 9, §4.5 · p. 9

5 CONCLUSION

The conclusion identifies input- and module-dependent vertical redundancy as the motivation for AdaVSkip’s independent self-attention and MLP routers. It reports improved performance–efficiency trade-offs over fixed-layer and early-exit methods, including when combined with visual-token compression.

5 CONCLUSION

  • Vertical redundancy varies across inputs and between self-attention and MLP, motivating independent visual-computation routing for each module at each layer.AdaVSkip uses lightweight routers and a progressive two-stage training scheme that updates only the routers while keeping the backbone frozen.Source: §5 · p. 9
  • Across three backbones and multiple benchmarks, AdaVSkip achieves better performance–efficiency trade-offs than fixed-layer and early-exit methods.Combining it with visual-token compression achieves better performance at lower computational cost than compression alone.Source: §5 · p. 10

Related papers

Loading 2609.15131v1…