PaperlayerView exact revision on arXiv

Source-linked AI summary

HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit

Hao Wu, Yingqi Fan, Jinyang Dai, Junlong Tong, Yunpu Ma, Xiaoyu Shen

arXiv:2602.23699v1cs.CVcs.CL

TL;DR

MLLMs face quadratic costs from visual tokens, while existing pruning methods misinterpret shallow layers and use rigid schedules. HiDrop aligns token reduction with hierarchical multimodal processing through Late Injection, adaptive pruning, and Early Exit. It compresses about 90% of visual tokens while matching original performance and accelerating training by up to 1.72×.

  • Problem

    Existing vision-token pruning methods misinterpret shallow-layer functions and rely on rigid schedules, limiting efficient MLLM processing.

  • Method

    HiDrop bypasses passive shallow layers with Late Injection, then applies Concave Pyramid Pruning and Early Exit using ILVAS and differentiable top-k selection.

  • Results

    HiDrop compresses ∼90% of visual tokens while matching original performance and accelerating training by up to 1.72×.

  • Takeaways & Limitations

    HiDrop aligns token pruning with hierarchical multimodal processing and achieves state-of-the-art efficiency–accuracy trade-offs.

Abstract

from arXiv · show

The quadratic computational cost of processing vision tokens in Multimodal Large Language Models (MLLMs) hinders their widespread adoption. While progressive vision token pruning offers a promising solution, current methods misinterpret shallow layer functions and use rigid schedules, which fail to unlock the full efficiency potential. To address these issues, we propose HiDrop, a framework that aligns token pruning with the true hierarchical function of MLLM layers. HiDrop features two key innovations: (1) Late Injection, which bypasses passive shallow layers to introduce visual tokens exactly where active fusion begins; and (2) Concave Pyramid Pruning with an Early Exit mechanism to dynamically adjust pruning rates across middle and deep layers. This process is optimized via an inter-layer similarity measure and a differentiable top-k operator. To ensure practical efficiency, HiDrop further incorporates persistent positional encoding, FlashAttention-compatible token selection, and parallel decoupling of vision computation to eliminate hidden overhead associated with dynamic token reduction. Extensive experiments show that HiDrop compresses about 90% visual tokens while matching the original performance and accelerating training by 1.72 times. Our work not only sets a new state-of-the-art for efficient MLLM training and inference but also provides valuable insights into the hierarchical nature of multimodal fusion. The code is released at https://github.com/EIT-NLP/HiDrop.

1 INTRODUCTION

HiDrop argues that MLLM visual-token processing is hierarchical: shallow layers mainly propagate vision tokens, middle layers perform sparse fusion, and deeper layers can discard vision tokens. It therefore combines Late Injection, adaptive Concave Pyramid Pruning with Early Exit, and implementation optimizations to improve efficiency while preserving performance.

  • Motivation: MLLMs commonly use progressive vision-token pruning because visual-token processing has quadratic computational cost.Progressive pruning preserves dense representations early and reduces tokens in deeper layers.
  • Layer-wise dynamics: Shallow layers primarily propagate vision tokens because intra-modal changes and cross-modal influence are negligible.The authors characterize these layers as propagators and attention sinks rather than true multimodal integrators.
  • Layer-wise dynamics: Visual information flow is non-uniform: middle layers are fusion hubs with substantial redundancy, while deep layers can discard vision tokens after integration.Rigid fixed-ratio schedules therefore miss the layer-dependent structure of multimodal processing.
  • HiDrop framework: Late Injection bypasses shallow layers and introduces the full vision-token set at the onset of active fusion.This delays visual input without prematurely removing potentially valuable information.
  • HiDrop framework: Concave Pyramid Pruning reduces tokens aggressively early in fusion, slows reduction later, and uses Early Exit to remove vision tokens before language-dominant layers.ILVAS identifies filtering layers, while differentiable top-k selects informative tokens.
  • Efficiency: HiDrop combines persistent positional identifiers, FlashAttention-compatible token selection, and parallelized vision computation to avoid hidden overhead.These mechanisms translate dynamic token reduction into practical acceleration.
  • Results: HiDrop compresses approximately 90% of visual tokens while matching original performance and accelerating training by up to 1.72×.The experiments are conducted on LLaVA-1.5-7B and report improved inference throughput.

2 UNMASKING THE PROCESSING DYNAMICS IN MLLMS

MLLMs process visual information hierarchically: shallow layers mainly propagate tokens, middle layers perform sparse fusion, and deep layers shift toward language-dominant reasoning. These dynamics challenge static pruning schedules and motivate behavior-grounded token reduction.

  • O(N^2) self-attention makes vision-token processing the primary computational bottleneck when visual token counts dominate text tokens.The architecture combines text and vision embeddings, with vision tokens often far outnumbering text tokens.
  • Static progressive pruning schedules ignore the distinct processing dynamics of shallow, middle, and deep MLLM layers.Existing strategies commonly apply predetermined linear or convex decay uniformly across layers.
  • Shallow Layers: Propagators: Shallow layers primarily propagate visual information: visual representations change little across consecutive layers, while text remains nearly invariant to the accompanying image.These findings indicate negligible intra-modal refinement and cross-modal influence before meaningful fusion begins.
  • Middle Layers: Sparse Fusion Hubs: Middle layers are sparse fusion hubs where textual representations respond strongly to visual input, but only a small subset of visual tokens grounds the text.Their combination of active fusion and high token redundancy makes middle layers a natural processing bottleneck.
  • Figure 3 indicates that lower p values enforce stronger pruning while model performance remains stable under high compression rates.The figure presents token-reduction curves and corresponding performance behavior across pruning strengths.
  • Deep Layers: Language-Dominant Reasoning: Deep layers transition to abstract, language-centric reasoning as the direct influence of visual tokens steadily diminishes.A training-free early-exit experiment examines the impact of discarding visual tokens at a selected layer.

3 HIDROP

HiDrop aligns vision-token computation with the hierarchical roles of shallow, middle, and deep MLLM layers. It delays vision-token entry, prunes adaptively during the middle-layer fusion window, exits vision processing in deep layers, and preserves efficiency through implementation mechanisms.

  • HiDrop partitions MLLM layers into shallow, middle, and deep stages, assigning Late Injection, Concave Pyramid Dropping, and Early Exit respectively.
  • Late Injection: Late Vision Injection bypasses initial text-only layers and introduces the full vision-token set at the onset of active fusion.The injection point is selected using a local minimum in visual layer-wise similarity; experiments identify layer 9.
  • Early Exit: Early Vision Exit discards all remaining vision tokens after a performance plateau indicates that deep reasoning no longer requires direct visual input.The experiments identify layer 25 as the exit point.
  • Vision Processing Window: HiDrop restricts vision processing to a middle-layer window, accelerating training and inference while preserving predictive accuracy.
  • Concave Pyramid Pruning: Concave Pyramid Pruning removes tokens rapidly at fusion onset and more gradually afterward to balance computational savings with information preservation.
  • Filtering Layers: ILVAS identifies filtering layers by measuring whether highly attended visual tokens remain important in subsequent layers.Local maxima of the ILVAS curve form the filtering-layer set, such as {10, 14, 16, 18}.
  • Adaptive Token Selection: Differentiable Top-K provides continuous token selection with a learnable pruning ratio, while hard thresholding yields discrete forward-pass selection.Combined with ILVAS, it determines both where and which visual tokens to prune.
  • Efficient Implementation: Persistent positional identifiers preserve visual-token positions across injection, pruning, and exit, while separate lightweight selection attention retains FlashAttention compatibility.

4 EXPERIMENT

Experiments evaluate HiDrop across multiple backbones, benchmarks, efficiency measures, and ablations. Results show strong performance retention under aggressive compression, with layer-selection and token-selection choices validated experimentally.

  • Main Results: HiDrop retains 98.3% and 96.5% of baseline performance while pruning 88.9% and 91.7% of vision tokens, respectively.Across 11 benchmarks, it outperforms competing in-LLM vision-token compression methods at all tested pruning ratios.
  • Efficiency: HiDrop reduces LLaVA-1.5-7B training time from 159.3 to 94.4 GPU hours, a 40.7% reduction.Inference FLOPs decrease from 3.82T to 0.42T, while prefill latency falls from 63.6 ms to 32.6 ms before further implementation optimizations.
  • Ablation Studies: The selected late-injection and early-exit window spans layers 9 to 25, supported by sweeps showing earlier injection adds cost and later exit can hurt accuracy.Layer 9 peaks in the injection sweep, while exits around layers 25–26 perform best.
  • Ablation Studies: Differentiable top-k raises average performance from 97.7% to 99.7% with two-stage training and from 97.5% to 98.1% with one-stage training.The larger two-stage gain motivates its use as the default configuration.
  • Ablation Studies: Persistent positional encoding achieves the best average performance among Persistent, Group, and Compacted PE schemes.The method is adopted by default because it preserves accuracy without overhead under dynamic token changes.
  • Ablation Studies: ILVAS-based filtering layers {10, 14, 16, 18} achieve higher average accuracy than the control set {12, 15, 18, 21}.Sweeps identify layer 14 and the pair {14, 18} as particularly strong choices.

5 CONCLUSION

HiDrop aligns vision-token reduction with the hierarchical processing dynamics of MLLMs through Late Injection, Concave Pyramid Pruning, and Early Exit. The study reports improved efficiency–accuracy trade-offs while offering insight into layerwise computation allocation.

  • HiDrop challenges the assumption that shallow layers actively integrate visual information, characterizing them instead as passive propagators.
  • Late Injection, Concave Pyramid Pruning, and Early Exit align token pruning with hierarchical multimodal integration.
  • The framework achieves state-of-the-art efficiency–accuracy trade-offs while providing insights into how MLLMs allocate computation across layers.

ETHICS STATEMENT

The work reports no ethical concerns because it focuses on methodological contributions and efficiency analysis without sensitive data, human subjects, or ethically risky applications.

  • The study reports no ethical concerns because it uses neither sensitive data nor human subjects.
  • Its scope is limited to methodological contributions and efficiency analysis without applications identified as ethical risks.

REPRODUCIBILITY STATEMENT

The paper documents experimental settings and provides ablations to support reproducibility, while describing prior token-reduction approaches and their placement relative to the LLM.

  • Experimental settings, dataset descriptions, training details, and hyperparameter selections are documented in the main text and appendix.
  • Extensive ablation studies are provided to justify the paper’s design choices.
  • The authors state that the full codebase and scripts will be released upon acceptance to facilitate replication.
  • LLMs were used only for language refinement, not for research ideation, methodological design, analysis, or result interpretation.
  • Training-free methods commonly prune vision tokens during inference using heuristic rules, creating a train–inference mismatch.
  • Prior token-reduction methods are categorized as Pre-LLM, In-LLM, or joint approaches according to where compression occurs.

C LLM BACKBONES

The experiments use two decoder-only LLM backbones within LLaVA-1.5, with detailed architectural settings summarized in Table 7. The method changes only the vision-token schedule and otherwise follows LLaVA defaults.

  • Table 7 presents the detailed settings of the LLM backbones.
  • The study evaluates MobileLLaMA 2.7B and Vicuna-7B-v1.5 as decoder-only LLM backbones within LLaVA-1.5.
  • Both backbones use 32 transformer blocks and 32 attention heads.
  • MobileLLaMA 2.7B uses hidden size 2560 and FFN dimension 6912, while Vicuna-7B-v1.5 uses 4096 and 11008, respectively.
  • Unless otherwise noted, the experiments follow LLaVA defaults and change only the vision-token schedule.

D BENCHMARKS

HiDrop is evaluated across 11 mainstream multimodal benchmarks using standard evaluation protocols and compared with representative visual-token compression methods under LLaVA-v1.5.

  • Benchmark suite: The evaluation covers 11 benchmarks spanning perception, reasoning, visual question answering, text understanding, hallucination, and comprehensive vision-language abilities.The suite includes MME-Perception, MMBench, MMBench-CN, GQA, VQAv2, ScienceQA-Iamge, VizWiz, TextVQA, POPE, SEED-Image, and MMStar.
  • Benchmark suite: The benchmarks test capabilities including scene understanding, logical reasoning, multimodal explanation, low-quality-image understanding, image-text interaction, and object hallucination.TextVQA specifically requires reading and reasoning about text in images, while POPE measures visual object hallucination.
  • Evaluation protocol: Unless otherwise noted, all benchmarks follow the official LLaVA evaluation protocol, while MMStar uses LMMS-Eval.
  • Baselines: Comparisons under LLaVA-v1.5 include FastV, PDrop, VoCo-LLaMA, and TwigVLM as representative vision-compression approaches.These methods respectively use attention-based early pruning, stage-wise similarity pruning, compact VoCo representations, or twig-guided pruning with self-speculative decoding.
  • Analysis: Figure 9 analyzes layer-wise representational dynamics through intra-modal refinement and cross-modal interaction intensity.

F EXTENDED ANALYSIS

The extended analysis examines whether layer-wise representational dynamics generalize across LLaVA backbones and model sizes, alongside complete results for injection and exit configurations.

  • Representational dynamics: Layer-wise analyses cover MobileLLaMA-2.7B, Vicuna-7B, LLaMA3.1-8B, and Vicuna-13B within the LLaVA-v1.5 framework.The reported analyses are intended to test whether the observed phenomena are universal across backbones and model sizes.
  • Configuration results: Table 8 provides complete per-benchmark results for Fig. 7’s injection and early-exit configurations.Ii denotes visual injection at layer i, and Ei denotes early visual exit at layer i.

G EXTENDED EXPERIMENTAL RESULTS

HiDrop selects its late-injection and early-exit window using layer-wise diagnostics and targeted sweeps, identifying layers 9–25 as a strong efficiency–accuracy compromise.

  • Diagnostic motivation: Layer 9 aligns with a local minimum in visual layer-wise similarity, while accuracy plateaus around layer 25 under deep-to-shallow masking.These diagnostics motivate layer 9 for visual-token injection and layer 25 for visual-token exit.
  • Late injection sweep: Injecting at layer 9 produces the sweep’s accuracy peak; earlier injection adds computation with negligible gains, whereas later injection degrades accuracy.
  • Fixed-entry span sweep: An injection at layer 9 with exit depth around layers 25–26 is optimal, while later exits add cost and earlier exits reduce accuracy.
  • Equal-depth window sweep: Equal-length window sweeps identify 8–24 and 9–25 as near-optimal spans, while 10–26 underperforms.
  • Final selection: Layer 26 matches baseline accuracy and layer 25 is only marginally lower, leading to the selected 9–25 window for balancing efficiency and accuracy.The authors expect training to recover the small gap at layer 25.

G.2 DIFFERENTIABLE TOP-k

Extended experiments refine HiDrop’s token-selection, positional-encoding, and filtering-layer choices, showing benefits from differentiable Top-k, Persistent PE, and ILVAS-based filtering layers.

  • Differentiable Top-k: Differentiable Top-k raises average score from 97.7% to 99.7% with PT+FT and from 97.5% to 98.1% with FT only.These comparisons replace hard Top-k under a progressive pruning schedule.
  • Differentiable Top-k: At 72 visual tokens versus the original 576, hard Top-k degrades clearly while differentiable Top-k preserves accuracy across benchmarks.The gain is especially evident on MMBench, SQA-I, and VizWiz.
  • Differentiable Top-k: The authors adopt PT+FT with differentiable Top-k because its smoother selection adapts to training signals, particularly under aggressive pruning.
  • Token scoring: The multi-round last-token scheme reaches 99.9% average versus 99.6% for the best all-token variant, while avoiding eager-attention cost.All-token strategies slightly improve some individual benchmarks but not the overall average.
  • Positional encoding: Persistent PE achieves 97.8% average, outperforming Group PE at 97.1% and Compacted PE at 96.9% under shallow–middle–deep compression.Persistent PE keeps fixed RoPE indices, whereas Compacted PE resets indices after pruning.
  • Filtering-layer selection: ILVAS profiles identify layers {10, 14, 16, 18} as filtering layers, and this set outperforms the control configuration {12, 15, 18, 21}.The selected set is reported as the best trade-off under the concave pyramid dropping policy.
Loading 2602.23699v1…