PaperlayerView exact revision on arXiv

Source-linked AI summary

DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models

Shuo Yang, Changbai Li, Linlin Yang, Huobin Tan, Rongyu Chen, Tongfei Chen, Tian Wang, Sheng Xu, Baochang Zhang

arXiv:2610.08341v1cs.CVcs.LG

TL;DR

Training-free visual-token pruning must balance reducing MLLM computation with retaining task-relevant content, but existing approaches can lack task guidance or rely on biased attention. DIPrune combines static semantic alignment and cross-layer semantic evolution in a rank-based pruning framework. The paper reports state-of-the-art results across multiple mainstream MLLMs under high compression ratios.

  • Problem

    Training-free visual-token pruning can be task-agnostic or rely on unreliable attention estimates, risking the removal of task-relevant content.

    Source: §1 · pp. 1–2

  • Method

    DIPrune formulates pruning as minimizing final task-loss distortion and uses rank-based dual importance scores for static semantic alignment and dynamic semantic evolution.

    Source: §1 · p. 2

  • Results

    DIPrune achieves state-of-the-art results under high compression ratios across multiple mainstream MLLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that evaluating both intra-layer static alignment and inter-layer dynamic evolution helps preserve task-critical tokens.

    Source: §7 · p. 10

    The error-bound derivation assumes that F is locally L_F-smooth.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Recent training-free pruning approaches for Multimodal Large Language Models (MLLMs) effectively cut computational overhead by exploiting visual redundancy or text-vision attention. However, they frequently suffer from semantic degradation due to their task-agnostic design or unreliable attention estimates. Based on our empirical analysis, we have found that this issue arises because salient tokens in shallow layers persistently suppress emerging semantic ones through numerical inertia, leading to premature discarding of signals crucial for deep reasoning. To address the aforementioned issue, from the task-oriented aspects, we first reformulate training-free pruning as a minimization of the distortion in the final task loss and derive a tractable, token-wise upper bound to serve as a surrogate objective. Specifically, this formulation inherently reveals a previously neglected inter-layer term that accounts for gradients across layers. Accordingly, for the implementation, we propose DIPrune, a rank-based framework that employs a dual importance scoring mechanism to jointly optimize intra-layer static feature saliency and inter-layer dynamic semantic evolution. Extensive experiments on LLaVA and Qwen-VL demonstrate that DIPrune consistently achieves state-of-the-art results.

1 INTRODUCTION

DIPrune addresses training-free visual-token pruning’s task-awareness and attention-reliability challenges by combining static semantic alignment with cross-layer semantic evolution. It identifies Early Inertial Bias as a reason shallow-layer signals can obscure tokens that become important for deeper reasoning.

1 INTRODUCTION

  • Existing pruning methods either ignore text instructions when removing redundant visual tokens or rely on attention estimates vulnerable to bias.The introduction identifies the need for task-aware and reliable estimates in training-free pruning.Source: §1 · p. 1
  • Early Inertial Bias describes shallow-layer tokens retaining high attention through numerical inertia and suppressing Emerging Tokens that gain semantic importance deeper in the model.The paper identifies Inertial Tokens as redundant signals and Emerging Tokens as increasingly important for task-specific reasoning.Source: §1 · pp. 1–2
  • The pruning objective minimizes final task-loss distortion under a token budget, and its tractable upper bound separates static alignment from cross-layer semantic evolution.The decomposition exposes an inter-layer term that accounts for gradients across layers.Source: §1 · p. 2
  • DIPrune uses rank-based dual importance scoring to jointly optimize static semantic alignment and dynamic semantic evolution.This framework is described as a way to counter early inertial bias while retaining task-relevant tokens.Source: §1 · p. 2
  • DIPrune achieves state-of-the-art results under high compression ratios across multiple mainstream MLLMs.The introduction also reports strong generalization capability and inference efficiency.Source: §1 · p. 2

2 RELATED WORK

The related work frames visual-token pruning as a response to the computational cost of growing visual sequences in MLLMs. It distinguishes attention-guided pruning from feature-redundancy approaches, each with limitations for retaining task-critical information.

2 RELATED WORK

  • High-resolution visual inputs increase token counts, making the quadratic self-attention cost a major inference-efficiency bottleneck.The passage gives LLaVA-v1.5’s 576 tokens and LLaVA-NeXT’s 2,880 as examples.Source: §2 · p. 3
  • Attention-based methods use Transformer attention to select tokens, while redundancy-based methods remove or merge tokens using visual-feature similarity.Examples include FastV and SparseVLM for attention guidance and VisionZip and DART for visual redundancy.Source: §2 · p. 3
  • Existing approaches struggle to reconcile critical-information retention with complex task adaptation because of attention bias or limited textual semantic guidance.Source: §2 · p. 3

3 MOTIVATION AND ANALYSIS

The analysis examines how token importance changes across layers and identifies Early Inertial Bias in attention-based pruning. Its observations indicate that low early-layer attention can mask tokens that matter for final reasoning.

3 MOTIVATION AND ANALYSIS

  • The analysis investigates how token importance varies across layers under attention-based pruning paradigms.Source: §3 · p. 3
  • Figure 2 visualizes attention-score and rank evolution for the top 20 tokens selected from the second layer.Source: Figure 2 · p. 3
  • Approximately 47% of final-layer key tokens have low scores in shallow layers, indicating that early scores can mask tokens important for final reasoning.The passage attributes this discrepancy to numerical inertia.Source: §3 · p. 4
  • Cross-layer attention visualizations show generic-background regions remaining highly attended while answer-critical regions gain attention with depth.Figure 1 follows attention from the second to the 12th layer.Source: §3 · p. 4

4 THEORETICAL ANALYSIS

The section casts pruning as choosing K visual tokens to minimize final-task-loss distortion, then derives a token-wise surrogate that separates within-layer feature content from cross-layer influence.

4.1 OPTIMIZATION OBJECTIVE UNDER BUDGET CONSTRAINT

  • Under a strict K-token budget, the objective selects a binary visual-token mask that minimizes distortion in final task loss.Source: §4.1 · p. 4

4.2 THEORETICAL ERROR DECOMPOSITION

  • Because the exact gradient is inaccessible and the mask is discrete, the analysis substitutes a tractable token-wise upper bound.Source: §4.2 · p. 4
  • Assuming local L_F-smoothness, the derivation bounds pruning perturbations and relaxes coupled gradient and feature terms into additive constraints.Source: §4.2 · p. 5
  • The bound separates intra-layer importance, based on token feature magnitude, from inter-layer importance, based on gradient energy through subsequent layers.Source: §4.2 · p. 6
  • Under budget K, retaining the K tokens with largest combined importance minimizes the error upper bound; a truncated local horizon approximates inter-layer dependencies at inference.Source: §4.2 · pp. 5–6

5 DIPRUNE FRAMEWORK

DIPrune ranks visual tokens using weighted static-alignment and dynamic-evolution scores, then retains the top K. Its components use text-to-visual attention and changes in token rank across layers.

5 DIPRUNE FRAMEWORK

  • DIPrune fuses static semantic-alignment and dynamic semantic-evolution scores in a rank-based framework to select top-K tokens and minimize the theoretical error bound.Source: §5 · p. 6

5.1 SEMANTIC ALIGNMENT SCORE

  • Cross-modal attention from text queries to visual keys supplies feature-saliency weights, rank-normalized to make semantic alignment independent of absolute attention scale.Source: §5.1 · p. 6

5.2 SEMANTIC EVOLUTION SCORE

  • Inter-layer rank displacement is clamped to score tokens with rising priority while suppressing spurious fluctuations; unselected tokens and KV caches are removed to reduce attention cost to O((K+M)^2).Source: §5.2 · pp. 6–8
  • The evaluation captions cover LLaVA-v1.5-7B across diverse benchmarks at different pruning ratios and LLaVA-NeXT-7B with 320 tokens retained.Source: Table 1 · p. 7, Table 2 · p. 7

6 EXPERIMENTS

Experiments evaluate DIPrune across model families, sequence lengths, pruning budgets, and efficiency settings. Results report strong retention and cross-architecture performance, while ablations examine pruning depth and dual-importance weights.

6.1 SETUP

  • Evaluation spans LLaVA-v1.5-7B/13B, high-resolution LLaVA-NeXT, Qwen2.5-VL, Video-LLaVA, and Qwen3-VL models.The setup uses LLaVA-v1.5 as the standardized backbone and includes high-resolution inputs and other architectures for generalization assessment.Source: §6.1 · p. 8, §6.3 · p. 8

6.2 MAIN RESULTS

  • DIPrune retains 99.9% of unpruned performance at 192 tokens (↓66.7%) and exceeds the best baseline by 2.3% at 128 tokens (↓77.8%) on LLaVA-v1.5-7B.At 64 tokens (↓88.9%), it retains 97.7%, compared with HoloV’s 97.6% at 192 tokens.Source: §6.2 · p. 8

6.3 GENERALIZATION AND ROBUSTNESS

  • At 88.9% compression on LLaVA-NeXT-7B, DIPrune retains 98.2% average performance versus 87.2% for FastV and 90.6% for SparseVLM.On Qwen2.5-VL-7B, it beats FastV and HoloV across budgets, with about a 3% average-performance lead over HoloV at each pruning ratio, including 93.3% vs. 90.5% at 88.9%; Table 3 reports this comparison.Source: §6.3 · p. 8, Table 3 · p. 8

6.5 EFFICIENCY ANALYSIS

  • At comparable TTFT, DIPrune retains 97.7% accuracy versus HoloV’s 93.7%, with 0.24 ms overhead (0.74% of TTFT); Table 4 reports the efficiency analysis.It reduces TTFT by 55.6% while ITT remains nearly identical across pruning methods; under matched pruning layers, it leads by +0.6% at L6 and +2.3% at L10.Source: Table 4 · p. 9, §6.5 · p. 9

6.4 ABLATION STUDIES

  • Pruning at layer 10 balances semantic preservation and computational savings: earlier pruning degrades performance, while layers 12 and later yield marginal gains but smaller savings.The tested dual-importance weights perform best at α = 0.6 and β = 0.4; excessive α neglects inter-layer dynamics, while excessive β lacks current-layer saliency anchoring.Source: §6.4 · p. 9

7 CONCLUSION & LIMITATIONS

The paper concludes that DIPrune combines intra-layer alignment and inter-layer evolution to preserve task-critical tokens. It also specifies limits for cluttered scenes, short queries, and settings favoring pre-LLM compression.

7 CONCLUSION & LIMITATIONS

  • DIPrune evaluates intra-layer static alignment and inter-layer dynamic evolution to preserve task-critical tokens, but alignment-based pruning may have a smaller margin in cluttered scenes or with extremely short queries.Because importance estimation uses semantic evolution within the LLM backbone, latency benefits may also be constrained when pre-LLM token compression is preferred.Source: §7 · p. 10

Supplementary Material This appendix is organized as follows:

  • The appendix organizes supplementary material into experimental settings, extended evaluations, further ablations, efficiency analysis, and visualizations of retained tokens.It assigns these topics to Appendices A and B, with the efficiency breakdown in B.3 and token visualizations in B.4.Source: Supplementary Material This appendix… · p. 14

A.1 BENCHMARKS

  • The evaluation suite covers eight benchmarks across general VQA, fine-grained perception and reasoning, hallucination evaluation, and instruction following, plus grounding, video QA, and generative tasks.The listed additional tasks include RefCOCO grounding, MSVD-QA and MSRVTT-QA video QA, captioning, OCR, chart understanding, and multi-turn reasoning.Source: §A.1 · p. 14

A.2 IMPLEMENTATION DETAILS

  • The evaluated models include LLaVA-v1.5-7B/13B, LLaVA-NeXT-7B, Video-LLaVA-7B, and Qwen2.5-VL and Qwen3-VL dense and MoE models.Experiments follow each model’s official evaluation protocols and default scripts.Source: §A.2 · p. 14
  • The default configuration prunes once at layer 10, with δ = 1, Mmax = 200, α = 0.6, and β = 0.4.The implementation uses manual text-to-vision attention and is compatible with SDPA and FlashAttention-2.Source: §A.2 · p. 14

A.3 COMPUTATIONAL COMPLEXITY ANALYSIS

The complexity analysis separates DIPrune’s attention construction, scoring, and pruning execution costs. It contrasts these overheads with the quadratic token dependence of attention in the LLM’s decoding layers.

A.3 COMPUTATIONAL COMPLEXITY ANALYSIS

  • Attention construction uses visual keys and a recent text-query window, typically 0.8× the text length; the decoupled computation remains compatible with FlashAttention-2 and SDPA.The windowed construction is described as reducing FLOPs while producing performance virtually identical to the full-text baseline.Source: §A.3 · p. 15
  • The scoring stage combines attention aggregation, rank calculation, and rank-displacement fusion, with estimated cost Nv · (Lwin · H + log Nv + 5).Because Nv is small relative to hidden dimension D in the stated example, scoring is described as negligible compared with the preceding matrix multiplication.Source: §A.3 · p. 15
  • Pruning execution selects top-k token indices in O(Nv) time, while gathering retained tokens is described as requiring nominally zero arithmetic FLOPs.In the decoding layers, the attention term depends quadratically on the retained-token count, so pruning reduces later inference burden.Source: §A.3 · p. 15
  • DIPrune’s overhead is dominated by lightweight attention construction and scales linearly with visual-token count and query-window size, rather than with full self-attention’s quadratic token cost.The stated approximation is 2 · Nv · Lwin · D + O(Nv · H) ≈ 2 · Nv · Lwin · D.Source: §A.3 · p. 16

A.4 ROBUSTNESS OF QUERY WINDOW STRATEGY

The query-window study finds that performance saturates around a 0.8 ratio and compares that setting with full-text attention across prompt lengths. The appendix also describes how decoupled attention produces per-token scores for the DIPrune pipeline.

A.4 ROBUSTNESS OF QUERY WINDOW STRATEGY

  • Against full attention, the 0.8 query-window ratio has performance gaps of −0.15, +0.44, and −0.18 for short, medium, and long TextVQA prompts, respectively, with +0.06 overall; Table 5 reports the ratio comparison.Performance saturates around 0.8 across tested ratios from 0.3 to 1.0, so 0.8 is the adopted default.Source: §A.4 · p. 16, Table 5 · p. 16

A.5 IMPLEMENTATION DETAILS OF DECOUPLED ATTENTION

  • At the pruning layer, decoupled attention uses visual keys and a recent text-query window, mean-pools attention across heads, then averages across query positions to score each visual token.The resulting scalar token score feeds the standard dual-importance scoring pipeline.Source: §A.5 · p. 16

B.1 MORE EXPERIMENTAL VALIDATION ON GENERALIZATION AND SCALABILITY

DIPrune generalizes across model scales, architectures, modalities, and task types, with advantages over baselines reported under aggressive pruning. Stratified results also identify shorter queries and medium-density scenes as more challenging regimes.

Model Scaling to Larger Models: LLaVA-v1.5-13B

  • At 128 tokens, DIPrune retains 99.9% of unpruned performance on LLaVA-v1.5-13B, versus 99.5% on the 7B model; at 64 tokens, it leads DART by +2.8%.The passage attributes this pattern to a plausible scale-dynamics synergy, not a confirmed mechanism.Source: §B.1 · p. 17

Cross-Modality Generalization: Video-LLaVA-7B

  • On Video-LLaVA-7B, DIPrune reduces the token budget while maintaining superior overall performance on video QA at 50% token retention.Source: §B.1 · p. 17, Table 7 · p. 18

Fine-Grained Task Generalization: Qwen2.5-VL-7B

  • On visual grounding, DIPrune outperforms baselines across RefCOCO, RefCOCO+, and RefCOCOg while preserving spatially precise tokens for coordinate regression.The passage links Salign to textual referring expressions and Sevol to tokens important in deeper-layer coordinate regression.Source: §B.1 · pp. 17–18, Table 8 · p. 18

Cross-Architecture Scalability: Qwen3-VL Series

  • At 88.9% pruning, DIPrune surpasses HoloV by up to +11.3 on POPE for Qwen3-VL-8B and +11.7 on MMBench for Qwen3-VL-30B-A3B.It outperforms HoloV across all pruning ratios on both the dense and MoE models.Source: §B.1 · pp. 18–19, Table 9 · p. 18

Generative Task Generalization

  • Across DetailCaps, CHAIR, ConvBench, OCRBench, and ChartQA, DIPrune reports superior performance across both models and pruning ratios, with captioning gains growing as pruning increases.The passage describes this as effectiveness beyond discriminative tasks.Source: §B.1 · p. 19, Table 10 · p. 19

Stratified Analysis on Query Length and Scene Density

  • At 88.9% pruning, DIPrune leads HoloV across GQA query lengths, but its margin is +2.0% for 3–5-word queries versus +4.1% for 6–11-word queries; HoloV is slightly better on medium-density CHAIRi scenes.The passage associates the smaller short-query margin with a weaker alignment signal and identifies cluttered scenes as challenging.Source: §B.1 · p. 19, Table 11 · p. 19

B.2 MORE ABLATION ANALYSIS

The ablations examine DIPrune’s proxies, rank-displacement settings, fusion weights, pruning layer, and performance across token budgets. Results support combining intra-layer alignment with inter-layer evolution, while showing that several settings involve task- or layer-dependent trade-offs.

Empirical Validation of Proxy Scores

  • Sevol alone captures 61.8% GradMass versus 16.0–16.5% for alternatives, while combining both terms yields the best accuracy, supporting their complementarity.Replacing Salign with raw feature magnitude degrades performance, with or without Sevol.Source: §B.2 · pp. 19–20, Table 12 · p. 20

Impact of Rank Displacement Step δ

  • On LLaVA-NeXT, δ = 1 gives optimal performance; wider rank-displacement windows degrade it, so the authors use δ = 1 by default.The authors suggest adjacent-layer signals are useful and wider windows may add irrelevant historical noise.Source: §B.2 · p. 20, Table 13 · p. 20

Impact of Rank Displacement Clipping Threshold (Mmax)

  • On LLaVA-NeXT-7B, Mmax = 200 balances TextVQA performance at 58.7% with stability on GQA and SQA; thresholds ≥300 degrade GQA while MME reaches 1807.The authors attribute the latter pattern to possible non-semantic noise at high thresholds.Source: §B.2 · p. 20, Table 14 · p. 20

Extended Sensitivity Analysis on Dual Importance Weights

  • On Qwen2.5-VL-7B, α = 0.6 and β = 0.4 yield peak performance, consistent with the reported optimum across architectures.The weights balance dynamic semantic evolution against stable saliency anchoring.Source: §B.2 · pp. 20–21, Table 15 · p. 21

Analysis on Pruning Layer Selection

  • The optimal pruning layer is a trade-off: pruning too early harms accuracy before multimodal representations mature, while pruning too late reduces computational savings.The recommended choice is a relative stage of representation maturity, identified by scanning candidate layers on a lightweight validation subset; the passage says this requires no gradients or fine-tuning.Source: §B.2 · p. 21

Performance Across Extended Token Budgets

  • At 16 tokens, DIPrune beats VisionZip by +4.08 on TextVQA and +7.65 on GQA, and HoloV by +3.44 on TextVQA and +6.00 on GQA.It achieves the best results at all four evaluated token budgets, with the performance gap widening as the budget decreases.Source: §B.2 · p. 22, Table 16 · p. 22

B.3 MORE EFFICIENCY ANALYSIS

DIPrune reports lower time-to-first-token latency across architectures, low pruning overhead, and reduced peak memory. Visualizations compare retained patches with FastV and illustrate the persistence of high-ranking shallow-layer tokens.

B.3 MORE EFFICIENCY ANALYSIS

  • DIPrune reduces TTFT by 55.6% on LLaVA-1.5, 56.9% on LLaVA-NeXT, and 35.8% on Qwen2.5-VL, with pruning overhead at most 0.74% of TTFT.It has the highest accuracy on all three models at comparable TTFT; ITT remains nearly identical across pruning methods.Source: §B.3 · p. 22, Table 17 · p. 22
  • Moving pruning from L6 to L10 improves DIPrune by +3.1%, versus +1.5% for VisionZip and +1.2% for HoloV.The passage suggests DIPrune benefits more from the richer inter-layer evolution signal at deeper layers.Source: §B.3 · p. 22
  • At 66.7% pruning, DIPrune achieves the lowest memory on all reported benchmarks, up to 22.9% less than SparseVLM, while maintaining comparable latency and higher accuracy.The passage attributes this to decoupled attention computation that keeps the main forward pass compatible with efficient kernels.Source: §B.3 · p. 22, Table 18 · p. 23

B.4 MORE SPARSIFICATION VISUALIZATION

  • Across pruning ratios of 66.7%, 77.8%, and 88.9%, Figure 6 compares whether DIPrune and FastV retain task-critical visual regions; Figure 7 shows shallow-layer high-ranking tokens tend to remain dominant.The visualization discussion reports DIPrune preserves relevant visual cues better than FastV, including at 88.9% pruning.Source: §B.4 · p. 23, Figure 6 · p. 24, Figure 7 · p. 25
Loading 2610.08341v1…