PaperlayerView exact revision on arXiv

Source-linked AI summary

Rescuing Your Pruning: Rethinking Visual Token Pruning and Post-Training Recovery in MLLMs

Peiran Yin, Hao Wu, Zihao Yang, Haozhe Hu, Xudong Wang, Xiaoyu Shen

arXiv:2610.33688v1cs.CV

TL;DR

Visual-token pruning reduces MLLM inference cost, but the extent to which post-training can recover pruning losses and how to allocate compression across stages remain open questions. The paper builds a unified post-training evaluation across pruning configurations and finds broad recovery, while joint-compression outcomes depend strongly on where tokens are removed. Its best reported joint configuration speeds up the LLM stage by 3.49× and TTFT by 1.54×.

  • Problem

    Visual-token pruning reduces MLLM inference cost, but how much pruning-related accuracy loss post-training can recover and how to compress pipeline stages jointly remain open questions.

    Source: §1 · p. 1

  • Method

    The study builds a 282K-sample task-diverse mixture and applies a unified post-training protocol to 36 pruning configurations across two backbones.

    Source: §1 · p. 2

  • Results

    Post-training improves every tested configuration and recovers most pruning-related performance loss, including much of the OCR and grounding ability training-free pruning nearly destroys.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Joint-compression outcomes depend on token-removal location: the best reported configuration speeds up the LLM stage by 3.49× but TTFT by only 1.54×.

    Source: §1 · p. 3

    On LLaVA-OV-1.5-8B, the adapted dense model reaches 150.3 grounding retention relative to the Dense-Free baseline, so post-pruning scores above 100 should not be read as outperforming that adapted dense model.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Visual token pruning substantially reduces the inference cost of multimodal large language models (MLLMs), yet existing methods are mostly training-free and plug-and-play. They treat the resulting performance drop as an inevitable cost of compression and pursue increasingly complex pruning rules. We rethink this view through two questions: how much of the loss can post-training rescue, and, building on this recovery, how can joint compression push pruning to its limit? We curate a 282K-sample post-training mixture covering general VQA, reasoning, OCR, and grounding, and post-train 36 configurations that drop or merge tokens under a unified protocol. Post-training rescues most of the loss, even for OCR and grounding, and its adaptation gain in closing the gap between the pruned and the dense model grows as the budget tightens. For joint compression, we find that the vision encoder, not the LLM, dominates prefill time in native-resolution MLLMs, so where tokens are removed matters more than how many: keeping 40% of tokens at the encoder runs faster than keeping 10% at the projector. Our best configuration prunes the encoder lightly and the projector aggressively, leaves the LLM intact, and keeps general understanding and reasoning at the dense level. It accelerates the LLM stage by 3.49x but the end-to-end time-to-first-token by only 1.54x, showing that LLM-stage speedups overstate end-to-end gains. We hope these findings offer insights for future research on joint visual token compression. Our data, protocol, and code will be released.

1 Introduction

The introduction asks how much pruning-related performance loss post-training can recover and how to allocate compression across the encoder, projector, and LLM. It presents broad recovery and joint-compression findings, emphasizing that token-removal location matters more than token count.

1 Introduction

  • Visual-token compression targets inference cost because visual tokens often far outnumber text tokens, and pruning can occur at the encoder, projector, or LLM by dropping or merging tokens.The introduction frames two questions: how much post-training can recover and how to compress the stages together.Source: §1 · p. 1
  • The study applies CAPT to 36 configurations using a 282K-sample task-diverse mixture, two backbones, three keep ratios, and 12 benchmarks under a unified protocol.It reports retention relative to the unpruned model.Source: §1 · p. 2
  • CAPT improves every tested configuration, recovering most pruning-related performance loss and much of the OCR and grounding ability that training-free pruning nearly destroys.At a 20% average keep ratio, general understanding and reasoning approach the original model in nearly every configuration.Source: §1 · p. 2
  • With CAPT, joint configurations keeping about 5% to 7% of visual tokens preserve dense-level general understanding and reasoning, unlike training-free methods even at a 25% budget.The configuration keeping the fewest tokens is not the fastest; where tokens are removed matters more than how many are kept.Source: §1 · p. 2
  • Keeping 40% of tokens at the encoder runs faster than keeping 10% at the projector, because projector pruning leaves encoder cost untouched.The introduction also recommends light encoder pruning paired with aggressive projector pruning and says compressing all three stages does not pay off.Source: §1 · p. 3
  • With joint compression, ToMe-70 with AIM-10 speeds up the LLM stage by 3.49× and TTFT by 1.54×, without degrading general understanding and reasoning.The introduction also reports that LLM-stage speedups overstate end-to-end gains.Source: §1 · p. 3

2 Related Work

Prior visual-token compression mainly uses dropping or merging, including methods that combine pruning across pipeline stages. The paper studies these mechanisms after post-training and examines stage-wise budget allocation under a common final token budget.

2 Related Work

  • Token dropping removes low-utility tokens using attention or feature similarity, while merging fuses redundant representations into retained tokens or compact representatives.Examples include FastV, AIM, FiCoCo, and PACT.Source: §2 · pp. 3–4
  • Unlike mostly training-free or plug-and-play studies, this work analyzes how dropping and merging behave after adaptation and systematically investigates joint encoder–projector–LLM compression.Its analysis considers how stage-wise budgets affect the accuracy–efficiency trade-off after adaptation.Source: §2 · p. 4
  • Recent methods combine compression across stages, but the interaction among stages and budget allocation under the same final token budget remain insufficiently understood.Examples include FiCoCo, FocusLLaVA [61], METEOR, and SAINT [16].Source: §2 · p. 4

3 Post-Training and Evaluation

The study constructs a compact, task-diverse post-training dataset and applies a uniform protocol that leaves pruning rules fixed while adapting model parameters around the pruning operation. It evaluates capability retention and latency using defined metrics and shared evaluation settings.

Post-Training Data

  • The approximately 282K-sample mixture covers general VQA, reasoning, OCR, and grounding, combining LLaVA-Icons, portions of six datasets, and visual grounding data.The original multi-turn conversations are reorganized into single-turn samples anchored by individual images.Source: §3 · p. 4

Training Protocol

  • The training protocol keeps each fixed-arithmetic pruning rule unchanged and cuts gradients at token selection, adapting the model upstream and downstream of pruning.Sequences are repacked after reductions, and retained tokens keep their pre-pruning indices for rotary embeddings.Source: §3 · p. 4
  • Experiments use Qwen2.5-VL-7B and LLaVA-OneVision-1.5-8B, with LLM-level methods unfreezing only the language model and other methods unfreezing every module.For Qwen2.5-VL, max_pixels is fixed at 451,584 to balance fine-grained detail and training cost.Source: §3 · p. 4

Evaluation Protocol and Metrics

  • Evaluation covers General, Reasoning, OCR, and Grounding on 12 benchmarks; GRR averages General and Reasoning retention, while ATR averages all four groups.Retention is the pruned-model score divided by the dense-model score, expressed as a percentage.Source: §3 · pp. 4–5

Latency Measurement

  • Latency is measured on one NVIDIA A100 GPU at batch size 1 using SDPA and the evaluation resolution, reporting TTFT from a single generated token and its stage breakdown.The accompanying stage timing passage averages each number over 5,000 TextVQA samples after five warm-up samples.Source: §3 · p. 5

4 Rescuing Pruning Across Levels and Mechanisms

Across pruning levels and mechanisms, post-training recovers much of the performance lost to pruning, but fine-grained tasks remain more vulnerable than general understanding and reasoning. The analysis attributes recovery to both adaptation and preserved visual information, with adaptation helping more as pruning becomes aggressive.

4.1 What Breaks and What Is Restored

  • All 36 configurations improve after post-training, and recovery gains grow with training-free degradation across the two backbones.The fitted slopes are 0.52 on Qwen2.5-VL-7B and 0.81 on LLaVA-OV-1.5-8B.Source: Figure 3 · p. 5, §4.1 · p. 5
  • Grounding rises from 4.0 to 64.8; fine-grained tasks gain far more but remain well below the dense model, while the GRR–ATR gap narrows from 33.3 to 17.2.Source: §4.1 · p. 6
  • After post-training, encoder- and projector-level dropping and merging differ by at most 1.4 ATR points on Qwen2.5-VL-7B, while LLM-level merging retains an advantage at 10% keep.At that budget, PACT exceeds PDrop by 12.5 ATR points on Qwen2.5-VL-7B and 9.6 on LLaVA-OV-1.5-8B.Source: §4.1 · p. 6

4.2 Where Does the Recovery Come From?

  • The pruning gap is decomposed into a recoverable component and a residual mainly reflecting information loss; the recoverable gain is further separated into data and adaptation gains.The attribution compares Free, Data-only, Prune-aware, and Dense-PT states.Source: §4.2 · pp. 7–8
  • At aggressive budgets, adaptation exceeds data gain in ATR and accounts for over 60% of recoverable gain at 10% keep, while the residual grows as information loss becomes severe.The residual more than doubles for PDrop on Qwen2.5-VL-7B; the comparison of Prune-aware and Dense-PT mainly attributes this uncompensated gap to information loss concentrated on fine-grained tasks.Source: §4.2 · pp. 8–9
  • On a RefCOCO example, post-training shifts pruning toward tokens inside the ground-truth region and strengthens downstream LLM attention to retained target-region evidence.The visualization compares six methods at 20% keep.Source: Figure 4 · p. 7, §4.2 · p. 9

5 How Far Can Post-Training Push Pruning?

The search fixes projector pruning at AIM-10, then tests additional compression under quality constraints. Encoder pruning contributes more to feasible end-to-end acceleration than LLM-stage pruning alone.

5 How Far Can Post-Training Push Pruning?

  • Under GRR ≥ 100 and ATR ≥ 85, the lowest feasible three-stage setting is ToMe-80, AIM-10, and PACT-70; PDrop fails even at 90% keep.Adding stages requires retaining more tokens at each level because losses accumulate; Fig. 5 marks the lowest feasible keep ratio.Source: Figure 5 · p. 9, §5 · p. 10
  • At a single compressed stage, encoder ToMe-70 and LLM PACT-50 meet the quality constraints; adding encoder pruning to AIM-10 permits further feasible joint compression.ToMe is used at the encoder because it matches SAINT in latency but performs better.Source: §5 · pp. 9–10
  • 1.54× end-to-end speedup: E70+P10 reduces TTFT from 213.06 to 138.61 ms while keeping GRR at the dense level.It is the fastest feasible configuration; its LLM stage is 3.49× faster and its vision side 1.11× faster.Source: §5 · pp. 10–11, Table 3 · p. 10
  • AIM-10 alone yields a 3.38× LLM-stage speedup but only a 1.38× end-to-end speedup, as the vision side remains nearly unchanged and takes about four fifths of TTFT.Dense Qwen2.5-VL-7B takes 213.06 ms TTFT, including 120.43 ms for the vision encoder, 0.65 ms for the projector, and 85.10 ms for the LLM.Source: §5 · pp. 9–10
  • In AIM-10, once tokens are aggressively pruned before the LLM, further LLM-level pruning leaves latency nearly unchanged or slightly higher because the LLM is already close to a length-independent latency floor.Source: §5 · p. 11

6 Conclusion

The paper concludes that compression-aware post-training recovers performance across pruning settings and enables aggressive joint pruning among nearly lossless configurations. It also describes the 282K-sample training mixture and reports no evaluation sample in its supplementary data.

6 Conclusion

  • Compression-aware post-training achieves broad performance recovery across pruning settings, and aggressive joint pruning reaches the highest speedup among nearly lossless configurations.The framework analyzes dropping and merging at encoder, projector, and LLM levels.Source: §6 · p. 11

A.1 Dataset Composition

  • The approximately 282K-sample mixture draws mainly on single-turn LLaVA-Icons and LLaVA-665K grounding data, with OCR, reasoning, and a small ShareGPT language-only supplement.LLaVA-665K provides region annotations on VG and COCO images.Source: §A.1 · p. 17

A.2 Train–Test Overlap

  • Among six supplementary datasets, only 8 AI2D images and 2 DocVQA images overlap evaluation images, and none share the same question, so no evaluation sample appears in the supplementary data.Source: §A.2 · p. 17

A.3 Data Ablation

Data ablations test source overlap and the contribution of OCR, grounding, and chart/diagram data, using same-size random removals as controls. The full mixture performs best on ATR, while targeted removals disproportionately harm the abilities those sources support.

A.3 Data Ablation

  • Each ablation compares a source-group removal with a uniformly random removal of the same number of samples to separate which data is removed from how much.Source: §A.3 · p. 17, Table 6 · p. 18
  • The full mixture achieves the highest ATR, while removing any task category hurts its corresponding ability more than a same-size random removal, most clearly for DocVQA without OCR data.GRR remains above the dense level in all post-trained runs; targeted supplements mainly serve fine-grained abilities.Source: §A.3 · pp. 17–18
  • Without grounding data, RefCOCO-family Acc@0.5 falls below 10% and ATR drops to 71.8, making grounding data irreplaceable in this ablation.Source: §A.3 · pp. 17–18
  • Removing all Direct-Eval sources leaves ATR on par with its random control, while GQA falls below that control and AI2D, TextVQA, and DocVQA stay above training-free levels.The ablation removes GQA, AI2D, TextVQA, and DocVQA, with an equal-size random-removal control.Source: §A.3 · pp. 17–18

B Cross-Ratio Generalization

Post-training transfers asymmetrically across keep ratios, with adaptation at a tight budget carrying over better to looser inference budgets. The search uses ToMe at the encoder because it matches SAINT's latency while achieving slightly higher post-training GRR and ATR.

B Cross-Ratio Generalization

  • 3.9 ATR points: the model trained at 10% keep loses this much when evaluated at 40%, compared with a 6.2-point loss for a model trained at 40% and evaluated at 10%.Matched training and inference ratios give the best ATR in every column; GRR varies by less than 4 points within each column.Source: B Cross-Ratio Generalization · pp. 18–19, Table 7 · p. 19

C Details of the Multi-Level Search

C.1 Encoder Operator Selection

  • ToMe is selected for encoder-level search because it reaches virtually the same prefill latency as SAINT while attaining slightly higher GRR and ATR after post-training.The comparison is at 40% keep on Qwen2.5-VL-7B.Source: §C.1 · p. 19, Table 8 · p. 19

C.2 LLM Latency Floor

LLM prefill latency has a length-independent floor, limiting the benefit of further LLM-side pruning once the visual sequence is short. The fastest feasible configuration therefore retains more tokens than some alternatives.

C.2 LLM Latency Floor

  • 15.9 ms + 0.067 ms per token fits LLM prefill latency; with AIM-10, only about 8 ms of the 25.15 ms latency remains sequence-length-dependent.The fit uses dense Qwen2.5-VL-7B and projector-level AIM at 40%, 20%, 10%, and 7% keep.Source: §C.2 · p. 19
  • Adding PACT-50 on top of AIM-10 raises TTFT from 154.54 to 158.67 ms, as further LLM pruning saves at most about 8 ms but adds operator cost.The LLM processes about 116 tokens under AIM-10.Source: §C.2 · p. 19
  • Figure 6 plots Qwen2.5-VL-7B LLM prefill latency against the number of tokens processed, with an affine fit and a shaded length-independent cost.The plotted points include dense and projector-level AIM at 40%, 20%, 10%, and 7% keep.Source: Figure 6 · p. 19
  • E70+P10 is the fastest feasible configuration despite retaining more tokens than E80+P10+L70 and P10+L50.Once the LLM approaches its latency floor, token count is not a reliable proxy for latency.Source: §C.2 · p. 20

D Details of Pruning Methods

The evaluated pruning methods span dropping and merging at the encoder, projector, and LLM stages. Their mechanisms range from similarity-based token selection to progressive pruning and token clustering.

D Details of Pruning Methods

  • The study evaluates SAINT and ToMe at the encoder, DivPrune and AIM at the projector, and PDrop and PACT inside the LLM.Source: D Details of Pruning Methods · p. 20
  • At the vision encoder, SAINT prunes using key-embedding similarity graphs and layer-adaptive rates, while ToMe iteratively matches and merges similar token pairs inside transformer blocks.Both are described as training-free methods.Source: D Details of Pruning Methods · p. 20
  • At the projector, DivPrune retains a diverse subset by formulating pruning as a Max-Min Diversity Problem, while AIM merges similar tokens before progressively pruning less important tokens inside LLM layers.Source: D Details of Pruning Methods · p. 20
  • Inside the LLM, PDrop progressively reduces visual tokens across layers, while PACT prunes less informative tokens and clusters nearby redundant tokens into representative embeddings.Source: D Details of Pruning Methods · p. 20

E Details of Datasets and Benchmarks

The appendix describes benchmarks spanning general visual understanding, reasoning, OCR, and grounding, then provides fuller results for pruning at each model stage. Across those stages, post-training improves performance, with extreme compression still leaving fine-grained abilities vulnerable.

E Details of Datasets and Benchmarks

F.1 Encoder-Level

  • Encoder-level SAINT and ToMe show broad post-training improvements, especially in OCR and grounding, while extreme compression retains an information-loss upper bound.Source: §F.1 · p. 22

F.2 Projector-Level

  • Projector-level DivPrune and AIM improve stably, with overall performance approaching or exceeding the Dense-Free baseline at 40% and 20% keep.Source: §F.2 · p. 22

F.3 LLM-Level

  • LLM-level PDrop and PACT improve general, reasoning, OCR, and grounding metrics after post-training; PACT is usually more stable, while fine-grained abilities remain vulnerable under extreme compression.Source: §F.3 · p. 22

G Grounding Details

  • Grounding details report IoU, Acc@0.5, and Center Acc by RefCOCO split; full pruning results cover encoder, projector, and LLM levels across two backbones.Tables 9–13 provide the detailed results; Free and Post distinguish training-free pruning from post-training.Source: G Grounding Details · p. 22, Table 9 · p. 23, Table 10 · p. 24

Related papers

Loading 2610.33688v1…