PaperlayerView exact revision on arXiv

Source-linked AI summary

DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models

Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, Yong Zhang

arXiv:2503.02175v2cs.CVcs.AIcs.LG

TL;DR

LMM inference becomes costly as visual tokens expand sequence length, while existing pruning approaches may require calibration or fine-tuning or preserve redundant tokens. DivPrune formulates pruning as a Max-Min Diversity Problem, and experiments across 16 datasets report state-of-the-art accuracy together with reduced latency and GPU memory usage.

  • Problem

    Visual tokens substantially increase LMM sequence length and quadratic inference costs, while existing pruning methods can require costly calibration or fine-tuning or use suboptimal importance metrics that retain redundancy.

  • Method

    DivPrune selects visual tokens by solving a Max-Min Diversity Problem that maximizes diversity and minimizes redundancy in the retained subset.

  • Results

    Across 16 image- and video-language datasets, DivPrune achieves state-of-the-art accuracy, with gains under pruning ratios of at least 80%, while improving tested LMMs' memory consumption and end-to-end latency.

  • Takeaways & Limitations

    High-diversity token subsets can represent original visual tokens effectively at high pruning ratios without fine-tuning, while providing practical efficiency benefits in tested LMMs.

  • Takeaways & Limitations

    Some baseline methods cannot match the requested TFLOP ratio finely, so the smallest exceeding ratio may give them a slight advantage over DivPrune.

Abstract

from arXiv · show

Large Multimodal Models (LMMs) have emerged as powerful models capable of understanding various data modalities, including text, images, and videos. LMMs encode both text and visual data into tokens that are then combined and processed by an integrated Large Language Model (LLM). Including visual tokens substantially increases the total token count, often by thousands. The increased input length for LLM significantly raises the complexity of inference, resulting in high latency in LMMs. To address this issue, token pruning methods, which remove part of the visual tokens, are proposed. The existing token pruning methods either require extensive calibration and fine-tuning or rely on suboptimal importance metrics which results in increased redundancy among the retained tokens. In this paper, we first formulate token pruning as Max-Min Diversity Problem (MMDP) where the goal is to select a subset such that the diversity among the selected {tokens} is maximized. Then, we solve the MMDP to obtain the selected subset and prune the rest. The proposed method, DivPrune, reduces redundancy and achieves the highest diversity of the selected tokens. By ensuring high diversity, the selected tokens better represent the original tokens, enabling effective performance even at high pruning ratios without requiring fine-tuning. Extensive experiments with various LMMs show that DivPrune achieves state-of-the-art accuracy over 16 image- and video-language datasets. Additionally, DivPrune reduces both the end-to-end latency and GPU memory usage for the tested models. The code is available $\href{https://github.com/vbdi/divprune}{\text{here}}$.

1. Introduction

Large Multimodal Models face high inference costs because visual tokens greatly increase sequence length, while existing pruning methods can require costly adaptation or retain redundant tokens. DivPrune instead maximizes diversity among retained visual tokens, supporting training-free pruning at high ratios with strong performance and practical efficiency gains.

  • Visual tokens can add thousands to LMM inputs, substantially increasing runtime and memory requirements because these costs scale quadratically with input size.
  • Reducing visual tokens by 50% to 95% has been shown to significantly improve LMM inference speed, reflecting substantial redundancy in visual information.
  • Existing methods may require model-specific calibration or fine-tuning, while attention-based pruning can overlook important tokens and retain similar, redundant tokens.
  • DivPrune formulates token pruning as a Max-Min Diversity Problem that maximizes minimum distances among selected tokens to reduce redundancy.
  • DivPrune is training-free, calibration-data-free, and plug-and-play, with applicability across LLM architectures and vision encoders and compatibility with KV caching.
  • Evaluated across 16 image- and video-language datasets, DivPrune achieves state-of-the-art performance with noticeable gains at pruning ratios of at least 80%.

2. Related Works

Related work improves LMM efficiency through architectural changes, token pruning, calibration, and fine-tuning. These approaches differ in where tokens are removed and in the additional optimization or calibration resources they require.

  • LMM efficiency research includes replacing transformer-based LLMs, using smaller LLMs, knowledge distillation, and skipping model blocks or layers.
  • Attention-based methods prune or merge visual tokens using attention sparsity or attention magnitudes at selected model layers.
  • Calibration-based methods determine pruning layers or ratios from LLM outputs or attention divergence on a calibration dataset.
  • Fine-tuning methods learn nested or compact visual-token representations, but their training requirements limit use across various models.

3. Proposed Method

DivPrune formulates visual token pruning as selecting a diverse subset that preserves essential information while reducing redundancy and computational cost. It uses diversity-based selection to retain representative tokens before passing them with textual tokens to the LLM.

  • LMM Processing: LMMs combine textual tokens with typically many more visual tokens before autoregressive processing by an integrated LLM.Visual inputs are converted into M vision tokens, generally with M ≫ N textual tokens.
  • Token Pruning Formulation: Token pruning selects ˜M visual tokens from M candidates while minimizing output differences and preserving information needed for accurate predictions.A mapping f produces the retained subset ˜Ev from the original visual-token set Ev.
  • Diversity-Based Selection: DivPrune reformulates pruning as a Max–Min Diversity Problem that maximizes the minimum distance among selected tokens to reduce redundancy.The method selects a subset with ˜M elements from Ev and maximizes the minimum pairwise distance between its elements.
  • Diversity-Based Selection: The token distance is defined using cosine distance, providing the pairwise measure used to evaluate diversity among candidate and selected tokens.The distance function d(γ, ω) is specified through cosine similarity between token representations.
  • Selection Procedure: The selection procedure initializes an empty selected subset, chooses an initial token by pairwise distance, then iteratively adds tokens based on their distances to the selected set.After selection, remaining visual tokens are discarded and the selected tokens are passed to the LLM with textual tokens.
  • Selection Procedure: DivPrune can also select visual-token features from intermediate LLM layers before subsequent layers process them.The method obtains high diversity for selected elements in both visual-token and intermediate-feature settings.

4. Experiments

Experiments across image- and video-language models show that DivPrune preserves accuracy better than competing pruning methods, especially under extreme compression, while reducing computation and memory. Ablations attribute this advantage to diversity-based selection and early pruning.

  • Image-language understanding: DivPrune significantly outperforms plug-and-play baselines across image-language datasets despite using lower TFLOPs.On LLaVA 1.5-7B, accuracy drops were 5.1% on GQA and 4.9% on MMB, versus at least 23.0% and 42.8% for VTW and FastV.
  • Image-language understanding: A 12.7% CIDEr drop with DivPrune on COCO contrasts with approximately 95% drops for VTW and FastV.DivPrune also stayed within 2% of the original model on MMMU and SQA while improving POPE and reducing the TFLOP ratio by 84.4%.
  • Image-language understanding: DivPrune outperforms PruMerge on all but one benchmark and exceeds FitPrune by up to 25.1% without calibration or fine-tuning.It achieves comparable or superior performance to fine-tuning-based baselines without fine-tuning.
  • Image-language understanding: On LLaVA 1.6-7B, DivPrune limits accuracy drops to 3.5%, 2.3%, 3.4%, and 1.6% on MMB, OKVQA, POPE, and SQA while reducing TFLOPs by 89%.Baseline pruning causes a 79% POPE F1 drop, whereas DivPrune drops 3.4%; DivPrune also improves MMMU performance.
  • Compression behavior: At extreme compression, DivPrune degrades more gradually than baselines, with the performance gap increasing as the TFLOP ratio approaches 10%.At higher TFLOP ratios, methods nearly converge toward original performance, while FitPrune slightly exceeds DivPrune by an insignificant margin and uses calibration data.
  • Video-language understanding: On video-language tasks, DivPrune achieves up to 12% higher accuracy than FastV and up to 19% higher accuracy than VTW.It remains highly competitive with the unpruned model using only 14.1% of the original model’s TFLOPs, and becomes more effective as visual context grows.
  • Ablation study: Pruning at Layer 0 yields higher task accuracy than pruning at Layers 1, 2, or 3 at a fixed TFLOP ratio of 19.61%.Layer 0 prunes visual tokens before the first decoder layer; later settings process one or more initial layers without pruning.
  • Ablation study: Cosine, ℓ1, and ℓ2 similarity measures perform comparably, while Min-Max and random selection perform 15.8% and 5.6% worse than Max-Min selection.Min-Max enforces high redundancy, whereas random selection provides diversity without guaranteeing its maximum.

5. Conclusion

DivPrune uses max-min diversity to reduce redundancy among selected visual tokens, maintaining representative subsets at high pruning ratios without fine-tuning. Across 16 image- and video-language datasets, it achieves state-of-the-art accuracy while improving memory consumption and end-to-end latency for tested LMMs.

  • DivPrune formulates visual token pruning as a max-min diversity problem, selecting a diverse subset that reduces redundancy.The selected tokens better represent the original visual-token set.
  • DivPrune maintains effective performance at high pruning ratios without requiring fine-tuning.
  • Across 16 image- and video-language datasets, DivPrune achieves state-of-the-art accuracy on the tested tasks.
  • DivPrune generalizes across different model sizes and architectures while improving memory consumption and end-to-end latency for tested LMMs.

A. Datasets, Tasks, and Metrics

The experiments cover 11 image-language and 5 video-language datasets spanning image captioning and several visual question-answering and reasoning tasks. Evaluation uses the corresponding task metrics and default lmms-evals system prompts.

  • The evaluation uses 11 image-language and 5 video-language datasets for image and video understanding.Three additional visual question-answering datasets are described elsewhere in the supplementary experiments.
  • Tasks include image captioning, visual reasoning, open-ended, closed-ended, and multiple-choice visual question answering.
  • The benchmarks use default system prompts from the lmms-evals evaluation package.

B. More Examples for Insights

Additional visualizations compare DivPrune with FastV using token distributions and max-min-distance histograms. DivPrune selects more widely separated tokens, reducing redundancy and improving representation of the original token set.

  • The analysis compares DivPrune and FastV with t-SNE visualizations and max-min-distance histograms over 1000 SeedBench samples.
  • FastV-selected tokens cluster densely, whereas DivPrune-selected tokens are more widely separated, reducing redundancy.
  • DivPrune includes tokens from top clusters that FastV omits, yielding a better representation of the original visual-token set.
  • On GQA samples, DivPrune reduces redundancy and achieves better representation than importance-based pruning, with higher max-min distances across 1000 samples.
  • Figure 4 plots average performance against the pruned model’s TFLOP ratio relative to the unpruned LLaVA 1.5-13B model.Average performance covers COCO CIDEr, OKVQA accuracy, POPE F1, and MMBench accuracy.

C. Results with Additional Datasets

Additional experiments evaluate DivPrune on TextVQA, VizWiz, and VQAv2 using LLaVA 1.5-7B. DivPrune generally outperforms the compared baselines, though its results relative to fine-tuning-based M3 vary by dataset.

  • Additional experiments evaluate visual token pruning on TextVQA, VizWiz, and VQAv2 with LLaVA 1.5-7B.The experiments use the same hyperparameters as the main-manuscript results.
  • DivPrune achieves higher accuracy than PruMerge on VizWiz and VQAv2 under matched pruning ratios.
  • DivPrune achieves higher task performance than calibration-based FitPrune on all three additional datasets.
  • Compared with fine-tuning-based M3, DivPrune performs worse on TextVQA, comparably on VQAv2, and better on VizWiz.

C.1. Different TFLOPs for the 13b Model

For LLaVA 1.5-13b, DivPrune outperforms all baselines, especially under high compression, while methods converge toward original performance at higher TFLOP ratios.

  • The evaluation averages performance across COCO, OKVQA, POPE, and MMBench using CIDEr, accuracy, F1, and accuracy metrics, respectively.The y-axis reports average performance across these four datasets, with CIDEr reaching a tested peak of 1.16.
  • DivPrune significantly outperforms all baselines at high compression, with the performance gap increasing when the TFLOP ratio falls to 25% or below.At higher TFLOP ratios, nearly all methods converge toward the original model’s performance.
  • The figure focuses on high-compression settings with TFLOP ratios of 40% or less.The reported x-axis covers only the high-compression scenario.

D. Qualitative Results

Qualitative examples show DivPrune preserving task-relevant outputs under severe compression, whereas FastV and VTW often produce irrelevant captions or incorrect VQA responses.

  • Image Captioning: Figure 6 compares prompts, ground-truth captions, and outputs from the original model, DivPrune, and baseline pruning methods on COCO examples.Three image-captioning examples are provided.
  • Figure 5 combines t-SNE visualizations of visual tokens from SeedBench and GQA with a histogram of selected-token Max-Min distances on GQA.
  • Image Captioning: At 12% TFLOP, DivPrune produces relevant image captions, while FastV and VTW generate irrelevant captions for the same images.
  • Visual Question Answering: In OKVQA examples, DivPrune outputs match the ground truth, unlike FastV and VTW, which produce incorrect responses or report insufficient information.

E. Hyper-Parameters of Baselines

The experiments use method-specific hyperparameters, including a 90.2% pruning ratio for DivPrune and distinct settings for FastV, VTW, M^3, and FitPrune.

  • DivPrune: DivPrune uses a 90.2% pruning ratio.
  • FastV: FastV uses K = 3 with R = 0.001 for 7B models and R = 0.023 for 13B models.
  • VTW: VTW uses K = 4 for LLaVA 1.5 models and K = 3 for LLaVA 1.6 models.
  • M^3 and FitPrune: M^3 uses S = 56, while FitPrune uses a 90% pruning ratio.
  • Qualitative Comparisons: Figures 6 and 7 provide qualitative comparisons for image captioning and visual question answering tasks.
Loading 2503.02175v2…