PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs

Qizhe Zhang, Mengzhen Liu, Lichen Li, Ming Lu, Yuan Zhang, Junwen Pan, Qi She, Shanghang Zhang

arXiv:2506.10967v2cs.CVcs.AI

TL;DR

MLLMs incur high inference costs because visual-token sequences are much longer than textual sequences, while existing pruning can retain duplicates or overlook instruction relevance. CDPruner uses instruction-conditioned similarity and DPP to maximize conditional diversity without training. It achieves state-of-the-art performance across diverse MLLMs and benchmarks while substantially reducing computation, latency, and memory, though it applies only to open-source MLLMs and some advanced models are more pruning-sensitive.

  • Problem

    MLLMs often process hundreds of visual tokens, making inference costly, while attention- and similarity-based pruning respectively retain duplicates or neglect instruction relevance.

  • Method

    CDPruner defines instruction-conditioned visual-token similarity and uses a determinantal point process to maximize conditional diversity in a training-free, model-agnostic pruning method.

  • Results

    CDPruner achieves state-of-the-art performance across diverse MLLM architectures and benchmarks while reducing inference latency and memory; on LLaVA-NeXT-7B, it cuts FLOPs by 95% and CUDA latency by 78%.

  • Takeaways & Limitations

    Selecting conditionally diverse tokens supports strong performance at high reduction ratios and facilitates more practical MLLM deployment.

  • Takeaways & Limitations

    CDPruner applies only to open-source MLLMs with accessible visual tokens, and advanced models such as Qwen2.5-VL and InternVL3 are more sensitive to pruning than the LLaVA series.

Abstract

from arXiv · show

In multimodal large language models (MLLMs), the length of input visual tokens is often significantly greater than that of their textual counterparts, leading to a high inference cost. Many works aim to address this issue by removing redundant visual tokens. However, current approaches either rely on attention-based pruning, which retains numerous duplicate tokens, or use similarity-based pruning, overlooking the instruction relevance, consequently causing suboptimal performance. In this paper, we go beyond attention or similarity by proposing a novel visual token pruning method named CDPruner, which maximizes the conditional diversity of retained tokens. We first define the conditional similarity between visual tokens conditioned on the instruction, and then reformulate the token pruning problem with determinantal point process (DPP) to maximize the conditional diversity of the selected subset. The proposed CDPruner is training-free and model-agnostic, allowing easy application to various MLLMs. Extensive experiments across diverse MLLMs show that CDPruner establishes new state-of-the-art on various vision-language benchmarks. By maximizing conditional diversity through DPP, the selected subset better represents the input images while closely adhering to user instructions, thereby preserving strong performance even with high reduction ratios. When applied to LLaVA, CDPruner reduces FLOPs by 95\% and CUDA latency by 78\%, while maintaining 94\% of the original accuracy. Our code is available at https://github.com/Theia-4869/CDPruner.

1 Introduction

MLLMs use hundreds of visual tokens, making inference costly because attention complexity grows quadratically with sequence length. CDPruner addresses weaknesses in attention- and similarity-based pruning by selecting instruction-relevant, conditionally diverse tokens with DPP, reducing LLaVA-NeXT-7B computation while preserving performance.

  • Motivation: Hundreds of visual tokens can exceed textual tokens by tens of times, making MLLM inference costly because attention complexity scales quadratically with token length.The burden grows further for video and high-resolution inputs.
  • Limitations of existing methods: Attention-based pruning retains duplicate tokens, whereas similarity-based pruning neglects user instructions and cannot dynamically align pruning with the current question.These weaknesses lead to suboptimal performance.
  • CDPruner: CDPruner maximizes conditional diversity by jointly considering visual-token similarity and instruction relevance, without requiring additional training.It reformulates token pruning with a determinantal point process to select the retained subset.
  • Practical advantages: CDPruner is plug-and-play and model-agnostic, does not require attention scores, and is compatible with efficient attention implementations and token-based MLLMs.These properties support application across different model architectures.
  • Results: 95% FLOPs reduction, 78% CUDA latency reduction, and 17% GPU memory reduction were achieved on LLaVA-NeXT-7B while maintaining 94% of original performance.The reported setting is training-free.
  • Results: CDPruner consistently achieves state-of-the-art performance across different reduction ratios in experiments on vision-language benchmarks.The contribution statement reports broad benchmark effectiveness.

2 Related work

Related work reduces MLLM inference cost through visual-token compression, especially by pruning tokens during inference. Existing approaches use attention or similarity, while DPP provides a framework for modeling list-wise diversity.

  • Visual token reduction: Visual-token reduction accelerates MLLM inference, while pre-fusion compression methods require architectural modifications and additional training.Training-free token pruning instead removes redundant visual tokens during inference.
  • Attention-based pruning: Attention-based methods assess visual-token importance through text-visual attention but can suffer attention shift and require attention scores incompatible with efficient implementations such as FlashAttention.These constraints can compromise pruning accuracy and implementation compatibility.
  • Determinantal point process: DPPs model diversity through pairwise similarity and have been adopted for list-wise diversity across multiple domains.Their origin is linked to the anti-bunching behavior of fermion systems.

3 Method

CDPruner prunes visual tokens by combining feature similarity with instruction relevance, then selecting a diverse subset through a determinantal point process. Its pipeline uses relevance-conditioned similarity and efficient MAP inference to support dynamic, model-agnostic pruning.

  • 3.4 CDPruner: CDPruner is training-free and model-agnostic, enabling application across token-based MLLMs without requiring attention scores.This design is compatible with efficient attention acceleration implementations and different visual encoders or language models.
  • 3.1 Visual token pruning: Visual token pruning reduces inference cost by retaining m tokens from the original n visual tokens while minimizing output discrepancy.The pruning objective treats m as smaller than n and measures discrepancy between outputs before and after pruning.
  • 3.2 DPP with token similarity: DPP models the diversity of retained visual-token subsets through a kernel matrix whose subset probability is proportional to det(L_S).L_S is the principal submatrix of the kernel corresponding to subset S.
  • 3.3 Instruction relevance: Instruction relevance is computed as cosine similarity between each visual-token embedding and the instruction embedding, then normalized to [0, 1].The embeddings may come from paired visual and text encoders or from a multimodal projector and language-model instruction tokens.
  • 3.4 CDPruner: CDPruner first computes similarity between visual tokens conditioned on their relevance to the user instruction, then uses DPP to select retained tokens.The method integrates feature similarity and instruction relevance into token-subset selection.
  • 3.4 CDPruner: The conditional kernel jointly incorporates feature similarity and instruction relevance, and MAP inference obtains the retained subset using greedy polynomial-time optimization.Cholesky decomposition reduces the overall time complexity to O(nm^2), with additional latency below 10ms per sample when m ≪ n.

4 Experiments

CDPruner is evaluated across diverse MLLM architectures, benchmarks, resolutions, and video settings, where it preserves performance under aggressive visual-token reduction. It also reduces inference costs and benefits from jointly modeling token diversity and instruction relevance.

  • Experimental setup: CDPruner is evaluated on LLaVA-1.5, LLaVA-NeXT, LLaVA-Video, and Qwen2.5-VL across image, video, and high-resolution settings.The evaluation includes multiple MLLM architectures and varied visual-token workloads.
  • Main results: At 64 retained visual tokens, CDPruner decreases LLaVA-1.5’s original performance by only 3.4%, outperforming VisionZip and DivPrune by 2.6% and 2.3%, respectively.This setting retains roughly one-tenth of the original visual-token length.
  • Main results: With only 5.6% of visual tokens retained, CDPruner maintains 94.3% of original LLaVA-1.5 performance and exceeds DivPrune by 3%.Attention- and similarity-based methods show noticeable degradation at this reduction level.
  • Video understanding: On LLaVA-Video, CDPruner maintains 98.6% of original performance after pruning 62.1% of visual tokens and preserves 95% performance after pruning 81.1%.With only 16 visual tokens retained per frame, it maintains 89.7% performance and improves over SparseVLM by 10%.
  • Advanced architectures: With 128 visual tokens on Qwen2.5-VL, CDPruner maintains 85.2% of original performance, compared with 79.9% for DivPrune.This result supports generalizability to an advanced MLLM architecture whose visual encoder and projector exclude methods requiring a [cls] token.
  • Efficiency analysis: Reducing LLaVA-NeXT’s visual tokens from 2,880 to 320 yields nearly a ×10 FLOPs reduction, while CDPruner reduces prefill and decode CUDA latency by ×6.6 and ×1.3.The analysis also reports reductions in KV cache and GPU memory, with CDPruner achieving the best efficiency while maintaining the highest performance.
  • Ablation study: The ablation shows that unconditioned DPP outperforms DivPrune, while adding instruction relevance produces further gains for CDPruner.This supports jointly modeling feature similarity and instruction relevance.

5 Conclusion

CDPruner is a training-free method that prunes visual tokens by maximizing instruction-conditioned diversity with DPP. The paper also provides a greedy MAP implementation and evaluates the method across multiple MLLM architectures.

  • 5 Conclusion: CDPruner defines instruction-conditioned similarity between visual tokens and selects a diverse subset by reformulating pruning with a determinantal point process.The method is designed for MLLM inference acceleration and does not require additional training.
  • 5 Conclusion: Experiments cover LLaVA, LLaVA-NeXT, LLaVA-Video, Qwen2.5-VL, and InternVL3, spanning image and video MLLM architectures.The evaluated architectures use varied visual encoders, resolutions, and tokenization schemes.
  • 5 Conclusion: The DPP MAP problem is NP-hard, so the implementation uses fast greedy inference with incremental Cholesky updates.The algorithm operates on a kernel matrix and repeatedly adds items until the retained subset reaches size m.
  • 5 Conclusion: The greedy DPP algorithm has overall complexity O(nm^2), and CUDA parallelization reduces additional inference latency to less than 10ms per sample.The latency is described as negligible for the intended setting.

B.2.1 General image benchmarks

The general image evaluation covers visual question answering, hallucination, multimodal perception and cognition, broad capability assessment, and multimodal integration.

  • General image benchmarks: VQAv2 evaluates open-ended visual question answering using images paired with multiple annotated answers.The test-dev split is used for evaluation.
  • General image benchmarks: GQA assesses compositional reasoning and visual understanding with scene-graph-annotated images and balanced question-answer pairs.The test-dev balanced split is used.
  • General image benchmarks: VizWiz evaluates visual question answering and unanswerable-question prediction in real-world accessibility images captured by blind users.The benchmark emphasizes poor image quality and ambiguous content.
  • General image benchmarks: POPE measures object hallucination through object-presence questions using precision, recall, and F1 score.Its images are sourced from COCO.
  • General image benchmarks: MME evaluates multimodal perception and cognition across 14 subtasks, including recognition, OCR, and cognitive abilities.Perception tasks include coarse- and fine-grained recognition.
  • General image benchmarks: MMBench and MM-Vet assess broad multimodal capabilities through diverse questions and six integrated capabilities, respectively.MMBench uses CircularEval, while MM-Vet uses ChatGPT-assisted evaluation.

B.2.2 Text-oriented benchmarks

The paper discusses text- and video-oriented evaluation alongside prior visual-token pruning methods. These benchmarks test diagram reasoning, scene text, charts, OCR, and temporal or long-form video understanding.

  • Text-oriented benchmarks: AI2D evaluates visual reasoning over grade-school science diagrams with structured labels, parses, and multiple-choice questions.The test split with mask is used.
  • Text-oriented benchmarks: TextVQA measures reading and reasoning about scene text in images such as signs, billboards, and product packaging.The task integrates OCR with visual question answering.
  • Text-oriented benchmarks: ChartQA tests multistep question answering over charts using visual interpretation and logical or arithmetic operations.Questions are human-written or generated from chart summaries.
  • Text-oriented benchmarks: OCRBench evaluates OCR capabilities across 29 datasets covering recognition, scene-text VQA, document VQA, extraction, and handwritten mathematics.It spans multiple text-related visual tasks.
  • Video benchmarks: MLVU, MVBench, LongVideoBench, and Video-MME evaluate long-video understanding, temporal reasoning, referring reasoning, and multimodal video comprehension.Their settings include videos with varied lengths, tasks, subtitles, and audio.
  • Prior token-pruning methods: Prior pruning methods use attention, visual-text similarity, clustering, diversity heuristics, or minimum pairwise distance to select or merge visual tokens.The cited approaches include FastV, PyramidDrop, SparseVLM, TRIM, VisionZip, DART, and DivPrune.

B.4 Implementation details

The implementation evaluates CDPruner with official or established evaluation codebases and reports strong retention under aggressive pruning on larger and advanced MLLMs.

  • Implementation details: The experiments use official LLaVA3 implementation for image benchmarks, official LLaVA-NeXT code with lmms-eval for video benchmarks, and VLMEvalKit for Qwen2.5-VL.These choices cover the paper’s image, video, and advanced-architecture evaluations.
  • Larger language models: Table 6 compares pruning methods on LLaVA-1.5-13B using average accuracy across 10 benchmarks and average percentage of performance maintained.The table distinguishes attention-based, attention-and-similarity-based, and similarity-based methods by background color.
  • Larger language models: With 77.8% of visual tokens removed, CDPruner retains 98.0% and 99.9% of original performance on LLaVA-1.5-13B and LLaVA-NeXT-13B, respectively.CDPruner consistently outperforms the other pruning approaches across reduction ratios in these experiments.
  • Advanced open-source MLLM: At 90% visual-token removal on InternVL3, CDPruner retains 83.9% of original performance, 3% higher than the second-best FastV.The evaluation fixes the input resolution to 896×896, yielding 1,280 visual tokens.

C.3 Efficiency analysis on larger language model

CDPruner remains efficient on LLaVA-NeXT-13B, reducing visual-token processing substantially while maintaining competitive performance. The section compares pruning methods across larger MLLMs and reports efficiency gains for higher-resolution inputs.

  • Efficiency analysis: 10× reduction in FLOPs results from reducing LLaVA-NeXT-13B visual tokens from 2,880 to 320.The analysis targets a larger model with higher computational demands.
  • Efficiency analysis: Tables 7 and 8 compare pruning methods on LLaVA-NeXT-13B and InternVL3-8B using average performance and performance-maintenance measures.The tables distinguish attention-based, attention-and-similarity-based, and similarity-based methods through colored backgrounds.
  • Efficiency analysis: CDPruner maintains competitive performance while reducing inference cost on the larger MLLM.The reported efficiency analysis evaluates performance on POPE.

C.4 Ablation study on balance factor

The balance factor θ controls the trade-off between diversity and instruction relevance in CDPruner. Its best setting varies across benchmarks, although the main version without this additional hyperparameter already performs strongly.

  • Balance factor: θ modulates the relative importance of relevance and diversity in CDPruner's modeling process.The factor is introduced because instruction information varies across benchmarks.
  • Balance factor: The transformed conditional kernel matrix is constructed as ˜L′ = diag(exp(α˜r)) · L · diag(exp(α˜r)).Here α is derived from θ to transform the relevance vector before constructing the kernel.
  • Balance factor: The DPP subset log-probability is updated after incorporating the transformed relevance and diversity terms.The supplied passage introduces the updated log-probability following the conditional-kernel construction.
  • Balance factor: Selecting the best θ for each dataset improves performance, with the optimal value varying across benchmarks.The ablation is conducted on LLaVA-1.5-7B with 64 visual tokens retained.
  • Balance factor: The main version without the balancing factor already achieves strong results, so tuning θ is optional for practical applications.The authors leave the choice to introduce and tune this hyperparameter to application needs.

D Additional visualization results

Additional relevance visualizations show that language-image-pretrained models capture correspondences between user instructions and image regions. These correspondences support instruction-guided visual token pruning.

  • Additional visualization results: Language-image-pretrained models effectively capture correspondence between user instructions and regions of interest in images.The visualizations use relevance scores to illustrate instruction-region correspondence.
  • Additional visualization results: Instruction-region correspondence is crucial for instruction-guided visual token pruning in MLLMs.

E Limitations

CDPruner requires access to encoded visual tokens during inference and therefore applies only to open-source MLLMs. More advanced open-source architectures can also be more sensitive to pruning, likely because they already compress visual tokens.

  • Limitations: CDPruner applies only to open-source MLLMs whose encoded visual tokens are accessible during inference.Blackbox models such as ChatGPT, Gemini, and Claude are outside this stated scope.
  • Limitations: Qwen2.5-VL and InternVL3 are applicable architectures, but they generally experience greater performance degradation after pruning than the LLaVA series.The authors associate this sensitivity with existing visual-token compression techniques such as pixel unshuffle.
  • Limitations: Enabling efficient inference within architectures that already incorporate visual token compression remains future work.
  • Limitations: Redundant visual inputs increase MLLM computational complexity and usage cost, motivating visual-token pruning without additional training.
Loading 2506.10967v2…