Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models

1 INTRODUCTION

• Visual tokens are often redundant and highly correlated, and processing hundreds per image increases computation, memory use, and inference latency.For tasks such as VQA, only task-relevant image regions may need processing.Source: §1 · pp. 1–2
• Prior evaluations vary in method, model, and task coverage and use inconsistent prompts, scoring metrics, and token-retention ratios.They also commonly omit runtime and end-to-end latency and may couple pruning implementations to specific architectures.Source: §1 · p. 2
• UniPruneBench offers a standardized evaluation platform and a modular interface that decouples pruning logic from model architecture.The interface is designed for integration with various LMMs.Source: §1 · p. 2
• The benchmark covers six ability dimensions, ten datasets, ten representative algorithms, and three LMM families, alongside runtime and prefilling-time metrics.The families are LLaVA-v1.5, Intern-VL3, and Qwen2.5-VL.Source: §1 · p. 2
• Random pruning often surpasses designed strategies, no method dominates across scenarios, OCR is most vulnerable, and aggressive pruning sharply degrades performance.The reported trends are consistent across all three model families.Source: §1 · pp. 2–3

2.1 LARGE MULTIMODAL MODEL

• LMMs combine language and vision components through different architectures and training strategies, with recent models targeting visual grounding, reasoning, and interactive tasks.Agent-based visual-tool methods are outside this paper’s scope.Source: §2.1 · p. 3

2.2 VISUAL TOKEN COMPRESSION BENCHMARK.

• A prior analysis evaluated four pruning baselines but provided no source code or reproducible scripts, whereas this work implements ten algorithms and evaluates broader tasks and metrics.The cited baselines are FastV, SparseVLM, random pruning, and pooling.Source: §2.2 · p. 3

3 UNIPRUNEBENCH

• The benchmark spans six evaluation dimensions and ten datasets, and tests ten representative methods across three categories and five models from three LMM families.The datasets are released under permissive licenses for research use.Source: §3 · p. 3

3.1 VISUAL TOKEN COMPRESSION METHODS

• Plug-and-play pruning methods are grouped by where pruning occurs: ViT-only, LLM-only, or hybrid.The benchmark also adopts random token pruning as a baseline.Source: §3.1 · p. 3

3.2 DATASETS

• The ten datasets cover comprehensive evaluation, mathematical reasoning, OCR, instruction following, multidisciplinary knowledge, and hallucination.Examples include MME and MMBench for comprehensive evaluation, MathVista and Math-Vision for mathematical reasoning, and OCRBench for OCR.Source: §3.2 · p. 4

3.3 BASE MODEL

• Five open-source models are evaluated across LLaVA-v1.5, InternVL3, and Qwen2.5-VL model families.The tested models are LLaVA-v1.5-7B, InternVL3-1B, InternVL3-8B, Qwen2.5-VL-3B, and Qwen2.5-VL-7B.Source: §3.3 · p. 4

3.4 EVALUATION METRICS

• The benchmark records accuracy and performance drop alongside total running time, compression-strategy time, and prefilling-phase time.Prefilling time covers processing image and text tokens before the first decoding step.Source: §3.4 · pp. 4–5

4.1 IMPLEMENTATION DETAILS

• Experiments use batch size 1, vary pruning ratios, and fix the intra-LLM pruning location at layer K = 2 for fair comparisons.They report average accuracy and relative performance; MME accuracy is normalized to 0–100.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• Random pruning consistently outperforms several designed methods, and six of eight methods fare worse on LLaVA-v1.5-7B at 66.7% and 77.8% pruning.The comparison names GPrune, VTW, and PruMerge among methods that random pruning surpasses.Source: §4.2 · p. 7
• No method dominates across models and ratios: DivPrune leads on Qwen2.5-VL-7B and InternVL3-8B, while LLaVA-v1.5-7B results shift with pruning level.Hybrid methods lead on LLaVA-v1.5-7B at 66.7% and 77.8% pruning but fare worse at 88.9%; ViT-only methods beat LLM-only methods on the other two models.Source: §4.2 · p. 7, Table 3 · p. 7
• Instruction-following tasks sometimes improve with pruning, whereas OCR degrades severely; these task-specific sensitivities recur across all three models.On InternVL3-8B, DivPrune raises MIA accuracy from 72.22% to 79.82%; the authors hypothesize this may reflect greater relative weight for textual inputs.Source: §4.2 · p. 8
• More aggressive pruning brings sharper accuracy losses; DivPrune nevertheless achieves the best results at 88.9% pruning.Under random pruning on Qwen2.5-VL-7B, average accuracy falls from 57.5% at 33% tokens to 50.1% at 11% tokens.Source: §4.2 · p. 8

4.3 ANALYSIS AND DISCUSSION

• Across nearly all benchmarks and compression methods, scaling InternVL3 from 1B to 8B improves accuracy and robustness to token reduction.The model-size comparison uses DivPrune, GPrune, and FastV at an 88.9% pruning rate.Source: §4.3 · p. 8, Figure 3 · p. 8
• At 88.9% pruning on MME, DivPrune and GPrune shorten prefill from 320 s to 185 s and 167 s, yielding 1.73–1.92× encoder and 1.62–1.68× end-to-end speed-ups.Compression-subroutine time stays below 0.5 s, less than 0.12% of total time.Source: §4.3 · p. 8, Table 4 · p. 9
• At the same 88.9% pruning rate, the best single-stage Pre-LLM method scores 64.04%, exceeding every two-stage 66.7% × 66.7% hybrid.The authors caution that chaining off-the-shelf criteria does not guarantee additive gains and may discard informative tokens in the second stage.Source: §4.3 · pp. 8–9, Table 5 · p. 9

5 CONCLUSION

• UniPruneBench evaluates visual token pruning across datasets, model families, algorithms, and system-level efficiency metrics.The benchmark is intended to address fragmented and non-standardized prior evaluations.Source: §5 · p. 9
• The experiments find random pruning competitive, no universally superior method, and task-specific pruning vulnerabilities.Source: §5 · p. 9
• The authors hope the benchmark will facilitate fair comparison and reproducibility and inspire advances in efficient multimodal learning and deployment.Source: §5 · p. 9

• The benchmark itself poses minimal direct ethical risk because it does not generate content or make decisions.Source: ETHICS STATEMENT · p. 10
• UniPruneBench relies on public datasets whose demographic, language, or visual-concept biases may affect evaluation outcomes.Source: ETHICS STATEMENT · p. 10
• Faster deployment in sensitive applications is a possible indirect misuse of efficiency results, prompting a call for responsible use.Source: ETHICS STATEMENT · p. 10
• Models evaluated on UniPruneBench may still produce harmful or inappropriate outputs to malicious or unsafe queries, and the benchmark does not mitigate those risks.Users should implement appropriate safeguards.Source: ETHICS STATEMENT · p. 10
• The benchmark aims to advance efficient multimodal modeling with transparency and reproducibility while minimizing ethical concerns.Source: ETHICS STATEMENT · p. 10

• The appendix provides UniPruneBench code, including standardized evaluation scripts and token-pruning implementations.Source: REPRODUCIBILITY STATEMENT · p. 10
• All datasets are publicly available, and the paper specifies preprocessing, prompt templates, token-retention ratios, and system-level measurement protocols.Source: REPRODUCIBILITY STATEMENT · p. 10
• Reproduction instructions cover LLaVA, Intern-VL, and Qwen-VL model families and the evaluated pruning methods.Source: REPRODUCIBILITY STATEMENT · p. 10
• The authors aim for the released benchmark and pipeline to enable fair comparison, further research, and transparency in reported findings.Source: REPRODUCIBILITY STATEMENT · p. 10

• LLMs were used for text refinement, language polishing, and structural editing, including clarity, grammar, concision, and logical flow.Source: A THE USE OF LARGE LANGUAGE MODELS · p. 14
• The authors independently designed, implemented, and verified all scientific content, experiments, and analyses.Source: A THE USE OF LARGE LANGUAGE MODELS · p. 14
• LLMs did not contribute to experimental results, numerical analyses, or the work's core intellectual content.Source: A THE USE OF LARGE LANGUAGE MODELS · p. 14

System Configuration

• Experiments used Python 3.12, PyTorch 2.5.1, Transformers 4.54.0, CUDA 12.4, and NVIDIA A100-40 GB GPUs.Source: B MORE IMPLEMENTATION DETAILS · p. 14

Model Configuration

• The evaluated official checkpoints were LLaVA-v1.5 (7B), Qwen2.5-VL (3B and 7B), and InternVL-3 (1B and 8B), without fine-tuning or weight modification.Qwen2.5-VL preprocessing used min pixels=256×28×28 and max pixels=1280×28×28.Source: B MORE IMPLEMENTATION DETAILS · p. 14

Method Implementation

• Pre-LLM pruning rebuilds a shorter embedding tensor after vision encoding, while intra-LLM pruning hooks at layer K=2 and truncates visual tokens and matching position and mask tensors.Pruning occurs before KV-cache construction, so generation length and memory footprint shrink proportionally.Source: B MORE IMPLEMENTATION DETAILS · p. 14

Metric Calculation

• Official VLMEvalKit scripts produce task scores, MME counts are normalized to 0–100, and relative performance measures accuracy retention across tasks and pruning strengths.Source: B MORE IMPLEMENTATION DETAILS · p. 14

Time Measurement

• Latency is reported as total benchmark time, prefill time before decoding, and pruning-subroutine time within prefill, averaged over three A100-40 GB runs at batch size 1.Source: B MORE IMPLEMENTATION DETAILS · p. 14

C MORE RESULTS

• The appendix benchmarks existing pruning methods on InternVL3-1B and Qwen2.5-VL-3B, reporting results in Tables 6 and 7, respectively.Source: C MORE RESULTS · p. 14, Table 6 · p. 15, Table 7 · p. 16
• The authors interpret the cross-architecture consistency as support for the reliability of their evaluation protocol and the transferability of benchmark insights.Source: C MORE RESULTS · p. 14
• Across InternVL3-1B and Qwen2.5-VL-3B, pruning methods’ relative strengths and limitations remain largely preserved across architectures.The comparisons are reported in Tables 6 and 7, respectively.Source: C MORE RESULTS · p. 14, Table 6 · p. 15, Table 7 · p. 16