PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods

Chenfei Liao, Wensong Wang, Zichen Wen, Xu Zheng, Yiyu Wang, Haocong He, Yuanhuiyi Lyu, Lutao Jiang, Xin Zou, Yuqian Fu, Bin Ren, Linfeng Zhang, Xuming Hu

arXiv:2510.07143v3cs.CV

TL;DR

General MLLM benchmarks are not designed to assess visual-token compression, leaving a task mismatch. The paper studies downsampling as a way to distinguish compression-relevant samples and introduces VTC-Bench; it finds benchmark noise and that downsampling can separate simple from difficult samples.

  • Problem

    Existing MLLM benchmarks are not designed for visual-token compression and can include overly simplistic, irrelevant samples.

    Source: §3.4 · p. 5

  • Method

    VTC-Bench uses downsampling to distinguish simple from difficult samples and select difficult cases from existing benchmarks.

    Source: §1 · p. 2

  • Results

    Across multiple widely used benchmarks, simple downsampling consistently outperforms many advanced methods, while the study finds benchmark noise and a downsampling-based difficulty distinction.

    Source: §3.5 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    VTC-Bench is intended as an additional reference for fairer evaluation of visual-token compression methods.

    Source: §1 · p. 2

    Filtering relies on Qwen2-VL and downsampling alone, which may limit generalizability and robustness across models and filtering criteria.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent efforts to accelerate inference in Multimodal Large Language Models (MLLMs) have largely focused on visual token compression. The effectiveness of these methods is commonly evaluated by measuring the accuracy drop on existing MLLM benchmarks before and after compression. However, these benchmarks are originally designed to assess general perception and reasoning abilities, rather than the specific challenges posed by visual token compression, leading to a fundamental task mismatch. In this work, we uncover a counterintuitive yet consistent phenomenon: simple image downsampling outperforms many advanced visual token compression methods across multiple widely used benchmarks. Through a comprehensive empirical study spanning eight popular benchmarks and multiple state-of-the-art compression techniques, we show that (i) current benchmarks contain substantial noise (task-irrelevant samples) for evaluating visual token compression, and (ii) downsampling can act as an effective data filter that distinguishes between simple and difficult samples with respect to compression sensitivity. Motivated by these findings, we propose VTC-Bench, an evaluation framework that explicitly leverages downsampling as a discriminator to denoise existing benchmarks, enabling a fairer and more meaningful additional assessment of visual token compression methods.

1 Introduction

Visual-token compression aims to reduce MLLM inference costs, but general benchmarks are not tailored to evaluate it. The paper reports that downsampling can outperform advanced methods and proposes VTC-Bench to identify difficult, compression-relevant samples.

1 Introduction

  • Current benchmarks contain task-irrelevant samples, and downsampling distinguishes simple from difficult samples with respect to compression sensitivity.The study reports significantly better accuracy for samples answered correctly than incorrectly by downsampling across compression methods and benchmarks.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Simple image downsampling consistently outperforms many advanced compression methods across multiple widely used benchmarks.Figure 1 summarizes the comparisons on Qwen2-VL-7B.Source: §1 · p. 1, Figure 1 · p. 2
  • VTC-Bench is an additional evaluation framework that uses downsampling to select difficult samples from existing benchmarks.It aims to provide a fairer assessment tailored to visual token compression.Source: §1 · p. 2
  • The paper identifies benchmark noise, introduces downsampling as a difficulty discriminator, and proposes a framework for evaluating visual token compression.Source: §1 · p. 2

2 Related Work

Related work covers training-free visual-token compression strategies and MLLM benchmarks that primarily target general perception, reasoning, or specific domains. The paper positions VTC-Bench as a dedicated additional evaluation framework for compression.

2.1 Visual Token Compression for MLLMs

  • Training-free compression methods identify redundant visual tokens through staged processes or improved metrics, with approaches also using importance maps, graphs, spectrum pruning, duplication, and greedy optimization.Source: §2.1 · pp. 2–3
  • Across most MLLM benchmarks, sophisticated compression methods underperform simple image-resolution reduction, motivating investigation of the phenomenon.Source: §2.1 · p. 3

2.2 MLLM Benchmarks

  • General MLLM benchmarks cover perception and reasoning or specific domains; EffiVLM uses existing datasets, while VTC-Bench offers a dedicated, challenging evaluation framework.VTC-Bench is presented as an additional tool for evaluating visual token compression.Source: §2.2 · p. 3

3 Experiments & Findings

Across general-purpose benchmarks, downsampling often outperforms advanced visual token compression, while fine-grained tasks show the value of methods that preserve detail. The experiments attribute the mismatch to easy, task-irrelevant samples and test downsampling as a way to identify compression-relevant difficult samples.

3.2 Experiments Setup

  • Experiments compare four compression methods across five compression ratios and eight benchmarks, using Qwen2-VL because it best supports downsampling.The benchmarks include six general-purpose datasets and two resolution-sensitive OCR benchmarks.Source: §3.2 · p. 4

3.3 Results Analysis

  • At 93.75% compression, downsampling scores 66.4% on MMBench, a 24.3% relative improvement over DART.At 99.00% compression, downsampling maintains 51.6% on MMBench while FastV and PruMerge+ decrease to approximately 24%; on ChartQA, VisionZip and FastV outperform downsampling at 93.75% and 88.89%.Source: §3.3 · pp. 4–5

3.1 Motivation

  • The study investigates why downsampling can outperform sophisticated methods, proposing that existing benchmarks include overly simple samples irrelevant to compression evaluation.The authors note that benchmarks are generally used without considering whether their data suit the visual token compression task.Source: §3.1 · p. 4, §3.4 · p. 5

3.4 Hypothesis

  • Downsampling separates simple Group B samples from difficult Group A samples, whose compression-method accuracies are substantially lower.On GQA at 75% compression, all methods exceed 87.6% accuracy on simple samples, while the difficult-sample maximum is 59.3% for VisionZip. Downsampling’s 100% accuracy on Group B and 0% on Group A provides reference points for assessing methods that preserve details it loses.Source: §3.4 · p. 6

3.5 Summary

  • The experiments support the hypothesis that existing benchmarks mix easy samples with compression-relevant difficult ones, and that downsampling can filter between them.The second experiment both validates the benchmark-noise hypothesis and demonstrates downsampling’s use as a filter for distinguishing sample difficulty.Source: §3.5 · p. 7

4 Evaluation Framework

VTC-Bench filters existing benchmarks using downsampling to identify compression-relevant difficult samples. On that subset, method comparisons differ from broad-benchmark comparisons, and VTC-Bench widens reported accuracy gaps.

4.1 Framework Construction

  • VTC-Bench filters existing benchmarks by first retaining samples answered correctly by the original Qwen2-VL, then using downsampling to label them simple or difficult.It evaluates compression methods on difficult samples and dynamically adapts the resulting subset to the compression ratio; it does not create new data.Source: §4.1 · pp. 7–8

4.2 Evaluation Results & Discussions

  • At 75% compression on ChartQA, the VisionZip–FastV gap widens from 8.8% to 16.2%; at 96% on GQA, it grows from 0.3% to 9.0%.On difficult Group A samples, the trend reverses: advanced methods outperform downsampling, which often leads broad-benchmark comparisons.Source: §4.2 · p. 8

5 Conclusion

The paper identifies task mismatch in evaluating visual token compression with general MLLM benchmarks: these benchmarks contain task-irrelevant samples, and downsampling can distinguish samples by compression difficulty. It proposes VTC-Bench as an additional framework for denoising existing benchmarks.

5 Conclusion

  • The study concludes that current benchmarks are noisy for visual token compression and that downsampling can filter samples by difficulty.The paper proposes VTC-Bench to denoise existing benchmarks by distinguishing simple and difficult samples through downsampling.Source: §5 · p. 9
  • The paper systematically analyzes task mismatch in current MLLM benchmarks for evaluating visual token compression methods.It aims to encourage further discussion about how to evaluate efficient MLLMs.Source: §5 · p. 9

6 Limitations

The framework’s sample filtering depends on Qwen2-VL and on downsampling as its sole criterion; its sample definitions also lack a formal theoretical basis. The method comparison is not exhaustive, though the authors say the selected methods support their core claims and the framework remains extensible.

6 Limitations

  • The sample-filtering mechanism depends entirely on Qwen2-VL, which may limit generalizability because models can differ in which samples they consider difficult.The authors plan to extend the framework to more base models with dynamic-resolution support.Source: §6 · p. 9
  • Filtering uses only downsampling, while an ensemble of baseline methods might provide a more robust definition of difficult samples.The authors identify a more robust filtering mechanism as future work.Source: §6 · p. 9
  • The identification of simple and difficult samples is experiment-based and lacks a formal theoretical framework or mathematical proof.Source: §6 · p. 9
  • The comparison omits some visual token compression methods, although the authors say the selected methods support their core claims and the framework remains open and extensible.Source: §6 · p. 9

A Experiment Details

Experiments use one A800 GPU, with all results from a single run. The appendix also specifies the compression implementations and software environment.

A Experiment Details

  • All experiments use one A800 GPU, and all reported results come from a single run.Source: A Experiment Details · p. 13
  • Downsampling resizes images before MLLM input using bicubic interpolation, while compression ratios are controlled through each method’s specified parameter.DART uses Reduction_Ratio; VisionZip, PruMerge+, and FastV use Budget.Source: A Experiment Details · p. 13
  • The experiments use Python 3.10, torch 2.6.0, torchvision 0.21.0, and torchaudio 2.6.0; the authors say they will release per-sample outputs and benchmark accuracy results.Source: A Experiment Details · p. 13

B Benchmark Details

The appendix describes benchmarks covering visual reasoning, perception, object hallucination, OCR, and chart question answering. Together, the listed benchmarks span general MLLM evaluation and resolution-sensitive OCR tasks.

B Benchmark Details

  • GQA evaluates visual reasoning and compositional question answering, while MMBench covers MLLM capabilities through multiple-choice questions across fine-grained dimensions.GQA has 22M questions; MMBench includes 3,217 questions across 20 dimensions and supports Chinese and English.Source: B Benchmark Details · p. 13
  • MME assesses MLLM perceptual and cognitive abilities, POPE evaluates object hallucination, and MMStar tests vision-dependent reasoning and perception.MME spans 14 subtasks; POPE uses random, popular, and adversarial sampling; MMStar covers six core abilities and 18 subdimensions.Source: B Benchmark Details · p. 13
  • OCRBench evaluates multimodal-model OCR capabilities, while ChartQA tests visual and logical reasoning over real-world charts.OCRBench includes 1,000 manually verified samples from 29 datasets; ChartQA includes human-written and automatically generated questions.Source: B Benchmark Details · p. 13

C Complete VTC-Bench Results

This section reports grouped VTC-Bench results and places the framework in context with VisionThink and EffiVLM-Bench. It also describes visual differences between samples grouped as difficult and simple.

C Complete VTC-Bench Results

D.1 VisionThink

  • Unlike VisionThink’s model-adaptive approach, this study examines downsampling’s effectiveness from a benchmark perspective and uses it to motivate VTC-Bench.VisionThink starts with a low-resolution image and uses reinforcement learning to decide whether higher resolution is needed, particularly for complex or OCR-heavy tasks.Source: §D.1 · p. 13

D.2 EffiVLM-Bench

  • EffiVLM-Bench evaluates training-free LVLM acceleration methods, while VTC-Bench addresses whether existing benchmarks suit visual token compression.EffiVLM-Bench supports same-model comparisons without adapting methods to different models; VTC-Bench is proposed as an evaluation framework rather than an open-source toolkit.Source: §D.2 · pp. 13, 16

E Visualization between Groups

  • Difficult samples tend to require multiple detail perceptions and comparisons, whereas simple samples rely on larger patterns or limited comparisons.Examples include reading all 12 months’ precipitation or counting tiny fruits versus identifying four medium-sized food-web patterns or comparing two values.Source: E Visualization between Groups · p. 16, Figure 3 · p. 16

F Statistical Analysis of Image Properties between Groups

The analysis finds no statistically significant separation between difficult and simple groups in the measured low-level image properties. These properties therefore do not explain or readily predict the groups’ compression-related difficulty.

F Statistical Analysis of Image Properties between Groups

  • Across MMStar and POPE, measured low-level image statistics show no statistically significant differences between difficult and simple groups.MMStar entropy ranges from 4.90 to 5.15 for both groups and brightness centers around 163–168; POPE entropy is approximately 7.40 and brightness around 110.Source: F Statistical Analysis of Image… · pp. 16, 18, Table 7 · p. 17, Figure 5 · p. 18
  • Difficult samples are not systematically brighter, more colorful, higher in contrast, or larger than simple samples.The analysis covers entropy, brightness, contrast, colorfulness, and original image dimensions across downsampling ratios from 2 to 5.Source: F Statistical Analysis of Image… · pp. 16, 18
  • Because these elementary properties do not distinguish the groups, the authors use downsampling as an operational proxy for identifying compression-challenging samples.They note that finding reasonable high-level difficulty metrics remains future work.Source: F Statistical Analysis of Image… · p. 18

G Inference Time Comparison

The section compares inference time for downsampling and DART under a shared code repository and A800 setup. It reports that token reduction had less speed impact than expected for DART, while downsampling showed more typical results.

G Inference Time Comparison

  • In the shared DART repository on one A800, token reduction had far less impact on DART speed than expected, while downsampling showed more normal results.The authors suggest additional metric-calculation overhead in sophisticated compression methods as the likely explanation.Source: G Inference Time Comparison · p. 18, Table 8 · p. 18
  • The authors attribute the unexpected DART timing pattern most likely to overhead from additional metric calculations in sophisticated compression methods.They treat this comparison as further evidence of downsampling’s efficiency in the evaluation setting.Source: G Inference Time Comparison · p. 18
  • Inference times are rough estimates calculated from training-log start and end times across all eight benchmarks.The comparison is limited to downsampling and one advanced method, DART.Source: G Inference Time Comparison · p. 18
Loading 2510.07143v3…