PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts

Ellis Brown, Jihan Yang, Shusheng Yang, Rob Fergus, Saining Xie

arXiv:2511.04655v1cs.CV

TL;DR

Multimodal benchmarks may reward linguistic priors and statistical patterns instead of visual understanding, making reliable evaluation difficult. The paper introduces test-set stress-testing and iterative bias pruning, finding pervasive shortcuts across four benchmarks and creating VSI-Bench-Debiased with a wider vision-blind gap.

  • Problem

    Multimodal models can achieve high benchmark scores without strong visual understanding by exploiting linguistic priors, biases, and superficial patterns.

  • Method

    The paper uses Test-set Stress-Test diagnostics on non-visual test-set inputs to assign bias scores, then applies Iterative Bias Pruning to filter highly biased samples.

  • Results

    Across VSI-Bench, CV-Bench, MMMU, and VideoMME, the framework uncovers pervasive non-visual shortcuts and produces VSI-Bench-Debiased with a wider vision-blind performance gap.

  • Takeaways & Limitations

    Robust multimodal benchmark design should adversarially audit evaluation test sets so scores better reflect genuine multimodal understanding.

  • Takeaways & Limitations

    TsT-LLM requires approximately 20 minutes per benchmark on 4×A100 GPUs and offers limited interpretability of the cues driving exploitability.

Abstract

from arXiv · show

Robust benchmarks are crucial for evaluating Multimodal Large Language Models (MLLMs). Yet we find that models can ace many multimodal benchmarks without strong visual understanding, instead exploiting biases, linguistic priors, and superficial patterns. This is especially problematic for vision-centric benchmarks that are meant to require visual inputs. We adopt a diagnostic principle for benchmark design: if a benchmark can be gamed, it will be. Designers should therefore try to ``game'' their own benchmarks first, using diagnostic and debiasing procedures to systematically identify and mitigate non-visual biases. Effective diagnosis requires directly ``training on the test set'' -- probing the released test set for its intrinsic, exploitable patterns. We operationalize this standard with two components. First, we diagnose benchmark susceptibility using a ``Test-set Stress-Test'' (TsT) methodology. Our primary diagnostic tool involves fine-tuning a powerful Large Language Model via $k$-fold cross-validation on exclusively the non-visual, textual inputs of the test set to reveal shortcut performance and assign each sample a bias score $s(x)$. We complement this with a lightweight Random Forest-based diagnostic operating on hand-crafted features for fast, interpretable auditing. Second, we debias benchmarks by filtering high-bias samples using an ``Iterative Bias Pruning'' (IBP) procedure. Applying this framework to four benchmarks -- VSI-Bench, CV-Bench, MMMU, and VideoMME -- we uncover pervasive non-visual biases. As a case study, we apply our full framework to create VSI-Bench-Debiased, demonstrating reduced non-visual solvability and a wider vision-blind performance gap than the original.

1. Introduction

Multimodal benchmarks can be solved through linguistic priors, biases, and superficial patterns rather than visual understanding. The paper proposes adversarially auditing test sets with diagnostic models and pruning biased samples to improve evaluation integrity.

  • Models can achieve high multimodal benchmark scores without using images, exploiting linguistic priors, biases, and superficial patterns instead.
  • The paper argues that benchmark designers should directly train on released test sets to expose test-specific shortcuts rather than relying only on blind evaluation.
  • Test-set Stress-Test uses k-fold cross-validation on non-visual test features to measure exploitability and assign sample-level bias scores.
  • TsT combines an LLM-based diagnostic for complex shortcuts with a lightweight Random Forest diagnostic for efficient, interpretable auditing.
  • Iterative Bias Pruning filters highly biased samples to mitigate non-visual shortcuts and refine benchmark quality.
  • Across VSI-Bench, CV-Bench, MMMU, and VideoMME, the framework finds pervasive shortcuts and produces VSI-Bench-Debiased with lower non-visual solvability and a wider vision-blind gap.

2. The Challenge: Non-Visual Shortcuts Undermine Multimodal Evaluation

Non-visual shortcuts make multimodal evaluation vulnerable when linguistic knowledge or benchmark statistics render visual inputs unnecessary. The paper identifies knowledge-based and statistical mechanisms through which models can obtain inflated scores without visual reasoning.

  • Non-visual shortcuts allow models to answer multimodal questions without using visual inputs, inflating scores and misrepresenting visual understanding.
  • 2.1. What Constitutes a Non-Visual Shortcut?: A pattern is exploitable when it makes the visual input redundant for a task intended to measure visual understanding, regardless of whether it is natural or artificial.
  • 2.2. Non-visual Shortcuts from Knowledge: Knowledge-based vulnerability varies across benchmarks: MMMU gains more from LLM scaling than vision, whereas VSI-Bench gains substantially from vision and little from blind scaling.
  • 2.1. What Constitutes a Non-Visual Shortcut?: World knowledge can bypass visual reasoning; for example, typical refrigerator heights may support size estimates without measuring the pictured refrigerator.
  • 2.3. Non-visual Shortcuts from Statistical Correlations: Statistical shortcuts include skewed counting answers, imbalanced spatial labels, category-position correlations, and predictable size distributions.
  • 2.3. Non-visual Shortcuts from Statistical Correlations: After fine-tuning on VSI-Train-10k, blind accuracy rose from 25.9% to 44.7% (+18.8 points), nearly matching the vision-enabled gain of +20.4 points.

3. Diagnosing Non-visual Shortcuts via Test-Set Stress-Testing

Test-set Stress-Testing (TsT) measures whether benchmark questions can be answered from non-visual test-set information, exposing shortcuts that may inflate multimodal scores. Across four benchmarks, TsT diagnostics reveal substantial and widespread exploitability while providing sample-level bias signals for auditing.

  • 3.1. Test-Set Stress-Test: The framework distinguishes evaluation failures caused by exploitable test-set artifacts from training failures caused by biased training data.TsT targets artifacts and regularities intrinsic to the particular released test set rather than relying on separate training data.
  • 3.1. Test-Set Stress-Test: TsT applies k-fold cross-validation directly to non-visual test-set features, producing overall non-visual solvability and per-sample bias scores s(x).Each held-out prediction comes from a model trained without that sample, while the aggregated accuracy estimates benchmark-level exploitability.
  • 3.2. LLM-Based TsT Diagnostic: TsT-LLM fine-tunes a language model with LoRA on question and answer-choice text only, requiring no manual feature engineering and generalizing across benchmark structures.The diagnostic ignores visual inputs and repeats the procedure across k folds.
  • 3.2. LLM-Based TsT Diagnostic: TsT-LLM requires GPU resources and longer training, and its strong shortcut detection offers limited interpretability for identifying specific causal cues.The reported setup takes approximately 20 minutes per benchmark on 4×A100 GPUs for k=5 folds.
  • 3.4. Empirical Validation: TsT Reveals Widespread Shortcut Susceptibility: +33.3 and +31.4 points on CV-Bench and VSI-Bench, respectively, while MMMU and VideoMME gain +8.6 and +6.4 points after blind test-set fine-tuning.These ΔTsT improvements quantify learnable non-visual shortcuts across both template-based and non-templated benchmarks.
  • 3.3. RF-Based TsT Diagnostic: TsT-RF trades TsT-LLM’s convenience for interpretability and efficiency, using hand-crafted features to identify cues that drive exploitability.On CV-Bench, TsT-RF reaches 75.5% versus TsT-LLM’s 73.4%; on VSI-Bench, it reaches 43.5% versus 56.4%.

4. Mitigating Non-Visual Shortcuts Guided by TsT Insights

The paper uses TsT-derived sample bias scores to iteratively prune non-visual shortcuts, producing a VSI-Bench version that better separates visual reasoning from statistical predictability.

  • 4.1. The Iterative Bias Pruning (IBP) Procedure: The framework computes sample-level scores s(x) with TsT diagnostics and uses them to guide data-driven benchmark refinement.Both TsT-LLM and TsT-RF can produce scores, while the VSI-Bench case study uses interpretable TsT-RF scores for targeted mitigation.
  • 4.1. The Iterative Bias Pruning (IBP) Procedure: IBP repeatedly removes small batches of the most biased samples, then recomputes sample bias scores because pruning can change the remaining dataset’s statistical landscape.The procedure balances bias reduction against dataset size through a removal budget and batch size, with smaller batches enabling more frequent re-diagnosis.
  • 4.1. The Iterative Bias Pruning (IBP) Procedure: The paper focuses on pruning because it is simple, reproducible, and has an unambiguous impact, while leaving reparative alternatives for future work.Question rewriting and answer rebalancing could introduce new, unquantified biases and therefore require careful validation.
  • 4.2. Case Study: Creating VSI-Bench-Debiased with IBP: The VSI-Bench case study reports less predictable answer distributions and reduced non-visual solvability after applying IBP.The resulting benchmark is intended to better compel genuine visual reasoning from evaluated multimodal models.
  • 4.2. Case Study: Creating VSI-Bench-Debiased with IBP: VSI-Bench-Debiased widens the fine-tuned vision-blind gap to +16.6 versus +12.4 on the original, while vision improves more than blind performance (+17.4 versus +11.7).The original benchmark shows nearly equal fine-tuning gains for vision and blind configurations, whereas the debiased version shows a larger vision gain.

5. Related Work

Prior work addresses multimodal shortcuts through model debiasing, adversarial benchmark construction, and diagnostic evaluation. TsT complements these approaches by auditing intrinsic exploitable patterns in the test set itself.

  • 5. Related Work: Earlier VQA research exposed language-prior exploitation, motivating distribution-shift benchmarks and training-time interventions such as adversarial regularization and counterfactual augmentation.These approaches primarily address biases learned during model training rather than intrinsic patterns in a released test set.
  • 5. Related Work: TsT targets test-set artifacts that can make visual input redundant, shifting attention from improving models to improving evaluation instruments.This differs from model-debiasing methods that modify training procedures to reduce learned biases.
  • 5. Related Work: The blind test provides a coarse dataset-level signal, whereas newer tools audit modality importance, local-versus-global visual reasoning, or unimodal bias dimensions.These diagnostics broaden benchmark auditing but address dimensions distinct from TsT’s test-set-specific exploitability analysis.
  • 5. Related Work: Adversarially designed benchmarks such as Winoground, Vinoground, Hateful Memes, and MMBench resist selected shortcuts through paired examples, confounders, skill coverage, or curation.Their design strategies complement post hoc auditing of intrinsic test-set patterns.

6. Conclusion

The paper argues that multimodal benchmarks can create an illusion of progress when models exploit non-visual shortcuts. It proposes adversarial test-set auditing and reports stronger visual-reasoning separation after bias-guided refinement.

  • 6. Conclusion: Blind models gain up to +33 points by learning test-set patterns alone, revealing pervasive non-visual shortcuts across four major benchmarks.The reported gains motivate treating benchmark auditing as part of robust multimodal evaluation.
  • 6. Conclusion: VSI-Bench-Debiased shows a 34% wider vision-blind gap after fine-tuning than the original benchmark.This result is presented as evidence that bias-score-guided refinement produces a more robust benchmark.
  • 6. Conclusion: The paper recommends rigorous test-set stress-testing so benchmarks measure genuine multimodal understanding rather than statistical pattern matching.The recommendation is framed as adversarial evaluation of the evaluation instruments themselves.

A. VSI-Train-10k Generation

The VSI-Train-10k set is generated by following the VSI-Bench curation pipeline with extracted scene attributes, benchmark templates, and rule-based question types.

  • A. VSI-Train-10k Generation: The authors extract object numbers, bounding boxes, and room sizes from ScanNet, ScanNet++, and arkitScenes training splits.
  • A. VSI-Train-10k Generation: They use VSI-Bench’s exact question templates to generate question-answer pairs for seven rule-based question types, excluding route planning.
  • A. VSI-Train-10k Generation: The resulting questions are randomly sampled to form a 10K in-distribution training set for demonstrating MLLM shortcut learning.

B.1. TsT-LLM Implementation Details

The TsT-LLM diagnostic fine-tunes Qwen2.5-7B-Instruct using five-fold cross-validation on textual inputs only, excluding visual information.

  • B.1. TsT-LLM Implementation Details: Qwen2.5-7B-Instruct is fine-tuned with LoRA separately in each of five cross-validation folds.The configuration uses rank r=128 and α=256.
  • B.1. TsT-LLM Implementation Details: The training uses a 5 × 10^-5 learning rate, cosine scheduling, batch size 32, and three epochs per fold.
  • B.1. TsT-LLM Implementation Details: Inputs contain only question text and, for multiple-choice items, answer choices; visual inputs are excluded.

B.2. TsT-RF Feature Extraction

The TsT-RF diagnostic extracts benchmark-specific non-visual features from question structure, answer spaces, metadata, and task characteristics.

  • B.2. TsT-RF Feature Extraction: Features include TF-IDF question representations, question length, and keyword presence.
  • B.2. TsT-RF Feature Extraction: Answer-space features encode answer format and the number of options for multiple-choice questions.
  • B.2. TsT-RF Feature Extraction: Metadata features capture question type and mentioned object categories.
  • B.2. TsT-RF Feature Extraction: Task-specific features are tailored to benchmark structures, including average object size for size-estimation tasks.

B.3. TsT-RF Feature Engineering

TsT-RF engineering captures textual, statistical, metadata, and task-specific patterns from template-based benchmark questions without visual input or validation leakage.

  • B.3. TsT-RF Feature Engineering: The engineered features target patterns that may correlate with ground-truth answers without requiring visual input.
  • B.3. TsT-RF Feature Engineering: VSI-Bench’s predictable templates enable regular-expression extraction of object categories, distances, directions, and related task elements.
  • B.3. TsT-RF Feature Engineering: Tables 6 and 7 organize VSI-Bench and CV-Bench TsT-RF features by task type using questions and answer choices only.
  • B.3. TsT-RF Feature Engineering: CV-Bench feature statistics are computed exclusively from training folds during five-fold cross-validation to avoid validation leakage.
  • B.3. TsT-RF Feature Engineering: IBP repeatedly computes sample bias scores, selects samples for removal under total-budget and batch-size constraints, and stops when the budget is exhausted or maximum bias falls below a threshold.

C.2. VSI-Bench-Debiased Dataset Statistics

Applying IBP to VSI-Bench removed highly shortcut-susceptible samples, producing a smaller dataset with category-dependent pruning rates tied to measured exploitability.

  • C.2. VSI-Bench-Debiased Dataset Statistics: 937 of 3,056 VSI-Bench questions were removed, leaving 2,119 questions after a 30.7% reduction.
  • C.2. VSI-Bench-Debiased Dataset Statistics: IBP removed samples identified as highly susceptible to non-visual shortcuts using TsT-RF sample-level bias scores.
  • C.2. VSI-Bench-Debiased Dataset Statistics: Removal fractions varied by category, with higher fractions removed from more biased categories and more samples preserved in less exploitable tasks.

D. TsT-RF Interpretability Analysis

TsT-RF exposes interpretable non-visual shortcuts in VSI-Bench’s object-size estimation task, showing that object-category statistics can enable prediction without visual estimation.

  • The feature set combined object category with count, frequency, mean log-size, and standard deviation features computed only from training folds.This design prevented validation-sample leakage during k-fold cross-validation.
  • 61.4% ± 2.4% accuracy versus a 34.0% majority baseline indicates strong non-visual exploitability under 5-fold TsT-RF cross-validation.
  • Object-category average size dominated feature importance at 0.968, while all other features together contributed less than 0.04.The diagnostic therefore largely memorized typical object sizes rather than using question structure, frequency, or size variance.
  • High-bias samples clustered around low-variance categories such as dishwashers, beds, and washers, unlike high-variance categories receiving lower bias scores.Reported coefficients of variation were 0.037 for dishwashers, 0.080 for beds, 0.058 for washers, versus 0.778 for ceiling lights and 0.755 for radiators.
  • Designers can mitigate this shortcut by removing low-variance object questions or sampling more diverse sizes within each category.This illustrates how TsT-RF feature analysis yields concrete benchmark revisions and complements TsT-LLM’s detection of more complex shortcuts.
Loading 2511.04655v1…