PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Multimodal Benchmarks Really Useful? Item-Level Multimodal Benchmark Diagnosis via Structure-Response Co-Calibration

Shiqi Zhang, Weixin Zeng, Ziheng Zhang, Jiuyang Tang, Xiang Zhao

arXiv:2610.14542v1cs.AI

TL;DR

Some multimodal benchmark items can be solved from a single modality or answer-option regularities, and existing diagnosis approaches rely on item structure or model responses alone. SRCoD anchors modality-conditioned response modeling with item-level structural estimates to produce interpretable dependence profiles. It outperforms intrinsic-structure and extrinsic-response baselines across complementary validation protocols.

  • Problem

    Some benchmark items can be answered from a single modality or answer-option regularities, while existing diagnosis methods use item structure or model responses alone.

    Source: §1 · pp. 1–2

  • Method

    SRCoD uses sample-wise multimodal interaction signals as structural priors for modality-conditioned IRT fitted to VLM responses, producing interpretable item-level modality-dependence profiles.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    SRCoD outperforms intrinsic-structure and extrinsic-response baselines across benchmark-refinement and human-labeled item validation.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Its calibrated profiles distinguish redundant, visual-dominant, textual-dominant, and synergistic evidence dependence at the item level.

    Source: §5 · pp. 9–10

    Experiments focus on multiple-choice vision-language benchmarks; SRCoD has not been validated on open-ended generation, interactive multimodal evaluation, or modalities beyond vision and language.

    Source: A Limitation · p. 13

Abstract

from arXiv · show

Multimodal benchmarks are central to evaluating whether models can reason over information from multiple modalities. However, existing literature has revealed that some benchmarks may contain shortcut items that can be answered from a single modality or from answer-option regularities alone, thus failing to reflect models' true cross-modal inference ability. Existing methods for assessing benchmarks typically rely either on intrinsic item structure or extrinsic model response behavior, which are not adequate for a comprehensive diagnosis. To fill in this gap, we propose SRCoD, a structure-response co-calibrated diagnosis framework for item-level diagnosis of modality dependence in multimodal benchmarks. SRCoD estimates sample-wise multimodal information structure from benchmark content and target answers, and uses this signal as a soft structural anchor for modality-conditioned item response modeling. The learned model produces a calibrated item-level diagnostic profile that reflects how strongly an item depends on different forms of modality evidence, enabling fine-grained benchmark analysis beyond aggregate accuracy. SRCoD also provides interpretable attribution of the diagnosis results. Extensive experiments under two complementary protocols, i.e., benchmark-refinement oriented validation and human-labeled item validation, demonstrate that SRCoD outperforms state-of-the-art benchmark diagnosis methods.

1 Introduction

Multimodal benchmark items can permit shortcut answers rather than test cross-modal reasoning, while existing diagnosis methods capture either item structure or model behavior alone. SRCoD combines these signals and is evaluated with benchmark-refinement and human-labeled protocols, outperforming the baselines across both.

1 Introduction

  • Shortcut items answerable from one modality or answer-option regularities may fail to test the intended cross-modal inference ability.Item-level diagnosis asks what evidence each item actually requires and whether it tests the intended ability.Source: §1 · p. 1
  • Intrinsic-structure methods miss whether models use an item's cross-modal evidence, whereas response-only methods can reflect the selected model pool's strengths.For example, models' chart-understanding limitations may obscure an item's actual evidence requirements.Source: §1 · p. 2
  • SRCoD anchors modality-conditioned IRT responses with sample-wise structural priors to produce interpretable item-level evidence profiles.The profile attributes item behavior to evidence factors while preserving intrinsic structure and calibrating it with empirical evaluation behavior.Source: §1 · p. 2
  • Across benchmark-refinement and human-labeled item validation, SRCoD outperforms intrinsic-structure and extrinsic-response baselines.The human-labeled validation set spans ScienceQA, MMMU, and SEED-Bench and checks whether methods recover each item's dominant evidence factor.Source: §1 · p. 2

2 Related Work

Prior work diagnoses multimodal benchmarks from intrinsic item structure or from extrinsic model responses, with different validation strengths and gaps. SRCoD connects the two and adds controlled human validation of item-level evidence sufficiency.

2 Related Work

  • PID-based methods decompose multimodal information, and sample-wise estimators such as LSMI make item-level interaction structure estimable.Prior human judgments assess estimated-signal plausibility, not controlled item-level evidence sufficiency.Source: §2 · p. 3
  • IRT-based extrinsic methods model responses under modality conditions and can validate downstream benchmark refinement, but lack ground-truth labels for natural items.Their assessments depend on model behavior rather than direct item-level diagnostic labels.Source: §2 · p. 3
  • SRCoD uses structural interaction signals as soft anchors for modality-conditioned IRT and introduces controlled human validation of evidence sufficiency.The protocol supplies item-level diagnostic labels for directly validating modality dependence.Source: §2 · p. 3

3 Method

SRCoD combines intrinsic item evidence structure with VLM responses under controlled modality conditions to estimate interpretable, response-calibrated modality-dependence profiles. Structural estimates anchor the response model, while empirical behavior calibrates the resulting item diagnoses.

3 Method

  • SRCoD produces a four-factor item profile—redundant, visual-dominant, textual-dominant, or synergistic—by anchoring modality-conditioned IRT with intrinsic structure and model responses.Each profile coordinate attributes item behavior to a distinct evidence factor.Source: §3 · p. 3, §3.5 · p. 5

3.1 Modality-Conditioned Response Model

  • The response model represents each VLM's correctness for an item under modality conditions using base, visual, textual, and joint coordinates.The active coordinates depend on which modalities are present.Source: §3.1 · p. 3

3.2 Prior From Intrinsic Item Structure

  • Sample-wise interaction estimates are converted into structural dependence scores, with entropy-based confidence controlling how strongly each prior anchors diagnosis.Near-uniform priors have little influence; low-entropy priors contribute more strongly.Source: §3.2 · p. 4

3.3 Structure-Anchored Item Parameters

  • A learned bridge projects structural interactions into response-coordinate space to anchor item parameters, while response-learned residuals can revise that anchor.This allows observed model behavior to adjust structural estimates when the two disagree.Source: §3.3 · pp. 4–5

3.4 Response-Calibrated Modality-Dependence Profile

  • SRCoD derives a calibrated profile from predicted correctness under no-modality, visual, textual, and joint conditions, interpreting its coordinates as evidence-factor attribution.The calibrated profile is described as less sensitive to diagnostic-pool composition than raw accuracy differences.Source: §3.4 · p. 5

3.5 Structure-Response Co-Calibration Objective

  • Training combines response-likelihood fitting with confidence-weighted structural-prior alignment, so ambiguous estimates are resolved mainly by model responses.Confident structural estimates act as stronger anchors; the final profile preserves intrinsic structure while empirical behavior corrects it.Source: §3.5 · p. 5

4 Experiments

Experiments assess SRCoD through benchmark-refinement and human-labeled item validation, then examine component ablations, model-pool robustness, and an interpretable diagnostic example.

4.1 Experimental Setup

  • Evaluation spans ScienceQA, MMMU, and SEED-Bench, using contaminated-pool ranking preservation and human-labeled four-factor validation against six baselines.The human-labeled set has 1,000 items per benchmark; the compared methods include structure-only, response-based, and psychometric baselines.Source: §4.1 · p. 6

4.2 Main Results

  • SRCoD preserves model rankings across contaminated candidate pools, outperforming IRT and Random on ScienceQA and achieving the best results on MMMU from 10% subset size onward.On SEED-Bench, SRCoD and M3IRT both perform strongly, with SRCoD reaching near-ceiling rank correlation from 10% onward.Source: Figure 2 · p. 7, §4.2 · p. 7
  • On human-labeled validation, SRCoD outperforms baselines; structure-only LSMI has relatively low macro-F1, while extrinsic-response methods perform better but still lag SRCoD.The validation compares accuracy and macro-F1 against human evidence-sufficiency references in a four-factor diagnostic space.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 Ablation Study

  • The full SRCoD model achieves the best accuracy and macro-F1 across all three benchmarks; removing entropy weighting or profile-level structural alignment reduces performance.Entropy weighting down-weights ambiguous structural priors, while profile-level alignment carries structural information into the final response-calibrated profile.Source: Figure 3 · p. 8, §4.3 · p. 8

4.4 Robustness to Model-Pool Subsampling

  • With four or six diagnostic models, SRCoD stays closer to full-pool performance and varies less across trials than M3IRT on all three benchmarks.The subsampling evaluation reports performance retention relative to the full-pool setting; shaded regions show one standard deviation.Source: Figure 4 · p. 9, §4.4 · p. 9

4.5 Qualitative Diagnostic Example

  • In the Wyoming-capital example, both the question text and highlighted map region independently identify the state, so SRCoD diagnoses the item as Redundant.The correct answer is Cheyenne.Source: Figure 5 · p. 9, §4.5 · p. 9

5 Conclusion

SRCoD combines sample-wise multimodal interaction signals with modality-conditioned response behavior to produce fine-grained item-level modality-dependence profiles. Experiments show stronger diagnosis than the compared baselines, with stability under limited model pools and preserved model rankings for compact subsets.

5 Conclusion

  • SRCoD co-calibrates intrinsic item structure with extrinsic model responses to diagnose modality dependence at the item level.It uses sample-wise multimodal interaction signals as structural priors and calibrates them with modality-conditioned response behavior.Source: §5 · p. 9
  • The resulting profiles attribute items as redundant, visual-dominant, textual-dominant, or synergistic.The paper also introduces a controlled human validation protocol and a diagnostic validation set spanning ScienceQA, MMMU, and SEED-Bench.Source: §5 · p. 10
  • Experiments report that SRCoD outperforms intrinsic-structure and extrinsic-response baselines, is more stable with limited model pools, and preserves rankings when selecting compact subsets.Future work may extend the framework to open-ended tasks and multimodal settings beyond vision and language.Source: §5 · p. 10

A Limitation

This material outlines SRCoD's scope and uncertainties, background concepts, and the human-label protocol used to construct its diagnostic validation set. The protocol tests item answerability under controlled evidence conditions and assigns four modality-dependence labels.

A Limitation

  • Experiments cover multiple-choice vision-language benchmarks, while model-pool composition and capabilities, estimated structure, and annotator judgments constrain the profiles; these are estimates, not definitive human evidence-sufficiency judgments.The paper has not validated SRCoD on open-ended generation, interactive evaluation, or modalities beyond vision and language.Source: A Limitation · p. 13

B.1 Partial Information Decomposition

  • PID decomposes target-relevant information from two sources into redundant, source-specific, and synergistic components, distinguishing evidence available from either modality or only jointly.For vision-language tasks, the sources correspond to visual and textual inputs and the target to the answer.Source: §B.1 · p. 13

B.2 Item Response Theory and Multimodal Extensions

  • IRT models binary responses using subject ability and item properties; MIRT extends this to latent vectors, while multimodal IRT conditions responses on modality availability.SRCoD adopts modality-conditioned response modeling but anchors item parameters with intrinsic item structure.Source: §B.2 · pp. 13–14

C Human Annotation Protocol

  • The human-labeled diagnostic set provides an independent evaluation signal and is not used to train SRCoD, tune its hyperparameters, or fit baselines.Source: C Human Annotation Protocol · p. 14

C.1 Candidate Items

  • The validation set samples multiple-choice vision-language items from ScienceQA, MMMU, and SEED-Bench, retaining items that a majority of three annotators answer correctly under the full multimodal condition.The set contains 1,000 sampled items per benchmark.Source: §C.1 · p. 14

C.2 Controlled Evidence Conditions

  • Controlled conditions test single-modality sufficiency: annotators receive visual input with answer options, or the textual question and options without the image.Each retained item is assigned five annotators per condition; no annotator sees both controlled conditions for the same item.Source: §C.2 · pp. 14–15

C.3 Label Assignment

  • Items receive one of four labels: Redundant when both conditions suffice, Visual-dominant or Textual-dominant when only that modality suffices, and Synergistic when neither suffices but full multimodal input does.A condition counts as sufficient when a majority of annotators answer correctly.Source: §C.3 · p. 15

C.4 Annotation Reliability

The controlled human annotations show moderate agreement for single-modality judgments and substantial agreement for full multimodal filtering. The annotated label proportions also differ across the three benchmarks.

C.4 Annotation Reliability

  • Fleiss’ Kappa is 0.76 for full multimodal filtering, 0.56 for visual-conditioned judgments, and 0.63 for textual-conditioned judgments; the controlled labels show moderate agreement.The authors attribute the greater ambiguity to judging answerability from a single modality.Source: §C.4 · p. 15

C.5 Label Distribution of the Diagnostic Validation Set

  • Label proportions vary across ScienceQA, MMMU, and SEED-Bench, supporting item-level diagnosis rather than assuming a fixed dataset-level modality structure.Source: §C.5 · p. 15

D.1 Evaluation Metric

The evaluation maps item-level modality-dependence profiles to four diagnostic labels and reports accuracy and macro-F1. The experiments use a shared computing environment for modality-conditioned evaluations.

D.1 Evaluation Metric

  • Evaluation labels are Redundant, Visual-dominant, Textual-dominant, and Synergistic; profile outputs are mapped to labels by selecting the largest coordinate or mapped score.Source: §D.1 · pp. 15–16
  • Macro-F1 complements accuracy because the diagnostic set is not perfectly class-balanced and weights the four categories equally.The authors use it to assess consistency across modality-dependence categories.Source: §D.1 · p. 16

D.2 Computational Resources

  • Modality-conditioned evaluations use the same computing environment, and Table 2 summarizes the computational resources used.Source: §D.2 · p. 16, Table 2 · p. 17

D.3 Datasets

The experiments use three multimodal benchmarks and a heterogeneous model pool to collect modality-conditioned responses. Standardized prompts and bounded parameterization support SRCoD’s response and structural modeling.

D.3 Datasets

  • ScienceQA, MMMU, and SEED-Bench cover science-oriented question answering, college-level multimodal reasoning, and general multimodal understanding, respectively.The experiments use vision-language question-answering items from these benchmarks.Source: §D.3 · p. 17

D.4 Models

  • A heterogeneous pool of closed-source and open-weight multimodal models supplies responses under modality conditions; no models are fine-tuned.The pool spans model families, parameter scales, and deployment settings.Source: §D.4 · p. 17, Table 3 · p. 17

D.5 Prompts

  • Standardized prompts keep answer options and output format fixed across conditions, and responses are reduced to binary correctness labels.Models select one option letter without explanations or chain-of-thought; invalid outputs count as incorrect.Source: §D.5 · pp. 18–19, Table 4 · p. 18

D.6 Implementation details

  • Structural coordinates are robustly normalized and aligned with response coordinates, while bounded latent abilities and discriminations stabilize the response surface.The implementation uses four base, visual, textual, and joint coordinates, with Adam optimization for 300 epochs.Source: §D.6 · p. 19

E.1 Redundant Items

The examples distinguish items answerable from either unimodal view, from text alone, from images alone, or only by combining both. The labels concern evidence sufficiency, not whether the modalities encode identical information.

E.1 Redundant Items

  • Redundant items allow both controlled views to recover the answer, sometimes through different cues, so the full item need not require cross-modal integration.In the rose-bud example, visual pest appearance and textual constraints with domain priors independently support aphids.Source: §E.1 · pp. 19–20

E.2 Textual-Dominant Items

  • Textual-dominant items are solvable from question wording and options while the image-option view is insufficient, so they provide limited evidence that visual information is required.The examples include cases where text or domain priors supply the decisive cue.Source: §E.2 · pp. 20–21

E.3 Visual-Dominant Items

  • Visual-dominant items are answerable from the image and options but not from the text and options, with decisive evidence such as an object attribute or chart trend.The ScienceQA kite example’s text refers only to “this object,” leaving the target unidentified without the image.Source: §E.3 · pp. 21–22

E.4 Synergistic Items

  • Synergistic items remain underdetermined in either unimodal view but become answerable when text identifies the target and the image supplies the evidence that grounds it.In the SEED-Bench example, the text identifies a chair by its relation to a blue couch, and the image shows that chair’s color.Source: §E.4 · pp. 23–24

F.1 Statistical Significance Analysis

On the human-labeled validation set, SRCoD’s improvements over M3IRT are statistically significant across all three benchmarks. Paired tests and bootstrap confidence intervals assess the item-level differences.

F.1 Statistical Significance Analysis

  • Paired item-level comparisons use bootstrap 95% confidence intervals and approximate-randomization tests, with McNemar’s test additionally reported for accuracy.McNemar’s test compares items classified correctly only by SRCoD with those classified correctly only by M3IRT.Source: §F.1 · p. 24
  • On ScienceQA, SRCoD improves accuracy by +0.037 and macro-F1 by +0.049 over M3IRT; accuracy gains are +0.061 on MMMU and +0.067 on SEED-Bench.For ScienceQA, the reported 95% confidence intervals are [0.015, 0.060] for accuracy and [0.026, 0.075] for macro-F1.Source: §F.1 · p. 24
  • Across validation items, SRCoD improves accuracy by +0.055 and macro-F1 by +0.058 over M3IRT, with positive confidence intervals and paired significance tests yielding p < 0.001.The aggregate results combine items across the three benchmarks.Source: §F.1 · p. 24

F.2 Comparison with Simple Fusion

SRCoD outperforms tested fixed-weight post-hoc profile fusion baselines on the human-labeled validation set. The authors attribute the difference to joint calibration of structural information and model responses.

F.2 Comparison with Simple Fusion

  • Against the strongest fusion baseline, SRCoD reaches average accuracy 0.61 and macro-F1 0.60, versus 0.53 and 0.50, respectively.The strongest fusion setting uses α = 0.25; SRCoD consistently outperforms all three tested settings across benchmarks.Source: Table 6 · p. 25, §F.2 · p. 25
  • With responses from 3B–12B multimodal models, SRCoD consistently outperforms M3IRT and shows smaller performance degradation on the subset.The comparison reports accuracy and macro-F1.Source: Figure 11 · p. 25
  • Unlike fixed-weight post-hoc fusion of independently estimated profiles, SRCoD incorporates structural information into item-parameter estimation and profile-level calibration.The authors describe this as allowing structural evidence to interact with observed model responses during optimization.Source: §F.2 · pp. 24–25

F.3 Robustness under a Small-Scale Model Pool

SRCoD remains stronger than M3IRT when diagnosis uses responses from a small-scale model pool. Its advantage persists across benchmark subsets, where M3IRT degrades more on some datasets.

F.3 Robustness under a Small-Scale Model Pool

  • The small-scale pool contains models with comparable sizes ranging from 3B to 12B parameters, a setting described as more challenging because their modality-conditioned responses can be noisier and more biased.Source: §F.3 · p. 25
  • SRCoD outperforms M3IRT on the full validation set across ScienceQA, MMMU, and SEED-Bench in both accuracy and macro-F1, and remains stronger on subsets.On ScienceQA and SEED-Bench, SRCoD maintains nearly the same performance while M3IRT degrades; on MMMU, both methods drop, but SRCoD remains ahead.Source: §F.3 · p. 26
  • SRCoD’s structural anchor makes its diagnosis less dependent on response-side evidence reliability alone, supporting more robust item-level diagnosis with limited, noisy model pools.Source: §F.3 · p. 26

F.4 Behavioral Validation on Held-Out Model Families

This section examines whether SRCoD’s predicted modality-dependence categories show expected behavioral patterns on held-out model families and discusses the method’s potential benefits, risks, and responsible-use considerations. It also describes the validation procedures, including paid human annotation without formal IRB review and MLLMs as diagnostic respondents, distinguishing this use from language polishing that did not affect the research claims.

F.4 Behavioral Validation on Held-Out Model Families

  • The leave-one-family-out evaluation excluded Qwen- or Ministral-family responses from fitting and item assignment, then assessed predicted categories on ScienceQA and SEED-Bench.Held-out-family accuracy was aggregated across no-modality, image-only, text-only, and image-text conditions; positive margins indicate the expected behavioral direction for a category.Source: §F.4 · p. 26
  • All 16 behavioral margins were positive across two held-out model families and two benchmarks, with each diagnostic group exhibiting its expected modality pattern.Synergistic items had higher average accuracy with joint image-text input than with either unimodal condition; the Visual-dominant, Textual-dominant, and Redundant groups also showed their expected patterns.Source: §F.4 · p. 26, Table 7 · p. 27

G Broader Impacts

  • The authors frame item-level diagnosis as a way to identify shortcut-prone or weak benchmark items, while warning that errors may lead developers to discard useful items or overemphasize certain dependencies.They recommend treating diagnosis as auxiliary evidence, reporting uncertainty, and combining automated results with human review before high-stakes decisions; benchmark filtering could also indirectly enable misuse in misinformation, surveillance, or biased decision support.Source: G Broader Impacts · p. 27

NeurIPS Paper Checklist

  • The paper reports reproducibility details and an anonymized code-and-dataset repository, discusses limitations in Appendix A, and reports statistical-significance analyses in Appendix F.1.Source: NeurIPS Paper Checklist · pp. 28–30

10. Broader impacts

  • The paper introduces a human-labeled diagnostic validation set for release through an anonymized repository and reports that crowdsourced annotators received an annotation protocol and shared no personal or sensitive information.Source: §10 · pp. 32–33
  • Paid human annotators constructed the diagnostic validation set without formal IRB review; the task was described as minimal-risk and collected no personal, sensitive, or health-related information.Participants answered multiple-choice benchmark items under controlled modality conditions.Source: §10 · p. 33
  • MLLMs supplied modality-conditioned responses for fitting and evaluating diagnosis methods, while their use for language polishing did not affect methodology or scientific claims.The model pool, prompts, modality conditions, and response normalization procedures are documented in Appendix D.Source: §10 · p. 34

Related papers

Loading 2610.14542v1…