PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing MCQA Benchmarks through Probability Landscapes

Minsoo Song, Chanjun Park

arXiv:2608.30372v1cs.CL

TL;DR

MCQA benchmarks are nearing saturation, while validating question quality and filtering flawed items remains labor-intensive. The paper proposes a two-level framework that analyzes probability landscapes and uses noise injection to prioritize items for review. It provides complementary benchmark-level characterization and item-level candidate prioritization, while external validation and model coverage delimit the conclusions.

  • Problem

    As MCQA performance approaches saturation, accuracy alone is insufficient for assessing benchmark quality, motivating systematic audits of existing datasets.

  • Method

    The framework uses Ptop1 and Hnorm with MPD for benchmark-level landscape analysis, and noise injection followed by targeted human review for item-level diagnostics.

  • Results

    The framework provides complementary aggregate landscape summaries and item-level flagging signals for comparing benchmark design and prioritizing questions for targeted human review.

  • Takeaways & Limitations

    The framework offers a practical foundation for benchmark auditing through macro-level landscape characterization and first-pass item-level review prioritization.

  • Takeaways & Limitations

    The external validation covers 373 exact-text matches with only 9 expert-identified flaws, yielding low and necessarily unstable recall estimates.

Abstract

from arXiv · show

As Large Language Models rapidly advance, performance on standard multiple-choice question answering (MCQA) benchmarks is reaching saturation. While the community has responded by developing increasingly difficult datasets, validating question quality and filtering flawed items remains a labor-intensive process. To provide a scalable diagnostic approach, we propose a two-component probabilistic framework for auditing MCQA benchmarks using model output distributions. First, for benchmark-level analysis, we characterize the probability landscape using the top prediction probability ($P_{top1}$) and normalized residual entropy ($H_{norm}$), summarized globally by Mean Pairwise Distance (MPD). Second, for item-level diagnostics, we introduce noise injection to reduce meaningful distractor competition, enabling us to flag candidate items for targeted human review and categorize residual failure patterns. Across four MCQA benchmarks, our landscape analysis reveals benchmark-level differences in model confidence and residual option competition. Concurrently, our noise-injection method flags potentially actionable item-level issues, showing alignment with expert error annotations from MMLU-Redux. These results suggest that our probability-based framework provides a lightweight audit lens for comparing macro-level benchmark structure and prioritizing individual items for targeted human review.

1 Introduction

As MCQA benchmarks approach saturation, accuracy alone provides limited insight into question structure and quality. The proposed probability-landscape framework combines benchmark-level distribution analysis with item-level candidate prioritization for targeted human review.

  • Motivation: MCQA benchmark saturation and labor-intensive dataset construction motivate cost-efficient auditing methods beyond aggregate accuracy.Existing analyses reveal limited headroom but provide limited information about the internal structure of benchmark questions.
  • Item-level diagnostics: Noise injection replaces distractors with unrelated city names to reduce meaningful distractor competition and narrow auditing to candidate items.The procedure is designed as a first stage before targeted human review and taxonomy-based categorization.
  • Interpretation: Probability-based metrics and perturbation failures are not treated as definitive evidence that benchmark items are flawed.Candidate flagging prioritizes review rather than automatically determining benchmark flaws.
  • Benchmark-level analysis: The framework represents each item with Ptop1 and Hnorm and summarizes benchmark-level probability-landscape dispersion using MPD.Hnorm captures residual option competition, while MPD summarizes landscape heterogeneity.
  • Benchmark-level analysis: Aggregate Hnorm and MPD provide complementary descriptive summaries of residual option competition and probability-landscape structure across evaluated models.The framework reports broadly consistent benchmark-level patterns across the well-calibrated models evaluated.
  • Item-level diagnostics: Cross-model noise-injection failures prioritize candidate items for targeted human review and separate perturbation-induced failures from potentially actionable benchmark issues.The framework includes a taxonomy and external evaluation against MMLU-Redux expert annotations.

2 Related Work

Related work studies benchmark saturation, dataset artifacts, distractor quality, and probabilistic signals of model behavior. This paper differs by using output distributions to diagnose benchmark design and prioritize items for expert review.

  • Benchmark saturation: Leaderboard-based saturation analyses identify limited benchmark headroom but do not directly diagnose structural causes within datasets.These approaches primarily rely on performance comparisons among leading models.
  • Dataset validity: Prior studies show that annotation artifacts and partial-input effects can produce non-trivial MCQA performance without genuine reasoning.Related work includes lexical cues, hypothesis-only baselines, and prediction from incomplete inputs.
  • Expert auditing: MMLU-Redux uses expert reannotation to identify erroneous and ambiguous items, whereas this work automatically prioritizes a smaller candidate set for targeted human review.The expert annotations serve as an external validation reference for the candidate signals.
  • Distractor quality: Traditional distractor analysis evaluates incorrect choices through human response frequencies, while proxy-model approaches require additional response collection or task-specific evaluation.Rarely selected distractors are typically considered ineffective in educational measurement.
  • Probabilistic signals: Prior probabilistic research analyzes confidence, calibration, entropy, and uncertainty primarily to understand model behavior or reliability.This paper instead applies probability distributions to auditing benchmark design and structural properties.

3 Methodology

The framework audits MCQA benchmarks through probability landscapes and controlled perturbations, using model confidence and residual distractor competition for benchmark-level analysis. Noise injection reduces semantic distractor competition to flag items that remain incorrect for targeted review.

  • Framework overview: The framework has two components: probability landscape analysis characterizes global benchmark structure, while perturbation-based diagnostics probe sensitivity to controlled dataset modifications.Noise injection is the primary item-flagging probe; masking, no-answer injection, and score bias assess benchmark-level robustness.
  • Probabilistic representation: Each question is represented by Ptop1 and Hnorm, capturing model confidence and competition among non-selected options.Ptop1 is the probability assigned to the top prediction, while Hnorm is computed over the renormalized residual distribution.
  • Probabilistic representation: Hnorm ranges from 0 to 1, with larger values indicating stronger distractor competition and values near 0 indicating concentrated residual competition.A low residual entropy means one non-selected option dominates the remaining probability mass.
  • Probability landscape cohesion: MPD summarizes global probability-landscape dispersion using Euclidean distances among question representations.Lower MPD indicates a narrower landscape, whereas higher MPD indicates greater heterogeneity in confidence and residual competition; MPD is descriptive rather than a direct quality measure.
  • Controlled perturbations: Token masking removes question semantics while preserving the prompt structure, isolating dependence on question information.High accuracy after masking suggests answers may be inferred from option patterns rather than question content.
  • Controlled perturbations: Noise injection replaces distractors with semantically unrelated city names to eliminate meaningful distractor competition.Items that remain incorrect after injection are flagged for inspection, although failures may also reflect limitations of the perturbation itself.

4 Experimental Setup

The experiments evaluate probability distributions from three small-scale instruction-tuned models on four MCQA benchmarks using standardized four-option prompts. The setup samples 1,000 items per dataset and extracts answer-choice probabilities from final-position logits.

  • Models: The study evaluates Qwen-2.5 7B, Gemma-2 9B, and Llama-3 8B, with larger-model experiments reported as supplementary scalability analyses.The supplementary models are Qwen-2.5 32B, Qwen-2.5 72B, and Llama-3.1 70B.
  • Benchmarks: The four benchmarks are MMLU, ARC, HellaSwag, and CommonsenseQA, with 1,000 randomly sampled instances per dataset.The samples provide a computationally manageable evaluation set described as representative.
  • Benchmarks: CSQA is standardized from five options to four using a deterministic positional rule that avoids removing the correct answer.Option E is removed by default; if E is correct, option D is removed and original E is relabeled D.
  • Prompting and probability extraction: Each item uses a standardized four-option A–D template, and a system instruction forces output of only the answer letter.This enables direct extraction of logits for each option token.
  • Prompting and probability extraction: Figure 2 shows ARC and CSQA concentrated near Ptop1 ≈1.0, whereas MMLU and HellaSwag span a wider region.The distributions are based on 1,000 questions per benchmark with coordinates averaged across three models.

5 Benchmark-Level Probability Landscape Analysis

The benchmark-level analysis represents each question through Ptop1 and Hnorm, then uses MPD to compare probability-landscape concentration and heterogeneity across benchmarks and perturbations.

  • Landscape representation: Ptop1 and Hnorm jointly represent each question’s probability landscape, capturing predicted-answer confidence and residual option competition.Each question is represented as a point in the joint Ptop1–Hnorm space.
  • Benchmark comparisons: ARC and CSQA cluster in the high-confidence region (Ptop1 ≈1.0), indicating near-deterministic probabilities for predicted answers.
  • Landscape representation: Lower MPD indicates a more concentrated landscape, whereas higher MPD indicates greater structural diversity across questions.
  • Benchmark comparisons: ARC has the most concentrated landscape (MPD = 0.404), while MMLU is more dispersed (MPD = 0.479), ordered ARC < CSQA < HellaSwag < MMLU.
  • Interpretation: Aggregate Hnorm and MPD provide complementary benchmark-level summaries, while near-zero cross-model item-level Hnorm correlations motivate candidate prioritization through noise-injection failures.
  • Perturbation effects: MPD remains generally stable under answer-position perturbations, with ARC decreasing from 0.404 to 0.357 under A- and D-biased settings.
  • Perturbation effects: Noise injection raises accuracy to nearly 100% by removing meaningful distractor competition, while masking and no-answer insertion substantially reduce accuracy.Relative no-answer accuracy drops are largest for MMLU (71.6%), followed by CSQA (61.2%), ARC (55.4%), and HellaSwag (50.2%).
  • Perturbation effects: Score bias causes only minor accuracy changes but alters MPD in some benchmarks.The representative suffix is “(Score: 10/10)”.

6 Noise-Injection Failure Analysis

The noise-injection audit narrows residual failures through cross-model screening and targeted human review, separating perturbation-induced cases from potentially actionable benchmark issues and comparing signals with expert labels.

  • Two-stage review: Residual failures are identified through cross-model noise-injection failures and then reviewed and categorized by human annotators.
  • Two-stage review: The taxonomy separates excluded perturbation-induced cases from potentially actionable benchmark issues, with fine-grained subtypes reserved for qualitative analysis.
  • Residual-failure outcomes: Of 30 residual failures, 21 are excluded and 9 are retained as actionable candidates in the final four-benchmark scope.The final-scope X2 count is 6; the earlier 11/36 count included a separate five-choice CSQA analysis.
  • Residual-failure outcomes: A GPT-4o-mini classifier flags 18.4% of CommonsenseQA items and 2.8% of MMLU items as additional candidates for targeted human review.
  • External validation: The 1,000-item MMLU subset has 373 exact-text overlaps with MMLU-Redux, including 9 expert-labeled flawed items.
  • External validation: Candidate signals enrich expert-identified flaws relative to random inspection, but recall remains low at 0.222–0.444.
  • External validation: The signals are intended to prioritize smaller candidate sets for human review rather than replace expert auditing.
  • External validation: Precision and recall estimates are directional because only 9 confirmed flaws occur in the overlap.

7 Conclusion

The paper presents a probabilistic framework for auditing benchmark structure and prioritizing items for targeted human review. Its signals are descriptive and scope-limited, requiring cautious interpretation rather than serving as definitive or exhaustive flaw detectors.

  • Accuracy alone becomes insufficient for assessing benchmark quality as MCQA performance approaches saturation.
  • Aggregate Hnorm and MPD characterize probability-landscape structure, while noise injection prioritizes items for targeted human review.
  • The framework offers complementary signals for comparing benchmark design and prioritizing questions for review.
  • The experiments exclude latest frontier and closed-source models, so findings are evidence from an open-model setting.
  • The signals should not be treated as direct measures of benchmark quality or exhaustive flaw detectors.
  • The primary scope covers four benchmarks, sampled subsets, and preprocessing choices, motivating broader future evaluation.

A Model-level MPD Detail

Model-level MPD analysis compares probability-landscape concentration across benchmarks and models. The benchmark ordering is consistently observed among evaluated small models, while prompt and implementation checks support the main comparison.

  • MMLU consistently ranks first in MPD among Gemma-2 9B, Llama-3 8B, and Qwen-2.5 7B.
  • The score-bias comparison finds a maximum accuracy spread of 1.10 percentage points across 20 model–benchmark combinations.
  • The main experiments use (Score: 10/10) as the representative score-bias prompt because differences across tested values are small.
  • The evaluated models use deterministic decoding with temperature 0, top-p 1, and one maximum generated token.

D Scaling Effects

Scaling generally produces more concentrated probability landscapes, but MPD does not change monotonically with parameter count. Benchmark ordering remains broadly similar despite model-specific deviations.

  • 70B models consistently produce lower MPD values than 8B models across all benchmarks.
  • The 32B Qwen model has substantially lower MPD than both the 7B and 72B variants.
  • Non-monotonic scaling suggests probability-landscape structure depends on model-specific architecture or training characteristics, not only capacity.
  • Benchmark-level ordering remains broadly similar for evaluated well-calibrated models, with individual deviations indicating calibration or inference sensitivity.

E CommonsenseQA: 4-Choice vs. 5-Choice

The CommonsenseQA comparison shows that adding a fifth choice uniformly reduces accuracy while changing Hnorm differently across models. The broader analyses also show that item-level Hnorm rankings are model-specific and that noise-injection failures can reflect either item issues or method limitations.

  • E CommonsenseQA: 4-Choice vs. 5-Choice: Accuracy decreases uniformly across all three models when CommonsenseQA adds a fifth choice.
  • E CommonsenseQA: 4-Choice vs. 5-Choice: Gemma-2 9B Hnorm decreases by −0.029, Llama-3 8B increases by +0.213, and Qwen-2.5 7B changes by +0.000.
  • E CommonsenseQA: 4-Choice vs. 5-Choice: The Hnorm rank ordering is preserved for Gemma-2 9B and Qwen-2.5 7B but disrupted for Llama-3 8B.
  • H Cross-Model Hnorm Consistency: Small-model mean item-level Hnorm correlations range from 0.028 on MMLU to 0.177 on CSQA, while ARC has an all-pair mean r of −0.010.
  • H Cross-Model Hnorm Consistency: Item-level Hnorm rankings are largely model-specific and are not reliable per-item diagnostics.
  • I Taxonomy Annotation Agreement: Fine-grained HellaSwag subtype agreement is weaker, with author–annotator κ values of 0.250 and 0.545 on a five-item subset.
  • Noise injection can fail when unrelated city names create semantic collisions or remove the contrastive structure required by NOT/EXCEPT questions.
  • Spreadsheet-converted fractions such as 1/6 → “6-Jan” can produce opaque answer options and are flagged as bad_options_clarity.

K LLM-Based Structural Classification

An independent GPT-4o-mini structural classifier audits benchmark items for unambiguous question- or option-clarity defects without requiring domain knowledge. Its flags reveal distinct artifact patterns across CommonsenseQA and MMLU, with coding artifacts converging with the noise-injection taxonomy.

  • Classifier and decision procedure: The classifier uses GPT-4o-mini through OpenRouter at temperature 0 to label items as ok, bad_question_clarity, or bad_options_clarity.Wrong-answer and multiple-correct-answer judgments are excluded because they require domain knowledge.
  • Classification results: Table 18 reports the LLM-based structural classification results using bq for bad_question_clarity and bo for bad_options_clarity.These abbreviations identify the two structural defect categories in the tabulated results.
  • Classification results: 18.4% of CommonsenseQA items were flagged, compared with 2.8% of MMLU items.CommonsenseQA flags reflect pronoun ambiguity, meaningless ConceptNet-generated distractors, and incomplete stems; MMLU flags reflect PDF/CSV extraction artifacts.
  • Classification results: MMLU’s coding-artifact pattern corresponds directly to the A3 encoding-artifact type in the noise-injection taxonomy.This correspondence provides convergent evidence from two independent methods.
  • Classifier and decision procedure: Items are flagged only for unambiguous structural defects, using a five-step hierarchy that prioritizes question clarity and options clarity.A consistency rule changed 77 self-contradictory outputs from flagged error types to ok.
Loading 2608.30372v1…