PaperlayerView exact revision on arXiv

Source-linked AI summary

What Does MMLU Actually Measure? A Psychometric Audit of Difficulty Structure in Aggregate Benchmark Scores

Dana Paquin, Riddhiman Jain

arXiv:2609.09372v1math.NTcs.CL

TL;DR

The paper asks whether MMLU’s single aggregate score measures a coherent general capability or conflates separable retrieval and reasoning constructs. It audits the benchmark with IRT and deterministic text-based complexity measures, finding distinct STEM and non-STEM ability structure and non-invariant difficulty mappings. The authors therefore argue for disaggregated reporting, while noting statistical, measurement, and scope limitations.

  • Problem

    MMLU’s single aggregate score is widely read as a unified measure of general capability, although the benchmark may draw unequally on factual retrieval and multistep reasoning.

  • Method

    The study fits a 2PL IRT model to roughly 14 million responses from 1,000 open-weights models and regresses item difficulties on deterministic structural-complexity indicators.

  • Results

    MMLU’s STEM and non-STEM abilities are distinct, with ρ = 0.965, and their structural-complexity-to-difficulty mappings are non-invariant across partitions.

  • Takeaways & Limitations

    Aggregate MMLU scores can weight retrieval capacity and reasoning stability unequally, supporting disaggregated benchmark reporting.

  • Takeaways & Limitations

    The evaluation targets open-weights models, structural complexity is significant but not dominant, and several findings are marginal under strict controls.

Abstract

from arXiv · show

Although MMLU is widely adopted as a benchmark for calibrating general AI capabilities, we psychometrically demonstrate that its aggregate score primarily evaluates a model's factual retrieval capacity rather than its reasoning ability. By calibrating item difficulty for 1,000 open-weights language models over 14,042 MMLU test items using Item Response Theory, we show that evaluating both abilities via a single test is inherently flawed. Difficulty is then regressed on a deterministic, text-extractable framework of structural complexity. Applying a joint Wald test with subject-clustered covariances demonstrates that the MMLU conflates fundamentally separable constructs. The mapping from structural complexity to difficulty is not invariant across the benchmark's STEM and non-STEM partitions. This finding has practical consequences. Aggregate leaderboard ranks track non-STEM accuracy more closely than STEM accuracy, so selecting a Top-50 model on the aggregate for a reasoning-intensive deployment displaces roughly 22% of the STEM-appropriate choices. Furthermore, when controlling for the multiple-choice guessing floor natively inside the response model, we find that higher-ability models continue to degrade more steeply under increased reasoning depth. The MMLU aggregate therefore weights retrieval capacity and reasoning stability unequally, inadvertently favoring models optimized for retrieval. We release our deterministic framework as a reproducible auditing instrument and recommend disaggregated reporting.

1 Introduction

MMLU reports one aggregate accuracy score across 14,042 multiple-choice items, but the paper argues that this score may conflate declarative retrieval with procedural reasoning. Using IRT and text-based structural predictors, it tests whether the benchmark measures separable abilities and whether item-difficulty mappings differ across STEM and non-STEM.

  • Motivation: MMLU’s aggregate score may conflate declarative fact retrieval with procedural multistep reasoning.Zipf Rarity and Entity Density represent retrieval demands, while WSCG represents sequential inferential load.
  • Approach: The audit calibrates item difficulty and model ability with a 2PL IRT model using roughly 14 million responses from 1,000 open-weights models.Estimated difficulties are then regressed on five deterministic, text-extractable dimensions and nine indicators.
  • Main result: MMLU’s STEM and non-STEM partitions do not measure a single interchangeable ability: a two-dimensional model estimates ρ = 0.965, below unity, and predicts held-out responses better.The matched-length disattenuated subscale correlation is 0.977.
  • Main result: The mapping from text structure to difficulty differs across STEM and non-STEM, with Entity Density surviving clustered, bootstrap, and Holm-corrected testing.The supplied passage identifies Entity Density as the only contrast surviving all stated controls.

2 Related Work

Prior work shows that aggregate benchmark scores can conflate factual retrieval with inference, while existing IRT-based benchmark analyses generally assume a single latent ability. This paper extends those critiques by testing whether item difficulty reflects separable constructs and whether text-based complexity behaves consistently across domains.

  • Prior critiques distinguish knowledge retrieval from inferential capacity and document models relying on surface heuristics despite strong benchmark scores.
  • Existing IRT-based NLP benchmarks and leaderboards assume unidimensionality, so they cannot detect separable abilities that contribute differently across domains.The cited work models one latent ability θj per model.
  • The proposed analysis uses a multi-group explanatory item-difficulty model, an error-extended Linear Logistic Test Model estimated separately for STEM and non-STEM item groups.The comparison concerns item groups rather than respondent populations and is an item-side analogue of measurement invariance.
  • Text-based difficulty prediction has precedent in educational measurement and NLP, while other benchmarks isolate particular reasoning types by constructing targeted item pools.
  • MMLU rankings are sensitive to semantically invariant perturbations, motivating an internal noise-floor analysis and refits across all 57 subjects.

3 Measurement Framework and Methodology

The framework calibrates MMLU item difficulty and model ability with a two-parameter IRT model, then predicts difficulty from deterministic structural-complexity indicators. It uses a fixed STEM/non-STEM partition and tests whether these complexity–difficulty relationships remain stable across domains.

  • A 1,000-model population yields approximately 14 million binary responses across 14,042 MMLU items, spanning 57 subjects and model families from 3.2 million to over 180 billion parameters.The population comes from the Open LLM Leaderboard and is substantially redundant rather than a designed sample.
  • Two-Parameter Logistic IRT estimates item difficulty bi, discrimination ai, and model ability θj using Stochastic Variational Inference with NumPyro and JAX.The model assigns normal priors to θj and bi and a HalfNormal prior to ai.
  • 0.9921 correlation between independently calibrated difficulty vectors supports approximately 0.9960 full-test reliability across all 14,042 items.The independent calibration uses two random cohorts of 500 models, indicating limited incidental measurement error from model sampling.
  • The fixed official partition contains 3,153 STEM items across 19 subjects and 10,889 non-STEM items across 38 subjects.The partition is fixed before estimating the path model to prevent post hoc selection bias.
  • Five dimensions and nine deterministic indicators extract structural complexity from prompt stems, including a weighted symbolic graph for sequential multistep reasoning.The graph uses lexical categories, symbolic operators, dependency arcs, and state-tracking edges.
  • WSCG Depth correlates 0.580 with token count, and residualizing the two WSCG indicators preserves the stratified fits with W(9) = 134.79.This contrasts with WSCG Nodes, whose token-count correlation is 0.985; option-set covariates add predictive power without eliminating the cross-domain contrast.

4 Results

MMLU difficulty is only weakly predictable from text globally, while structural features relate differently to difficulty in STEM and non-STEM items. Cluster-aware tests support domain-specific slope differences, although the strength of evidence varies across inference procedures.

  • 4.1 Global Path Analysis and Baselines: 3.57%: the nine-indicator framework explains little global difficulty variance, despite eight significant predictors and an extreme omnibus test.Variance inflation factors range from 1.025 to 2.379, so the small R2 is not attributed to multicollinearity.
  • 4.1 Global Path Analysis and Baselines: 56.5% in STEM versus 32.6% outside it: the indicators recover different shares of the text-predictable difficulty ceiling.The dense encoder ceiling is 0.1269 pooled and 0.1330 in STEM; option-set indicators raise the STEM share to 89.9%.
  • 4.2 Domain-Stratified Bifurcation: 0.0732 versus 0.0361: structural complexity explains roughly twice the cross-validated difficulty variance in STEM as outside STEM.The ratio is 2.46× in logit-squared units, with bootstrap CI [1.80, 3.36].
  • 4.2 Domain-Stratified Bifurcation: Entity Density most robustly diverges across domains, with β = 0.140 in STEM but b = 0.0002 outside STEM.The contrast survives Holm correction and wild cluster bootstrap; Syntactic MDD shows a similar but less robust split.
  • 4.3 Testing Structural Homogeneity: W(9) = 28.21, p = 8.8 × 10^-4 under CR2, rejecting structural homogeneity after subject-clustered correction.CR2 inflates slope standard errors by a median factor of 1.65; CR1 also rejects, with W(9) = 33.13 and p = 1.3 × 10^-4.
  • 4.3 Testing Structural Homogeneity: The wild cluster bootstrap makes the joint test marginal at p = 0.060, while one-contrast tests remain decisive for Entity Density.Entity Density has pHolm = 0.014; the bootstrap reference distribution is wider than the nominal chi-square calibration.

5 Consequences for Leaderboard Validity

MMLU’s aggregate score creates practical leaderboard asymmetries because it weights non-STEM performance more heavily and conflates retrieval capacity with reasoning stability. Native guessing-floor control confirms that higher-ability models degrade more steeply with reasoning depth, although the true interaction is modest.

  • 5 Consequences for Leaderboard Validity: Because 77.55% of MMLU items are non-STEM, aggregate scoring is arithmetically weighted toward non-STEM performance before structural non-invariance adds measurement asymmetry.The section treats the resulting asymmetry as a practical cost for downstream model selection.
  • 5.1 Reasoning Stability: A naive reasoning-sensitivity regression reports stronger negative ability correlations, but 3PL simulations reproduce much of those values through guessing-floor compression.The generative null accounts for 116%, 69%, and 30% of the observed correlations across the three samples.
  • 5.1 Reasoning Stability: Native 3PL modeling estimates a negative ability-by-depth interaction of γ1 = −0.086 logits per standardized WSCG-depth unit.The interaction remains negative after setting the guessing floor to c = 0.25, but is much smaller than the naive correlation suggests.
  • 5.2 Consequences for Downstream Model Selection: The aggregate tracks non-STEM accuracy more closely than STEM accuracy, with weighted Kendall’s τ of 0.9887 versus 0.9634.Subsampling non-STEM to STEM’s test length reduces the matched gap to 0.9756.
  • 5.2 Consequences for Downstream Model Selection: 22% of STEM-appropriate choices are displaced when practitioners select models from the aggregate Top 50.The displacement estimate has a bootstrap CI of [16%, 28%].

6 Discussion

The discussion argues for treating MMLU as a multidimensional benchmark rather than a unified ability measure. It recommends text-based, deterministic auditing and disaggregated reporting while acknowledging statistical, construct-invariance, and scope limitations.

  • 6 Discussion: Future benchmarks should report separate retrieval and reasoning-capacity indices alongside aggregate scores.The proposed indices are computable from item text before model evaluation and avoid stochastic LLM-annotator dependence.
  • 6.2 Limitations: The framework’s structural complexity signal is significant but not dominant, and it is weakest where abstract STEM content is strongest.Mathematical content is overwhelmingly ASCII, leaving the highest symbolic weight tier inert, while Zipf rarity can fall back to the population mean.
  • 6.2 Limitations: The evaluation covers open-weights models, so whether the observed sensitivities extend to closed frontier models remains open.This is a scope boundary of the reported population rather than a conclusion about closed models.
  • 6.2 Limitations: Several findings are marginal against strict controls, including the between-domain R2 gap at p = 0.08 and the Top-50 displacement control at p = 0.09.The joint homogeneity test under a wild cluster bootstrap is also marginal at p = 0.060.
  • 6.2 Limitations: Rejecting structural homogeneity concerns item-side mapping and does not formally establish person-side measurement non-equivalence.The design therefore does not strictly prove that a single latent θ is incomparable across models.

7 Conclusion

Using IRT, multi-group analysis, and confirmatory two-dimensional modeling, the paper finds that MMLU’s STEM and non-STEM partitions measure separable abilities with structurally different difficulty mappings. These differences affect leaderboard selection and reasoning-stability estimates, motivating disaggregated benchmark reporting.

  • 7 Conclusion: MMLU’s partitions do not measure one interchangeable ability: latent correlation is ρ = 0.965, matched-length disattenuated correlation is 0.977, and the two-dimensional model improves held-out prediction.The structural distinction is supported by confirmatory modeling rather than only exploratory factor recovery.
  • 7 Conclusion: Structural mappings from text complexity to item difficulty differ across partitions, with the homogeneity rejection surviving item-level and subject-clustered inference.The reported Wald statistics are W(9) = 138.36 for item-level inference and W(9) = 28.21 under CR2.
  • 7 Conclusion: Aggregate Top-50 selection displaces 22% of STEM-appropriate choices, while native guessing-floor control estimates γ1 = −0.086 for reasoning-depth sensitivity.Together these results connect construct heterogeneity to model-selection consequences and procedural reasoning stability.
  • 7 Conclusion: The aggregate weights declarative retrieval capacity more heavily than procedural reasoning stability, thereby favoring models optimized for retrieval.The paper releases its framework to support auditing and future disaggregated reporting.

A Sensitivity, Diagnostics and Clustering

Sensitivity and diagnostic analyses largely preserve the paper’s structural conclusions across weighting schemes, model specifications, filtering thresholds, and clustering choices. They also reveal non-normal, heteroskedastic residuals and marginal results under some stricter controls.

  • Sensitivity and robustness: All 123 weight perturbation schemes preserve structural homogeneity rejection, STEM-over-non-STEM R2, and positive Syntactic MDD divergence.Homogeneity rejection remains at maximum p = 1.26 × 10−77, while STEM R2 ranges from 0.077–0.083 versus 0.031–0.043.
  • Diagnostics and robustness: Cross-validation yields R2_cv of 0.0732 and 0.0361 versus in-sample values of 0.0792 and 0.0380, while individual subject pools show small-N failure modes.These results indicate stable partition-level fits but weaker reliability for some subject-level estimates.
  • Diagnostics and robustness: Residuals are right-skewed and heteroskedastic, with Jarque–Bera, Breusch–Pagan, and White tests rejecting normality or homoskedasticity.The reported diagnostics motivate resampling rather than normal-theory inference.
  • Diagnostics and robustness: Rasch refitting correlates with the 2PL baseline at r = 0.894 and strengthens the structural contrast to 0.1038 versus 0.0676.Filtering items with ai < 0.2 raises STEM R2 to 0.0862 while leaving non-STEM invariant.
  • Diagnostics and robustness: Filtering uninformative items disproportionately removes STEM items, with 31.9% of removals versus a 22.5% STEM pool share.Slope equality remains rejected at every tested discrimination threshold.
  • Clustering and size-matched nulls: Whole-subject-block nulls widen the R2-gap null to mean 0.0076 with p = 0.080, while the block rank-displacement null is 22.4% with p = 0.609.These controls make several conclusions marginal against stricter clustering-aware comparisons.

B Functional Form and Option Structure

The structural indicators explain item difficulty better than simple length or readability, but their relationships with difficulty differ across STEM and non-STEM items. Choice-based indicators add predictive information, especially in STEM, and jointly remain significant.

  • Functional form: Syntactic MDD and Entity Density rise monotonically in STEM but follow non-monotone curves outside STEM.Spline fits significantly improve non-STEM fit for Syntactic MDD and Concreteness, but not inside STEM.
  • Option structure: Option indicators explain R2cv = 0.0684 globally and R2cv = 0.1196 in STEM when combined with stem indicators.The combined 17-indicator model remains decisive with W(17) = 138.86, while option indicators alone are non-invariant with W(8) = 51.97.
  • Option structure: Difficulty scales with distractor structure: key-stem similarity has b = −1.477, while maximum distractor similarity has b = 1.056.These are the two strongest option-based members reported.

C Confirmatory Two-Dimensional IRT

A two-dimensional IRT model separates STEM and non-STEM abilities while outperforming the unidimensional model, but the companion explanatory response model is not trustworthy because its item residuals overwhelm the difficulty variation.

  • Confirmatory model: The two-dimensional IRT model improves held-out response prediction over the unidimensional model while estimating STEM and non-STEM abilities separately.Each item loads only on its designer-assigned subject dimension, with latent correlation 0.965 and no posterior mass above 0.99.
  • Confirmatory model: The two-dimensional specification assigns each model separate STEM and non-STEM abilities drawn from a bivariate normal with a free correlation.The model is estimated by SVI and uses designer-fixed subject assignments for item loadings.
  • Model limitation: The explanatory response model is not reported because its 4.93-logit residual scale exceeds the 2.24-logit difficulty standard deviation.Its credible intervals are also an order of magnitude narrower than robust standard errors, and two interaction signs reverse as step budgets increase.

D Label Errors, Contamination and Transfer

Label errors and cohort composition do not explain the STEM–non-STEM contrast, while respondent-side DIF is more prevalent in STEM across the tested model families.

  • Label errors and contamination: Label re-annotation covers 5,984 items and flags 374 items, or 6.25%, with a lower STEM rate of 4.70% than the 6.98% non-STEM rate.Flagged items are harder on average, with mean b = +1.86 versus −0.10.
  • Transfer: The difficulty contrast persists across model cohorts, with early and recent recalibrations retaining distinct STEM–non-STEM patterns.Difficulty vectors correlate at r = 0.832, equal to 86% of the attainable r = 0.967 ceiling.
  • Transfer: STEM has higher DIF rates than non-STEM for Mixtral, Solar, and Yi by 7.5, 7.5, and 7.2 percentage points, respectively.The excess is significant for all three families, with all z > 7.4, and is reproduced after purification.
Loading 2609.09372v1…