PaperlayerView exact revision on arXiv

Source-linked AI summary

What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks

Meera Desai, Sang T. Truong, Hanna Wallach, Alex Chouldechova, A. Feder Cooper, Jean Garcia-Gathright, Daniel E. Ho, Abigail Z. Jacobs, Sanmi Koyejo, Nicholas Pangakis, Angelina Wang

arXiv:2609.08812v1cs.CY

TL;DR

AI benchmark scores are widely used, but it is unclear whether benchmarks measure the concepts they claim to measure. The paper compares benchmark rankings and item-level response patterns across 56 benchmarks and 53 models, finding weak within-concept convergence for many safety benchmarks and limited discrimination among capability concepts.

  • Problem

    Benchmarks inform model development and governance, but it is often unclear whether their scores measure the concepts they purport to measure.

    Source: §1 · p. 1

  • Method

    The paper compares model rankings on benchmarks with the same versus different assigned concepts and uses item-level IRT analyses to make analogous comparisons.

    Source: §1 · p. 2

  • Results

    Safety rankings often correlate weakly within assigned concepts, while capability rankings correlate as strongly within as across concepts, suggesting capability concepts may not be discriminable.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Systematic convergence and discrimination analyses can reveal patterns unavailable without item- and benchmark-level scores and surface validity issues in individual benchmarks.

    Source: §5 · p. 10

    The analyses require item- and benchmark-level scores across many benchmarks and models, involve substantial collection effort (approximately 1050 H200 GPU-hours), and may reflect a snapshot in time.

    Source: §5 · p. 11

Abstract

from arXiv · show

Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reasoning, refusal). We adapt convergent and discriminant validity from the social sciences into an approach for interrogating AI benchmarks, applying it to 56 capability and safety benchmarks across 53 models. We label benchmarks with substantively similar purported concepts to a shared assigned concept, and ask whether model rankings on benchmarks with the same assigned concept correlate more strongly than rankings on benchmarks with different assigned concepts. We ask analogous questions at the item level using item response theory (IRT) models. We find that correlations between model rankings on benchmarks with the same assigned safety concepts are often weak, suggesting these concepts may be conceptualized inconsistently across benchmarks. For assigned capability concepts (e.g., reasoning, knowledge), model rankings are often as strongly correlated among benchmarks with the same assigned concept as between benchmarks with different assigned concepts, suggesting these capability concepts may not discriminate well from one another. In some cases, benchmarks that share design elements (e.g., score format) correlate more strongly than benchmarks with the same assigned concept. Finally, some individual benchmarks correlate more strongly with benchmarks assigned a different concept than with benchmarks sharing their own assigned concept, suggesting they may measure a different concept than they purport to. For example, BBQ-accuracy correlates more strongly with benchmarks labeled reasoning than with benchmarks that share its assigned concept, bias. To support future empirical work on benchmark validity, we release our extensive dataset of model outputs and scores at the item- and benchmark-level.

1 Introduction

The paper adapts convergence and discrimination analyses to examine whether AI benchmarks measure the concepts they claim to measure. Across the studied benchmarks, safety rankings often correlate weakly within assigned concepts, while capability rankings correlate similarly within and across concepts.

1 Introduction

  • Benchmark concepts are often underspecified, and low correlation can indicate invalidity or distinct but valid ways of conceptualizing a concept.Few, if any, benchmarks have been validated using construct validity, so comparisons may use benchmarks that are themselves invalid.Source: §1 · p. 2
  • The approach compares sets of benchmarks with similar or different assigned concepts rather than relying on one potentially invalid benchmark as a standard.For individual benchmarks, it compares model rankings with those on benchmarks assigned the same or a different concept.Source: §1 · p. 2
  • Safety rankings often correlate weakly within assigned concepts, while capability rankings correlate as strongly within as across concepts.Shared design elements sometimes align with stronger correlations than shared concepts; BBQ-accuracy aligns more with reasoning than with other bias benchmarks.Source: §1 · p. 2
  • The study contributes an approach and releases item- and benchmark-level outputs and scores for 53 models across 56 benchmarks.The authors also report empirical findings about the concepts benchmarks purport to measure.Source: §1 · pp. 2–3

2 Related work

Related work establishes construct validity as a framework for interrogating benchmarks and describes emerging empirical studies of benchmark agreement. The paper distinguishes its broader focus on benchmark validity across capability and safety concepts, including item-level analyses, from prior work with narrower aims.

2 Related work

  • Prior work has applied construct validity and its convergent and discriminant lenses to AI benchmark design and interrogation.The paper situates its approach among work raising concerns about benchmark validity and adapting these social-science concepts.Source: §2 · p. 3
  • Empirical studies have examined benchmark agreement, but this paper broadens the inquiry across capability and safety concepts and adds item-level analyses using IRT.Ren et al. (2024) studied safety-benchmark rankings against general capability; Burnell et al. (2023a) examined latent model ability, while this paper asks what benchmarks measure.Source: §2 · p. 3
  • Prior IRT work in AI has characterized model ability or supported benchmark item selection and difficulty analysis; this paper uses IRT diagnostically to test shared-trait prediction across benchmarks.It asks whether jointly estimating a latent trait on two benchmarks with the same assigned concept improves prediction of item-level model scores.Source: §2 · p. 3

3 Methodology

The methodology assembles model scores across capability and safety benchmarks, assigns shared concepts to substantively similar claims, and compares benchmark rankings and item-level response patterns. It also examines design effects and tests whether individual benchmarks align more closely with an alternative concept.

3.1 Dataset collection

  • The dataset contains 56 capability and safety benchmarks across 53 models, spanning 17 benchmarks in four capability concepts and 39 in seven safety concepts.The collection draws on SafetyPrompts and HELM and expands coverage across concepts within practical constraints.Source: §3.1 · p. 3
  • Benchmarks are assigned shared concepts by grouping substantively similar purported concepts, using HELM categories where available and inductive grouping otherwise.The labels are often high-level because benchmark authors describe purported concepts with varying specificity.Source: §3.1 · p. 4
  • For benchmarks with more than 1,000 items, the study samples 1,000, collects zero-shot outputs at temperature 1, and applies each benchmark paper's scoring metrics.Exact-match tasks use output-constraining system prompts to increase scorable responses.Source: §3.1 · p. 4

3.2 Analysis

  • Benchmark-level convergence and discrimination are assessed by comparing Spearman correlations between model rankings for pairs sharing versus not sharing an assigned concept.The analysis adapts MTMM considerations because benchmarks do not follow a fully crossed design and their purported concepts are often underspecified.Source: §3.2 · pp. 4–5
  • Item-level analyses compare how well responses are predicted by a shared latent trait versus separate benchmark-specific traits, using paired 1PL IRT models and held-out AUC.∆AUC is the separate-model AUC minus the shared-model AUC; positive values indicate better prediction by separate traits.Source: §3.2 · p. 5
  • The item-level IRT models assume benchmarks are approximately unidimensional, an assumption that may not hold for every benchmark.Lack of item-level convergence alone is not sufficient to establish invalidity; it flags a benchmark for closer scrutiny.Source: §3.2 · p. 5
  • The design-effect analyses test whether rankings align more by demographic target or shared design, and whether shared score format aligns more strongly than shared concept.The study uses hierarchical clustering and a partial Mantel test to assess the role of score format.Source: §3.2 · pp. 5–6
  • For hypothesized mislabeling, the relabeling statistic compares a focal benchmark's mean absolute correlations with benchmarks assigned the hypothesized versus current concept.Positive values favor the hypothesized concept; uncertainty is estimated by model-family cluster bootstrap and tested with a one-sided p-value.Source: §3.2 · p. 6

4 Findings

Across 56 benchmarks and 53 models, model rankings often fail to converge or discriminate according to assigned concepts; in some cases, benchmark design or an alternative concept better explains their relationships.

4 Findings

  • The analyses compare model rankings across benchmarks, after excluding four saturated and four format-sensitive benchmarks from the subsequent analyses.The remaining rankings agreed with HELM’s on 10 shared benchmarks (mean Spearman correlation 0.74).Source: §4 · p. 6

4.1 Convergence of benchmarks with the same assigned concept

  • Safety benchmarks labeled refusal, safety detection, and bias show wide correlation ranges that often approach or fall below zero, while capability benchmarks with the same assigned concept converge more consistently.Item-level results broadly agree: a shared latent trait predicts capability responses about as well as separate traits, while safety responses are better predicted by separate traits.Source: §4.1 · pp. 6–7, Figure 1 · p. 7

4.2 Discrimination between benchmarks with different assigned concepts

  • Rankings across different capability concepts are often as strongly correlated as rankings within the same concept, while some safety benchmarks correlate more strongly with capability benchmarks; refusal and over-refusal are a notable exception.Summarization is the exception to the capability pattern; refusal and over-refusal rankings are strongly inversely correlated and their item scores require separate latent traits.Source: §4.2 · pp. 7–8, Figure 2 · p. 8

4.3 Role of demographic target and score format

  • Correlations are higher within bias benchmarks across demographic targets than across benchmarks for the same target, and score format predicts similarity more strongly than assigned concept.With three-way format coding, both shared concept and format independently predict higher correlation; with binary coding, shared LLM-judge scoring is a stronger predictor than concept.Source: §4.3 · p. 9, Figure 3 · p. 10

4.4 Interrogating individual benchmarks

  • BBQ-accuracy aligns more with reasoning than bias benchmarks, and DecodingTrust-Fair aligns more with knowledge than bias; OR-Bench favors its over-refusal label over tested capability concepts.The authors suggest DecodingTrust-Fair’s relationship with knowledge may reflect its income-prediction items and demographic parity scoring.Source: §4.4 · pp. 9–10, Figure 3 · p. 10

5 Discussion and conclusion

The authors argue that convergent and discriminant validity can systematically reveal patterns across benchmarks and surface validity issues in individual benchmarks. Interpretation is constrained by underspecified concepts, analytic assumptions, and the effort required to collect broad item- and benchmark-level scores.

5 Discussion and conclusion

  • Systematic convergent and discriminant validity analyses reveal patterns across benchmarks and can surface substantial validity issues in individual benchmarks.The analyses rely on model scores collected at both item and benchmark levels across many models.Source: §5 · p. 10
  • Developers could iteratively apply the relabeling test, and researchers can analyze a new benchmark using a subset of models from the released dataset at relatively low cost.The authors acknowledge that broad data collection is resource-intensive, while the smaller-scale approach is relatively low cost.Source: §5 · pp. 10–11
  • Underspecified benchmark concepts make it difficult to distinguish measurement failure from conceptual disagreement, so developers should document intended concepts and their boundaries.Source: §5 · p. 11
  • The analyses do not control for general model ability and assume item-level unidimensionality and shared properties within assigned concepts, assumptions that may fit safety concepts less well.For safety concepts, weak correlations may reflect variation in task, user, environment, or normative standard rather than poor measurement.Source: §5 · p. 11
  • Validity analyses require item- and benchmark-level scores across many models and benchmarks; this study used approximately 1050 H200 GPU-hours and may reflect a snapshot in time.The authors report robustness across model generations and release their dataset as a step toward ongoing monitoring.Source: §5 · p. 11

6 Ethics

The authors report that the study collected no new human-subjects data and disclose LLM assistance. The appendix defines “benchmark” and notes candidate benchmarks excluded for missing licenses.

6 Ethics

  • The work analyzes existing benchmark data and collected model outputs and scores, with no new human-subjects data; the authors state their intent is to strengthen safety evaluation.Source: §6 · p. 11
  • Claude and ChatGPT assisted with prose editing, some appendix tables, and code for data processing and visualization.Source: §6 · p. 11

A Benchmarks collected and interrogated

A.1 Benchmarks excluded for missing licenses

  • LLM decision bias, LLM implicit bias, and SAFE were excluded from the candidate pool because their licenses were missing.Source: §A.1 · p. 18

A.2 Excluding benchmarks with unreliable model rankings

The appendix screens benchmarks for reliable model rankings using saturation and comparisons with HELM, excluding benchmarks whose rankings appear sensitive to output-format compliance. It also reports item-level agreement patterns and describes the item-level analysis set.

A.2 Excluding benchmarks with unreliable model rankings

  • Because the analyses compare model rankings, benchmarks were screened for unreliable rankings using saturation and HELM comparisons, including a check for format-driven rankings on free-text exact-match tasks.Eight benchmarks were excluded, four by each screen, and the remaining 48 were used in the analyses.Source: §A.2 · pp. 18–19

A.2.1 Saturation

  • Four benchmarks—DecodingTrust Stereotype, CIVICS, BoolQ, and IMDB—were classified as saturated because their normalized top-to-median score gap was below 0.05.Source: §A.2.1 · p. 19

A.2.2 Comparison against HELM: pipeline validation and format non-compliance

  • Across six shared multiple-choice benchmarks, median RMSE against HELM was 0.071 despite differences in prompting and temperature.EntityMatching was an exception (RMSE 0.403), driven by a comparison between HELM base Llama models and the chat variants evaluated here.Source: §A.2.2 · p. 19
  • WikiFact, Dyck, bAbI, and Synthetic Reasoning (Abstract) were excluded after their shared-model rankings disagreed with HELM’s, with output-format and scoring issues documented for excluded tasks.For bAbI, 52 items require two-step routes despite an instruction to answer with a single word; 47 evaluated models scored zero on all 52.Source: §A.2.2 · pp. 19–20
  • Mean Spearman correlation between our model scores and HELM’s was 0.25 for four excluded benchmarks versus 0.74 for ten retained benchmarks.Median RMSE against HELM was 0.246 for the four excluded benchmarks and 0.111 for the four retained free-text benchmarks.Source: §A.2.2 · p. 20
  • Item-level judgments agree closely on DataImputation and GSM8K but diverge on the four excluded benchmarks.Source: §A.2.2 · p. 20, Figure A.5 · p. 22
  • Figure A.4 reports near-diagonal score agreement for some benchmarks, preserved rankings despite score discrepancies for others, and substantial divergence on some tasks.Source: Figure A.4 · p. 21

A.3 Benchmarks included in the item-level analyses

  • The item-level analyses cover assigned safety and capability concepts; items with responses from fewer than five models were excluded from response matrices.Source: §A.3 · p. 23

A.4 Final list of benchmarks used in the correlation and item-level analyses

The appendix lists the benchmarks and scoring choices used in the analyses, alongside the staged inclusion counts and exclusion reasons. The benchmark-level analyses use 48 benchmarks, while item-level analyses use 37.

A.4 Final list of benchmarks used in the correlation and item-level analyses

  • The analyses use 48 benchmarks at the benchmark level and 37 at the item level, reduced from 56 collected benchmarks.Table A.1 tracks benchmark counts at each inclusion stage; the two screens are disjoint.Source: §A.4 · p. 23, Table A.1 · p. 23
  • A benchmark enters the remaining analyses only after passing saturation and format-compliance checks; item-level analyses also require binary correct/incorrect responses.The benchmark list records scoring and aggregation choices and each benchmark’s outcome on the screens.Source: §A.4 · p. 23, Table A.2 · p. 24
  • Exclusions outside the staged counts include three benchmarks omitted before collection for missing licenses, four saturated benchmarks, and four saturated variants of retained benchmarks.The excluded benchmarks and variants are listed in Table A.3.Source: §A.4 · p. 27, Table A.3 · p. 28

B Correlation and item-level benchmark discrimination matrices

The benchmark-level correlation and item-level discrimination analyses mostly align, but several benchmark pairs show different relationships under the two approaches. The appendix also documents assigned-concept groupings and the evaluated models.

B Correlation and item-level benchmark discrimination matrices

  • CALM (bias) and EntityMatching (reasoning) correlate at r = 0.57, yet their item-level ∆AUC is 0.063.For this pair, item-level response patterns favor separate benchmark-specific traits despite the aggregate correlation.Source: B Correlation and item-level benchmark… · p. 28
  • SGBench-mcq (refusal) correlates with Synth reasoning at r = 0.51 and LSAT at r = 0.50, while their ∆AUC values are 0.035 and 0.030, respectively.Both ∆AUC values are well above the reported mean between safety and capability benchmarks.Source: B Correlation and item-level benchmark… · p. 28
  • Pairs with high aggregate correlation tend to have small ∆AUC, and pairs with low correlation tend to have large ∆AUC, though some pairs diverge across analyses.Figure B.1 shows benchmark-level ranking correlations; Figure B.2 shows item-level discrimination across IRT-eligible benchmarks.Source: B Correlation and item-level benchmark… · p. 28, Figure B.1 · p. 29, Figure B.2 · p. 30

C Assigned concepts

  • Assigned Reasoning includes benchmarks described as testing mathematical, analytical, legal, and commonsense reasoning, as well as entity matching.Table C.1 presents benchmarks alongside their self-reported concepts.Source: C Assigned concepts · p. 31, Table C.1 · p. 35

D.1 List of evaluated models

  • Table D.1 lists evaluated models grouped by creator.The supplied passage identifies the table and its grouping, but does not enumerate model names.Source: Table D.1 · pp. 36–37

D.2 Evaluation implementation details

The evaluation implementation specifies sampling, decoding, response scoring, and judge-model choices. The item-level analysis compares shared-ability and benchmark-specific 1PL models.

D.2 Evaluation implementation details

  • Benchmarks exceeding 1,000 items were sampled down to 1,000, using stratified sampling for demographic subgroups and random sampling otherwise.GenMO preserved male/female narrative pairs, while CALM sampling stratified demographic–template combinations.Source: §D.2 · p. 37
  • Evaluations were zero-shot at temperature 1, with output-constraining system prompts for exact-match label tasks to increase scorable responses.The prompts were custom unless the benchmark paper specified one; the choice followed an ablation of few-shot and zero-shot prompting and prompt inclusion.Source: §D.2 · p. 37
  • For LLM-judge benchmarks, the authors used the original paper’s judge when available and otherwise substituted Qwen3-30B as the default judge.Judges were queried at temperature 0, and a sensitivity analysis compared Qwen3-30B with Llama-3.3-70B-Instruct as the default.Source: §D.2 · pp. 37–38

D.3 IRT model specification

  • The item-level analysis compares 1PL models using one shared ability versus benchmark-specific latent traits; θ_m is model m’s latent trait and b_i is item i’s difficulty.The equation specifies the probability of a model response being 1.Source: §D.3 · p. 40

D.4 Ablation: system prompts and few- vs. zero-shot prompting

The ablation compares few-shot counts and system-prompt use on Civil Comments. A system prompt supports near-perfect compliance at zero shots, whereas few-shot prompting without one reduces non-compliance but introduces sensitivity to examples.

D.4 Ablation: system prompts and few- vs. zero-shot prompting

  • With a system prompt, compliance is near-perfect even at zero shots and accuracy stays stable across shot conditions; without one, zero-shot compliance is near zero.The ablation used phi-3-5-mini-instruct on Civil Comments and varied 0, 2, or 4 examples.Source: §D.4 · p. 40, Figure D.1 · p. 41
  • Without a system prompt, adding few-shot examples reduces non-compliance but introduces sensitivity to example choice and ordering.Predictions also shift substantially as the number of shots changes in the no-system-prompt condition.Source: §D.4 · p. 40, Figure D.1 · p. 41
  • The authors therefore use zero-shot prompting with a system prompt as the default, aiming for high compliance without few-shot example-choice variance.Source: §D.4 · p. 40

D.5 Robustness to model era and LLM-judge choice

All nine results hold under both judges, and eight of nine replicate across model-era cohorts. The OR-Bench negative control is the exception: its statistic stays negative, but its confidence interval no longer excludes zero in the smaller cohorts.

D.5 Robustness to model era and LLM-judge choice

  • Robustness tests recomputed the main statistics after perturbing model-set composition and the default LLM-judge choice.The judge sensitivity analysis re-scored four benchmarks with Llama-3.3-70B-Instruct and substituted those scores into the matrix.Source: §D.5 · pp. 42–43
  • Eight of nine results replicate in both model-era cohorts, with the convergence gap rising from +0.21 to +0.38 and the partial Mantel contrast from +0.43 to +0.84.The cohorts split the 53 models by public release date, before October 2024 (n = 27) and October 2024 onward (n = 26).Source: §D.5 · pp. 42–43, Table D.4 · p. 43
  • The OR-Bench negative-control statistic remains negative in both cohorts (−0.38 and −0.36), but its confidence interval no longer excludes zero at the halved sample size.The authors attribute this to reduced statistical power, not a reversal.Source: §D.5 · pp. 42–43, Table D.4 · p. 43
  • All nine results hold under both judges, and the score-format versus assigned-concept Mantel contrast is essentially identical under both.For refusal × over-refusal, the mean correlation shifts from −0.42 to −0.48; the OR-Bench relabeling statistic shifts from −0.52 to −0.58.Source: §D.5 · p. 43, Table D.5 · p. 44

E Benchmark use in recent commercial model releases

The section reports benchmark use in recent commercial model releases, including cases where BBQ-accuracy is the only bias metric or one of two bias metrics used.

E Benchmark use in recent commercial model releases

Loading 2609.08812v1…