Source-linked AI summary
Benchmark^2: Systematic Evaluation of LLM Benchmarks
Qi Qian, Chengsong Huang, Jingwen Xu, Changze Lv, Muling Wu, Wenhao Liu, Xiaohua Wang, Zhenghua Wang, Zisu Huang, Muzhao Tian, Jianhan Xu, Kun Hu, He-Da Wang, Yao Hu, Xuanjing Huang, Xiaoqing Zheng
TL;DR
LLM benchmarks increasingly guide research and deployment, but their own quality and reliability remain insufficiently assessed. BENCHMARK2 introduces three complementary metrics and evaluates them across 15 benchmarks and 11 models. The study finds substantial quality variation and shows that selective construction can retain comparable evaluation performance using only 35% of the original data.
Problem
Benchmark reliability is rarely questioned even though benchmarks guide LLM research and deployment decisions and may produce inconsistent rankings, weak performance separation, or rank-inconsistent items.
Method
BENCHMARK2 assesses benchmark quality with CBRC, DS, and CAD, evaluating 15 benchmarks across mathematics, reasoning, and knowledge domains with 11 LLMs from four families.
Results
35% of the original data achieves comparable evaluation performance after quality-based instance filtering, while the study identifies substantial quality variation among existing benchmarks.
Takeaways & Limitations
The framework provides practitioners with metrics for assessing benchmark reliability and constructing smaller evaluations that preserve evaluation fidelity.
Takeaways & Limitations
The evaluation covers three domains and text-based benchmarks, and CAD requires multiple model sizes within a family, limiting applicability to single-variant proprietary models.
Abstract
from arXiv · showhide
The rapid proliferation of benchmarks for evaluating large language models (LLMs) has created an urgent need for systematic methods to assess benchmark quality itself. We propose Benchmark^2, a comprehensive framework comprising three complementary metrics: (1) Cross-Benchmark Ranking Consistency, measuring whether a benchmark produces model rankings aligned with peer benchmarks; (2) Discriminability Score, quantifying a benchmark's ability to differentiate between models; and (3) Capability Alignment Deviation, identifying problematic instances where stronger models fail but weaker models succeed within the same model family. We conduct extensive experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains, evaluating 11 LLMs across four model families. Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction based on our metrics can achieve comparable evaluation performance with substantially reduced test sets.
1 Introduction
BENCHMARK2 frames benchmark quality assessment as a necessary complement to evaluating LLMs themselves, addressing inconsistent rankings, weak discrimination, and rank-inconsistent items. It introduces three metrics and shows that quality-based instance selection can preserve evaluation performance with substantially less data.
- Benchmark quality matters because benchmarks guide research and deployment decisions, yet their reliability is rarely questioned despite the proliferation of available evaluations.
- Existing benchmarks can produce conflicting model rankings, compress performance differences, or contain items where weaker models outperform stronger ones.
- BENCHMARK2 proposes CBRC for alignment with external benchmarks, DS for model-separation ability, and CAD for detecting violations of within-family capability hierarchies.
- The study evaluates benchmark quality across 15 benchmarks and 11 LLMs spanning four model families and reports substantial variation among existing benchmarks.
- 35% of the original data suffices for comparable evaluation performance after filtering instances with the proposed quality metrics.
2 Related Work
Prior work identifies problems in benchmark-driven evaluation, including contamination, saturation, statistical uncertainty, and overreliance on single-number metrics. BENCHMARK2 complements these efforts with quantitative metrics specifically targeting benchmark reliability, discriminability, and capability alignment.
- Existing research highlights benchmark-driven research risks and the need for more robust evaluation practices.
- Related work addresses data contamination, benchmark saturation, statistical significance in model comparisons, and limitations of single-number metrics.
- HELM advances holistic evaluation across multiple dimensions, while BENCHMARK2 focuses specifically on assessing benchmark quality.
- BENCHMARK2 contributes systematic quantitative measures of benchmark reliability, discriminability, and capability alignment rather than proposing another evaluation paradigm.
3 Methodology
The methodology evaluates benchmark quality through complementary measures of cross-benchmark agreement, model discriminability, capability alignment, and selective-evaluation stability. It also defines within-family hierarchies and uses them to identify problematic instances and assess reduced benchmarks.
- Framework: The framework models benchmark quality using model scores across benchmarks and proposes complementary quality-assessment approaches.The approaches assess ranking consistency, discriminability, and capability alignment.
- Cross-Benchmark Ranking Consistency: CBRC measures how closely a benchmark’s model ranking correlates with other benchmarks in the same domain using Kendall’s τ.Values range from -1 to 1; CBRC > 0.7 indicates high consistency, 0.4–0.7 moderate consistency, and < 0.4 low consistency.
- Discriminability Score: DS combines normalized score spread with the proportion of model pairs showing practically significant differences.The minimum meaningful difference is set to ϵ = 0.02; higher DS indicates better discriminability, with DS > 0.4 considered good and DS < 0.2 minimal.
- Capability Alignment Deviation: CAD evaluates instance-level inversions where stronger models fail but weaker models succeed, using within-family capability hierarchies rather than global model ordering.The hierarchy orders models by parameter count within each family, and the inversion rate is transformed exponentially so higher CAD indicates better alignment.
- Capability Alignment Deviation: The CAD transformation uses λ > 0 to control sensitivity, with λ = 12 in the experiments; CAD > 0.6 indicates good quality, 0.4–0.6 acceptable quality, and < 0.4 significant issues.The score ranges from 0 to 1, where 1 indicates perfect alignment and values approaching 0 indicate severe hierarchy violations.
- Stability Score: Stability measures whether selective benchmark evaluation preserves model rankings across bootstrap samples of the selected instances.The procedure uses K = 100 iterations, samples r · |B| instances each time, and compares rankings with Kendall’s tau; Stability > 0.7 is high, 0.5–0.7 moderate, and < 0.5 low.
4 Experimental Setup
The experiments cover 15 benchmarks across mathematics, general reasoning, and knowledge and understanding, evaluated on 11 models from four families. The setup uses diverse model lineages, within-family comparisons, held-out validation, and standardized reproducible evaluation procedures.
- Benchmark Coverage: The study evaluates 15 benchmarks spanning mathematics, general reasoning, and knowledge and understanding.Each domain contributes five benchmarks.
- Model Coverage: The evaluation includes 11 models from four families with clear capability hierarchies based on model size.The families are DeepSeek-R1-Distill-Qwen, Llama-3.1-Instruct, Qwen2.5-Instruct, and Qwen3.
- Model Coverage: Model selection provides architectural diversity and 1–3 within-family comparison pairs, yielding 10 pairs per benchmark instance.This design supports capability alignment analysis across multiple scales.
- Validation: Held-out validation uses Qwen2.5-Base models that were excluded from computing CAD, DS, and CBRC.These base models share architecture with Qwen2.5-Instruct but differ fundamentally in training because they lack instruction tuning.
- Evaluation Protocol: Evaluation follows benchmark-specific standardized prompts, greedy decoding, and exact-match or execution-based scoring where appropriate.These choices target reproducibility while respecting original benchmark specifications.
5 Results
Results show substantial quality variation across benchmarks and a trade-off between discriminability, alignment, and stability. Selecting approximately 35% of instances with combined quality criteria preserves model rankings while improving the balance of evaluation properties and generalizing to held-out models.
- Overall Quality Patterns: Benchmark quality varies substantially across domains, with mathematics spanning BQS 0.55–0.79 and knowledge benchmarks spanning BQS 0.51–0.58.General reasoning exhibits a quality-discriminability trade-off rather than a uniformly strong profile.
- Domain Profiles: AIME 2024 achieves DS = 0.74 and CAD = 0.85, whereas MATH-500 shows low discriminability at DS = 0.16.Within general reasoning, ARC reaches CAD = 0.87 while BBH reaches DS = 0.25 and SIQA has CAD = 0.23.
- Quality Patterns: High discriminability and high capability alignment rarely co-occur, while objective evaluation criteria consistently achieve higher CAD scores.These patterns indicate that benchmark quality dimensions capture distinct properties.
- Model Performance: Within-family performance increases with model scale, including DeepSeek averages from 28.5% (1.5B) to 47.8% (7B) to 63.1% (32B).Comparable scaling patterns occur for Llama, Qwen2.5, and Qwen3.
- Selective Evaluation: Using 35% of the original instances yields average Kendall’s τ = 0.93 while preserving relative model ordering and causing mostly minor rank shifts.The positive selective-benchmark deltas for larger models indicate stronger differentiation among capable models; shifts occur mainly among mid-tier models.
- Held-Out Validation: Held-out validation preserves mathematics rankings perfectly with Avg |∆Rk| = 0.0, while General Reasoning and Average score 1.0 and Knowledge & Understanding scores 1.3.The 1.5B held-out model retains rank 14 across domains.
- Selection Ratio: The 35% selection point balances ranking τ of 0.93, stability of 0.69, and DS = 0.47, improving stability over the full benchmark’s 0.59.Ranking consistency rises from 0.88 at 10% to 0.93 at 35%, while discriminability decreases as more instances are added.
- Metric Ablation: The CAD+DS combination achieves ranking consistency 0.93, stability 0.69, and discriminability 0.47, balancing objectives better than either metric alone.CAD-only selection gives stability 0.61 and DS 0.32, whereas DS-only selection gives DS 0.48 and stability 0.50.
6 Discussion and Future Directions
The framework identifies benchmark quality differences and supports selective construction using complementary quality metrics. It also highlights circularity concerns in cross-benchmark comparison and scope limits for CAD.
- Benchmark quality findings: AIME 2024 achieves the highest reported BQS at 0.79, while AIME, OmniMath, and OlympiadBench exceed 0.7 in DS.These benchmarks combine broad score ranges with high discriminability.
- Recommendations: DS > 0.2 and CAD > 0.6 are recommended minimum thresholds for benchmark developers.The framework also recommends objective evaluation, selective construction with CAD+DS, and monitoring family-specific CAD variation.
- Methodological considerations: CBRC faces circularity concerns because benchmark quality is assessed using other benchmarks as references.The authors mitigate this by using widely adopted references, aggregating across benchmarks, and complementing CBRC with CAD and DS.
- Methodological considerations: CAD requires multiple model sizes within a family, limiting applicability to single-variant proprietary models.Held-out validation across four model families supports generalization while preserving reliable capability ordering.
7 Conclusion
BENCHMARK2 evaluates LLM benchmark quality with three complementary metrics and finds substantial quality variation across widely used benchmarks. Selective construction preserves evaluation fidelity using only 35% of original instances.
- Framework: BENCHMARK2 combines Cross-Benchmark Ranking Consistency, Discriminability Score, and Capability Alignment Deviation.Together, these metrics assess benchmark reliability, model differentiation, and within-family capability ordering.
- Selective construction: 35% of original instances suffice for selective benchmark construction while maintaining evaluation fidelity.The reduced benchmarks provide greater efficiency than full benchmarks.
- Findings: Evaluation across 15 benchmarks and 11 models reveals significant quality variations among widely used benchmarks.The study presents the framework as a tool for assessing benchmark reliability.
Limitations
The study’s validation is limited to selected domains, text-based benchmarks, and 11 models across four families. Broader domains, modalities, and model coverage remain future work.
- Domain scope: The evaluation covers mathematics, reasoning, and knowledge understanding, while validation in code generation, translation, and dialogue remains future work.The metrics are domain-agnostic by design, but additional-domain validation is still needed.
- Modality scope: The study evaluates text-based benchmarks, leaving vision-language, audio-language, and video-understanding benchmarks for future extension.This limitation reflects the growing prevalence of multimodal language models.
- Model scope: The evaluation includes 11 models across four families, and broader coverage including proprietary systems would improve generalizability.The stated limitation concerns the breadth of model representation.
Ethics Statement
The study uses public benchmarks and models, standardized evaluation tooling, and fixed inference settings. Its methods include CAD transformation and BQS aggregation, followed by performance comparisons across domains.
- Ethics and study setting: The work evaluates existing public benchmarks and models without new data collection or human subjects research.Models are accessed through official APIs or publicly released weights.
- Evaluation infrastructure: EvalScope provides standardized benchmark implementations and consistent evaluation protocols, while vLLM supports model deployment and inference.These frameworks are used throughout the experiments.
- Inference configuration: Greedy decoding and a maximum of 16384 new tokens are used for reproducibility and long-form reasoning outputs.Experiments run on NVIDIA A100 80GB GPUs, with deployment varying by model size.
- CAD transformation: CAD(Bi) = e^−λ·inv_rate(Bi) transforms raw inversion rates into interpretable scores, with λ = 12 selected from criterion-based analysis.The selected value achieves a total score of 0.68 in the reported analysis.
- BQS aggregation: BQS aggregates normalized CBRC, DS, and CAD, assigning CAD weight 0.4 and CBRC and DS weights 0.3 each.CBRC is normalized from [−1, 1] to [0, 1] before aggregation.
- Performance results: Qwen3-32B achieves 87.0% on MATH-500, 67.2% on AMC 22-24, 85.7% on DROP, and 95.0% on ARC.These results are reported across mathematics and general reasoning benchmarks.
E Statistical Reliability Analysis
Bootstrap confidence intervals quantify uncertainty in all metrics, revealing different stability profiles across CBRC, CAD, and DS.
- 95% confidence intervals are computed for all metrics using bootstrap sampling with 1000 iterations.Table 14 reports the resulting intervals across benchmarks.
- CBRC estimates show moderate uncertainty, with typical confidence-interval widths of 0.3–0.5.
- CAD estimates are more stable, with typical confidence-interval widths below 0.1 because CAD aggregates many instance-level comparisons.
- The DS metric is more variable, particularly for smaller benchmarks such as AIME 2024, whose confidence interval is [0.54, 1.19].
F Cross-Benchmark Correlation Analysis
Cross-benchmark correlations vary by domain but are generally strong among related evaluation instruments. Mathematics includes both near-perfect benchmark alignment and moderate correlations for AIME 2024, while reasoning and knowledge show strong pairs.
- Mathematics: Mathematics benchmarks generally correlate strongly, with MATH-500 and AMC 22-24 at τ = 0.88 and OlympiadBench and OmniMath at τ = 0.99.
- Mathematics: AIME 2024 has moderate correlations with other mathematics benchmarks, at approximately τ ≈0.62–0.71, reflecting its unique difficulty level.
- General Reasoning: DROP and BBH show the highest general-reasoning correlation at τ = 0.85, while SIQA and CommonsenseQA align at τ = 0.80.
- Knowledge & Understanding: The Knowledge & Understanding domain has a relatively uniform correlation structure, including τ = 0.80 between IFEval and EQ-Bench.