PaperlayerView exact revision on arXiv

Source-linked AI summary

BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices

Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, Mykel J. Kochenderfer

arXiv:2411.12990v1cs.AIcs.LG

TL;DR

AI benchmarks support high-stakes assessment, but their quality and usability vary, complicating interpretation of model performance. This paper evaluates 24 benchmarks against a 46-criterion lifecycle framework and finds substantial quality differences, limited statistical reporting, and poor reproducibility, while offering a checklist and repository to improve comparability.

  • Problem

    AI benchmarks are widely used to assess models and inform downstream selection and policy, but their design and usability vary in quality.

  • Method

    The paper develops a framework of 46 best practices across the benchmark lifecycle and evaluates 24 AI benchmarks against it.

  • Results

    MMLU scored 5.5 and GPQA 11.0 in the assessment, while most benchmarks also lacked statistical significance reporting and easy result replication.

  • Takeaways & Limitations

    The paper provides a best-practice checklist and living assessment repository to support minimum quality assurance and benchmark comparability.

  • Takeaways & Limitations

    The assessment equally weights criteria despite differences in effort and contribution to overall benchmark quality, and its four score categories may miss nuances.

Abstract

from arXiv · show

AI models are increasingly prevalent in high-stakes environments, necessitating thorough assessment of their capabilities and risks. Benchmarks are popular for measuring these attributes and for comparing model performance, tracking progress, and identifying weaknesses in foundation and non-foundation models. They can inform model selection for downstream tasks and influence policy initiatives. However, not all benchmarks are the same: their quality depends on their design and usability. In this paper, we develop an assessment framework considering 46 best practices across an AI benchmark's lifecycle and evaluate 24 AI benchmarks against it. We find that there exist large quality differences and that commonly used benchmarks suffer from significant issues. We further find that most benchmarks do not report statistical significance of their results nor allow for their results to be easily replicated. To support benchmark developers in aligning with best practices, we provide a checklist for minimum quality assurance based on our assessment. We also develop a living repository of benchmark assessments to support benchmark comparability, accessible at betterbench.stanford.edu.

1 Introduction

AI evaluation is increasingly important as systems enter consequential settings, but benchmark quality varies and lacks structured comparative assessment. BetterBench addresses this gap with a 46-criterion framework, evaluations of 24 benchmarks, developer guidance, and a living repository.

  • Motivation: Benchmarks support model comparison, performance measurement, progress tracking, weakness identification, model release reporting, and policy evaluation.Their usefulness depends on benchmark quality, including interpretability, clarity of purpose, and usability.
  • Contributions: BetterBench introduces a 46-criterion framework for assessing AI benchmark quality across the benchmark lifecycle.The criteria were derived from expert interviews and domain literature.
  • Contributions: The study evaluates 16 foundation-model and 8 non-foundation-model benchmarks, finding quality differences across both categories.
  • Contributions: The paper identifies prevalent benchmarking issues and provides a minimum-quality checklist to help developers align with best practices.
  • Contributions: A living repository of benchmark assessments enables users to analyze benchmark quality and appropriateness for their usage contexts.

2 Related Work

Prior work identifies concerns about what benchmarks measure, how evaluations are used, and whether benchmark practices support valid, interpretable, accessible assessments. BetterBench extends these efforts into a lifecycle-spanning framework with practical improvement tools.

  • AI Benchmarking Practices and Challenges: Existing research highlights concerns about benchmark validity, narrow scope, saturation, and the tendency to overgeneralize reductive benchmark results.
  • AI Benchmarking Practices and Challenges: Data curation, documentation, and FAIR principles support evaluation best practices but do not provide a framework for assessing AI benchmarks themselves.
  • AI Benchmarking Practices and Challenges: BetterBench assesses foundation-model and non-foundation-model benchmarks across design, implementation, documentation, and maintenance.Its criteria include domain-expert involvement, evaluation-script availability, licensing, and user feedback channels.
  • Benchmarking Best Practices in Other Fields: Best practices from hardware, environmental quality, and bioinformatics benchmarking emphasize downstream utility, validity, interpretability, and accessibility.

3 Methodology

The methodology combines literature review, stakeholder interviews, lifecycle-based criteria, validation, scoring, and continuous publication of benchmark assessments. It applies the resulting procedure to 24 benchmarks and maintains an updateable public platform.

  • Methodology: The assessment defines 46 criteria from literature review and interviews with five stakeholder groups, organized around a five-stage benchmark lifecycle.The interviews included 20+ policymakers, model developers, benchmark developers, model users, and AI researchers.
  • Methodology: The team translated lifecycle tasks and stakeholder objectives into criteria distinguishing developer-controlled, context-dependent, and externally constrained aspects.
  • Methodology: Three authors independently scored one benchmark for calibration, clarified scoring guidelines, and revised the assessment using stakeholder feedback.
  • Assessment Scope: The evaluation covers 16 foundation-model and 8 non-foundation-model benchmarks, prioritizing commonly used benchmarks and developer-released information sources.
  • Scoring: Each criterion receives 0, 5, 10, or 15 points, with stage averages and a weighted usability score; a mean of 10 or higher indicates reasonable quality.
  • Continuous Updates: A supplementary website continuously publishes assessments, accepts community feedback, and supports an up-to-date database for selecting benchmarks by application context.

4 Assessment Criteria

The framework organizes 46 assessment criteria across benchmark lifecycle stages, covering design, implementation, documentation, and maintenance. It emphasizes clear goals, reproducible infrastructure, accessible documentation, and ongoing support.

  • The framework separates assessment criteria by benchmark lifecycle phase, excluding retirement because active benchmarks cannot be assessed for retirement.The four assessed stages are design, implementation, documentation, and maintenance.
  • Benchmark Design: Benchmark design criteria require clear goals and scope, including the tested capability, its task translation, and its real-world relevance.These choices should incorporate use cases, user personas, domain experts, and domain literature.
  • Benchmark Implementation: Implementation criteria emphasize working evaluation code, accessible evaluation materials, reproducibility scripts, and support for both API-based and local models.The framework also considers safeguards against contamination and gamification, including unique identifiers and encrypted instances.
  • Benchmark Documentation: Documentation criteria require accessible information about benchmark construction, data collection or test environments, tasks, rationales, metrics, and statistical significance.These elements support practical use and interpretation of benchmark results.
  • Benchmark Maintenance: Maintenance criteria call for regular usability checks, user feedback channels, responsible contact information, or an explicit retirement statement.Without maintenance, even a well-designed, implemented, and documented benchmark can cease to be useful.

5 Other Design Considerations

The paper identifies context-dependent design choices that were excluded from scoring but should receive deliberate consideration. These include benchmark scope, sensitivity to small improvements, modality, versioning, dynamics, gameability, and positionality.

  • General vs. specific benchmarks: Benchmark developers must choose between general concepts and specific domains because broad benchmarks may reveal foundational characteristics but face real-world applicability and reliable-testing challenges.
  • Detecting small improvements: Benchmarks should reliably detect 1% improvements, with evaluation-data size and task diversity likely affecting sensitivity to small progress.
  • Multi-modal assessment: Multimodal benchmarks can assess capabilities across modalities but increase complexity in mapping concepts to modalities and handling output formats.
  • Versioning: Minor benchmark changes should be versioned clearly, while major updates should receive new releases.The paper gives AgentBench v0.1 and v0.2 as an example of major-version updates.
  • Dynamic vs. static benchmarks: Dynamic benchmarks may reduce saturation and contamination but sacrifice result comparability, while static benchmarks retain those vulnerabilities.The trade-off depends partly on the evaluation setting and task.
  • Gameability: Benchmarks should resist performance gains that do not reflect improved underlying capability, because existing evaluations can be manipulated.
  • Positionality statement: Developers may provide positionality statements when background, experiences, or biases significantly influenced benchmark construction.These statements add context and transparency but were not included in the assessment.

6 Quantitative Results

Across 24 assessed benchmarks, implementation was generally the weakest lifecycle stage, and reproducibility and statistical interpretation were especially underdeveloped. Design and usability scores were related for foundation-model benchmarks and the full sample, while mean design and usability differences between benchmark categories were not statistically significant.

  • Average scores were 3.75 for including a replication script and 5.62 for reporting statistical significance, revealing weaknesses in reproducibility and result interpretation.
  • Implementation was the weakest lifecycle stage for both foundation-model and non-foundation-model benchmarks, followed by maintenance.
  • The overall average design score was 10.7 and the weighted average usability score was 8.7 across all assessed benchmarks.Usability was defined as the weighted average of implementation, documentation, and maintenance scores.
  • Design and usability score differences between foundation-model and non-foundation-model benchmarks were not statistically significant at the 95% confidence level.
  • Statistically significant correlations linked design and usability scores for foundation-model benchmarks alone and for all benchmarks combined.The paper reports these correlations at the 95% confidence level.

7 Discussion

The assessment reveals substantial quality differences and recurring weaknesses across AI benchmarks, especially in statistical reporting and reproducibility. The authors argue that practical guidance and modest improvements can raise benchmark quality and usability.

  • MMLU scored 5.5 versus GPQA’s 11.0, illustrating substantial quality differences that are often not acknowledged when developers report benchmark results.The paper notes that both benchmarks appear in model communications and evaluation frameworks without explicit discussion of their differing quality or limitations.
  • 14 of 24 benchmarks omitted repeated model evaluations, statistical significance, or uncertainty reporting needed to distinguish genuine model differences from noisy results.Intra-model variance helps users determine whether observed inter-model differences reflect performance or noise.
  • 17 of 24 benchmarks lacked easy-to-run scripts for reproducing the initial paper’s results, while 4 provided scripts for only part of the results.The paper links this implementation weakness to reduced reproducibility and limited scrutiny of benchmarking processes.
  • Documentation and a point of contact are relatively easy additions that can improve benchmark usability, accountability, and ease of use.The paper presents these changes as examples of small interventions with potentially significant practical benefits.
  • The authors provide a checklist of best practices to help benchmark developers establish minimum quality standards.The checklist responds to strong quality discrepancies and developers’ limited knowledge of what constitutes a good benchmark.

8 Open Challenges in AI Benchmarking

Open challenges constrain the validity, usefulness, and comparability of AI benchmarks beyond what benchmark developers can address alone. These challenges include saturation, contamination, construct validity, and inconsistent reporting standards.

  • Rapid AI progress can saturate benchmarks within months, requiring checks of current model performance and whether the benchmark remains challenging.The paper frames saturation as an open problem requiring assessment of whether the underlying concept has already been solved.
  • Benchmark contamination remains a shared responsibility because benchmark developers cannot enforce model developers’ use of safeguards such as canary strings.Contamination has been demonstrated in both foundation-model and non-foundation-model settings, so cross-stakeholder work is needed to preserve benchmark validity.
  • Construct validity is difficult when properties such as factual accuracy arise from interactions between models and users rather than models alone.The paper notes that ecologically valid user interactions are difficult to incorporate because model developers provide limited transparency into actual user behavior.
  • Most benchmarks provide only relative signals because construct-validity difficulties prevent absolute measurement, while reporting practices remain unstandardized.Model developers can report whichever benchmarks they choose without explaining their rationale, producing inconsistent reporting.

9 Limitations

The assessment uses equal weighting and coarse score categories, which can obscure differences in criterion fulfillment and benchmark quality. Evaluating construct validity itself is outside the assessment’s scope.

  • 9 Limitations: Equal weighting and four score categories may miss nuances and allow developers to focus on easily fulfilled criteria.A benchmark barely meeting a criterion receives the same 10-point score as one almost entirely fulfilling it.
  • 9 Limitations: Determining whether a benchmark validly assesses its target concept requires domain-expert analysis beyond this assessment’s scope.

10 Impact Statement

The framework and checklist aim to improve benchmark quality, transparency, and comparability, while the repository supports informed benchmark selection. The assessment provides minimum quality assurance rather than determining suitability for a specific use case, and some criteria may disadvantage less-resourced researchers.

  • 10 Impact Statement: The framework and checklist aim to encourage higher-quality, more practicable, and more transparent AI evaluations.
  • 10 Impact Statement: The living repository promotes transparency and comparability, helping users make informed benchmark choices without establishing use-case suitability.
  • 10 Impact Statement: Criteria such as involving domain experts may disadvantage researchers with fewer resources or connections, limiting diversity in benchmark creation.

NeurIPS Checklist

The checklist documents claims, limitations, impacts, reproducibility, statistical reporting, asset handling, and human-subject considerations. It records when items are inapplicable and requests justifications for answers.

  • NeurIPS Checklist: The checklist asks authors to mark each item Yes, No, or N/A and provide a justification referencing the paper or describing the answer.
  • NeurIPS Checklist: Authors report that the work describes its claims, limitations, broader impacts, and ethics-review compliance.
  • NeurIPS Checklist: The paper states that code, data, instructions, and statistical-significance results will be provided or reported where applicable.
  • NeurIPS Checklist: The checklist records asset citations, licensing choices, released assessment results, and the absence of personal-data use.
  • NeurIPS Checklist: The authors report using voluntary, uncompensated information-gathering interviews without formal participant instructions or IRB approval.

A Stakeholders

The paper maps benchmark stakeholders to lifecycle stages and describes how the assessment, checklist, and repository support their decisions. It evaluates 24 benchmarks across foundation-model and non-foundation-model categories.

  • A Stakeholders: Benchmark developers can use the checklist before deployment, the assessment to identify shortcomings, and the website to compare related benchmarks.
  • A Stakeholders: Model users can compare benchmark quality and prioritize a higher-quality benchmark when multiple results fit their application context.
  • A Stakeholders: AI researchers can use the repository to study benchmarking challenges, pursue open evaluation questions, and understand the benchmark landscape.
  • A Stakeholders: Regulators can use the checklist or assessment scores when setting benchmark requirements for model-compliance evidence.
  • A Stakeholders: The benchmark lifecycle comprises design, implementation, documentation, maintenance, and retirement stages.
  • A Stakeholders: The evaluation covers 16 foundation-model benchmarks and 8 non-foundation-model benchmarks.

D Sensitivity Analysis Details

The mean usability-score difference between foundation-model and non-foundation-model benchmarks is not statistically significant at the 95% confidence level.

  • The mean usability-score difference between foundation-model and non-foundation-model benchmarks is not statistically significant at the 95% confidence level.

E Additional Results

The paper reports lifecycle-stage scores for 24 benchmarks and aggregates implementation, documentation, and maintenance into usability scores using a discrete assessment framework.

  • The repository provides individual benchmark scores and justifications for further inspection.
  • Scores are reported for each benchmark across design, implementation, documentation, and maintenance lifecycle stages.The corresponding barplots are Figures 9–12, with benchmarks ordered ascending and labeled as foundation-model or non-foundation-model benchmarks.
  • Stage scores average applicable criterion points, weighting criteria equally within each lifecycle stage.Benchmark retirement is excluded from the assessment because the evaluated benchmarks remain actively used and are not saturated.
  • Usability is a weighted average of implementation, documentation, and maintenance scores, while design is reported separately.The weighting uses the numbers of applicable criteria in each stage.
  • Each criterion receives 0, 5, 10, or 15 points for unaddressed, acknowledged, partially addressed, or fully addressed criteria, with N/A for irrelevant criteria.The discrete scale is intended to distinguish degrees of benchmark quality while supporting quantitative analysis.

G Methodology Flow Diagram

The methodology derives benchmark best practices from literature and stakeholder input, organizes them into a lifecycle checklist, and applies explicit scoring and quality-assurance safeguards.

  • The assessment is intended as a minimum quality-assurance checklist rather than a sufficient standard for constructing high-quality benchmarks.
  • The checklist covers benchmark purpose, scope, tasks, use cases, domain expertise, literature, metrics, documentation, implementation, and maintenance.It includes criteria such as defining the tested capability, explaining task translation, documenting assumptions and limitations, and reporting statistical uncertainty.
  • The framework was tested on BetterBench itself to verify usability and provide an example of a completed checklist.
  • The repository improves transparency and comparability, but the assessment does not determine whether a benchmark suits a specific use case.
  • The framework’s best practices were derived through literature review and interviews with more than 20 policymakers, developers, users, and researchers.The process identified five stakeholder groups and developed a five-stage benchmark lifecycle model.
Loading 2411.12990v1…