Source-linked AI summary
Lessons from the Trenches on Reproducible Evaluation of Language Models
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, Andy Zou
TL;DR
Reliable LM evaluation is undermined by setup sensitivity, inconsistent comparisons, and insufficiently documented practices. Drawing on three years of developing lm-eval, the paper codifies challenges, recommends best practices, and shows how to operationalize them in reproducible evaluation infrastructure. The authors argue that rigorous evaluation requires looking beyond isolated benchmark scores and reporting the methodological details and uncertainty that shape them.
Problem
LM evaluation lacks reliable, reproducible, and transparent methods despite benchmark results influencing research, deployment, and governance decisions.
Method
The paper synthesizes practical evaluation challenges and best practices from developing lm-eval, then documents their implementation in a flexible evaluation library.
Results
The paper documents how prompt and implementation choices can substantially alter results, including ARC-Easy accuracy of 72.4 ± 1.80% with cloze evaluation versus 26.5 ± 1.78% with MMLU-style evaluation for GPT-NeoX.
Takeaways & Limitations
Meaningful LM evaluation requires contextualized scores, disclosed code and prompts, statistical analysis, and attention to the setup-specific practices that affect conclusions.
Takeaways & Limitations
Independent evaluation is constrained because some models are inaccessible, while many APIs lack log-probabilities and make comprehensive assessment expensive.
Abstract
from arXiv · showhide
Reliable evaluation of language models (LMs) remains an open challenge. Re- searchers and engineers face methodological issues such as the sensitivity of models to evaluation setup, difficulty of proper comparisons across methods, and the lack of reproducibility and transparency. Evaluation difficulties are exacer- bated by the fracturing and siloing of information about conventions and common practices. In this paper we draw on three years of experience in evaluating large lan- guage models (LMs) as developers of the popular Language Model Evaluation Harness (lm-eval) (Gao et al., 2023) framework to provide guidance and lessons for the field moving forward. We document a variety of challenges faced by prac- titioners and provide concrete instances where these challenges or the absence of best practices have come into effect. We make recommendations to the field for improving evaluation rigor and confidence, and attempt to codify much of the tacit or folk knowledge surrounding LM evaluation, for a solid ground to move forward.
1 Introduction
The paper addresses unreliable and fragmented LM evaluation by documenting practical challenges, recommending best practices, and showing how lm-eval operationalizes them. Its goal is rigorous, reproducible evaluation infrastructure that guides default practice while preserving customization.
- Motivation: Evaluation results influence research direction, model deployment decisions, and governance, making improper practices consequential.Numerical benchmark scores inform judgments about relative capabilities and suitability, but do not fully constitute model behavior.
- Contribution: The paper draws on experience maintaining lm-eval to document evaluation challenges, recommend best practices, and operationalize them in a flexible library.The framework is intentionally non-opinionated about which benchmarks to run, but provides carefully chosen default configurations for how evaluations should be conducted.
- Contribution: lm-eval is designed as research infrastructure for studying evaluation, locating de facto best practices, and enhancing reproducibility.Its design has evolved with open-source community needs and changing understanding of best practices.
- Contribution: The paper formalizes tacit evaluation knowledge by systematically presenting common challenges together with concrete examples.This effort targets implementation details and conventions that are often scattered or barely documented.
- Contribution: It provides recommendations for best practices and documents their incorporation into newer lm-eval versions to ease adoption.The library serves as an example of how the recommendations can be operationalized.
2 Why is Evaluating Language Models Hard?
LM evaluation is difficult because correctness is often hard to determine, incentives can undermine rigor, and rapidly changing model uses outpace benchmark conventions. These problems create trade-offs among evaluation approaches, restrict independent access, and fragment methodological practice.
- 2.1 The Key Problem: The Key Problem is that semantically equivalent answers can have different syntax, while the models used to judge equivalence are themselves being evaluated.The paper identifies this as a fundamental methodological challenge for LM evaluation.
- 2.1 The Key Problem: Evaluation responses to the Key Problem use closed-domain tasks, model-based grading, or human labor, each with significant trade-offs.Closed-domain methods enable mechanical checking; automated grading improves reproducibility and cost; human judgment is expensive and difficult to scale.
- 2.1 The Key Problem: The paper focuses on multiple-choice question answering because it is common in the literature and the authors have the most experience with it.The authors expect much of their analysis to apply elsewhere but encourage area-specific documentation of best practices.
- 2.2 Social Dynamics of Evaluation: Model-development incentives favor favorable evaluations, including selective task and prompting choices, while limited access can leave developers’ published numbers as the only available evidence.API-based evaluation may also lack log-probabilities and impose costs that disproportionately burden independent researchers.
- 2.2 Social Dynamics of Evaluation: Even when independent evaluation is possible, cherry-picking tasks, flattering prompts, and single-run estimates can produce technically accurate but misleading comparisons.The paper links these pressures to benchmark selection becoming an optimization target rather than solely a measure of progress.
- 2.3 Fast-changing Progress and Conventions: Many benchmarks predate in-context learning and chat interaction, so their use with current open-domain models may have unclear validity and fragmented methodology.Retrofitted benchmarks may lack ground-truth implementations, leaving practitioners to derive their own settings.
3 Evaluation Details are Lost in Communication
LM evaluation details are often lost as benchmarks are reimplemented and methodological knowledge is transmitted across researchers, making reported comparisons difficult to reproduce. The paper documents how seemingly minor choices affect scores and rankings, recommends clearer reporting and code release, and describes shared infrastructure as one way to operationalize these practices.
- Benchmark reimplementation can obscure methodological details, producing irreproducible or non-equivalent comparisons and reducing trust in others’ evaluations.
- 3.1 Challenges: Prompt wording and few-shot examples can dramatically change performance, sometimes by more than 20%, while the best prompt style varies by model.
- 3.1 Challenges: Three independent MMLU implementations produced widely different scores and changed model ranking order despite ostensibly evaluating the same benchmark.
- 3.1 Challenges: Many benchmark quirks require individual handling, including ARC’s single five-choice question, MMLU’s document-level micro average, and HumanEval’s three documents without example tests.
- 3.1 Challenges: Crucial methodological details are often insufficiently reported, leaving evaluation setups underdetermined when code is unavailable or prior work is itself undocumented.
- 3.2 Best Practices: The authors recommend releasing exact evaluation code, reporting prompts and methodological details thoroughly, and creating reporting standards.
- 3.3 Operationalizing Best Practices: lm-eval operationalizes these practices through configurable prompts and common infrastructure intended to reduce duplicated effort and increase confidence in evaluation scores.
4 Evaluations are Not Just Numbers
Benchmark scores are proxies whose interpretation depends on evaluation choices, uncertainty, and task validity rather than numbers alone. The paper shows that prompting formats and reporting practices can materially change model comparisons, and recommends statistical, realistic, and qualitative analyses.
- What, if Anything, Does a Benchmark Measure?: Benchmark scores are proxies for broader capabilities, so numerical results alone do not constitute conclusions about model behavior.The paper argues that overemphasizing benchmark numbers can hinder evaluation, research, and governance.
- Lack of Reporting on Uncertainty and Variance: Stochastic model behavior and omitted variance information can produce misleading performance pictures.The authors highlight that evaluation outcomes depend on random events and arbitrary decisions during training and evaluation.
- Lack of Reporting on Uncertainty and Variance: A single-run estimate versus a 95% confidence interval across 10 runs can yield very different conclusions about relative model performance.The same concern applies to small score increases on benchmarks such as HumanEval with only 164 examples.
- The Benchmark Lottery: On ARC-Easy, GPT-NeoX scores 72.4 ± 1.80% with cloze evaluation but 26.5 ± 1.78% with MMLU-style evaluation.Exploratory analysis attributes the at-random performance on ARC-C and MMLU to failure to follow the task and output format under MMLU-style prompting.
- Best Practices: The paper recommends reporting variance and statistical significance, evaluating models in realistic settings, and performing qualitative error analysis.These practices are presented as ways to contextualize durability, improve validity, and understand superficial versus fundamental errors.
5 Conclusion
The paper documents widespread challenges and tacit benchmarking knowledge, recommends mitigation practices, and incorporates those practices into lm-eval’s design.
- 5 Conclusion: The paper synthesizes ubiquitous LM evaluation challenges, formalizes scattered benchmarking folk knowledge, and presents recommendations for mitigating their effects.It also discusses how these practices were incorporated into lm-eval.
Checklist
The checklist records whether the paper reports reproducibility materials, limitations, ethics, theoretical assumptions, experiments, and asset-use information.
- Authors and Contributions: The paper indicates that its main claims reflect its contributions and scope, and that limitations are discussed in Appendix F and throughout the main text.It also reports discussion of potential negative societal impacts in Appendix G.
- Theoretical Results: The paper marks theoretical-result assumptions and proofs as not applicable.Both checklist items are explicitly answered N/A.
- Experiments: For experiments, the paper indicates that reproducibility materials and training details were addressed, and that 95% bootstrap confidence intervals were reported for case studies.The checklist also asks about compute resources and reports affirmative answers for those disclosures.
- Existing and New Assets: The paper reports citing directly referenced benchmark datasets and notes that lm-eval provides citations for implemented tasks not otherwise used or referenced.License, new-asset, consent, and personally identifiable information items are marked N/A.
- Human Subjects: Human-subject and crowdsourcing checklist items are marked not applicable, including participant instructions, risks, compensation, and total spending.The participant-risk item explicitly records N/A, as do the compensation-related items.
A Library Design
lm-eval is organized around modular evaluation Tasks and extensible language-model integrations, connected through standardized Requests. Tasks package data, prompting, model inputs, output processing, and metric calculation for reproducible sharing.
- lm-eval supports two extensible implementations: evaluation Tasks and integrations with novel language-model implementations.
- Tasks: Tasks use a common API so new evaluations can be implemented, collected, shared, and reproduced across practitioners.
- Tasks: Task definitions can use YAML configuration or Python subclassing, with components for data sources, prompts, Requests, post-processing, and metrics.
- LMs: The LM API lets arbitrary software libraries and autoregressive architectures extend a provided interface for model evaluation.
- LMs: Requests map string inputs to strings or probabilities, keeping tokenizers and model-specific integrations inside the LM class.
- Request Types: The framework supports conditional loglikelihood, perplexity, and generation-based Requests as three core measurement types.
- Request Types: These primitives cover major LM evaluation approaches, while subtle implementation decisions require explicit documentation for reliable comparisons.
B Formalizing Measurements
The paper formalizes common LM measurements to expose implementation choices that can materially affect evaluation. It describes token-level probability calculations, perplexity, and the complementary roles of generative and loglikelihood-based evaluation.
- B Formalizing Measurements: The appendix formalizes common LM measurement approaches because papers often omit implementation details that can vitally affect results.
- B.1 Preliminaries: An autoregressive LM maps token prefixes to vocabulary distributions, represented internally as logits and converted to log probabilities.
- B.2 Ranking-Based Multiple Choice QA: Conditional loglikelihood computes log P(y|x), the probability of a target string conditioned on an input, in a single LM call.
- B.2 Ranking-Based Multiple Choice QA: The procedure concatenates input and target tokens, extracts logits for target positions, applies log-softmax, and sums target-token log probabilities.
- B.3 Perplexity evaluation: Perplexity is the exponential of the average negative loglikelihood per token over a dataset.
- B.3 Perplexity evaluation: Perplexity evaluation tokenizes each document, computes document loglikelihood, and averages across the dataset’s total token count.
- B.4 Generative Evaluation: Loglikelihood tasks measure understanding and option ranking, but do not directly assess coherent, relevant text generation.
- B.4 Generative Evaluation: Generation-based evaluation is increasingly necessary because many language-model APIs restrict or omit access to log probabilities.
C.1 Comparisons Across Evaluation Settings
Evaluation scores can change when prompts, answer formats, or scoring continuations change, even for the same benchmark. lm-eval makes these settings explicit and easy to modify through configuration files.
- C.1 Comparisons Across Evaluation Settings: The experiment compares ARC and MMLU under two literature-used evaluation settings that vary prompts and answer strings for loglikelihood scoring.
- C.1 Comparisons Across Evaluation Settings: ARC’s cloze setup scores candidate completion strings, whereas MMLU’s original style scores generation of the correct answer letter.
- C.1 Comparisons Across Evaluation Settings: ARC-Challenge is compared using Brown et al.’s Cloze prompt versus an MMLU-style prompt with explicit multiple-choice options.
- C.1 Comparisons Across Evaluation Settings: MMLU’s Hybrid variant keeps the MMLU-style prompt but scores answer strings instead of answer letters as continuations.
- C.1 Comparisons Across Evaluation Settings: The experiment provides configuration files and uses lm-eval version 0.4.2 to support replication across ARC-Challenge and MMLU settings.
- C.1 Comparisons Across Evaluation Settings: Task configuration specifies datasets and splits, prompt mappings, target and choice mappings, metrics, aggregation, and task version metadata.
- C.1 Comparisons Across Evaluation Settings: The doc_to_* fields map examples to prompts, gold targets, and answer choices, while acc and acc_norm apply different loglikelihood normalizations.
- C.1 Comparisons Across Evaluation Settings: Configurations support inheritance and define both original and Hybrid MMLU variants through distinct prompt and continuation mappings.
D Best Practices Checklist for Language Model Evaluation
The checklist emphasizes transparent, comparable, artifact-rich, and statistically grounded LM evaluation. It pairs exact prompt and output sharing with qualitative inspection and significance testing.
- The paper recommends concrete measures to improve current LM evaluation practices despite their difficulty.
- Researchers should share exact prompts or full evaluation code, use the same prompts for comparisons, and disclose prompt engineering.
- Results from other implementations should not be copied unless the same code was verified, because experimental differences can mislead comparisons.
- Model outputs should accompany evaluation code so others can recalculate scores, test significance, and evaluate alternative metrics without equivalent compute access.
- Researchers should qualitatively inspect a small batch before scaling evaluation to catch generation bugs and understand score behavior.
- Statistical significance testing and multiple random seeds can increase the reliability, validity, and utility of reported results.
F Limitations
The paper restricts its scope and does not discuss measurement validity in great detail because of length constraints.
- The paper limits its scope and omits an in-depth discussion of measurement validity due to space constraints.
G Impacts
The authors identify no strong potential negative societal impacts and argue that healthier benchmarking practices could support safer assessment and deployment of LMs.
- The authors do not see strong potential negative societal impacts from their work.
- They believe healthier, more reliable benchmarking can improve LLMs’ societal impacts and guide careful capability assessment.
- They hope these recommendations reduce the likelihood of deploying LLMs in unsafe and unsuitable scenarios.