Source-linked AI summary
The Benchmark Lottery
Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, Oriol Vinyals
TL;DR
The paper examines how accepted benchmark choices can shape perceived algorithmic superiority beyond fundamental algorithmic differences. It introduces the benchmark lottery, analyzes task-selection effects across machine-learning evaluation settings, and discusses ways to make benchmarking more robust.
Problem
Accepted benchmarks can influence research direction and comparisons, while differing tasks, metrics, and aggregation choices make algorithmic evaluations fragile.
Method
The paper frames benchmark evaluation through the benchmark lottery, analyzes task-selection and benchmark-suite effects, and discusses benchmark-design practices across machine-learning domains.
Results
The analyses show that selecting different benchmark tasks or subsets can substantially alter algorithms’ relative performance and favor models aligned with accepted evaluation setups.
Takeaways & Limitations
Benchmark results should be interpreted as reflecting benchmark-task choices and values, not solely fundamental algorithmic superiority.
Takeaways & Limitations
Strong empirical evidence for model superiority requires repeated trials across sources of variation, but obtaining it can be prohibitively expensive.
Abstract
from arXiv · showhide
The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of "a benchmark lottery" that describes the overall fragility of the ML benchmarking process. The benchmark lottery postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior. On multiple benchmark setups that are prevalent in the ML community, we show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks, highlighting the fragility of the current paradigms and potential fallacious interpretation derived from benchmarking ML methods. Given that every benchmark makes a statement about what it perceives to be important, we argue that this might lead to biased progress in the community. We discuss the implications of the observed phenomena and provide recommendations on mitigating them using multiple machine learning domains and communities as use cases, including natural language processing, computer vision, information retrieval, recommender systems, and reinforcement learning.
1 Introduction
The paper frames benchmark evaluation as a lottery in which task selection, community forces, benchmark reuse, and field-specific practices can shape which algorithms are perceived as superior. It analyzes these influences and proposes recommendations for more accurate model comparisons.
- Benchmarks encode judgments about which tasks and values matter, and community adoption can tie algorithmic development to those success metrics.
- The benchmark lottery describes how factors beyond algorithmic superiority influence which methods emerge as successful or are perceived as better.
- Community forces and task-selection biases may overemphasize some model types and hinder others if left unchecked.
- Selecting different subsets of tasks in benchmark suites can substantially change which algorithms win, while averaging favors well-rounded models.
- Benchmark effects differ across fields: established NLP and computer-vision benchmarks contrast with freer evaluation practices in recommender systems and reinforcement learning.
- The paper examines benchmark life cycles, task selection, community bias, stateful reuse, benchmark fitting, and recommendations for mitigating these issues.
2 Background
The paper situates benchmark evaluation within the difficulty of measuring progress and the possibility that perceived algorithmic superiority reflects evaluation choices. It introduces benchmark life cycles and examines how benchmark structure can shape research directions.
- Measuring progress requires choices about evaluation setups, tasks, data, and benchmarks, making algorithmic superiority difficult to establish with full confidence.
- Models may be perceived as superior because benchmarking factors influence which ideas receive follow-up research, tuning, and adaptation.
- Life of a benchmark: A benchmark can begin as an example task or shared task, gain traction through submissions and leaderboards, and later mature, degenerate, or be deprecated.
- Life of a benchmark: Figure 1 represents ranking disagreement across SuperGLUE task subsets: the x-axis gives subset size, lines give Top-k values, and A/B gives unique rankings over possible combinations.
3 Task selection bias
The paper shows that model rankings are highly sensitive to which benchmark tasks and datasets are selected. Across VTAB, LRA, and RL Unplugged, aggregated scores can disagree with rankings from subsets or individual tasks, making benchmark selection a lottery.
- Relative model performance is highly sensitive to the choice of tasks and datasets used for measurement.
- Visual Task Adaptation Benchmark (VTAB): VTAB rankings from task-category subsets do not always correlate perfectly with the full score; the structured category has Kendall correlation ≈0.7.Different subcategories can also fully disagree on the top-ranked model.
- Visual Task Adaptation Benchmark (VTAB): Individual VTAB tasks show stronger disagreement with the aggregated score, with mean Kendall correlation ≈0.60 and some negative correlations.
- Long Range Arena: In LRA, the identity of the top-three models changes frequently when the subset of evaluation tasks changes.The benchmark ranks eleven efficient Transformer models across six tasks and compares rankings for task combinations.
- RL Unplugged: For RL Unplugged, individual-task rankings widely disagree with aggregated rankings, with average Kendall correlations of ≈0.49 for median and ≈0.54 for mean Atari scores.Mean and median aggregation strategies also disagree with each other.
- Score and rank aggregation: Aggregating multiple tasks is difficult because task difficulty reflects data size, class count, distribution shift, performance range, and duplicated domains.The paper notes that simple averaging can overrepresent particular task aspects and domains.
4 Community bias
Community norms influence which benchmarks are accepted as evidence of progress, reinforcing benchmark choices beyond researchers’ formal freedom to select tasks. This community bias can favor incremental work and make performance comparisons depend on canonical tasks rather than only on algorithmic quality.
- Researchers may technically choose any dataset, but community norms determine which benchmarks are acceptable for measuring progress.These norms are implicit and lack hard rules about permissible benchmarks or tasks.
- Peer-review criticism and demands for specific datasets push researchers toward safe, canonical benchmarks such as GLUE.The paper states that benchmark canonicality need not reflect test quality, metrics, evaluation paradigms, or what a benchmark truly measures.
- Once a benchmark becomes well established, community endorsement can turn evaluation on it into an unspoken rule.
- Benchmark errors can matter in close comparisons: label errors were identified in ten widely used computer-vision, language, and audio datasets, including 6% of ImageNet validation examples.
- Community bias can make new approaches struggle against established, carefully tuned models, pushing research toward incremental work aligned with common expectations.
- GLUE example: GLUE contains eight datasets, seven of which are matching tasks requiring relationships between sequences.This formulation favors models such as Transformers with cross-attention in the encoder and may conflate problem-formulation advantages with broader language understanding.
5 Benchmarks are stateful
Benchmarks are stateful: prior submissions, test-set feedback, and accumulated task-specific practices shape later model development and can weaken the validity of comparisons. As benchmark use expands, success may reflect access to prior information and specialized recipes as well as algorithmic quality.
- Stateful benchmarks: When the number of adaptive test-set interactions becomes large, test error may no longer indicate model quality, although empirical studies report mixed evidence about adaptive overfitting in practice.The cited theoretical result states that once i ≫ N, test results are no longer an indication of model quality; other studies found less adaptive overfitting than predicted.
- Validity of reported progress: Benchmark progress can therefore create a gap between test error and population error, raising questions about whether benchmark gains represent progress on the original problem.The paper connects adaptive modeling choices to possible effects on generalization and statistical validity.
- Stateful benchmarks: Prior models’ test errors shape subsequent model development, while repeated access to holdout results can undermine the separation between training and evaluation.The paper describes test-set access, copied implementations, hyperparameters, and checkpoints as forms of buying more lottery tickets.
- Accumulated know-how: Later participants face benchmark-specific tricks, specialized strategies, and accumulated know-how that may not transfer beyond a narrow task or setup.The paper gives MNLI pre-training for RTE and STS as an example of a task-specific practice that became standard.
6 Rigging the lottery: making your own luck
The paper calls “rigging the lottery” the distortions that arise when researchers can choose inconsistent tasks, datasets, metrics, or evaluation protocols. Examples from recommender systems and reinforcement learning show that both absent standards and divergent setups can make comparisons difficult and favor methods suited to selected evaluations.
- Recommender systems: In recommender systems, the lack of canonical test splits, metrics, and evaluation methods makes ranked comparisons difficult across papers.Common datasets such as MovieLens and Amazon Reviews are used with different evaluation choices, sometimes producing contradictory results.
- Recommender systems: Different recommender-system goals and user reactions require distinct evaluation setups, further complicating comparisons across platforms and methods.The paper distinguishes retrieval, ranking, and multitask recommendation settings and notes that user reactions can differ across algorithms.
- Absence of consensus: Without a unified benchmark, researchers may select experimental setups that fit their models, potentially hindering progress while creating an illusion of progress.The paper presents this as the opposite risk of standardized benchmarks that can bias progress.
- ALE evaluation inconsistencies: ALE studies vary in episode termination, skipframe and evaluation schedules, summary statistics, and stochasticity injection, making direct comparisons difficult.The paper notes that some results omit variance or standard error, while random-seed variation can produce statistically different distributions in reinforcement learning.
7 What can we do?
The paper recommends shared guidelines for creating and using benchmarks, broader evaluation practices, and support for rigorous testing to reduce benchmark-induced distortions. It also identifies computational cost, variance estimation, overfitting, and context-dependent model choice as important constraints.
- Guidelines for creating benchmarks: Benchmark guidelines should standardize dataset selection, metrics, goals, and reporting practices because these choices can shape research directions and benchmark outcomes.The paper emphasizes that datasets and metrics are essential benchmark components whose careless selection can contribute to task-selection bias and benchmark rigging.
- Model selection: Benchmark rankings should account for simplicity, adaptability, and context because top-scoring models are often overly complex or specialized.The paper rejects the assumption that one ideal model exists independently of application requirements.
- Evaluation resources: Rigorous evaluation is often limited by high computational costs, including the expense of analyzing many performance-variance sources.The paper suggests reproducibility challenges and conference tracks that provide expertise, infrastructure, and computational resources.
- Guidelines for benchmark usage: Clear usage guidelines can reduce inconsistent evaluation setups and help limit researchers’ ability to rig benchmarks through selective choices.The proposed guidance includes specifying evaluation setups and reporting requirements.
- Statistical significance and benchmark reuse: Single fixed test sets and limited split analysis can underestimate variance, producing false positives about model improvements, although ImageNet evidence provides a counterexample.The paper also discusses creeping overfitting and recommends changing evaluation datasets, withholding test sets, and limiting test-set queries.
8 Epilogue
Benchmarks and datasets have supported substantial machine learning progress, but their flaws can create benchmark-induced model selection bias. The paper presents the benchmark lottery as a framework for recognizing these risks and motivating further study.
- Epilogue: Benchmarks and datasets have been responsible for much recent progress in machine learning, while static and inflexible benchmarks can create risks for future research.The epilogue frames the benchmark lottery as one narrative for understanding benchmark-induced model selection bias and calls for dedicated studies of related issues.
A SuperGLUE: Ranking of models on different combinations of tasks
SuperGLUE model rankings vary across combinations of tasks, with no strong correspondence to rankings on the full benchmark. Thus, the model ranked first overall does not consistently rank first on task subsets.
- Task-combination rankings: SuperGLUE rankings show no strong pattern across different task combinations relative to the full benchmark.Models are ordered by their rank when evaluated on all tasks, then compared across task combinations.
- Task-combination rankings: The model ranked first on all SuperGLUE tasks does not consistently perform best on other task combinations.This illustrates why selecting the top-ranked model as an absolute winner can be misleading.
B SuperGLUE: Rank correlation between SuperGLUE score and individual tasks
SuperGLUE’s mean score only imperfectly tracks individual-task rankings, and selecting different task subsets can substantially change which model appears strongest.
- Rank correlation: 0.648 average Kendall rank correlation links the eight individual SuperGLUE tasks with the benchmark’s mean score across 55 models.The correlation is imperfect, and six of eight tasks have different winning models.
- Practical implication: Using the SuperGLUE mean score to select a model for adaptation can be sub-optimal for a practitioner’s specific application context.The task-level disagreement means the best choice depends on which task matters for the application.
- Task-selection effects: Different SuperGLUE task selections produce very different relative rankings among model architectures.Universal, Mixture-of-Experts, Adaptive Softmax, and Switch models take turns ranking first across configurations.
- Task-selection effects: Lightweight Convolutions, Dynamic Convolutions, and Transparent Attention never appear in any Top-5 ranking across the reported configurations.Funnel and Weighted models appear only rarely.
D VTAB: Agreement on top-ranked models across sub-categories and tasks
VTAB model rankings disagree across both sub-category combinations and task subsets, with many distinct winners emerging from different selections.
- Evaluation setup: The VTAB analysis evaluates disagreement across three sub-categories—Natural, Specialized, and Structured—and 19 tasks.The number of selected categories or tasks is the independent variable in Figure 6.
- Ranking disagreement: VTAB rankings disagree across sub-category and task subsets, including disagreement among the top-1, top-2, and top-3 models.Figure 6 varies the number of selected sub-categories or tasks and tracks unique model rankings.
- Task subsets: 12 different models win among the 19 individual VTAB tasks, while 20 different winners emerge across two-task subsets.The two-task result corresponds to 20/171 distinct winner outcomes.
E LRA: Comparing the top-scoring models on all possible task subsets
The LRA analysis reports Top-1, Top-2, and Top-3 model rankings for every possible task-subset combination, extending the earlier subset comparison to the complete set of combinations.
- Complete task-subset analysis: Top-1, Top-2, and Top-3 LRA results are reported for all possible combinations of benchmark tasks.The complete results appear in Table 3 and correspond to the full version of an earlier table.
- Ranking comparison: Table 3 compares which models occupy the top three positions as the selected LRA task subset changes.Its stated purpose is to show the top three performers conditional on task selection.
F Popular public benchmarks for evaluating recommend systems
The cited materials identify publicly available datasets used for recommender-system evaluation and present them as a list of popular offline benchmarks.
- Public evaluation datasets: Table 4 lists publicly available datasets used by the community to evaluate recommender systems.The table is described as covering popular offline datasets.
G Benchmarking checklist for the review process
The section proposes a reviewer and area-chair checklist intended to reduce benchmark lottery effects. It asks reviews to examine evaluation choices, variance, significance, practical benefits, and constraints beyond headline performance.
- Benchmarking checklist for reviewers and area chairs: The proposed checklist is intended to reduce benchmark lottery effects during the review process.It frames checklist use as a response to benchmark effects rooted in multiple aspects of evaluation.
- Benchmarking checklist for reviewers and area chairs: Reviews should justify requests for particular baselines, tasks, or benchmarks beyond treating datasets as mandatory.
- Benchmarking checklist for reviewers and area chairs: Evaluation should consider efficiency, fairness, and simplicity alongside commonly reported metrics such as accuracy.
- Benchmarking checklist for reviewers and area chairs: Reviewers should assess criticism of methods that underperform on only a subset of tasks and methods that deviate from current trends.The checklist asks whether such criticism has logical elaboration and rational justification.
- Benchmarking checklist for reviewers and area chairs: Reviews should examine variance across multiple splits or random seeds, appreciate significance analyses, and account for computational limits on additional experiments.The checklist also asks whether improvements are secured by tricks unrelated to the main contributions.