Source-linked AI summary
What Will it Take to Fix Benchmarking in Natural Language Understanding?
Samuel R. Bowman, George E. Dahl
TL;DR
NLU benchmarks often produce very high scores despite unreliable annotations, model failures on simple targeted cases, and socially relevant biases. This position paper proposes four criteria for valid, reliable, statistically powerful, and responsible benchmarks, arguing that these problems should be addressed directly rather than by abandoning IID evaluation. It concludes that progress requires better dataset design, annotation reliability, benchmark size, and bias handling.
Problem
Popular NLU benchmarks can yield near-human or superhuman scores while failing to reveal important model weaknesses and socially relevant biases.
Method
The paper proposes four benchmark criteria and surveys directions including expert–crowdworker collection, larger-scale validation, and auxiliary bias datasets.
Results
The paper concludes that adversarial filtering does not address the underlying benchmark problems and that they should instead be addressed directly within static, IID evaluation.
Takeaways & Limitations
Restoring useful NLU evaluation requires progress in dataset design, annotation reliability, statistical power, and safeguards against harmful social bias.
Takeaways & Limitations
The paper acknowledges that community structure and incentive design make it difficult to ensure researchers report potentially unflattering bias metrics.
Abstract
from arXiv · showhide
Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. The recent trend to abandon IID benchmarks in favor of adversarially-constructed, out-of-distribution test sets ensures that current models will perform poorly, but ultimately only obscures the abilities that we want our benchmarks to measure. In this position paper, we lay out four criteria that we argue NLU benchmarks should meet. We argue most current benchmarks fail at these criteria, and that adversarial data collection does not meaningfully address the causes of these failures. Instead, restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, the reliability with which they are annotated, their size, and the ways they handle social bias.
1 Introduction
NLU benchmarks often combine near-ceiling scores with unreliable data and known model failures, weakening their ability to measure progress. The paper proposes four criteria and argues these problems should be addressed directly rather than through adversarial filtering alone.
- 1 Introduction: State-of-the-art systems approach maximum benchmark scores despite serious known issues, leaving little room to measure further improvements.The benchmarks use roughly IID splits and commonly rely on crowdsourced data with longstanding quality concerns.
- 1 Introduction: A trustworthy evaluation ecosystem requires benchmarks whose performance implies robust in-domain task performance and whose examples are accurately and unambiguously annotated.The proposed criteria call for stronger dataset design, collection, and test-set validation.
- 1 Introduction: Adversarial filtering selects examples that machine-learning models misclassify, producing short-term challenges for current systems but not necessarily useful benchmarks.The approach begins with candidate examples, often crowdsourced, and retains cases on which one or more models fail.
- 1 Introduction: Selecting model failures can reward systems that make different rather than fewer errors, slowing progress and encouraging spurious discovery claims.Examples exposing new errors may be absent from the evaluation set, allowing leaderboard gains without better underlying performance.
- 1 Introduction: The paper proposes four benchmark criteria: valid coverage of relevant language phenomena, consistent labels, adequate statistical power, and disincentives for harmful bias.Suggested directions include hybrid crowdworker–expert collection, larger-scale validation, and auxiliary bias datasets.
2 Background
Popular NLU benchmarks can show near-human or superhuman scores while models still fail simple targeted tests and exhibit socially relevant biases. The paper therefore seeks comprehensive, reliable, and responsible evaluation of labeled, general language-understanding tasks.
- 2 Background: GLUE and SuperGLUE reached near-human or above-human performance on many tasks, leaving insufficient headroom for continued progress.SuperGLUE excluded many submitted tasks because BERT already matched or exceeded majority-vote crowdworker performance.
- 2 Background: High leaderboard scores do not guarantee detection of needed improvements because crowdsourced benchmarks often have limited quality control.The paper notes that benchmark metrics may neither have achievable perfect performance nor distinguish systems with superhuman metric scores.
- 2 Background: Existing benchmarks provide little protection against socially relevant biases and may incentivize models that reproduce problematic biases.The paper links this concern to reliance on crowdsourced or naturally occurring text data.
- 2 Background: The paper targets labeled benchmarks for relatively general language-understanding skills and aims to support comprehensive, reliable understanding within specified tasks, language varieties, and domains.It distinguishes abstract tasks from concrete benchmark datasets and metrics, emphasizing the need to strengthen their correspondence.
- 2 Background: The paper retains social-bias evaluation while setting aside computational and data efficiency, arguing that community benchmarks should identify and disincentivize harmful biases.Ethical concerns motivate treating bias as part of responsible benchmark design.
3 Four Challenges
Effective NLU benchmarks must validly measure the full range of task-relevant language behaviors, use reliable annotations, provide sufficient statistical power, and discourage harmful social biases. The paper argues that common data-collection paradigms, including adversarial filtering, do not straightforwardly satisfy these requirements.
- Validity: A benchmark comparison should provide strong evidence that the higher-scoring system is actually better at the tested task.If the benchmark fully meets this standard, improvements should translate to similar improvements on other valid evaluations for the same task and domain.
- Validity: Effective benchmarks should cover the full range of linguistic variation and provide a plausible means of testing all expected task-related behaviors.They should also avoid annotation artifacts that let systems reach near-human performance without demonstrating the required language abilities.
- Validity: Naturalistic data distributions often fail to isolate relevant behaviors, especially for multi-text tasks and single-input tasks dominated by factual knowledge.For reading comprehension and inference, no natural distribution often efficiently isolates the target behaviors; for coreference and Cloze, naturalistic data can be dominated by world knowledge.
- Dataset design: Existing datasets often contain repetitive, easy cases and fail to isolate key behaviors, while expert-authored datasets can overrepresent phenomena already prioritized by researchers.Expert control can be useful for diagnostic datasets but is counterproductive when broad coverage is needed to guide priorities and progress.
- Dataset design: Adversarial filtering selects examples that models misclassify, but it neither guarantees valid measurement nor preserves broad coverage of already-solved skills.Its mode-seeking behavior can reduce dataset diversity, making validity harder to achieve; adversarial competitions may instead complement conventional evaluations.
- Reliable annotation: Reliable annotation is essential because legitimate disagreement can make models predict majority-label behavior rather than match individual human judgments.Reported ambiguity affects 20% of examples across several textual-entailment datasets and 36% of short-answer annotations in Natural Questions.
4 Sketching a Solution
The paper sketches task-specific reforms for benchmark validity, annotation reliability, statistical power, and bias measurement, while emphasizing substantial costs and unresolved implementation challenges.
- 4.1 Improving Validity: Building valid benchmarks will require significant, task-specific research into data collection methods.The authors expect improvements in crowdsourcing and non-expert annotation to play a major role.
- 4.1 Improving Validity: Expert augmentation can fill coverage gaps and mitigate annotation artifacts in initially high-quality crowdsourced datasets.Examples include Build-it-Break-it, the Open Reading Benchmark, and contrast sets.
- 4.1 Improving Validity: Small interventions during crowdsourcing can improve data quality without significantly limiting annotators’ creative freedom.OCNLI used bonus payments for examples avoiding overused words and constructions, though convincing validation remains difficult.
- 4.2 Improving Reliability: Redundant annotations can reduce mistaken labels, while careful planning and pilot work can address ambiguous guidelines.Ambiguous examples may be discarded during validation or represented through annotator label distributions and multiple supported answers.
- 4.3 Improving Statistical Power: 10-way annotation of 500,000 NLI examples would cost over $1 million at a $15/hr pay rate, even under optimistic timing assumptions.The estimate uses at least 45 seconds to label an existing example and at least one minute to create a new one; it ignores platform fees and assumes only 10% disagreement-related discard.
- 4.3 Improving Statistical Power: Large datasets have no clear low-cost collection route, and gamification introduces expert-time, protocol, and ethical constraints.The paper argues that benchmark investment should be weighed against wasted research time and computational resources.
- 4.4 Disincentives for Biased Models: Because harmful social bias lacks a one-size-fits-all definition, benchmarks cannot be guaranteed never to reward biased models.The authors nevertheless argue that omitting tests for plausible harms is a political choice.
- 4.4 Disincentives for Biased Models: Auxiliary expert-constructed datasets and metrics could isolate specific biases and expose them alongside primary task evaluation.This approach would not prevent subtle bias rewards in the primary metric, but could highlight and penalize them while adapting to changing harms.
5 Related Work
Related work addresses reporting precision, dataset documentation, statistical power, social bias, benchmark history, task validity, and dataset design from complementary perspectives.
- Evaluation and Dataset Design: Prior work advocates more precise reporting of test-set and metric capabilities, explicit dataset documentation, and careful dataset design.These efforts emphasize statistical power and exposing harmful mismatches between data and applications.
- Benchmarking Critiques: Church and Hestness argue that quantitative benchmarking’s success helped create conditions for widespread use of poor-quality benchmarks.Their work revisits the motivations for benchmarking in NLP.
- Social Bias: Related scholarship challenges researchers to focus social-bias analysis on specific harms and populations rather than broad formulations.The paper characterizes its own position as broad and not fully meeting that challenge.
- Task Validity: Ongoing debates question which tasks best test substantial language understanding, including recent arguments for grounding of various kinds.Multiple task-specific papers contribute to this debate.
6 Conclusion
The conclusion argues that NLU benchmarking is broken and that leaving IID evaluation is not the remedy; instead, the four criteria should be addressed directly.
- 6 Conclusion: The paper presents four criteria for faithful, useful, and responsible measures of language ability.It argues that adversarially filtered departures from IID evaluation do not address these criteria.
- 6 Conclusion: Important open questions remain about integrating expert effort into crowdsourcing and ensuring bias metrics are built and used when needed.The paper does not identify a clear institutional model for either challenge.
Ethical Considerations
The paper advocates reforms to benchmarking practices that inadequately address social bias and proposes measures intended to contribute to harm reduction without claiming complete solutions.
- Ethical Considerations: The authors argue that current benchmarking practices have helped create a false sense of security among developers of applied systems.They offer no complete and satisfactory solution, but propose measures that should contribute to reducing harm.