PaperlayerView exact revision on arXiv

Source-linked AI summary

What Will it Take to Fix Benchmarking in Natural Language Understanding?

Samuel R. Bowman, George E. Dahl

arXiv:2104.02145v3cs.CL

TL;DR

Popular NLU benchmarks can score systems highly even when those systems fail simple tests, leaving uncertain whether the benchmarks can measure future improvements. This position paper proposes four criteria for useful benchmarks and argues that moving away from IID evaluation does not address them.

  • Problem

    Popular NLU benchmarks approach human-level scores while high-scoring systems fail simple tests, leaving their ability to distinguish future improvements uncertain.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper proposes four benchmark criteria: coverage of relevant language phenomena, consistent labels, adequate statistical power, and disincentives for potentially harmful bias.

    Source: §1 · p. 2

  • Results

    The authors argue that concerns about standard benchmarks are justified and that departures from IID evaluation do not address the proposed criteria.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors argue benchmark concerns can and should be addressed directly within static, IID evaluation.

    Source: §1 · p. 2

    The paper identifies community structure and incentive design as a challenge to implementing its approach.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. The recent trend to abandon IID benchmarks in favor of adversarially-constructed, out-of-distribution test sets ensures that current models will perform poorly, but ultimately only obscures the abilities that we want our benchmarks to measure. In this position paper, we lay out four criteria that we argue NLU benchmarks should meet. We argue most current benchmarks fail at these criteria, and that adversarial data collection does not meaningfully address the causes of these failures. Instead, restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, the reliability with which they are annotated, their size, and the ways they handle social bias.

1 Introduction

The paper argues that saturated IID benchmarks cannot reliably measure NLU progress despite serious system shortcomings, and that adversarial data collection does not fix the underlying problems. It proposes four benchmark criteria and research directions for meeting them.

Benchmarking problem

  • IID benchmarks are nearing their achievable scores, yet their top-performing systems still have known problems and have not reached human-level competence.This combination leaves little room for benchmark scores to register further improvements.Source: §1 · p. 1
  • Untrustworthy evaluation discourages newcomers and gives specialists room to cherry-pick settings that mask a lack of progress.These concerns have fueled interest in replacing IID evaluation with adversarial data collection.Source: §1 · p. 1

Limits of adversarial evaluation

  • Adversarial filtering can reward models for making different rather than fewer errors, slowing progress and encouraging spurious discovery claims.A system may rise on the leaderboard by producing fewer errors than the adversary or by producing errors that are not represented in the evaluation set.Source: §1 · p. 2

Proposed criteria and directions

  • The paper proposes benchmarks that test relevant language phenomena, use consistently labeled data, provide adequate statistical power, and disincentivize potentially harmful bias.Suggested directions include hybrid crowdworker–expert collection, larger-scale data validation, and auxiliary bias-metric datasets.Source: §1 · p. 2

2 Background

Popular NLU benchmarks report near-human or superhuman scores even as high-scoring systems fail simple tests, leaving uncertain whether benchmarks can measure further progress. The paper argues for benchmarks that better support reliable evaluation while addressing harmful social bias.

The Problem

  • Popular benchmarks are saturated or exceed human performance, yet high-scoring systems can fail simple tests and current metrics may not detect needed improvements.Crowdsourced datasets with limited quality control offer no guarantee that perfect metric performance is achievable or that systems with superhuman scores can be distinguished.Source: §2 · p. 2
  • Current benchmarks do little to discourage harmful social biases and may incentivize models to reproduce them, motivating community benchmarks that identify and disincentivize potentially harmful bias.The paper notes that systems built for scientific purposes can be shared and deployed in settings where bias causes real harm.Source: §2 · pp. 2–3

The Goal

  • The paper proposes four benchmark criteria to advance machines’ comprehensive and reliable understanding of everyday text within a specific task, language variety, and topic domain.Its focus is on labeled-data tasks designed to test relatively general language-understanding skills.Source: §2 · p. 3
  • A task is a language skill in an input–output format, whereas a benchmark grounds that task in a text domain through a dataset and evaluation metric.Because no general proof can establish that a concrete benchmark faithfully measures an abstract task, the paper argues for strengthening their correspondence.Source: §2 · p. 3
  • The paper sets aside computational and data efficiency and few-shot learning, but retains social bias because ethical concerns warrant benchmarks that discourage harmful biases.It focuses on settings where researchers seek excellent task performance and are willing to collect data or expend other resources.Source: §2 · p. 3

3 Four Challenges

The paper argues that effective NLU benchmarks must satisfy four criteria, and that no current benchmark for a difficult broad-domain task satisfies all four.

3.1 Validity

  • A valid benchmark should cover the domain’s linguistic variation, test the language-related behaviors required by the task, and prevent near-human scores through annotation artifacts.The authors say clear improvements on a fully valid benchmark should translate to similar improvements on other valid evaluations of the same task and domain.Source: §3.1 · p. 4
  • Natural distributions can be product-context-specific or dominated by factual knowledge, expert-authored data may overfocus familiar phenomena, and standard crowdsourcing often produces easy repetitive cases.These approaches can therefore fail to isolate key behaviors or provide broad coverage for measuring progress.Source: §3.1 · pp. 4–5
  • Adversarial filtering can remove artifact-easy examples but may also eliminate already-solved skills and reduce dataset diversity; periodic adversarial competitions could complement conventional evaluations.The authors contrast filtering examples on which a model fails with competitions that compare the difficulty of collecting valid adversarial examples for different systems.Source: §3.1 · p. 5

3.2 Reliable Annotation

  • 20% of examples across several textual entailment datasets were significantly ambiguous, and 36% of Natural Questions short-answer annotations differed significantly from the majority answer.Reliable labels require avoiding careless mistakes, underspecified guidelines, and legitimate annotator disagreement, which can let models predict typical labels better than individual annotators.Source: §3.2 · p. 5

3.3 Statistical Power

  • Two orders of magnitude more evaluation data are needed to reliably detect an improvement from 98% to 98.1% than to detect the same relative gain from 80% to 81%.With a reliable, difficult dataset, a few thousand examples may suffice to detect a 1% absolute improvement under typical NLU conditions.Source: §3.3 · pp. 5–6

3.4 Disincentives for Biased Models

  • Benchmarks should favor models without socially relevant biases, but harmful bias is difficult to define comprehensively because salient attributes vary across contexts and political attitudes change.The authors argue that bias-evaluation choices can align researchers with particular groups and may bring them into conflict with institutions.Source: §3.4 · p. 6

4 Sketching a Solution

Section 4 sketches improvements across benchmark validity, annotation reliability, statistical power, and social-bias evaluation. The authors argue that better benchmarks will require new collection methods, sufficient investment, and stronger incentives to measure and report harms.

4.1 Improving Validity

  • Validity improvements may combine broad-coverage crowdsourcing by non-experts with expert augmentation or researcher interventions, though demonstrating validity remains difficult.Experts can add examples to fill coverage gaps or correct artifacts in crowdsourced data; small interventions such as bonus payments to discourage overused words and constructions may improve quality without sharply limiting creative examples.Source: §4.1 · pp. 6–7

4.2 Improving Handling of Annotation Errors and Disagreements

  • Redundant annotations and careful piloting can reduce mistaken labels and ambiguous guidelines; benchmarks can discard ambiguous examples or represent annotator disagreement, with the latter requiring more judgments.For multiple-choice QA, models can be tested on ambiguous phenomena using examples where one answer is debatably correct but all others are unequivocally wrong; alternatives include predicting annotator label distributions or accepting any annotator-supported answer, as in SQuAD.Source: §4.2 · p. 7

4.3 Improving Statistical Power

  • Adequately powered benchmarks may require costly annotation: 500,000 examples with ten annotations each would cost over $1 million at $15/hour, reinforcing concentration on high-resource languages and tasks.Gamification may reduce labor costs but requires expert time, constrains data-collection protocols, and raises ethical issues; the authors urge comparing benchmark investment with the cost of wasted research effort and compute.Source: §4.3 · p. 7

4.4 Disincentives for Biased Models

  • Because no robust strategy guarantees bias-free evaluation data, benchmarks can pair primary task metrics with expert-constructed auxiliary datasets that measure specific biases and help maintainers adapt to changing harms.The authors note that many papers omit potentially unflattering bias metrics, even when SuperGLUE requires reporting DNC Winogender results; they suggest community and publication policies or aggregated reporting to encourage disclosure, while emphasizing that measurement alone cannot prevent harms in deployment.Source: §4.4 · p. 8

5 Related Work

Related work calls for more transparent, carefully designed dataset and evaluation practices, while debating what language-understanding benchmarks should measure and how they can address poor-quality evaluation and social harms.

Related Work

6 Conclusion

The paper argues that NLU benchmarks should meet four criteria and that moving away from IID evaluation does not address them. It also identifies unresolved questions about integrating expert input into crowdsourced data collection and ensuring bias metrics are used when needed.

6 Conclusion

  • The paper proposes four criteria for benchmarks to provide faithful, useful, and responsible measures of language ability.It argues that NLU benchmarking is broken.Source: §6 · p. 9
  • Departing from IID evaluation, including through adversarial filtering, does not address the criteria; the paper instead outlines ways to address them directly.Source: §6 · p. 9
  • Open questions include how to integrate expert effort into crowdsourced data collection and how to ensure bias metrics are built and used when most needed.Source: §6 · p. 9

Ethical Considerations

The paper advocates reforming benchmarking practices that have largely failed to address social bias and helped create false security among applied-system builders. It proposes measures intended to reduce harm, while acknowledging that it offers no complete, satisfactory solutions.

Ethical Considerations

  • The paper proposes reforms to reduce harm from benchmarking practices that neglect social bias, without claiming to offer complete solutions.The paper says those practices have helped create a false sense of security among people building applied systems.Source: Ethical Considerations · p. 9

Related papers

Loading 2104.02145v3…