Source-linked AI summary
Dynabench: Rethinking Benchmarking in NLP
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams
TL;DR
NLP models can achieve outstanding benchmark performance while failing simple challenge examples and real-world scenarios because benchmarks are imperfect proxies for desired capabilities. Dynabench introduces dynamic, human-and-model-in-the-loop data collection and benchmarking; across initial tasks, later rounds become more adversarial while models improve, including a final target model reaching 94% on HATECHECK. The platform supports more robust evaluation and ongoing research into model robustness, distributional shift, and human-machine interaction.
Problem
NLP benchmarks can be inadequate proxies for broad language capabilities, allowing models to perform well while failing simple challenge examples and real-world scenarios.
Method
Dynabench is an open-source, web-based platform that collects and validates examples through repeated interaction between human annotators and target models, then uses the data for evaluation and training.
Results
Across four rounds, examples become increasingly adversarial while model error rates decline; the final round target model achieves 94% on HATECHECK.
Takeaways & Limitations
Dynabench supports dynamic leaderboards and evaluation across accuracy, computational resources, inference time, fairness, and other dimensions, while opening research directions in robustness and distributional shift.
Takeaways & Limitations
Dynamic benchmarking is expensive, and annotators may overfit to target models, causing cyclical progress and making evaluation across all rounds and high-quality static tests important.
Abstract
from arXiv · showhide
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-in-the-loop dataset creation: annotators seek to create examples that a target model will misclassify, but that another person will not. In this paper, we argue that Dynabench addresses a critical need in our community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios. With Dynabench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and informative benchmarks. We report on four initial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic benchmarking as a new standard for the field.
1 Introduction
NLP models now reach benchmark milestones rapidly, yet benchmark success can coexist with failures on simple challenges and real-world scenarios. Dynabench proposes dynamic human-and-model-in-the-loop benchmarking to expose these weaknesses and generate data for improvement.
- Benchmark milestones that once took decades are now routinely reached within a few years on newer datasets.The paper attributes NLP progress to increased computational power and algorithmic breakthroughs, including attention, Transformers, and pretrained language models.
- Models with super-human benchmark performance can still fail simple challenge examples and falter in real-world scenarios.The paper links this mismatch partly to benchmark tasks that contain unwanted statistical and social biases.
- Dynabench is an open-source, web-based platform for dynamic data collection and model benchmarking.The paper presents it as a partial solution to the need to rethink benchmarking.
- Its guiding hypothesis is that dynamic evaluation and data collection with humans and models in the loop can accelerate progress beyond static benchmarking.The platform uses interaction between annotators and target models rather than relying only on traditional static datasets.
- Across multiple rounds, annotators create examples against state-of-the-art models, revealing shortcomings and producing training and assessment data.The resulting interaction supplies a concrete measure of model performance with humans and data for developing stronger models.
- The paper describes the platform, reports findings from four initial tasks, and addresses objections before discussing future plans.
2 Background
Standard NLP benchmarks can saturate quickly while models remain weak on robust, generalizable language capabilities. Existing challenge, adversarial, and diagnostic evaluations expose this gap, motivating Dynabench as one path toward combining development with stress-testing.
- GLUE saturated within a year, and SuperGLUE already has models rather than humans at the top of its leaderboard.These achievements do not establish human-level natural language capabilities.
- Challenge test sets show that state-of-the-art NLP models struggle with compositionality and social or linguistic stress tests.
- Models remain vulnerable to adversarial inputs, including added relevant sentences in question answering and character changes in text classification.
- Prior work uses iterative human-and-model-in-the-loop annotation to find unknown unknowns and support never-ending or lifelong learning.
- Crowdsourced data supports large-scale dataset creation, but NLP systems are ultimately intended to handle natural and ecologically valid data.
- The mismatch between benchmark and challenge-set performance suggests that standard evaluation overstates model ability, motivating Dynabench’s combination of development and robustness stress-testing.
3 Dynabench
Dynabench is an open-source platform for dynamic, human-and-model-in-the-loop data collection and benchmarking. Its initial tasks illustrate how iterative interaction can expose model shortcomings, support stronger training data, and assess progress across rounds.
- Dynabench: Dynabench collects task data over multiple rounds against target models interacting with human annotators, then validates examples that models get wrong.Collected data supports both evaluation of state-of-the-art models and training of stronger ones.
- Dynabench: Humans can create model-fooling examples, but Dynabench also supports examples eliciting model uncertainty, collaborative feedback, and negotiation.The common requirement is live interaction between models and humans during evaluation and data collection.
- Dynabench: Dynabench emphasizes expert linguists because stronger models make valid model-fooling examples harder to find and require more exploration of decision boundaries.The paper predicts that linguists with expertise in exploring model decision boundaries will become essential if the proposed virtuous cycle develops.
- Motivation: Adversarial human-and-model-in-the-loop data collection can address benchmark failures that coexist with super-human task scores and real-world model brittleness.Related efforts found that best model performance remained far from human performance despite adversarial training improving some blind spots.
- Dynabench: The platform’s central change is dynamic dataset creation, enabling evaluation across rounds and different conditions while retaining existing benchmark metrics.
- Initial Tasks: Across four initial tasks, validated model error rates remain high, while task-specific results show models improving as later-round data becomes increasingly adversarial.In hate-speech classification, the final-round target model achieved 94% on HATECHECK, outperforming the original authors’ models.
4 Caveats and Objections
The paper acknowledges practical and methodological objections to dynamic benchmarking, including unnatural distributions, model overfitting, changing benchmarks, limited generative-task support, and higher costs. It proposes mitigations such as combining adversarial and non-adversarial data, retaining earlier evaluation rounds, using model ensembles, and accepting evolving benchmarks.
- Dynamic adversarial collection can produce unnatural datasets and distributional shift, especially when used alone.The paper recommends combining adversarially collected data with non-adversarial, preferably naturally collected, data.
- Annotator overfitting may create cyclical progress by causing models to forget weaknesses exposed in earlier rounds.The paper recommends evaluating all dynamic rounds alongside high-quality static test sets rather than discarding older tests.
- Data distributions may become overly dependent on the target model, but ensembles spanning architectures and random seeds can mitigate this risk.
- Changing benchmarks complicate direct comparisons, but evolving datasets are already treated as distinct evaluation settings.Dynabench anticipates benchmark saturation and embraces iterative dataset updates.
- Dynabench currently emphasizes classification and span extraction because generation metrics make correctness and successful fooling harder to establish.The paper suggests reframing generation as multiple choice with hard negatives or asking annotators whether outputs are good enough.
- Dynamic benchmarking costs more because failed model-fooling attempts and validation add expense, with costs likely rising across successive rounds.The authors report approximately $0.50 per validated SNLI example, $0.64 for MultiNLI, and closer to $1.00 for ANLI examples.
5 Conclusion and Outlook
Dynabench is presented as an open platform for dynamic benchmarking, with future plans spanning broader participation, richer evaluation, and research on human–machine interaction. The authors frame these capabilities as steps toward democratizing model evaluation and improving robustness.
- 5 Conclusion and Outlook: The platform’s future directions include studying ensembles, distributional shift, annotator efficiency and expertise, and robustness to targeted interactive attacks.It also supports broader analyses of dynamic data collection and human–machine interaction.
- 5 Conclusion and Outlook: Dynabench aims to let anyone run a human-in-the-loop evaluation or data-collection task using a target model, contexts, and annotators.Only these three components are needed to get started.
- 5 Conclusion and Outlook: The current platform is text-only and focused on English, with multilinguality and multimodality identified as future goals.This marks an explicit boundary of the current iteration.
- 5 Conclusion and Outlook: Live evaluation could score models across accuracy, computational resources, inference time, fairness, and other dimensions rather than relying on one test-set number.The authors also envision dynamic utility-based leaderboards and automatic inclusion of new state-of-the-art models.
- 5 Conclusion and Outlook: Authors could eventually link model inference points so others can interact with models, examine capabilities, and identify failures for subsequent improvement.The paper presents this as a way to democratize evaluation even when training remains inaccessible.