Source-linked AI summary
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, Douwe Kiela
TL;DR
Dynaboard addresses accuracy-centered and self-reported evaluation by evaluating submitted NLP models in the cloud and combining performance with other metrics in customizable rankings. In the reported comparisons, SuperGLUE's ranking is roughly preserved, while added evaluation axes can change model positions, including T5's lower rank associated with throughput and memory.
Problem
Static leaderboards often rely primarily on accuracy despite users valuing other factors, while self-reported results and inaccessible or differently evaluated models complicate reproducibility and comparison [52].
Source: §1 · pp. 1–2Method
Dynaboard evaluates submitted models in the cloud and aggregates performance and other metrics into a utility-based Dynascore that users can customize.
Source: §1 · p. 2, §5.2 · p. 7Results
SuperGLUE's ranking is roughly preserved, but T5 ranks lower because of relatively low throughput and high memory use, while DeBERTa still performs best across the additional evaluation axes.
Source: §5.5 · p. 10Takeaways & Limitations
Dynaboard provides a customizable ranking that factors prediction costs into model comparisons, supporting a more diverse and comprehensive evaluation of model quality.
Source: §6 · p. 11The fixed ϵ threshold can change Dynascores and rankings, and the authors note that small performance differences may not be statistically significant on many NLP datasets [8].
Source: §5.4 · p. 10
Abstract
from arXiv · showhide
We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform. Our platform evaluates NLP models directly instead of relying on self-reported metrics or predictions on a single dataset. Under this paradigm, models are submitted to be evaluated in the cloud, circumventing the issues of reproducibility, accessibility, and backwards compatibility that often hinder benchmarking in NLP. This allows users to interact with uploaded models in real time to assess their quality, and permits the collection of additional metrics such as memory use, throughput, and robustness, which -- despite their importance to practitioners -- have traditionally been absent from leaderboards. On each task, models are ranked according to the Dynascore, a novel utility-based aggregation of these statistics, which users can customize to better reflect their preferences, placing more/less weight on a particular axis of evaluation or dataset. As state-of-the-art NLP models push the limits of traditional benchmarks, Dynaboard offers a standardized solution for a more diverse and comprehensive evaluation of model quality.
1 Introduction
Dynaboard directly evaluates submitted models in the cloud and combines performance with other evaluation axes in rankings users can customize. It aims to address accuracy-only evaluation and reproducibility, accessibility, and compatibility concerns within one end-to-end platform.
1 Introduction
- Accuracy-centered leaderboards can misalign with users who also value compactness, fairness, or efficiency, while leaving other pressing issues out of view [16] [63] [68] [51] [17].The passage frames leaderboard viewers as consumers with differing preferences, drawing on Ethayarajh and Jurafsky [17].Source: §1 · pp. 1–2
- Self-reported results, limited access to state-of-the-art models, and evaluations on different data complicate reproducibility, accessibility, and comparison across older and newer models [52].Access may require computational resources or technical expertise, and older models may not have been evaluated on the same data.Source: §1 · p. 2
- Dynaboard evaluates submitted models on demand in the cloud, enabling older and newer models to be compared and addressing reproducibility, accessibility, and backwards-compatibility concerns.The platform integrates with Dynabench, which supports human-and-model-in-the-loop dataset creation [34].Source: §1 · p. 2
- Dynaboard supports accuracy-style metrics alongside memory use, throughput, robustness, and fairness, then ranks models with a customizable Dynascore.Users can adjust weights for metrics or datasets and see rankings change in real time; they can also interact with models to inspect predictions and identify challenging examples.Source: §1 · p. 2
- Dynaboard is presented as an end-to-end system that addresses problems other platforms handle only in subsets, with low submission overhead and flexible model comparisons.The authors describe it as a standardized solution for diverse, comprehensive evaluation as NLP models push traditional benchmarks' limits.Source: §1 · p. 2
2 Objectives
The objectives identify gaps in reproducibility, access, compatibility, prediction-cost reporting, and utility-aware ranking. Dynaboard pairs cloud evaluation with a customizable leaderboard to address evaluation access and ranking preferences.
2 Objectives
- Self-reported results without trust guarantees, random-seed variation, metric implementation differences, and possible hyperparameter overfitting complicate reproducibility.Reported state-of-the-art gains may also fail to be statistically significant [8].Source: §2 · p. 2
- Open-sourcing weights alone may not make a model accessible when researchers lack inference-time compute resources.The passage links broad access to understanding weaknesses, progress on long-tail data, and collecting adversarial datasets.Source: §2 · p. 2
- When dynamic benchmarks add evaluation sets, older models become incomparable to newer ones if they have not been evaluated on the same data.The stated objective is to evaluate models on demand rather than rely on predictions from one point in time.Source: §2 · p. 2
- The current reporting paradigm cannot evaluate older models on new automated metrics, limiting comparisons when improved metrics become available.The passage cites BLEU and ROUGE as metrics with known flaws and BLEURT [64] and BERTScore [77] as examples of newer metrics.Source: §2 · pp. 2–3
- Prediction costs such as memory, latency, throughput, and robustness have real-world implications, yet most leaderboards treat them as zero [17].A highly accurate model may be unsuitable for an embedded systems engineer if it is too large.Source: §2 · p. 3
- Because leaderboard viewers have different utility functions, rankings should provide a default while letting users customize how performance, memory, throughput, and fairness count [17].Dynaboard's backend supports on-demand evaluation, while its frontend offers a dynamic leaderboard whose scoring function users can adjust.Source: §2 · p. 3
3 Related Work
Prior platforms address subsets of benchmarking needs, but the cited comparisons describe gaps in combining evaluation access, compatibility, prediction-cost collection, and nuanced model comparison. Other efforts report selected costs or study particular trade-offs.
3 Related Work
- DAWNBench reports selected prediction costs, such as time to reach an accuracy target, but relies on self-reported results and retains reproducibility and compatibility issues.The passage also identifies accessibility as an unresolved issue for DAWNBench.Source: §3 · p. 3
- CodaLab2 and EvalAI ease access and reproduction, but automatically evaluating models on new datasets or metrics and collecting costs on standard hardware are not straightforward.The passage says backwards and forwards compatibility are possible in theory, but difficult to scale.Source: §3 · p. 3
- Explainaboard [39] and Robustness Gym [23] support nuanced comparisons through data slices and error types, but do not directly upload models to collect costs or automate compatibility.The approaches are described as potentially compatible and complementary with Dynaboard.Source: §3 · p. 3
- Other efforts include studying accuracy–latency trade-offs in speech translation and reporting or limiting prediction costs in challenges such as EfficientQA [45].The authors describe such practices as few and hope Dynaboard can lower the overhead of creating multi-metric benchmarks.Source: §3 · p. 3
4 Backend: Evaluation-as-a-Service
The backend evaluates uploaded models consistently across datasets and shared virtual hardware, while supporting multiple evaluation axes and repeat evaluation as datasets and metrics evolve. The four initial task benchmarks combine task-specific datasets and configurable scoring choices.
4 Backend: Evaluation-as-a-Service
- Uploaded models are scored through a standardized API on the same data and virtual hardware, and the same model instance is evaluated across evaluation sets rather than cherry-picked results.The evaluation-as-a-service pipeline supports standard test sets, stress tests [47] [11], and checklists [59], with model cards requested for documentation [46].Source: §4 · p. 4
- All uploaded models can be accessible online for interaction, and users can rank them by selecting datasets and metrics and their relative weights.Models are evaluated on combinations of adversarial and non-adversarial datasets, and top models can be used for dynamic adversarial evaluation [34].Source: §4 · p. 4
- Dynaboard is designed to let metrics, datasets, and dynamic in-the-loop evaluation expand over time, making re-evaluation of older models essential as preferences and evaluation resources change.The authors explicitly do not claim to have found the one right approach to model evaluation.Source: §4 · p. 4
4.1 Implementational details
- Dynalab reduces submission overhead through a model handler with specified input and output formats; a BERT sentiment model requires roughly 10 lines of code.Its library and command-line tool initialize, test, and deploy model codebases in a few commands.Source: §4.1 · p. 4
4.2 Metrics
- The reported axes include task-dependent canonical performance, throughput measured as examples per second on CPU with batch size 1, and memory use averaged over a fixed interval.The default weighting assigns half to canonical performance and divides the other half among remaining metrics; memory use is converted to memory saved before aggregation.Source: §4.2 · pp. 4–5
- Robustness uses perturbations such as typographical errors and local paraphrases, while the black-box fairness metric perturbs gender or race/ethnicity cues and inspects predictions.The authors call fairness measurement imperfect and note the risk of spurious fairness claims, value-hacking, or a false sense of fairness; fairness perturbations are described further in Appendix A.Source: §4.2 · p. 5
4.3 Tasks
- The four initial tasks are NLI, QA, Sentiment Analysis, and Hate Speech, with task owners setting included datasets and relative dataset and metric weights.Leaderboard ranking uses only a subset of each task's full evaluation-as-a-service dataset collection; Table 1 lists the datasets.Source: §4.3 · pp. 5–6, Table 1 · p. 6
5 Frontend: Dynamic Leaderboard
Dynaboard combines performance and other model metrics in a customizable, utility-based ranking. Its exchange rates and weights support comparisons across metrics, while its results and limitations show why the ranking depends on task, model set, and user preferences.
5 Frontend: Dynamic Leaderboard
- Static accuracy- or F1-based rankings cannot reflect individual preferences or prediction costs; Dynaboard aggregates metrics and lets users adjust the scoring function.Its evaluation-as-a-service frontend reports metrics such as memory use and throughput for ranking models.Source: §5 · p. 6
5.1 Background
- Metrics are treated as goods to maximize, and MRS measures the rate of trade-off between a metric and performance along a fixed-utility curve.Metrics such as memory used can be transformed into goods such as memory saved.Source: §5.1 · p. 6, Figure 1 · p. 7
5.2 Converting between Metrics
- Dynaboard converts metrics into performance units using AMRS, estimated from two-dimensional indifference curves and assumptions for models that cannot share one curve.The estimates are a starting point and change as models are added; the method is intended to avoid problems with z-scores for highly skewed metrics.Source: §5.2 · pp. 7–8
5.3 Weights and Customization
- The default Dynascore assigns half its weight to performance and splits the rest across other metrics; users can adjust weights to reflect their preferences.Changing weights can change the model ranking.Source: §5.3 · pp. 8–9, Figure 2 · p. 9
5.4 Limitations & Future Work
- AMRS estimates are approximate, and if all models have identical performance, AMRS cannot be calculated.The estimates are intended as starting points for user adjustment; using another base metric is possible in principle.Source: §5.4 · p. 9
- Ignoring performance increases below 1 × 10^-4 avoids undefined MRS, but the threshold can change Dynascores and rankings.Some models and tasks also require human-in-the-loop evaluation rather than fully automatic evaluation.Source: §5.4 · p. 10
5.5 Results
- Across added evaluation axes, DeBERTa performs best, while T5 falls below its SuperGLUE standing because of low throughput and high memory use.In the sentiment leaderboard, shifting weight to throughput and memory replaces T5 with FastText; Dynascores are not comparable across tasks.Source: Table 2 · p. 8, Figure 2 · p. 9, §5.5 · p. 10
6 Conclusion
Dynaboard provides an end-to-end platform for holistic benchmark evaluation. It addresses evaluation shortcomings and lets users include prediction costs and preferences in model rankings.
6 Conclusion
- Dynaboard provides an end-to-end solution for hosting benchmarks and evaluating model quality more holistically.The platform addresses reproducibility, accessibility, and backwards/forwards compatibility, and collects prediction costs for model rankings that users can customize.Source: §6 · pp. 10–11
- The platform addresses shortcomings in reproducibility, accessibility, and backwards/forwards compatibility.Models can be submitted for on-demand evaluation, and prediction costs are factored into rankings.Source: §6 · p. 11
- Users can customize rankings to better align with their preferences, including by factoring prediction costs into the overall ranking.Source: §6 · p. 11
7 Broader Impact
Dynaboard aims to broaden benchmark incentives beyond accuracy, while its fairness measurements and Dynascore have specific scope and interpretation limits.
7 Broader Impact
- Multi-metric benchmarks may incentivize models that are greener and fairer, qualities largely ignored by the dominant accuracy-focused paradigm.The authors say incentive structures shape the problems the community works on and the models it builds.Source: §7 · p. 11
- The fairness method is an initial attempt that assumes equal treatment across genders and races, a first step toward equitable treatment that can make measurements noisier.The authors note that fairness perturbations do not always preserve classification labels or apply equally to all members of a group.Source: §7 · p. 11
- The fairness evaluation covers only selected gender and race/ethnicity categories, and gender-neutral perturbations were set aside because heuristic substitutions produced too much noise.The selected race/ethnicity groups reflected data availability and may reflect US-centric bias in existing datasets.Source: §7 · pp. 11–12
- For hate speech, perturbations may overestimate model sensitivity because the training data did not include corresponding examples about men, straight people, or white people.Source: §7 · p. 12
- Strong performance on the fairness perturbations does not establish that a model is fair, though poor performance indicates room for improvement.Source: §7 · p. 12
- A Dynascore changes with the models on a leaderboard and its metric weights, so it should be reported with at least the dataset, weights, and timestamp.The authors recommend contextual reporting because users may otherwise treat the default Dynascore like accuracy or F1.Source: §7 · p. 12
A Fairness metric
The fairness metric uses post hoc black-box perturbations of evaluation examples and measures whether model predictions change. Heuristic substitutions are designed to preserve the input distribution, but can introduce noise.
A Fairness metric
- The post hoc black-box method applies to existing Dynabench data and uploaded models using only their predictions.Source: A Fairness metric · p. 17
- The method substitutes words signaling gender or race/ethnicity with words referring to another group, then measures label flips as the difference in microaverage accuracy.The targeted substitutions aim to preserve the original input data distribution and minimize effects on classification labels.Source: A Fairness metric · p. 18
- Race/ethnicity perturbations substitute listed names across Asian/Pacific Islander, Black, Hispanic, and white groups, leaving unlisted names unchanged.The lists use the 200 most frequent names selected from a US dataset and were augmented with names from prior literature.Source: A Fairness metric · p. 18
- Gender perturbations cover names and noun phrases, with paired pronoun and noun substitutions chosen to avoid random replacements that could make sentences ungrammatical.Name perturbations use statistically likely genders, while noun-phrase pairs include words referring exclusively to women or men.Source: A Fairness metric · p. 18
- Named-entity recognition filters examples with familiar entities, but heuristic perturbations can still create unusual inputs and occasional classification-relevant noise.The authors report fewer than five noisy examples from NER errors and three noisy examples among 122 spot-checked sentences involving explicit gender information.Source: A Fairness metric · p. 18
B Robustness metric
The robustness metric evaluates prediction stability under typographical transformations. It uses the percentage of unchanged predictions before and after perturbation.
B Robustness metric
- The evaluation applies TextFlint’s Contraction, Keyboard, OCR, Punctuation, SpellingError, Typos, and WordCase transformations, focusing on typographical errors.Source: B Robustness metric · p. 18
- Robustness is measured as the percentage of unchanged predictions before and after perturbation.Source: B Robustness metric · p. 19
- The metric assumes a robust model should not change its predictions under these input perturbations, so higher values indicate lower prediction differences.Source: B Robustness metric · p. 19
C Model Details
The model suite was chosen to illustrate Dynaboard’s evaluation breadth and the state of the art across its metrics, using default training hyperparameters rather than tuning. The section specifies task-specific training data and model configurations, including a larger ELECTRA model and a train-set majority baseline.
C Model Details
- The authors selected diverse models to demonstrate the platform and current performance across metrics, without hyperparameter tuning.They used default training hyperparameters.Source: C Model Details · p. 19
Training Data
- NLI used ANLI [49] upsampled 2x with MNLI [74]; QA used SQuAD [57] and Adversarial QA [1], while hate-speech and sentiment models used Learning from the Worst [71] and Dynasent [55].Hate-speech training used rounds 0–4 with respective upsampling ratios 1, 5, 100, 1, and 1; Dynasent round 2 was upsampled 3x.Source: C Model Details · p. 19
Transformers
- Transformer models used HuggingFace base versions except ELECTRA, which used the large discriminator to test framework capacity for larger models.The training scripts’ default hyperparameters were used.Source: C Model Details · p. 19
T5
- T5 models used early stopping, and sequence classification predicted the first token of one of strings "0" and "1" for 2-way tasks or "0", "1", and "2" for 3-way tasks.Text classification used 3 epochs to match the BERT-style model default; T5’s default was 2 epochs.Source: C Model Details · p. 19
FastText
- FastText classifiers used the large-dataset website defaults, including 25 epochs, 2-grams, 50 dimensions, and hierarchical SoftMax.The initial learning rate was 1 and the bucket size was 200000.Source: C Model Details · p. 19
BiDAF
- BiDAF used the AllenNLP trainer’s default settings.Source: C Model Details · p. 19
Majority Baseline
- The majority baseline selected each task’s most frequent training-set label; for NLI, it always returned neutral.The ANLI label counts were 52,111 entailed, 68,789 neutral, and 41,965 contradictory.Source: C Model Details · p. 19