Source-linked AI summary
Show Your Work: Improved Reporting of Experimental Results
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, Noah A. Smith
TL;DR
Current experimental reporting makes model comparisons difficult because test scores alone can obscure computational budgets and variability. The paper introduces expected validation performance as a function of computation budget and recommends richer reporting, showing that comparisons and reproducibility can change under different budgets.
Problem
Test-set scores and single performance estimates do not provide enough evidence for reliable model comparisons, especially when experiments use different computational budgets.
Method
The paper estimates the expected validation performance of the best model obtainable under a given computation budget and presents reporting recommendations.
Results
Different computational budgets can change conclusions about relative model performance, while reported test-score differences may be much smaller than test–validation gaps.
Takeaways & Limitations
Reporting validation performance, computation budgets, hyperparameter sensitivity, and random-initialization sensitivity supports fairer comparisons, reproducibility, and improved scientific understanding.
Takeaways & Limitations
Fair comparison remains difficult because models may have different hyperparameter spaces, application costs, and unmeasured human effort invested in narrowing those spaces.
Abstract
from arXiv · showhide
Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e.g., accuracy) on held-out test data, compared to previous results. In this paper, we demonstrate that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. We argue for reporting additional details, especially performance on validation data obtained during model development. We present a novel technique for doing so: expected validation performance of the best-found model as a function of computation budget (i.e., the number of hyperparameter search trials or the overall training time). Using our approach, we find multiple recent model comparisons where authors would have reached a different conclusion if they had used more (or less) computation. Our approach also allows us to estimate the amount of computation required to obtain a given accuracy; applying it to several recently published results yields massive variation across papers, from hours to weeks. We conclude with a set of best practices for reporting experimental results which allow for robust future comparisons, and provide code to allow researchers to use our technique.
1 Introduction
Test-set scores alone can mislead model comparisons because conclusions depend on computational budget and hyperparameter search. The paper introduces expected validation performance curves and reporting recommendations to make comparisons more reproducible.
- Test-set improvements are commonly used to indicate that one NLP or machine-learning method is superior.
- Expected validation performance of the best model is reported as a function of computational budget, using models already evaluated during development.The curve covers budgets n ∈ {1, . . . , N} and requires no computation beyond hyperparameter search or optimization.
- The expected-best model can change with budget, so model comparisons may support different conclusions under different amounts of computation.The paper also uses these curves to estimate the budget required to obtain a target accuracy and to infer computation used for previous results.
- The proposed calculation has strictly less error than the bootstrap because it has a closed-form solution without resampling error.Both approaches retain the same finite-sample error from the available validation values.
- The paper recommends improved reporting for reproducibility and hyperparameter sensitivity, but an analysis of 50 randomly selected EMNLP 2018 papers found none reported all suggested items.
2 Background
The paper frames reproducibility as obtaining the same comparative conclusion across independent implementations, while noting that computation and hyperparameter-selection practices affect that conclusion. It describes model families, hyperparameter spaces, and search strategies used to formalize these comparisons.
- Reproducibility here means that different research groups should follow procedures leading to the same conclusion about which method performs better.That conclusion can depend on the amount of computation applied.
- A model family is an approach subject to comparison and hyperparameter selection, with its own space of possible hyperparameter values.A hyperparameter value is a k-tuple specifying one value for each of k hyperparameters.
- Hyperparameter spaces may include random seeds, hidden-layer sizes, and learning rates, so model selection can vary beyond the model architecture itself.
- Grid search and uniform sampling select hyperparameters systematically, whereas Bayesian and manual search adapt choices after earlier experiments.Adaptive strategies may find better results quickly but are generally less reproducible and harder to parallelize; manual search is least reproducible.
3 Expected Validation Performance Given Budget
The paper estimates expected validation performance of the best model as a function of computation, using intermediate model evaluations rather than only the final test score. This diagnostic supports comparisons of computational cost and performance while avoiding additional experimentation under random search.
- Estimation: Under random search, validation performances from hyperparameter configurations are treated as i.i.d. draws, making the maximum a random variable whose expectation can be estimated.The approach uses the empirical distribution of observed validation performances and does not require further experimentation.
- Expected validation performance: Expected validation performance of the best model is estimated for each computation budget n, rather than reporting only the maximum after the full budget B.The budget can represent the number of hyperparameter evaluations, and can generally be converted to GPU hours or monetary cost.
- Limitation: The approach does not claim to extrapolate expected performance beyond B, the number of hyperparameter values to which the algorithm was actually applied.Its asymptotic-looking curve reflects increasing computation only within the evaluated budget range.
- Properties: At n = 1, expected maximum performance equals the mean observed validation performance, while for all n the resulting curve is a lower bound on the selected model’s validation performance.As n increases, the expected maximum approaches the observed maximum v*_B and the curve appears to asymptote.
- Comparison with bootstrap: The method computes expected maximum performance in closed form, avoiding the bootstrap’s resampling error while retaining finite-sample error from the observed evaluation set.The paper also estimates variation from the resulting distribution and recommends reporting variability, with shading restricted to observed ranges when necessary.
4 Case Studies
The case studies show that expected validation performance changes with computational budget, altering model rankings and revealing widely varying effort behind reported results.
- Case-study overview: The method estimates which approach performs best at a given budget and the budget required to match previously reported validation performance.The authors apply it across text classification, contextual embeddings, and two published model comparisons.
- Initial classification study: Fewer than 10 hyperparameter trials favor logistic regression, while larger budgets let the CNN reach higher overall expected validation accuracy.Logistic regression also has lower variance at every budget, supporting faster prototyping.
- Contextual representations: For contextual embeddings, feature extraction is preferable between approximately 6 hours and 1 day, fine-tuning wins at 10 days, and GloVe is preferable below 2 hours.The comparison uses time as budget, scaled by each model’s average training duration.
- SciTail: Different SciTail models require budgets ranging from 2 trials for the n-gram baseline to 20 for DGEM to reach their reported performance in expectation.With different computation budgets, the top-performing model changes, although the neural models are similar.
5 Recommendations
The paper proposes a structured reporting checklist because comprehensive details about experimental setup and variability are rarely reported, limiting reproducible comparisons.
- Checklist: The checklist combines relevant reproducibility guidance with recommendations specific to reporting experimental results in a structured format.The authors emphasize reporting rather than requiring additional experiments.
- Checklist coverage: None of fifty randomly sampled EMNLP 2018 papers reported all checklist items, although every item appeared in at least one paper.This establishes incomplete coverage rather than complete absence of reporting.
- Checklist coverage: 74% of analyzed papers reported at least some best hyperparameter assignments, while 10% or fewer reported search bounds, evaluation-trial counts, or variability measures.The full coverage analysis appears in the appendix’s Table 1.
- Budget-aware comparisons: Claims about relative model performance should be qualified by computational expense, because superiority may depend on the budgets being compared.Equal trial counts do not necessarily make comparisons fair when model families have different hyperparameter spaces or per-application costs.
6 Discussion: Reproducibility
The discussion links reproducibility to matching conclusions across implementations and argues that validation reporting can reduce dependence on leaderboard test evaluations.
- Leaderboards: Standardized test sets and public leaderboards have helped mitigate replication concerns, but leaderboards can obscure tuning budgets and create reproducibility issues.They may hide the work required to apply a model to a new dataset.
- Leaderboards: On GLUE, top-model test-score differences can be about a tenth of a percent, while test–validation differences can be one percent or larger.This makes validation performance useful for comparisons without repeatedly submitting implementations to a leaderboard.
- Recommendations: The paper recommends that leaderboards report validation performance for models evaluated on test sets.This recommendation targets comparison and verification without spending additional test evaluations.
7 Related Work
Related work identifies weaknesses in machine-learning evaluation, reproducibility, and documentation; this paper formalizes those concerns into reporting recommendations.
- Evaluation practice: Prior critiques describe empirical gains being incorrectly attributed to modeling choices when they instead arise from hyperparameter tuning or other evaluation factors.The paper presents its recommendations as an actionable response to these concerns.
- Reproducibility: Research on standard data splits found that shuffling training, validation, and test partitions can reduce performance and disrupt model rankings.These findings connect reporting details to reproducibility across comparisons.
- Documentation: Datasheets and model cards are proposed elsewhere to document dataset collection and cleaning, model assumptions, training data, and intended use.The paper positions its experimental reporting recommendations alongside these documentation standards.
8 Conclusion
The paper identifies shortcomings in experimental NLP reporting and introduces a technique for estimating expected validation performance as a function of computation budget.
- The paper shows that current experimental NLP practice fails to support a simple standard of reproducibility.
- It introduces a technique for estimating a method’s expected validation performance as computation budget changes.
- It presents recommendations for reporting experimental findings.
A EMNLP 2018 Checklist Survey
Table 1 reports the presence of checklist items across 50 randomly sampled EMNLP 2018 modeling papers.
- Table 1 summarizes checklist-item presence across 50 randomly sampled EMNLP 2018 papers involving modeling experiments.
B Hyperparameter Search Spaces for Section 4.2
Appendix B provides the SST fine-grained classifier hyperparameter search spaces and best assignments for CNN and logistic regression models.
- CNN classifier: Table 2 covers the SST fine-grained CNN classifier search space and best assignments.
- Logistic regression: Table 3 covers the SST fine-grained logistic regression search space and best assignments.
C Hyperparameter Search Spaces for Section 4.3
Appendix C lists hyperparameter search spaces and best assignments for models evaluated in Section 4.3, including SST, SciTail, and SQuAD systems.
- SST binary: Table 4 covers the SST binary BCN GloVe search space and best assignments.
- SST binary: Table 5 covers the SST binary BCN GloVe with frozen ELMo search space and best assignments.
- SST binary: Table 6 covers the SST binary BCN GloVe with fine-tuned ELMo search space and best assignments.
- SciTail: Table 7 covers the SciTail DAM search space and best assignments.
- SciTail: Table 8 covers the SciTail ESIM search space and best assignments.
- SciTail: Table 9 covers the SciTail n-gram baseline search space and best assignments.
- SciTail: Table 10 covers the SciTail DGEM search space and best assignments.
- SQuAD: Table 11 covers the SQuAD BiDAF search space and best assignments.