Source-linked AI summary
With Little Power Comes Great Responsibility
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, Dan Jurafsky
TL;DR
NLP experiments often lack adequate statistical power, limiting reliable detection of small model differences and increasing the risk of exaggerated findings. The paper meta-analyzes NLP studies and datasets, applies simulation-based power analysis, and finds underpowered experiments across benchmark and human-evaluation settings while offering guidance and notebooks for improvement.
Problem
Statistical power has largely been ignored in NLP, making it difficult to distinguish statistical noise from meaningful improvements and increasing the chances of exaggerated findings.
Method
The paper meta-analyzes existing NLP papers and datasets and uses simulation-based power analysis across several experimental settings.
Results
Underpowered experiments are widespread in NLP, including benchmark comparisons and human evaluations, while typical machine-translation test sets have approximately 75% power to detect 1 BLEU point differences.
Takeaways & Limitations
Power analyses, larger test sets and samples, and broader sharing of code, data, and experimental details can support more reliable NLP evaluation.
Takeaways & Limitations
The machine-translation estimates are based on a single language pair, and better estimates could be obtained by fitting the distribution to a new domain.
Abstract
from arXiv · showhide
Despite its importance to experimental design, statistical power (the probability that, given a real effect, an experiment will reject the null hypothesis) has largely been ignored by the NLP community. Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements, and increase the chances of exaggerated findings. By meta-analyzing a set of existing NLP papers and datasets, we characterize typical power for a variety of settings and conclude that underpowered experiments are common in the NLP literature. In particular, for several tasks in the popular GLUE benchmark, small test sets mean that most attempted comparisons to state of the art models will not be adequately powered. Similarly, based on reasonable assumptions, we find that the most typical experimental design for human rating studies will be underpowered to detect small model differences, of the sort that are frequently studied. For machine translation, we find that typical test sets of 2000 sentences have approximately 75% power to detect differences of 1 BLEU point. To improve the situation going forward, we give an overview of best practices for power analysis in NLP and release a series of notebooks to assist with future power analyses.
1 Introduction
The paper argues that statistical power has received too little attention in NLP, making underpowered experiments common and significant findings more vulnerable to exaggeration. It documents this problem across benchmark comparisons and human evaluations, and recommends stronger planning and reporting practices.
- Motivation: Statistical power is the probability that a test detects a true effect, but NLP has paid relatively little attention to it.Power depends on the test, threshold, effect size, variance, and sample size.
- Motivation: Underpowered experiments make meaningful improvements harder to detect and significant findings more likely to exaggerate or reverse true effects.These distortions are described as Type-M and Type-S errors.
- Findings: Evidence indicates that underpowered experiments are widespread in NLP, including human evaluations with too few items and/or raters.The paper examines how these issues affect NLP research specifically.
- Findings: Some widely used benchmark datasets, including MRPC and SST-2, are too small to properly measure future progress against top-performing models.The paper frames this as a failure to detect small effects reliably.
- Findings: A survey finds a general lack of statistical evaluation and detailed reporting in recent NLP papers.The paper also introduces a machine-translation power-analysis approach.
- Recommendations: The paper recommends prior power analyses, larger samples, shared raw data, analysis parameters, and reproducible code.These practices are intended to support more thoughtful sample-size decisions and stronger evidence of progress.
2 Power Analysis for NLP
The paper presents a reusable, simulation-based framework for estimating statistical power in NLP. It makes assumptions explicit, generates datasets under those assumptions, applies the planned test repeatedly, and estimates power from the proportion of significant results.
- Framework: The proposed approach uses simulations because many NLP tasks do not resemble standard experiments and require task-specific power analyses.The framework is described as modular, reusable, transparent, and adaptable.
- Framework: Power analyses should state assumptions explicitly because no single approach is correct for every setting.Assumptions determine the number of subjects, ratings, or test-set items needed.
- Procedure: The simulation procedure generates many datasets from assumed or estimated parameters, runs each through a significance test, and reports the significant proportion.This proportion estimates the probability of detecting a true effect under the chosen experimental setting.
- Procedure: The algorithm defines a generative process using the number of items, hypothesized effect, and other relevant parameters, then selects a test, threshold, and repetitions.The statistical test determines the detail required in the data-generating process.
- Example: For preference comparisons, samples are simulated from a binomial distribution and tested against the null proportion p = 0.5.Estimated power is the proportion of simulated differences found significant.
3 Comparing Models on Accuracy
The section develops power analysis for paired classifier comparisons and finds that small benchmark test sets often cannot reliably detect typical improvements over state of the art.
- Power analysis for accuracy: Small test sets can yield large minimum detectable effects, so only large improvements may achieve at least 80% power.Power analysis depends on the comparison design and expected model agreement.
- Significance test for comparing classifiers: McNemar’s test compares paired classifiers using the items where their predictions disagree.The test assesses whether the imbalance between the two disagreement types is statistically significant.
- Power analysis for accuracy: Power increases with test-set size, expected accuracy difference, and agreement between model outcomes.These quantities determine the simulated data-generating process for paired accuracy comparisons.
- Estimating power parameters: Validation data, historical regressions, or middle-of-the-road assumptions can provide the expected accuracy difference and agreement needed for power analysis.Validation and test comparisons for SQuAD 2.0 showed strong correlations for both quantities.
- Findings for benchmark datasets: The mean reported improvement is below the estimated MDE for WNLI, MRPC, and SST-2, leaving most future comparisons on these datasets inadequately powered.The analysis further warns that significant improvements on datasets such as MRPC may be exaggerated and fail to generalize.
- Implications: Larger test sets may be necessary for well-powered state-of-the-art claims, while MRPC and SST-2 may be candidates for retirement as comparison datasets.The recommendation follows from the large improvements required to reach 80% power on saturated tasks.
4 Machine Translation
The section adapts simulation-based power analysis to BLEU, a corpus-level machine-translation metric, and estimates that typical 2,000-sentence test sets have about 75% power to detect a 1-point BLEU difference.
- Motivation: BLEU power analysis is more difficult because BLEU is corpus-level and machine-translation evaluation data are scarce.Much parallel data has already been used in training, limiting direct estimation of performance variation.
- Significance testing: The analysis uses a randomization test that swaps paired model outputs on random subsets and compares the resulting BLEU differences.Repeated swaps produce a null distribution for testing the observed difference.
- Generative process: Individual swap effects are modeled with a mixture containing zero effects and Laplace-distributed nonzero effects.The parameters include the expected BLEU difference, the zero-effect proportion P0, and a variance-control parameter b0.
- Empirical estimates: The parameter estimates come from FAIRSEQ models evaluated on WMT English-German test sets from 2016–2019.The analysis fits a Delta-Laplace mixture to effects from swapping individual output pairs.
- Limitations and future work: The parameter estimates are based on a single language pair, and larger held-out test sets would support validation and better-powered future comparisons.The same simulation process can be adapted to new language pairs, but its specific estimates may not generalize perfectly.
- Power results: Approximately 75% power is expected for detecting a 1 BLEU-point difference with a typical 2,000-sentence test set.Power increases substantially with both test-set size and the BLEU difference.
5 Likert-Scale Human Evaluations
The meta-analysis finds that NLP human evaluations often use small samples and omit key reporting details, complicating assessment of their statistical power. Simulations based on estimated variation indicate that common designs are frequently underpowered for detecting small model differences.
- Meta-analysis: 69% of evaluations used 100 or fewer items, while only 18% used more than 200.
- Meta-analysis: 57% of experiments collected 3 annotations per item, the modal number of unique annotators.
- Meta-analysis: 34% omitted ratings per item and 28% omitted total workers, limiting evaluation of experimental validity.
- Power analysis: The simulations model item and rater variation using hierarchical mixed-effects estimates, then assess significance and power across sample sizes.
- Power analysis: Under high variance, 3 workers and 100 items are underpowered unless the effect is at least 0.2 on the [0, 1] scale.
- Power analysis: Under low variance, detecting a 0.05 effect requires an unusually large number of ratings, at least 10 per item for 100 items.
6 Overall Recommendations
The recommendations call for power analyses before evaluation, appropriately sized test sets and human studies, and broader release of data, models, code, and experimental details. These practices are intended to make comparisons more interpretable and adequately powered.
- Power analyses should precede baseline comparisons because underpowered evaluations do not provide convincing evidence of progress.
- For new datasets and shared tasks, choose the test-set size according to the minimum detectable effect size.
- Expand or retire tasks such as MRPC and SST-2 when their test sets lack adequate power for typical improvements.
- Model owners should release fine-tuned checkpoints, or leaderboard owners should publish validation predictions from submitted models.
- Human evaluations should share anonymized raw data, analysis parameters, replication code, and pre-collection power analyses.
7 Conclusion
The paper concludes that underpowered experiments are widespread in NLP and that small-sample comparisons provide unreliable evidence about improvements. It recommends larger test sets, power analyses, and wider sharing of code, data, and experimental details.
- Underpowered experiments are widespread in NLP and can make significant improvements exaggerate or reverse the true effect.
- Larger test sets, simple power analyses, and broader sharing practices would support a higher standard of experimental methodology.
A Further Discussion of Significance Testing, Power Analysis, and Post-Hoc Analysis
The discussion explains how statistical power analysis can be estimated through simulations under explicit assumptions, while distinguishing valid prospective analysis from problematic post-hoc analysis. It situates these recommendations within continued use of NHST and a narrower focus on claims about particular trained-model comparisons.
- Significance testing: NHST remains common in NLP, where equivalent baseline performance is typically treated as the null hypothesis.
- Significance testing: Sharing code and data, sensitivity analyses, attention to null findings, and preregistration can improve NHST validity.
- Scope: The paper focuses on claims about particular trained-model instantiations rather than broader claims about expected performance.
- Power analysis: Power depends on sample size, significance threshold, effect size, variance, statistical test, and other experimental parameters.
- Post-hoc analysis: Post-hoc power based on estimates from the experiment is not recommended because it adds no information beyond transforming the observed p-value.
- Power analysis: Simulation-based power analysis generates repeated datasets under an assumed process, applies the chosen significance test, and estimates power from significant outcomes.
- Power analysis: The generative process should reflect the statistical test and encode assumptions about the comparison, such as a preference proportion for a binomial test.
B Type-M and Type-S errors
Underpowered experiments do more than miss true effects: among significant findings, they can exaggerate effect magnitude or reverse its sign. These Type-M and Type-S risks grow as sample size, model differences, and agreement rates decrease.
- Underpowered experiments can systematically distort the literature when significant findings are preferentially published.The distortion combines missed true effects with exaggerated or reversed significant effects.
- Type-M error measures exaggeration of the true effect among significant findings, while Type-S error measures the probability that the observed sign is reversed.Both errors are conditional on obtaining a significant result.
- Lower power makes significant findings more likely to reflect atypically large observed effects and, under two-sided tests, to have the wrong sign.With small samples relative to effect size, significance requires an unusually large deviation from the null.
- Type-M and Type-S errors increase with smaller test sets, smaller model differences, and lower agreement rates.These conditions correspond to lower statistical power when comparing classifiers.
C Numerical Example of a McNemar’s Test Simulation
A simulated McNemar’s-test example shows how test-set size determines whether a small classifier improvement can be evaluated reliably. With 500 examples, power is very low and significant effects are strongly exaggerated; 2,000 examples perform much better.
- For 500 examples, a 2-percentage-point improvement with 90% model agreement yields approximately 0.25 power at α = 0.05.The simulation treats the two models as disagreeing on 10% of examples, with a 6% versus 4% split favoring the new model.
- A 500-example test set produces a Type-M factor of 1.9, implying significant observed differences near 0.04 for a true difference of 0.02.The estimate applies to experiments that obtain a significant result under the stated assumptions.
- 2,000 examples provide nearly 80% power with a Type-M factor of 1.1 under the same assumed improvement and agreement.The authors describe this design as comfortable for proceeding with and reporting the evaluation.
- A 4-percentage-point improvement or 97.5% agreement would make the comparison adequately powered even with the smaller test set.Greater effects or higher agreement improve the information available to McNemar’s test.
D SQuAD 2.0 Analysis and Results
The SQuAD 2.0 analysis uses leaderboard predictions to assess whether validation-based power estimates guide test-set comparisons. Validation and test estimates correlate strongly, and only a minority of purported improvements meet both power and significance criteria.
- Data and setup: After filtering anomalous leaderboard submissions, the SQuAD 2.0 analysis retains 144 models.The filtering removes six submissions with test EM below 50% and one with a development–test gap suggesting development-set training.
- Validation as a guide: Validation and test estimates of pairwise accuracy differences and agreement rates correlate above 0.99.This supports using validation measurements as a guide for power analysis, while lower development agreement can slightly underestimate test agreement.
- Power and leaderboard outcomes: Three of the remaining 124 submissions were significant on test data despite insufficient validation-based power, and all were surpassed within five days.Most other submissions were significantly worse than the current state of the art, reinforcing that real improvements are uncommon and generally small.
- Regression-based estimation: The effect-size regression achieves R^2 = 0.69 for GLUE comparisons and R^2 = 0.67 for SQuAD 2.0 state-of-the-art improvements.The authors treat these fits as priors for likely future effect sizes rather than perfect predictions.
- Caveats: Regression-based power estimates may fail when overlap is not linear or when validation performance is inflated by overfitting and tuning.The paper therefore cautions that validation–test correlations and predicted effect sizes are dataset- and setting-dependent.
E.4 Extended Results
Extended results compare minimum detectable effects and unpaired accuracy analyses across datasets. They caution that predicted future improvements can be smaller than detectable effects, making further comparisons of some datasets difficult to justify.
- Table 9 compares minimum detectable effects from a two-sample proportion test and the Lachenbruch methodology, alongside predicted and observed leaderboard effect sizes.The table also reports standard errors, sample sizes, and current versus next state-of-the-art differences.
- Predicted state-of-the-art improvements are lower than historical averages because those averages include early large gains.In some cases, predicted effects are smaller than the lowest Lachenbruch minimum detectable effect.
- If predicted improvements fall below the minimum detectable effect, the usefulness of further comparisons on that dataset may warrant reconsideration.This follows from the mismatch between likely future gains and the dataset’s detectable effect threshold.
- When samples are unpaired, the analysis uses a binomial framework and tests the null hypothesis H0: p1 = p2 against a two-sided alternative.The two-sample proportion test can calculate power, minimum detectable effect, or required sample size.
F Additional Metrics
The appendix extends power-analysis guidance beyond accuracy to metrics such as F1, Matthews correlation, and BLEU, using randomization-based simulation approaches. It also examines how assumptions about metric behavior affect power estimates.
- Recall, Precision, F1, Matthew’s correlation: F1 power analysis uses a randomization test on paired data to assess model differences.Because F1 is not a simple sum over instances, the analysis requires assumptions about per-class confusion-matrix probabilities and label frequencies.
- GLUE benchmark: 26.7% of observed GLUE improvement effects met the minimum detectable effect under the binomial power calculation.The share was 30% under the midpoint Lachenbruch calculation and 26.7% using its upper bound.
- Machine translation: BLEU power simulations model individual sample effects as a mixture of a zero-effect Delta distribution and a Laplace distribution.The mixture uses P0 for the zero-effect proportion and location and scale parameters for the remaining effects.
- Machine translation: The BLEU data-generation procedure sets µ = −2 · ∆B/(n · (1 −P0)) so that swapping half the samples has average effect −∆B.The procedure assumes values for ∆B, n, P0, and b0, with b0 estimated from empirical observations.
- Machine translation: Lower P0 requires larger effects and test sets for adequate power, whereas higher P0 permits detection with smaller samples.As the Laplace scale b increases, even large effects may remain undetectable with test sets of 5,000 samples.
H.1 Meta-analysis of human ratings for EMNLP 2019
The authors analyze human-evaluation practices and simulate power under mixed-effects and head-to-head comparison designs. Their survey finds limited significance testing and little evidence of prospective power analysis.
- Dataset and sample: 97 human-evaluation experiments from 86 relevant EMNLP 2019 workshop papers were analyzed.The set included 51 Likert experiments and 38 other experiments.
- Reported practices: 24% of experiments reported significance testing, while 63% of human-rating experiments bolded or starred the best results.Only one surveyed paper performed a power analysis to estimate sample size, and its procedure could not be determined precisely.
- Ordinal ratings: Power simulations use linear mixed-effects models with random intercepts and slopes for workers and items.The model includes fixed intercept and model-effect parameters plus worker, item, and residual variance parameters.
- Ordinal ratings: The simulations vary annotator counts, item counts, and effect sizes under high- and low-variance settings.Each setting simulates 200 experiments and counts significant positive effects with t > 1.96; the authors caution against copying the code blindly.
- Head-to-head comparisons: Head-to-head power analysis can simulate win, lose, and draw probabilities, or use richer mixed-effects models for repeated judgments.The richer approach can model win-or-lose outcomes logistically, with ties excluded or split.