Source-linked AI summary
Appendix - Recommended Statistical Significance Tests for NLP Tasks
Rotem Dror, Roi Reichart
TL;DR
NLP researchers need valid significance tests to draw conclusions from experimental results, including comparisons between algorithms. This appendix recommends tests for common tasks and evaluation measures, with assumptions to guide selection. It advises using parametric tests when normality holds and non-parametric alternatives otherwise, while noting that coreference measures based on precision and recall are not valid targets for the listed parametric tests.
Problem
Researchers need valid significance tests to draw conclusions from NLP experiments, particularly when comparing algorithms.
Source: Statistical Significance Test Table · p. 1Method
The appendix presents tests for common NLP evaluation measures and explains the assumptions behind its recommendations.
Source: Statistical Significance Test Table · p. 1Results
Parametric tests are recommended when normality holds because they have higher statistical power; otherwise, the authors recommend non-parametric tests.
Source: Statistical Significance Test Table · p. 1Takeaways & Limitations
For algorithms tested on the same dataset, use the paired version of the significance test, such as the matched-pair t-test.
Source: Statistical Significance Test Table · p. 1Coreference measures are functions of precision and recall, so the listed parametric tests are not valid for them.
Source: Assumptions and Comments · p. 3
Abstract
from arXiv · showhide
Statistical significance testing plays an important role when drawing conclusions from experimental results in NLP papers. Particularly, it is a valuable tool when one would like to establish the superiority of one algorithm over another. This appendix complements the guide for testing statistical significance in NLP presented in \cite{dror2018hitchhiker} by proposing valid statistical tests for the common tasks and evaluation measures in the field.
Statistical Significance Test Table
The table recommends valid significance tests for NLP evaluation measures and explains their assumptions. Test choice depends on normality and whether the algorithm outputs are paired.
Statistical Significance Test Table
- The table presents valid tests and their assumptions for each evaluation measure, basing its recommendations on considerations discussed in.A test is valid here if it bounds the probability of a Type I error by a predefined constant.Source: Statistical Significance Test Table · p. 1
- Parametric tests have higher statistical power when their normality assumption holds; otherwise, use non-parametric tests without that assumption.The authors mark measures for which normality is unlikely in the parametric-test column.Source: Statistical Significance Test Table · p. 1
- For precision, recall, and F-score, the t-test is suitable for recall but not precision or F-score, according to [14].For other measures, the authors suggest checking normality with statistical tests; they list a parametric test only when normality seems likely.Source: Statistical Significance Test Table · p. 1
- For comparisons on the same dataset, use the paired version of the significance test, such as the matched-pair t-test.Implementations of paired versions of the listed tests and other tests are available in the cited repository.Source: Statistical Significance Test Table · p. 1
- The listed statistical measures of agreement include Krippendorf’s alpha and Cohen’s kappa; the passages separately mention a Wilcoxon signed-rank test without identifying it as an agreement measure.Source: Statistical Significance Test Table · pp. 2–3
Assumptions and Comments
The comments qualify test recommendations with assumptions about the evaluation measure and data. They also specify settings where particular tests or interpretations apply.
Assumptions and Comments
- A t-test assumes that an average count of correct classifications is normally distributed.Source: Assumptions and Comments · p. 3
- Bootstrap assumes the dataset represents the whole population, requiring a sample large enough to cover many use-cases and the domain satisfactorily.Source: Assumptions and Comments · p. 3
- For Pearson correlation, the Fisher transformation approximately follows a normal distribution with mean F(ρ) and standard deviation 1; it can be used in a z-test for significance.Here, n is the sample size.Source: Assumptions and Comments · p. 3
- Unlabeled and labeled attachment scores are treated as accuracy measures.Source: Assumptions and Comments · p. 3
- For language models with very large prediction vocabularies, the authors prefer non-sampling non-parametric tests such as the Wilcoxon signed-rank test.Source: Assumptions and Comments · p. 3
- Coreference measures, being functions of precision and recall, are not valid targets for the listed parametric tests.Source: Assumptions and Comments · p. 3