Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets

1 Introduction

• Comparisons across languages, domains, and genres have become standard as heterogeneous written-language data and NLP evaluation setups have grown.The paper notes the expectation that new algorithms be evaluated on as many such setups as possible.Source: §1 · p. 1
• Replicability Analysis addresses two questions: how many datasets show an algorithm's superiority, and which datasets they are.Its framework adapts statistical methods for counting and identification to NLP evaluation setups.Source: §1 · pp. 1–2
• Per-dataset significance testing does not guarantee control of the chance of making at least one erroneous claim as evaluation sets accumulate.The paper argues this approach becomes more error-prone as the number of datasets grows.Source: §1 · p. 2
• The authors demonstrate the framework on multi-domain dependency parsing, multilingual POS tagging, cross-domain sentiment classification, and word similarity prediction.They report that current NLP practices for counting and identifying effects can be error-prone and contrast them with the proposed approach.Source: §1 · p. 2
• The framework aims to support statistically sound conclusions while summarizing many experiments in a handful of interpretable numbers.The authors hope it will encourage adding standard evaluation setups when appropriate, potentially reaching hundreds of comparisons per study.Source: §1 · p. 2

Multiple Comparisons in NLP

• Multiple-dataset comparisons are common in NLP, including multilingual machine translation, domain adaptation, and lexical-semantic tasks.Examples include language-pair evaluations, domain-specific datasets, and word-embedding comparisons on word-pair judgments.Source: §2 · p. 3
• As comparison tables accumulate performance and significance figures, the section argues that growing comparison counts require more intricate analysis.The usual summaries may also include cross-comparison statistics such as average performance.Source: §2 · p. 3

Existing Analysis Frameworks

• Demšar’s methods test whether at least one dataset shows a significant difference, whereas this paper aims to count and identify datasets where one algorithm significantly outperforms another.The paper says its questions provide more intricate information, especially when datasets come from different sources.Source: §2 · p. 3
• Søgaard (2013) modified Demšar’s tests using a Gumbel-distribution assumption to estimate effects across datasets, but did not answer the counting or identification questions.The survey notes this was, to the authors’ knowledge, the only earlier work addressing statistical properties of evaluation across multiple datasets.Source: §2 · p. 4

3 Preliminaries

• The setup compares algorithms A and B across datasets that may come from different languages or domains, using one or more evaluation measures.For concision, the paper assumes one measure throughout, while noting the framework can be extended to multiple measures.Source: §3.1 · p. 4

3.1 Hypothesis Testing

• In a single-dataset hypothesis test, rejecting the null supports concluding that A outperforms B in that setup; non-rejection is insufficient evidence for that conclusion.The test controls type I error at a predefined significance level while seeking high power.Source: §3.1 · p. 4

3.2 The Multiplicity Problem

• Testing each dataset separately at the nominal significance level can produce more erroneous claims as the number of datasets grows, making naive significance counts error-prone.The paper distinguishes controlling the chance of any type I error from addressing less restrictive error criteria.Source: §3.2 · pp. 4–5

3.3 Partial Conjunction Hypotheses

• The partial conjunction test asks whether at least u of N null hypotheses are false, where u is specified in advance.This is a compromise between requiring an effect on every dataset and merely finding evidence of an effect on at least one; counting and identification are treated separately, and estimated counts may exceed identified datasets.Source: §3.3 · p. 5, §3.2 · p. 5

4 Replicability Analysis for NLP

• The framework treats findings across studies differing in language, domain, or genre as replications, while noting that dependent-dataset dependencies are assumed to be undetermined.The authors say the nature of such dependencies had not been analyzed in NLP work to their knowledge.Source: §4 · pp. 5–6, §4.3 · p. 8

4.1 The Partial Conjunction p−value

• The partial conjunction framework tests whether algorithm A is superior to B on at least u of N datasets, combining their p-values to assess that claim.Its lower bound on the number of datasets with an advantage, at confidence level 1−α, answers the counting question.Source: §4.1 · p. 6, §4 · p. 6

4.2 Dataset Counting (Question 1)

• The estimated count k̂ is the largest u whose partial conjunction null is rejected, with P(k̂ > k) ≤ α when the p-value combination is valid for the datasets’ dependence.Use k̂Fisher when datasets are independent; use k̂Bonferroni when independence cannot be assumed.Source: §4.2 · p. 7

4.3 Dataset Identification (Question 2)

• Holm’s procedure returns a rejection list identifying datasets while bounding the probability of any false rejection at α for any dependency among datasets.It rejects the hypotheses with the smallest p-values using thresholds that depend on the number of evaluation datasets.Source: §4.3 · pp. 7–8

Independent datasets

• 0.963, 0.001, and 0.021 were the estimated overestimation probabilities for k̂count, k̂Bonferroni, and k̂Fisher, respectively, with 100 independent null datasets.Across 1,000 repetitions, only the Bonferroni and Fisher estimates were below 0.05.Source: §5 · p. 8

Dependent datasets

• 0.943, 0.046, and 0.234 were the estimated overestimation probabilities for k̂count, k̂Bonferroni, and k̂Fisher, respectively, with dependent datasets.The simulation used 100 p-values from 34 independent and two positively correlated groups of 33 test statistics, with correlations ρ = 0.2 and ρ = 0.5.Source: §5 · p. 9
• The dependent-data simulation supports using Bonferroni rather than Fisher when datasets are dependent, even when some datasets are independent.All null hypotheses were true in this 1,000-repetition simulation.Source: §5 · p. 9

6.1 Data

• The study examines four NLP applications: multi-domain dependency parsing and multilingual POS tagging with independent datasets, plus cross-domain sentiment classification and word-similarity prediction with dependent datasets.The POS evaluation covers 23 of 44 shared languages and copies word-level accuracy for the low-resource setup; sentiment uses 12 domain pairs, and word similarity evaluates 12 English word-pair datasets.Source: §6 · p. 9, §6.1 · pp. 9–10

6.2 Statistical Significance Tests

• The analyses use paired bootstrap for parsing, Wilcoxon signed-rank for POS tagging, McNemar for sentiment, and Steiger’s test for word-similarity correlations.The tests are selected for the respective evaluation settings, including paired sentence-level accuracy for POS and paired nominal data for sentiment.Source: §6.2 · p. 10

6.3 Results

• Holm identification named all seven domains for Mate over SpaCy, only MZ for Mate over Redshift, and six languages for MIMICK over Char→Tag.The identified languages were Tamil, Hungarian, Basque, Indonesian, Chinese, and Czech.Source: §6.3 · p. 12
• For dependent sentiment and word-similarity datasets, kBonferroni estimates were 6 and 2 for sentiment, and 6 and 4 for word similarity, at α = 0.05 and 0.01, respectively.The text characterizes kBonferroni as more conservative and says it controls the probability of overestimating the number of datasets with effect.Source: Table 1 · p. 10, §6.3 · p. 12
• Across the applications, counting was too conservative in some independent setups but could be too optimistic when evaluation datasets were dependent.The sentiment example gives 10 versus 6 out of 12 setups for counting and the proposed analysis at α = 0.05.Source: §6.3 · p. 13
• In independent setups, replicability analysis estimated more affected datasets than counting for Mate–Redshift (5 vs. 2) and MIMICK–Char→Tag (16 vs. 11).For Mate–Redshift, kBonferroni estimated 1; for MIMICK–Char→Tag, it estimated 6, while kFisher exploits dataset independence.Source: Table 1 · p. 10, §6.3 · pp. 11–13

• The framework addresses both estimating the number of datasets with an effect and identifying those datasets.Source: §7 · p. 13
• For dependent datasets with unknown dependency type, the framework uses general procedures; specified dependency assumptions can permit more powerful alternatives.Examples include Simes-based partial conjunction and Hochberg’s or Hommel’s procedures under positive dependency properties.Source: §7 · p. 13
• Multiple evaluation sets can support cross-language and cross-domain research, but sets reflecting very similar linguistic phenomena may complicate algorithm comparisons.Source: §7 · p. 13
• The analysis is most useful for heterogeneous datasets from different languages or domains; technically different but combinable datasets may be better treated as one large dataset.For the latter case, the paper points to asking whether at least one dataset shows evidence for an effect.Source: §7 · p. 13