PaperlayerView exact revision on arXiv

Source-linked AI summary

The Effect of Natural Distribution Shift on Question Answering Models

John Miller, Karl Krauth, Benjamin Recht, Ludwig Schmidt

arXiv:2004.14444v1cs.LGcs.CLstat.ML

TL;DR

It is unclear whether strong SQuAD scores indicate generalization beyond Wikipedia, and repeated test-set evaluation raises concern about adaptive overfitting. The paper builds four new test sets and finds no evidence of adaptive overfitting, but SQuAD models show non-trivial performance drops under natural distribution shifts.

  • Problem

    It is unclear whether strong SQuAD scores reliably indicate generalization beyond Wikipedia, while repeated evaluation may produce adaptive overfitting.

    Source: §1 · p. 1

  • Method

    The authors replicate SQuAD test-set creation to build four new test sets from Wikipedia, New York Times articles, Reddit posts, and Amazon product reviews.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    Despite no evidence of adaptive overfitting, SQuAD models show non-trivial drops under natural shifts while the human baseline is largely unchanged; Reddit drops averaged 14.0 F1 points.

    Source: §1 · pp. 2–3, §7 · p. 15

  • Takeaways & Limitations

    Evaluation should explicitly incorporate distribution shifts rather than remain limited to the standard i.i.d. setting.

    Source: §7 · p. 15

    Why performance drops vary across datasets remains unexplained, and stratifying by answer type or reasoning required is insufficient.

    Source: §7 · pp. 15–16

Abstract

from arXiv · show

We build four new test sets for the Stanford Question Answering Dataset (SQuAD) and evaluate the ability of question-answering systems to generalize to new data. Our first test set is from the original Wikipedia domain and measures the extent to which existing systems overfit the original test set. Despite several years of heavy test set re-use, we find no evidence of adaptive overfitting. The remaining three test sets are constructed from New York Times articles, Reddit posts, and Amazon product reviews and measure robustness to natural distribution shifts. Across a broad range of models, we observe average performance drops of 3.8, 14.0, and 17.4 F1 points, respectively. In contrast, a strong human baseline matches or exceeds the performance of SQuAD models on the original domain and exhibits little to no drop in new domains. Taken together, our results confirm the surprising resilience of the holdout method and emphasize the need to move towards evaluation metrics that incorporate robustness to natural distribution shifts.

1 Introduction

The study tests whether SQuAD models generalize beyond the original test set, finding little evidence of adaptive overfitting but substantial, domain-dependent performance drops. Humans show much smaller declines on the new domains and outperform the best SQuAD models on each new test set.

Study design

  • The study replicates SQuAD’s data-creation process to build four test sets: one in Wikipedia and three in New York Times articles, Reddit posts, and Amazon product reviews.The evaluation addresses whether leaderboard performance generalizes beyond the Wikipedia distribution measured by standard SQuAD evaluation.Source: §1 · pp. 1–2

Results

  • Across many models, original and replicated SQuAD test-set F1 scores differed little, even when compared with the publicly answered development set, extending this finding to NLP.The authors report this as no evidence of adaptive overfitting.Source: §1 · pp. 2–3
  • 3.8, 14.0, and 17.4 average F1-point drops occurred on New York Times, Reddit, and Amazon, respectively, despite a shared pipeline that isolates changes in source text.The figure summarizes drops across the new datasets and shows that their magnitude depends on the corpus.Source: Figure 1 · p. 2, §1 · p. 3
  • Humans’ average F1 fell 0.1 on New York Times, 2.9 on Reddit, and 3.0 on Amazon, while their scores exceeded the best SQuAD models on each new test set.On the original SQuAD data, human accuracy was on par with the best SQuAD models.Source: Figure 1 · p. 2, §1 · p. 3

Contribution

  • All four new test sets are freely available online to support future research.Source: §1 · p. 3

2 Background

This section defines the SQuAD task and evaluation metrics, then frames generalization as the challenge of maintaining performance on new examples from the same or different distributions. It distinguishes adaptive overfitting from distribution shift and sampling variability.

2.1 Stanford Question Answering Dataset

  • SQuAD examples pair a passage and question with one or more answer spans; the paper focuses on v1.1 rather than v2.0, which adds unanswerable questions.Because the v1.1 test set is not public, analyses otherwise use its public development set, while also reporting public test-set evaluation numbers.Source: §2.1 · p. 3
  • Exact match (EM) scores predictions that exactly match a gold answer, while F1 measures maximum token overlap between a predicted span and a gold answer.Source: §2.1 · p. 3

2.2 A Model for Generalization

  • Held-out SQuAD performance is intended to indicate generalization to unseen examples, so the paper tests whether high leaderboard scores persist on new data from the same or different distributions.Source: §2.2 · p. 4
  • The loss difference between a new sample and the original test set decomposes into adaptivity, distribution, and generalization gaps; the new Wikipedia set’s average Student’s t-confidence interval is ±0.6 F1.The adaptivity gap reflects test-set adaptation bias, the distribution gap reflects a change from D to D′, and the generalization gap reflects sampling variation on S′; independent sampling makes the latter typically small and well-controlled.Source: §2.2 · p. 4
  • The experiments measure both adaptivity and distribution gaps across a wide range of SQuAD models using new test sets sampled from varied distributions.Source: §2.2 · p. 5

3 Related Work

The related work positions the study as extending adaptive-overfitting investigations to heavily scrutinized NLP and as isolating input-text corpus as the distribution-shift factor in question answering.

Adaptive data analysis

Datasets for question answering

Generalization in question answering

  • Prior QA generalization studies vary several dataset dimensions, whereas this work varies only the input text corpus and finds model F1 drops with essentially constant human F1.For example, human performance differs by 10 F1 points between SQuAD v1.1 and NewsQA (Trischler et al., 2017).Source: §3 · p. 5

Adversarial robustness

  • Unlike adversarial-corruption and example-perturbation studies, this work focuses on naturally occurring distribution shifts and also evaluates its model testbed on adversarial shifts for comparison.Related approaches include Jia and Liang (2017), Ribeiro et al. (2018), Kaushik et al. (2019), and Gardner et al. (2020).Source: §3 · p. 6

4 Collecting New Test Sets

The authors built four new SQuAD test sets by curating passages from Wikipedia, New York Times, Reddit, and Amazon reviews, then collecting crowdworker question-answer pairs with a protocol designed to replicate SQuAD.

4.1 Passage Curation

  • Wikipedia passages came from 48 non-overlapping articles sampled from the original SQuAD article list; New York Times passages came from 2015 articles and were length-matched to SQuAD.Wikipedia text was taken from around June 16, 2016, to limit temporal variation; short New York Times paragraphs were sometimes merged before subsampling.Source: §4.1 · p. 6
  • Reddit posts from January 2016 and Home and Kitchen Amazon reviews were cleaned and subsampled to match SQuAD passage-length statistics.Reddit posts were restricted to safe-for-work content, stripped of Markdown, and had links replaced with a token; Amazon reviews came from McAuley et al. (2015).Source: §4.1 · pp. 6–7

4.2 Crowdsourcing Question-Answer Pairs

  • Crowdworkers used a near-identical SQuAD protocol and interface, asking and answering up to five questions per paragraph, with at least two additional answers per question.The team reused SQuAD's task instructions, worker qualifications, and time-per-task, while adjusting the hourly rate for inflation; it applied SQuAD-style cleaning afterward.Source: §4.2 · p. 7
  • 1.5 F1 was the average score difference between original and new Wikipedia sets despite two protocol deviations, and 95% of models were within 2.7 F1.The deviations were interfacing directly with MTurk rather than the discontinued Daemo platform and limiting workers to five paragraphs per task.Source: §4.2 · p. 7

4.3 Human Evaluation

  • Graduate student and postdoc authors answered about 1,000 questions on each new test set and the original SQuAD development set to establish a stronger human baseline.Each first labelled 500 training examples and compared their answers with the ground truth before answering the evaluation questions.Source: §4.3 · p. 8

5 Main Results

Across more than 100 SQuAD models, scores on a new Wikipedia test closely tracked the original test, while performance fell under natural distribution shifts; humans outperformed models on the shifted test sets.

5 Main Results

  • The evaluation covered over 100 SQuAD leaderboard models, run with the same configuration as their original submissions.The collection included XLNet, BERT, and BiDAF.Source: §5 · p. 8

5.1 Adaptive Overfitting

  • F1 scores on the new Wikipedia test closely replicated original SQuAD test scores, providing little evidence of adaptive overfitting.The observed linear fit contrasted with the concave curve expected under adaptive overfitting.Source: §5.1 · p. 8, Table 2 · p. 9

5.2 Robustness to Natural Distribution Shifts

  • All models lost F1 on new test sets; BERT dropped around 2.1 F1 points on New York Times, while XLNet lost nearly 10 F1 and 40 EM points there.BERT dropped around 11.9 F1 points on Amazon and 11.5 F1 points on Reddit; XLNet was a clear outlier despite generalizing well to new Wikipedia data.Source: §5.2 · p. 9

5.1 Adaptive Overfitting

  • SQuAD development and test F1 scores closely matched except for three models explicitly trained on the development set; the linear fit slope was 0.97.The exceptions were Common-sense Governed BERT-123 (April 21), Common-sense Governed BERT-123 (May 9), and XLNet-123++.Source: §5.1 · pp. 8–9, Figure 3 · p. 10

5.2 Robustness to Natural Distribution Shifts

  • 3.8 F1 points for New York Times, 14.0 points for Reddit, and 17.4 F1 for Amazon separate the observed linear fits from perfect robustness, though higher SQuAD v1.1 performance predicts higher shifted-set performance.The relationship between original and shifted-set F1 scores is well-captured by a linear fit.Source: §5.2 · p. 10
  • Human performance substantially exceeded model performance on every shifted test set, ruling out a shift in Bayes error rate as the sole explanation for model score drops.The results indicate substantial room for improvement on the new test sets.Source: §5.2 · p. 10

6 Further Analysis

Further analyses find that differences in answer types, syntactic divergence, and reasoning explain little or none of the performance gaps across test sets. Training on more data yields mixed robustness gains: some models improve substantially, but all remain below human performance and robustness.

6 Further Analysis

  • 0.94 was the R2 statistic in the probit domain, versus 0.89 in the linear domain, where visual inspection also favored the probit-domain fit.Source: Figure 4 · p. 11

6.1 Are The New Test Sets Harder Than The Original?

  • Answer-type shifts, including more person answers in NYT and adjective phrases in Amazon, did not explain performance differences, and reweighting by answer type explained only a small fraction of them.Source: §6.1 · pp. 11–12, Figure 5 · p. 13
  • Datasets had similar syntactic-divergence distributions, though Reddit and Amazon had more low-divergence pairs; reweighting by divergence explained only a small fraction of performance differences.Source: §6.1 · p. 12, Figure 6 · p. 14
  • Amazon and Reddit had more questions requiring world knowledge to resolve lexical variation, while NYT had more multi-sentence reasoning, but these differences did not explain drops and category reweighting explained virtually none.Source: §6.1 · pp. 12–13

6.2 Are Models Trained with More Data More Robust to Natural Distribution Shifts?

  • The MRQA 2019 Shared Task (Fisch et al., 2019) trained models on six QA datasets, including SQuAD, and evaluated them on 12 held-out datasets that also differed in collection, question distribution, and question-passage relationship.Source: §6.2 · pp. 13–14
  • On Reddit and Amazon, Delphi (Longpre et al., 2019) achieved higher F1 scores than any SQuAD model, but many more-data models gained little robustness and all remained below human F1 and robustness.The reported comparison is for the best MRQA model in the testbed; the passage does not provide exact F1 values.Source: §6.2 · p. 14

7 Discussion

SQuAD models show little evidence of adaptive overfitting on the original benchmark, yet remain vulnerable to natural distribution shifts that leave human performance largely unchanged. The findings support evaluating robustness beyond the standard i.i.d. setting.

Discussion

  • Despite years of test-set reuse, the authors find no evidence of adaptive overfitting on SQuAD.They conclude that benchmarks such as SQuAD can support progress longer than first-principles reasoning might suggest.Source: §7 · p. 14
  • SQuAD models generalize to same-distribution examples but suffer substantial, nearly uniform drops on shifted datasets, while a strong human baseline remains largely unchanged.The authors argue that evaluation should explicitly incorporate distribution shifts rather than focus only on the standard i.i.d. setting.Source: §7 · pp. 14–15
  • Differences in reasoning demands do not explain the observed performance drops, despite Reddit and Amazon having more world-knowledge cases and New York Times and Amazon requiring more multi-sentence reasoning.The comparison used a random sample of 192 examples from each dataset.Source: Table 4 · p. 15
  • MRQA models trained on five datasets beyond SQuAD show increased robustness across the new datasets, especially when their original SQuAD F1 is high, but still lag human performance.Figure 7 presents their human and model F1 scores on the original test set and the new datasets.Source: Figure 7 · p. 15
  • Future work should develop dataset-comparison metrics because stratifying by answer type or reasoning required appears insufficient to explain why performance drops differ across datasets.The authors specifically ask why models perform well on New York Times but experience larger drops on Reddit and Amazon.Source: §7 · pp. 15–16
  • The authors also call for studying when additional training data improves robustness and how to collect and use it, noting that Delphi (Longpre et al., 2019) benefits substantially while some MRQA models remain near the SQuAD models’ linear trend line.They identify this as an open empirical and theoretical question.Source: §7 · p. 16

A Evaluation Metrics

The evaluation uses F1 overlap and exact match against the ground-truth answers, following Rajpurkar et al.’s (2016) script conventions for normalization.

A Evaluation Metrics

  • F1 is the maximum bag-of-words harmonic-mean precision/recall score between the prediction and any ground-truth answer.The evaluation script ignores punctuation and the articles “a”, “an”, and “the” when computing F1.Source: A Evaluation Metrics · p. 21
  • Exact match is the percentage of predictions that exactly match any one of the ground-truth answers.The evaluation script ignores punctuation and the articles “a”, “an”, and “the” when computing exact match.Source: A Evaluation Metrics · p. 21

B Comparing Natural and Adversarial Distribution Shift

Adversarial AddSent and AddOneSent evaluations retained broad linear relationships with original SQuAD scores but showed greater variability than natural distribution shifts. AddSent’s fit remained weaker after probit scaling.

AddSent

  • AddSent appends up to five generated sentences to each passage and evaluates the altered example with the lowest model score.The added sentences are made similar to the correct answer, while crowdworkers remove ungrammatical or contradictory ones.Source: B Comparing Natural and Adversarial… · p. 22
  • AddSent F1 fit had R² 0.72, below the natural-shift sets’ 0.99, 0.97, 0.91, and 0.89, with greater variability around the trend line.In probit space, AddSent F1 R² was 0.82 versus 0.99, 0.96, 0.94, and 0.94 for New Wikipedia, New York Times, Reddit, and Amazon, respectively.Source: B Comparing Natural and Adversarial… · pp. 22–23, Figure 8 · p. 22, Figure 9 · p. 23

AddOneSent

  • AddOneSent randomly selects one of five altered passages without model access, and its performance likewise follows a linear trend with more variability than natural shifts.The reported linear-fit slopes are 1.48 for F1 and 1.34 for EM; R² statistics are 0.79 and 0.80, respectively.Source: B Comparing Natural and Adversarial… · p. 23, Figure 10 · p. 24

C Additional Analysis and Results … C.6 Do Shifts in Reasoning Required Distributions Predict Performance Drops?

The appendix extends the distribution-shift analysis with exact-match comparisons and tests whether annotator agreement or changes in answer, syntax, and reasoning distributions account for performance losses. It finds that category and syntactic reweighting explain little of the drops, while reasoning-based predictions closely follow a no-change baseline.

C.1 Exact Match Scatterplots

  • EM drops averaged 4.6, 5.75, 20.0, and 24.8 on new Wikipedia, New York Times, Reddit, and Amazon, respectively.The EM results show little evidence of Wikipedia overfitting and the same drop ranking as F1: smaller on New York Times, larger on Reddit, and largest on Amazon.Source: §C.1 · p. 24, Figure 11 · p. 25

C.2 Linear Fits in the Probit Domain

  • For wide-ranging scores, probit models can fit better than linear models; the paper reports a better fit for Amazon F1 scores.The probit scale transforms F1 scores using the inverse Gaussian CDF.Source: §C.2 · p. 25, Figure 12 · p. 26, Figure 13 · p. 27

C.3 Does Annotator Agreement Correlate with Performance Drops?

  • Across datasets, models show proportionally smaller performance drops on easier or less ambiguous questions with higher annotator agreement.The analysis uses label agreement as a measure of question difficulty and ambiguity.Source: Figure 14 · p. 28

C.4 Do Shifts in Answer Category Distributions Predict Performance Drops?

  • Answer-category shifts predict only 2-3 F1 points of Reddit loss, versus the observed 14.0 F1 points.Across the new datasets, category-based predictions are exceedingly optimistic and explain little of the observed performance drops.Source: Figure 15 · p. 29

C.5 Do Shifts in Syntactic Divergence Predict Performance Drops?

  • Syntactic-divergence shifts predict 1.9 F1 points of Reddit loss, versus the observed 14.0 F1 points.Across the new datasets, syntactic-distribution predictions explain only a small amount of the observed performance differences.Source: Figure 16 · p. 30

C.6 Do Shifts in Reasoning Required Distributions Predict Performance Drops?

  • Reasoning-category reweighting predictions closely follow the y = x no-change baseline across the distribution-shift datasets.The analysis finds that changes in reasoning-required distributions explain little of the observed performance differences.Source: Figure 17 · p. 31

C.7 Does Manual Data Curation Reduce Performance Drops? · D Dataset collection details. · D.1 Passage Length Statistics

Manual filtering barely changed average model F1, while differences in paragraph length did not explain the performance drops across datasets. The appendices describe the collection pipeline and compare paragraph-length distributions.

C.7 Does Manual Data Curation Reduce Performance Drops?

  • Models gained an average 0.86 F1 points on filtered New Wikipedia data but lost 0.09 F1 points on filtered Amazon data.The curated subsets contained 5,574 New Wikipedia questions and 6,471 Amazon questions.Source: Figure 18 · p. 32
  • Human filtering preserved the model rank order and the linear trend seen on the full datasets.Annotators automatically marked questions valid when all three agreed on the answer; additional manual review removed questions or answers judged malformed, unanswerable, incorrect, or ungrammatical.Source: Figure 18 · p. 32

D Dataset collection details.

  • The collection appendix provides further details about the data-collection pipeline, including manual review of New Wikipedia and Amazon questions and answers.Reviewers inspected 1,894 additional New Wikipedia questions and 1,839 Amazon questions; they removed 85 and 46 questions, respectively, and removed answers for 444 and 282 questions, respectively.Source: Figure 18 · p. 32, D Dataset collection details · p. 32

D.1 Passage Length Statistics

  • The passage-length analysis split paragraphs into sentences, words, and characters with spaCy and computed length-distribution histograms for each dataset.The cited passage reports use of spaCy.Source: §D.1 · p. 32
  • Small differences in passage-length distributions did not explain the observed performance drops.The analysis reweighted SQuAD F1 scores stratified by words or sentences per paragraph according to each new test set’s length distribution.Source: §D.1 · p. 32, Figure 22 · p. 36, Figure 23 · p. 36
  • Character-length histograms matched exactly across datasets; Wikipedia word and sentence distributions matched closely, while Amazon and Reddit passages had slightly more words and longer sentence-count tails.The character match followed sampling designed to match character length; new Wikipedia also matched SQuAD in sentence counts.Source: Figure 19 · p. 33, Figure 20 · p. 34, Figure 21 · p. 35

D.2 MTurk Experiment and UI Examples · E Complete Model Testbed and Results Tables · E.1 Models Evaluated

The appendices document the crowdwork protocol and UI, then describe a broad, consistently reproduced model testbed for evaluating performance across distribution-shift datasets. The model evaluation also includes five MRQA systems, while some competition submissions could not be run.

D.2 MTurk Experiment and UI Examples

  • Workers needed a 97% HIT acceptance rate, at least 1000 HITs, and a U.S. or Canadian location; pay was $9.60 per hour.The task guidance allotted four minutes per paragraph for question asking and one minute per question for answering, with a 6.52% inflation adjustment from 2016 to 2019.Source: §D.2 · p. 37
  • Figures 24–27 show the Ask and Answer task directions and example HITs; the UI otherwise matched SQuAD’s original setup, but workers handled at most five paragraphs.In the Ask task workers posed questions, and in the Answer task they answered questions generated during Ask.Source: §D.2 · p. 37, Figure 24 · p. 38, Figure 25 · p. 39

E Complete Model Testbed and Results Tables

  • The complete testbed section reports model results on four distribution-shift datasets and the adversarial distributions discussed in Section B.The supplied passage describes the section’s scope, not its results.Source: E Complete Model Testbed and Results… · p. 42

E.1 Models Evaluated

  • Over 100 SQuAD leaderboard models were evaluated in their original CodaLab configurations and listed by SQuAD v1.1 Test F1 score.The reproduced configuration included weights, hyperparameters, command-line arguments, and execution environment.Source: §E.1 · p. 42
  • Five MRQA Shared Task models were evaluated on the new test sets in the same configurations as their original submissions (Fisch et al., 2019).The remaining competition models were not publicly accessible or otherwise could not run on CodaLab.Source: §E.1 · p. 46

E.2 Full Results Tables

The appendix compiles F1 and exact-match results for models and human annotators across four new test sets, alongside comparisons with original SQuAD performance, and summarizes model scores on adversarial shifts.

E.2 Full Results Tables

Related papers

Loading 2004.14444v1…