PaperlayerView exact revision on arXiv

Source-linked AI summary

The Effect of Natural Distribution Shift on Question Answering Models

John Miller, Karl Krauth, Benjamin Recht, Ludwig Schmidt

arXiv:2004.14444v1cs.LGcs.CLstat.ML

TL;DR

The paper asks whether strong SQuAD leaderboard performance reflects generalization beyond the original test set. It builds four matched test sets across Wikipedia, New York Times, Reddit, and Amazon, finding no adaptive overfitting but substantial model drops under some natural shifts. Human performance changes little, motivating evaluation that explicitly incorporates distribution shifts.

  • Problem

    SQuAD evaluation measures new examples from Wikipedia, leaving open whether leaderboard performance generalizes beyond that distribution and survives repeated test-set reuse.

  • Method

    The authors replicate SQuAD data creation to build four test sets covering Wikipedia, New York Times articles, Reddit posts, and Amazon product reviews, using closely matched crowdsourcing procedures.

  • Results

    Models show no evidence of adaptive overfitting but drop 3.8 F1 points on New York Times, 14.0 on Reddit, and 17.4 on Amazon, while humans remain largely unchanged.

  • Takeaways & Limitations

    SQuAD supports progress beyond what test-set reuse alone might suggest, but evaluation should explicitly include robustness to natural distribution shifts.

  • Takeaways & Limitations

    The paper does not explain why robustness differs across New York Times, Reddit, and Amazon, and common answer-type or reasoning stratifications are insufficient.

Abstract

from arXiv · show

We build four new test sets for the Stanford Question Answering Dataset (SQuAD) and evaluate the ability of question-answering systems to generalize to new data. Our first test set is from the original Wikipedia domain and measures the extent to which existing systems overfit the original test set. Despite several years of heavy test set re-use, we find no evidence of adaptive overfitting. The remaining three test sets are constructed from New York Times articles, Reddit posts, and Amazon product reviews and measure robustness to natural distribution shifts. Across a broad range of models, we observe average performance drops of 3.8, 14.0, and 17.4 F1 points, respectively. In contrast, a strong human baseline matches or exceeds the performance of SQuAD models on the original domain and exhibits little to no drop in new domains. Taken together, our results confirm the surprising resilience of the holdout method and emphasize the need to move towards evaluation metrics that incorporate robustness to natural distribution shifts.

1 Introduction

SQuAD leaderboard progress may not reflect generalization beyond Wikipedia or resistance to natural distribution shifts. The paper addresses both concerns with four new test sets and finds no adaptive overfitting but substantial, corpus-dependent model degradation.

  • Motivation: SQuAD progress has produced large leaderboard gains, but broader generalization remains uncertain.The best 2020 models scored more than 40 F1 points above the 2016 baseline.
  • Research questions: The study asks whether SQuAD models overfit the test set and whether they withstand natural distribution shifts.The standard evaluation uses new examples from Wikipedia, while deployed systems may face other domains.
  • Approach: The authors replicate SQuAD construction to create four test sets spanning Wikipedia, New York Times, Reddit, and Amazon product reviews.The new tests preserve the original domain while adding three natural source-text shifts.
  • Findings: There is no evidence of adaptive overfitting: scores on the original and replicated Wikipedia tests differ little across many models.This conclusion also holds when comparing the public development set with the new test set.
  • Findings: Models lose an average of 3.8 F1 points on New York Times, 14.0 on Reddit, and 17.4 on Amazon.All datasets use the same generation pipeline, so the reported degradation is attributed to source-text changes rather than annotation-procedure differences.
  • Findings: Humans remain substantially more robust than models, with average F1 decreases of 0.1 on New York Times, 2.9 on Reddit, and 3.0 on Amazon.Human scores on the original SQuAD data are on par with the best SQuAD models and exceed the Mechanical Turk baseline.
  • Implications: The results indicate that SQuAD progress has not been matched by comparable progress in robustness to natural distribution shifts.The new test sets are released freely for future research.

2 Background

SQuAD is an extractive question-answering benchmark evaluated with exact match and F1. The paper formalizes the difference between performance on the original and new samples as adaptivity, distribution, and sampling gaps.

  • SQuAD: SQuAD examples contain a passage, a question, and one or more answer spans within the passage.The benchmark is an extractive question-answering dataset.
  • Evaluation: Exact match measures exact agreement with a ground-truth answer, while F1 measures maximum token overlap with an answer.Both metrics evaluate predicted answer spans against one or more ground-truth answers.
  • Scope: The study focuses on SQuAD v1.1 because its longer-public test set may have faced more adaptive reuse.The original test set is not public, so the analysis otherwise uses the public development set.
  • Generalization: The paper tests whether high leaderboard performance predicts strong results on unseen examples from the same or different distributions.This frames benchmark validity as a generalization question rather than a narrow held-out-score question.
  • Model for generalization: The performance difference between the original sample S and new sample S′ is decomposed into adaptivity, distribution, and generalization gaps.The setup compares losses on S sampled from D with losses on S′ sampled from potentially different D′.
  • Model for generalization: The adaptivity gap captures test-set reuse, the distribution gap captures changing D to D′, and the generalization gap captures sampling variation in S′.The paper empirically measures the adaptivity and distribution gaps across models and new datasets.

3 Related Work

Related work documents limited adaptive overfitting on established machine-learning benchmarks and several approaches to question-answering generalization. This paper differs by examining naturally occurring distribution shifts while also comparing adversarial shifts.

  • Adaptive data analysis: Replication studies report little or no adaptive overfitting on MNIST, CIFAR-10, ImageNet, and classification competitions.These findings motivate testing whether the pattern also holds for SQuAD and natural language processing.
  • Adaptive data analysis: Prior work offers explanations for why adaptive overfitting may not arise in standard machine-learning workflows.The paper positions its results as another empirical point for understanding overfitting.
  • Question-answering datasets: Existing question-answering research includes datasets beyond SQuAD, which this paper selects because of its long-running leaderboard and submission format.These properties make SQuAD useful for studying adaptive overfitting in NLP.
  • Generalization in question answering: Other studies evaluate SQuAD-trained models across datasets, under controlled distribution shifts, or in shared-task settings.This work extends that generalization literature with new test sets built from naturally different source domains.
  • Distribution shifts: Related work also studies adversarial corruptions and semantically meaningful perturbations, whereas this paper focuses on naturally occurring shifts.The authors additionally evaluate their model testbed on adversarial shifts for comparison in an appendix.

4 Collecting New Test Sets

The paper constructs four new SQuAD test sets across Wikipedia, New York Times, Reddit, and Amazon domains while closely replicating the original data-generation and annotation procedures. This design enables comparisons across text corpora with limited procedural confounding.

  • Passage Curation: Four test sets use Wikipedia, New York Times articles, Reddit posts, and Amazon product reviews as source corpora.The datasets were generated through passage curation, crowdsourced question-answer creation, and post-processing designed to replicate SQuAD.
  • Passage Curation: The collection pipeline samples and preprocesses passages to match key SQuAD characteristics, including passage-length statistics.Reddit and Amazon passages were subsampled to match the original SQuAD passage-length statistics; Wikipedia sampling avoided overlap with SQuAD training and development articles.
  • Crowdsourcing Question-Answer Pairs: The study uses the same MTurk interface, instructions, worker qualifications, task timing, and inflation-adjusted hourly rate as the original SQuAD process.These choices were intended to keep annotation procedures nearly identical across the new datasets and the original benchmark.
  • Crowdsourcing Question-Answer Pairs: Each paragraph received up to five initial question-answer pairs from one worker, followed by at least two additional answers per question from separate workers.The protocol also applied SQuAD-style answer cleaning and post-processing after collection.
  • Crowdsourcing Question-Answer Pairs: The authors answered approximately 1,000 questions on each new test set and the original data to establish a stronger human baseline.This supplements the MTurk answers because workers may not know SQuAD’s typical answer style and receive no performance feedback.

5 Main Results

The evaluation tests more than 100 SQuAD models for adaptive overfitting and robustness to controlled natural distribution shifts. Models closely reproduce original Wikipedia performance, but performance declines substantially on New York Times, Reddit, and Amazon data, while higher original SQuAD scores remain predictive.

  • Evaluation Setup: More than 100 SQuAD leaderboard models are evaluated with the original submissions’ weights, hyperparameters, arguments, and execution environments.The testbed includes state-of-the-art BERT and XLNet models alongside older models such as BiDAF.
  • Adaptive Overfitting: New Wikipedia test-set scores closely replicate original SQuAD test-set scores across the model collection, providing little evidence of adaptive overfitting.The observed relationship is approximately linear rather than the concave curve expected under adaptive overfitting.
  • Adaptive Overfitting: SQuAD development-set and test-set F1 scores also closely match, except for three models explicitly trained on the development set.The three exceptions are Common-sense Governed BERT-123 (April 21), Common-sense Governed BERT-123 (May 9), and XLNet-123++.
  • Robustness to Natural Distribution Shifts: Higher F1 on original SQuAD predicts higher F1 on each shifted test set, although the gap from perfect robustness varies by dataset.The gaps between perfect robustness and the observed linear fits are 3.8 F1 points for New York Times, 14.0 for Reddit, and 17.4 for Amazon.
  • Human Baseline: Human performance is substantially higher than model performance on every distribution-shifted test set and exceeds the linear fits.This leaves substantial room for model improvement on the new test sets.

6 Further Analysis

The paper tests whether dataset difficulty measures and additional training data explain or reduce performance drops under natural distribution shifts. Standard answer-type, syntactic-divergence, and reasoning analyses explain little, while some but not all MRQA models improve robustness.

  • Further Analysis: The analysis tests whether common difficulty measures and more or more diverse training data explain or improve robustness to distribution shifts.It compares answer diversity, syntactic divergence, reasoning requirements, and models trained beyond SQuAD.
  • Are The New Test Sets Harder Than The Original?: Answer-type reweighting explains only a small fraction of performance differences across test sets.
  • Are The New Test Sets Harder Than The Original?: Syntactic-divergence distributions are broadly similar across datasets, although Reddit and Amazon contain more pairs with small divergence.
  • Are The New Test Sets Harder Than The Original?: Reasoning-category reweighting explains virtually none of the observed F1 changes, despite differences in world-knowledge and multi-sentence reasoning requirements.
  • Are Models Trained with More Data More Robust to Natural Distribution Shifts?: MRQA models trained on six datasets can improve performance on Reddit and Amazon, but many show little robustness improvement and all remain below humans.Delphi achieves higher F1 than any SQuAD model on Reddit and Amazon and lies substantially above the SQuAD-model linear fit.

7 Discussion

The discussion finds that SQuAD remains resistant to adaptive overfitting, while models still lose robustness under natural distribution shifts. Humans remain largely stable, and the results motivate evaluation beyond the standard i.i.d. setting.

  • Discussion: Despite years of test-set reuse, SQuAD shows no evidence of adaptive overfitting and continues to support progress longer than expected.
  • Discussion: SQuAD models generalize well within the original distribution but suffer non-trivial, nearly uniform drops on every shifted test set.
  • Discussion: Human performance remains largely unchanged across the new test sets, leaving a substantial robustness gap between humans and models.
  • Discussion: The findings support incorporating natural distribution shifts into evaluation rather than relying only on standard i.i.d. model evaluation.
  • Discussion: Answer type and reasoning requirements do not sufficiently explain why performance drops are much larger on Reddit and Amazon than on New York Times.

A Evaluation Metrics

The evaluation uses F1 overlap and exact match to measure extractive question-answering predictions, with preprocessing that ignores selected punctuation and articles.

  • F1 Score: F1 measures the harmonic-mean overlap between predicted and ground-truth answer bags of words, taking the maximum over reference answers.
  • Exact match: Exact match measures the percentage of predictions that exactly match at least one ground-truth answer.
  • Evaluation procedure: The evaluation script ignores punctuation and the articles “a”, “an”, and “the” when computing F1 and exact match.

B Comparing Natural and Adversarial Distribution Shift

The paper compares natural distribution shifts with adversarial attacks on SQuAD models, finding that adversarial shifts produce more variable model behavior than natural shifts. Probit scaling improves linear fits, but adversarial relationships remain less tightly aligned.

  • AddSent: Adversarial AddSent evaluation produces more variability around the trend line than natural distribution shifts.For AddSent, F1 and EM fits have R2 values of 0.73 and 0.74.
  • AddSent: 0.72 R2 for AddSent F1 is lower than 0.99, 0.97, 0.91, and 0.89 for the New Wikipedia, New York Times, Reddit, and Amazon shifts.The natural-shift fits therefore track original-test performance more closely.
  • Probit scaling: 0.82 R2 in the probit domain for AddSent F1 remains below 0.99, 0.96, 0.94, and 0.94 for the four natural shifts.Probit scaling improves the adversarial fit but does not eliminate the difference from natural shifts.
  • AddOneSent: AddOneSent shows the same broad linear pattern as AddSent, with F1 and EM slopes of 1.48 and 1.34.AddOneSent randomly selects one altered passage rather than choosing the worst-scoring one.

C Additional Analysis and Results

Additional analyses show that exact-match drops preserve the main cross-dataset pattern, while alternative scaling and stratification analyses examine possible explanations for performance differences. The appendix also reports model–human comparisons and robustness analyses across difficulty-related factors.

  • Exact Match Scatterplots: 4.6, 5.75, 20.0, and 24.8 average EM-point drops occur on New Wikipedia, New York Times, Reddit, and Amazon, respectively.The ranking remains the same: a small New York Times drop followed by larger Reddit and Amazon drops.
  • Scaling analyses: Linear and probit scaling are compared for model and human F1 and EM scores across the original and new test sets.The figures report slopes and R2 statistics for both scaling choices.
  • Difficulty analyses: Model and human F1 scores are also compared after stratifying examples by label agreement, which roughly corresponds to question difficulty and ambiguity.The figure describes agreement levels based on whether labellers provide identical normalized answers.
  • Distributional analyses: F1 comparisons are stratified by answer-type, syntactic-divergence, and reasoning-required distributions.These analyses reweight SQuAD development-set scores using distributions from the new test sets.

C.7 Does Manual Data Curation Reduce Performance Drops?

The additional analyses examine whether manual curation, text length, or paragraph structure account for the observed performance drops. The reported results indicate that these factors explain little or none of the differences.

  • Manual data curation: Additional human filtering removes malformed, unanswerable, incorrect, ungrammatical questions and answers from New Wikipedia and Amazon subsets.Questions with unanimous agreement among three MTurk annotators are automatically marked valid.
  • Passage length statistics: Paragraph character lengths match exactly across the original and new datasets because sampling preserves character length.The comparison is shown in Figure 19.
  • Passage length statistics: Wikipedia paragraph word-length histograms match closely, while Amazon and Reddit paragraphs contain slightly more words.These differences do not explain the observed performance drops.
  • Passage length statistics: New datasets have a slightly longer tail in sentences per paragraph, except that New Wikipedia matches SQuAD v1.1.The paper reports that these sentence-count differences do not explain performance drops.
  • Reweighting analyses: Changes in word-per-paragraph and sentence-per-paragraph distributions do not explain the observed F1 differences.The analysis predicts new-test F1 by reweighting SQuAD v1.1 development-set bucket scores.

D.2 MTurk Experiment and UI Examples

The study reproduces the SQuAD crowdsourcing setup with specified worker requirements, modified paragraph limits, and example interfaces. It evaluates a broad, consistently configured model testbed, including over 100 leaderboard submissions and named systems such as XLNet and BERT.

  • Worker details: Crowdworkers needed a 97% HIT acceptance rate, at least 1000 HITs, and residence in the United States or Canada.Workers were allocated four minutes per paragraph for asking questions and one minute per question for answering.
  • UI examples: The new interface matches the original SQuAD setup except that each worker handles at most five paragraphs rather than all paragraphs in an article.Figures 24–27 show directions and examples for the Ask and Answer tasks.
  • Model testbed: The evaluation covers over 100 SQuAD leaderboard models using the original submission configurations.The configurations include weights, hyperparameters, command-line arguments, and execution environments.
  • Model testbed: The testbed includes XLNet, SpanBERT, tuned BERT variants, and numerous BiDAF, MARS, MEMEN, and Mnemonic Reader systems.The supplied model list includes single-model and ensemble submissions.
  • Model testbed: The model inventory also includes InfoWord, Commonsense Governed BERT, Reinforced Mnemonic Reader, AttentionReader+, FusionNet, and other leaderboard submissions.These entries identify the evaluated systems and, where available, their references and submission bundles.

E.2 Full Results Tables

This section reports model and human F1 and exact-match results across four new test sets, comparing them with original SQuAD performance. It also includes adversarial distribution-shift results and ranking comparisons.

  • Full results: The results cover SQuAD models, five MRQA models, and three human annotators across the New Wikipedia, New York Times, Reddit, and Amazon test sets.F1 results appear in Tables 5–8, while exact-match results appear in Tables 9–12.
  • F1 results: F1 tables compare original SQuAD scores with New Wikipedia, New York Times, Reddit, and Amazon test-set scores.The tables report ranks, new ranks, relative rank differences, and confidence intervals.
  • Exact-match results: Exact-match tables provide the corresponding comparisons for New Wikipedia, New York Times, Reddit, and Amazon test sets.They likewise report original and new rankings, relative rank differences, and confidence intervals.
  • Adversarial results: Adversarial results summarize model F1 and exact-match scores under the AddSent and AddOneSent distribution shifts.The adversarial results are reported in Tables 13–16.
Loading 2004.14444v1…