Source-linked AI summary
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
Patrick Lewis, Pontus Stenetorp, Sebastian Riedel
TL;DR
ODQA benchmark scores do not reveal how much performance reflects memorization versus generalization. The paper analyzes train-test overlap in three datasets and evaluates models on overlap-stratified subsets, finding substantial overlap and much stronger performance on memorizable data.
Problem
Single aggregated ODQA test scores provide limited evidence about whether models memorize training questions, reuse familiar answers, or generalize to novel questions and answers.
Method
The paper analyzes answer and question overlap in three ODQA datasets, then evaluates open-book, closed-book, and nearest-neighbor models on stratified test subsets.
Results
60% of test answers overlap with training data, 30% of test questions have training duplicates, and nearest-neighbor models outperform a BART closed-book model.
Takeaways & Limitations
Overall QA accuracy cannot properly characterize ODQA behavior, so evaluation should emphasize behavior-driven analyses rather than single-number scores.
Takeaways & Limitations
The T5-11B model saw approximately 10% more training data, leaving possible unaccounted-for overlap, and TriviaQA results are omitted because its split differed.
Abstract
from arXiv · showhide
Ideally Open-Domain Question Answering models should exhibit a number of competencies, ranging from simply memorizing questions seen at training time, to answering novel question formulations with answers seen during training, to generalizing to completely novel questions with novel answers. However, single aggregated test set scores do not show the full picture of what capabilities models truly have. In this work, we perform a detailed study of the test sets of three popular open-domain benchmark datasets with respect to these competencies. We find that 60-70% of test-time answers are also present somewhere in the training sets. We also find that 30% of test-set questions have a near-duplicate paraphrase in their corresponding training sets. Using these findings, we evaluate a variety of popular open-domain models to obtain greater insight into what extent they can actually generalize, and what drives their overall performance. We find that all models perform dramatically worse on questions that cannot be memorized from training sets, with a mean absolute performance difference of 63% between repeated and non-repeated data. Finally we show that simple nearest-neighbor models out-perform a BART closed-book QA model, further highlighting the role that training set memorization plays in these benchmarks
1 Introduction
The paper examines whether ODQA benchmarks measure memorization, novel questions with familiar answers, and generalization to novel answers. Analyses of three datasets reveal substantial train-test overlap, motivating overlap-stratified model evaluation.
- Motivation: ODQA benchmarks provide limited insight into what kinds of questions models answer well and how questions and answers are distributed.This makes standard test results difficult to understand and contextualize.
- Evaluation framework: Three competencies range from recalling answers to seen questions, through answering novel questions with familiar answers, to answering novel questions with unseen answers.The paper stratifies test sets to assess these behaviors separately.
- Findings: 58-71% of test answers also occur in training data across WebQuestions, Natural Questions, and TriviaQA.Thus, most test data does not probe answer generalization.
- Findings: 28-34% of annotated test questions have paraphrases in training, mostly near-duplicates differing by one or two words.These questions primarily probe memorization of question-answer pairs.
- Model evaluation: Train-overlapping examples contribute the bulk of overall performance across the evaluated ODQA models.The study evaluates both open-book and closed-book approaches.
- Model evaluation: Simple nearest-neighbor models outperform a BART closed-book model despite having virtually no capacity to generalize beyond training data.This comparison further highlights the role of training-set memorization.
2 Datasets
The study covers three open-domain QA datasets containing factual questions and short answers, with different sources, answer formats, and split sizes. Their train, development, and test partitions were randomly created, and overlap analyses exclude development data.
- Dataset overview: The analysis uses WebQuestions, TriviaQA, and Open Natural Questions, all containing factual natural-language questions and short multi-token answers.The datasets differ in question style and answer format.
- WebQuestions: WebQuestions contains 3,778 train and 2,032 test question-answer pairs, with Freebase entities as answers.Questions were mined from a search engine and answers were annotated by crowdworkers.
- TriviaQA: TriviaQA contains 78,785 train, 8,837 development, and 11,313 test question-answer pairs scraped from trivia websites.Wikipedia entities and their aliases are accepted as correct answers.
- Open Natural Questions: Open Natural Questions uses search-engine questions with short Wikipedia answer spans and has 79,168 train, 8,757 development, and 3,610 test pairs.The open-domain version restricts short answers to fewer than six tokens.
- Splits: All three datasets have randomly split canonical partitions without exact duplicate questions, while overlap analyses exclude development data.This isolates train-test overlap when assessing training memorization.
3 Test-Train Overlaps
The paper measures answer overlap automatically and question overlap through manual annotation of candidate training questions. Both analyses show substantial train-test reuse, including many near-duplicate question formulations.
- Definitions: Answer overlap exists when a training pair shares an answer reference with a test pair, whereas question overlap requires a paraphrased training question with the same answer.The analysis considers these as distinct forms of train-test overlap.
- Answer overlap: Answer normalization lower-cases references, strips punctuation, removes articles, and normalizes whitespace before matching.The normalized references are used to search for overlapping answers.
- Answer overlap: 58% of WebQuestions, 63.6% of Natural Questions, and 71.7% of TriviaQA test pairs have answer overlaps.TriviaQA has more answer references per question on average, which is associated with higher expected overlap.
- Question overlap: 1,000 test question-answer pairs per dataset were manually checked for question overlap using up to 50 training questions with similar answer references.The candidate-selection function was designed for high recall and a tight lower bound.
- Question overlap: Three expert annotators marked whether candidate training questions paraphrased the test question and had the same answer.The annotation process directly assessed semantic duplication rather than relying only on automatic rules.
- Results: 27.5-33.6% of annotated test questions had a training duplicate, with Natural Questions averaging 2.8 duplicates per overlapping test question.A 100-example two-way annotation sample had 93% agreement and Cohen’s Kappa of 0.85.
4 Implications for Modelling
The paper stratifies ODQA evaluation by question and answer overlap to distinguish memorization, answer classification, and QA generalization. Across these splits, performance is substantially higher on overlapping data, while simple nearest-neighbor retrieval can outperform larger generative models.
- Question Memorization: Question memorization tests whether models can recognize training questions and recall their associated answers.The overlap subset includes both near-identical questions and subtler duplicates.
- Answer Classification: Answer classification isolates novel questions whose answers appeared in training, excluding question duplicates because they are easier and inflate scores.This regime can be addressed by a classifier over training-set answers.
- QA Generalization: QA generalization evaluates questions with no overlapping training answers, where models cannot rely on memorizing training data.Higher-frequency answers are expected to be underrepresented in this split.
- Results: 45% average Exact Match separates answer classification from question memorization, with open-book models handling answer-overlap-only questions better than closed-book models.The BART model reaches only 10.2% accuracy on this set.
- Results: 25% average absolute performance drops from overall scores on QA generalization, showing that aggregate accuracy can obscure performance on non-overlapping questions.The FID model falls from an expected half of Natural Questions-style questions to about one third after repeated questions and answers are accounted for.
- Nearest-Neighbor Models: Dense nearest-neighbor retrieval outperforms BART on Natural Questions and TriviaQA despite having limited capacity to generalize beyond training data.The dense model also outperforms DPR on TriviaQA and WebQuestions for the question-overlap subset.
5 Related Work
Prior work has examined learned behaviors and overlap in several question-answering settings. The closest studies identify memorization or answer overlap, but this paper focuses on systematic analysis across popular ODQA benchmarks.
- Related Work: Diagnostic evaluation has been used in natural language understanding, while visual question answering studies have found memorized question-answer relationships.The cited examples include GLUE and analyses of visual question answering models.
- Related Work: Other related studies analyze closed-book TriviaQA predictions, Wikipedia-title overlap in Natural Questions, and non-overlapping knowledge-base QA subsets.These works provide adjacent analyses of memorization and overlap rather than the same three-dataset ODQA study.
6 Conclusion
The paper argues that overall QA accuracy does not adequately reveal whether ODQA models memorize training data or generalize to novel questions and answers. Its analysis supports behavior-driven evaluation of popular ODQA benchmarks.
- Conclusion: 60% of test-set answers overlap with training answers, and about 30% of test-set questions have at least one training duplicate.The study stratifies seven ODQA models by different amounts of training-set overlap.
- Conclusion: Overall QA accuracy cannot properly characterize model behavior on these datasets.The paper recommends greater emphasis on behavior-driven evaluation rather than single-number accuracy.
A.1 Additional Question Overlap Examples
This section provides additional examples of test-train overlapping questions across Open Natural Questions, TriviaQA, and WebQuestions.
- Table 5 gives additional examples of test-train overlapping questions in Open Natural Questions.
- Table 6 presents examples of test-train overlapping questions in TriviaQA.
- Table 7 presents examples of test-train overlapping questions in WebQuestions.