Source-linked AI summary
Know What You Don't Know: Unanswerable Questions for SQuAD
Pranav Rajpurkar, Robin Jia, Percy Liang
TL;DR
Extractive reading-comprehension systems often guess when contexts do not support an answer, while existing datasets rarely test this reliably. SQuAD 2.0 adds adversarially written unanswerable questions to SQuAD, and existing models achieve only 66.3% F1, demonstrating a challenging benchmark for answerability judgment.
Problem
SQuAD’s guaranteed-answer design lets models select related spans without checking whether the context entails an answer.
Method
SQuAD 2.0 combines SQuAD 1.1 with 53,775 crowdwritten unanswerable questions designed to remain relevant and contain plausible answer types.
Results
66.3% F1 was achieved by the best model on SQuAD 2.0, versus 89.5% F1 for humans, confirming the dataset is substantially harder for existing models.
Takeaways & Limitations
SQuAD 2.0 provides a challenging benchmark intended to drive reading-comprehension models that recognize when questions cannot be answered from context.
Takeaways & Limitations
The dataset assumes unanswerable questions remain relevant and contain plausible answer types, since otherwise simple word-overlap or type-matching heuristics could identify them.
Abstract
from arXiv · showhide
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuAD 2.0, the latest version of the Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD 2.0 is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0.
1 Introduction
SQuAD 2.0 addresses the limitation that reading-comprehension systems can exploit datasets where answers are guaranteed to appear in context. It combines SQuAD 1.1 with adversarially written unanswerable questions to test whether systems can abstain when no answer is supported.
- Background: SQuAD is a widely used reading-comprehension benchmark that has driven substantial progress in machine reading comprehension and model architectures.Some systems surpass human-level exact-match accuracy on the benchmark, despite remaining far from true language understanding.
- Motivation: SQuAD systems can rely on context and type-matching heuristics, while guaranteed answer existence limits the need to recognize unanswerable questions.Models also lack robustness to distracting sentences, motivating a dataset that tests more than span selection.
- Dataset construction: SQuAD 2.0 combines answerable questions from SQuAD 1.1 with 53,775 new unanswerable questions about the same paragraphs.The new questions were written to remain relevant while presenting plausible answers of the type requested by the question.
- Results: 66.3% F1 is achieved by a state-of-the-art model on SQuAD 2.0, compared with 89.5% F1 human accuracy.The human-model gap is 23.2 points, whereas the same architecture reaches 85.8% F1 on SQuAD 1.1.
- Results: SQuAD 2.0’s unanswerable questions are more challenging than automatically generated questions produced through distant supervision or rule-based methods.This supports the dataset’s goal of evaluating whether systems know when they do not know.
2 Desiderata
SQuAD 2.0 aims to make unanswerable questions difficult to identify through superficial heuristics by ensuring they remain relevant to the context and have plausible answer types represented in it.
- Desiderata: Unanswerable questions should remain relevant to the context topic, preventing word-overlap heuristics from distinguishing them from answerable questions.The dataset also targets large size, diversity, and low noise.
- Desiderata: Each unanswerable question should have a context span whose type matches the requested answer type, blocking type-matching heuristics.For example, a question asking which company was founded in 1992 should have some company mentioned in the context.
3 Existing datasets
Existing reading-comprehension datasets provide several kinds of negative examples, but many are easy to identify, excluded, scarce, misannotated, or poorly suited to extractive user-facing systems. Other datasets rely on rule-based edits, sentence ranking, or multiple-choice formats that differ from extractive question answering.
- Extractive reading comprehension datasets: 65% of negative examples in the Zero-shot Relation Extraction dataset lack a plausible answer, making them easy to identify.The dataset generates negative examples through distant supervision.
- Extractive reading comprehension datasets: TriviaQA excludes retrieved documents without the correct answer, while TFIDF examples pair questions with other paragraphs based on lexical overlap.TFIDF pairings are not guaranteed to contain plausible answers.
- Extractive reading comprehension datasets: 9.5% of NewsQA questions are unanswerable, but some are misannotated or out-of-scope, and negative examples are excluded from the final dataset.NewsQA produces unanswerable questions because workers receive only article summaries, making the strategy hard to scale.
- Extractive reading comprehension datasets: RULEBASED questions lack diversity because they replace entities, numbers, nouns, and adjectives through limited lexical transformations.The procedure uses similar words and WordNet antonyms to edit SQuAD questions.
- Sentence selection datasets: 3,047 questions and 1,473 answers comprise WikiQA, whose sentences are not guaranteed to contain plausible answers or be highly relevant.WikiQA pairs Bing query-log questions with Wikipedia sentences, while lexical baselines are highly competitive on QASENT.
- Multiple-choice datasets: Multiple-choice datasets such as MCTest and RACE are less suited to user-facing systems because options may be unavailable and questions emphasize interpretation or summarization.Their questions can include a “none of the above” option and often differ from extractive questions.
4 SQuAD 2.0
SQuAD 2.0 augments SQuAD 1.1 with adversarially written unanswerable questions designed to reference paragraph entities while lacking supported answers. The resulting negative examples are diverse, and human validation found 93% of sampled examples were truly unanswerable.
- Question construction: Crowdworkers wrote up to five unanswerable questions per paragraph that referenced paragraph entities while ensuring plausible answers were present.Workers used entire SQuAD 1.1 articles and saw existing questions as inspiration.
- Dataset composition: SQuAD 2.0 combined the new questions with SQuAD 1.1 data using the same article partitions for train, development, and test splits.The resulting splits had roughly equal answerable and unanswerable questions, while training had about twice as many answerable questions.
- Human validation: 4.8 answers per question were collected on average, with final answers selected by majority vote to reduce crowdworker noise.Ties favored answering questions, and shorter answers were preferred.
- Negative-example analysis: 93% of 100 randomly sampled development-set negative examples were indeed unanswerable.The examples covered diverse phenomena beyond negation, antonymy, and entity changes, making SQuAD 2.0 more diverse than RULEBASED.
5 Experiments
The experiments evaluate three no-answer reading-comprehension architectures on SQuAD 2.0 and compare performance with automatically generated negative examples. DocQA + ELMo reaches 66.3 F1, while plausible crowdworker answers fool systems and humans in roughly half of wrong unanswerable-question responses.
- SQuAD 2.0 evaluation: Three architectures predict both answer choices and whether a question is unanswerable, abstaining when the predicted unanswerability probability exceeds their decision threshold.The models are BiDAF-No-Answer and two DocumentQA No-Answer variants, with and without ELMo.
- SQuAD 2.0 evaluation: 66.3 F1 is achieved by DocQA + ELMo on SQuAD 2.0’s test set, 23.2 points below human performance at 89.5 F1.An always-abstain baseline scores 48.9 test F1, leaving existing models closer to that baseline than to humans.
- Automatically generated negatives: The experiments also test whether TFIDF- or RULEBASED-generated negative examples can produce a challenging dataset comparable with SQuAD 2.0.All three architectures are trained and tested on SQuAD 1.1 augmented with these automatically generated examples, using articles associated with SQuAD 2.0 unanswerable questions.
- Plausible-answer distractors: Roughly half of wrong answers on unanswerable questions exactly match crowdworkers’ plausible but incorrect answers for both computer systems and humans.The result indicates that these plausible answers function as effective distractors.
6 Discussion
SQuAD 2.0 frames reading comprehension as determining whether a context entails an answer, paralleling textual entailment and relation extraction. Its adversarial unanswerable questions remain difficult for models trained on similarly distributed examples, supporting the dataset’s goal of deeper language understanding.
- Discussion: SQuAD 2.0 requires models to determine whether a paragraph entails that a particular span answers a question, connecting reading comprehension to textual entailment and relation extraction.Textual entailment classifies hypotheses as entailed, contradicted, or neutral, while relation extraction assesses whether relationships between entities are entailed by text.
- Discussion: SQuAD 2.0 adversarial examples remain difficult even for models trained on examples from the same distribution.By contrast, Jia and Liang’s adversarial examples are not easily effective against models trained on similar examples.
- Discussion: SQuAD 2.0 is a challenging, diverse, large-scale dataset that forces models to recognize when questions cannot be answered from context.The authors aim to encourage reading-comprehension models that know what they do not know and thereby understand language more deeply.
A Supplementary material · A.1 Crowdsourcing details
Crowdworkers received task instructions and used an interface to create unanswerable questions for each paragraph. They wrote a question first, then highlighted a plausible answer in the paragraph.
- A.1 Crowdsourcing details: Workers wrote an unanswerable question for each paragraph and then highlighted a plausible answer in that paragraph.The crowdsourcing task instructions and writing interface are shown in Figures 2 and 3.
A.2 Plausible answers as distractors
The study measures how often systems are fooled by crowdworker-provided plausible answers on unanswerable questions. It evaluates these false-positive predictions against the plausible answers using exact match and F1 scores.
- A.2 Plausible answers as distractors: The analysis targets cases where systems predicted an answer to an unanswerable question.These cases are defined as false positive errors.
- A.2 Plausible answers as distractors: False positive errors were isolated on the development set before evaluating system predictions.The procedure was applied separately for each system.
- A.2 Plausible answers as distractors: Exact match and F1 scores were measured between system predictions and the plausible answers.The resulting scores are reported in Table 5.