Source-linked AI summary
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang
TL;DR
Existing reading-comprehension datasets were either too small or semi-synthetic, limiting evaluation of data-intensive models. The paper introduces SQuAD and finds its logistic-regression model reaches 51.0% F1 versus 86.8% for humans, leaving substantial room for improvement.
Problem
Existing reading-comprehension datasets were either too small for modern data-intensive models or semi-synthetic rather than explicit comprehension questions.
Method
The paper constructs SQuAD from crowdsourced questions on Wikipedia passages, analyzes question diversity using dependency and constituency trees, and evaluates a logistic-regression model.
Results
51.0% F1 for the best model versus 86.8% for humans, indicating a substantial performance gap.
Takeaways & Limitations
SQuAD provides a challenging benchmark with ample opportunity for developing more expressive reading-comprehension models.
Takeaways & Limitations
All experimental results reported in the paper are on SQuAD v1.0.
Abstract
from arXiv · showhide
We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com
1 Introduction
SQuAD addresses limitations in reading-comprehension datasets with a large, high-quality collection of crowdworker questions over Wikipedia articles whose answers are passage spans. Its span-selection format supports diverse questions, while baseline experiments show substantial room beyond current machine performance.
- Motivation: Reading comprehension requires machines to understand natural language and use world knowledge to answer questions about text.The paper illustrates this with locating relevant text and interpreting the relation expressed by “under.”
- Motivation: Existing reading-comprehension datasets were either high-quality but too small for modern data-intensive models or large but semi-synthetic.The paper cites MCTest and related datasets as examples of these limitations.
- Dataset: 107,785 question-answer pairs on 536 Wikipedia articles make SQuAD almost two orders of magnitude larger than previous manually labeled datasets.Questions were posed by crowdworkers, and every answer is a text span from its corresponding passage.
- Dataset: SQuAD requires selecting answers from all passage spans rather than choosing among listed options, while retaining diverse questions and answer types.The larger candidate space makes answer selection more demanding, although span-based answers are more constrained than interpretative standardized-test questions.
- Evaluation: 51.0% F1 is achieved by the best logistic-regression model, versus 20% for the sliding-window baseline and 86.8% human F1.Lexicalized and dependency-tree-path features matter; model performance worsens with answer-type and syntactic-divergence complexity, unlike human performance.
2 Existing Datasets
Existing datasets span reading comprehension, deeper reasoning, open-domain question answering, answer extraction, and cloze prediction, with different task formulations and data-creation strategies. SQuAD’s cloze comparison emphasizes longer, non-entity answers that are entailed by passages rather than merely suggested.
- Reading comprehension: Earlier reading-comprehension datasets include 600 curated 3rd–6th grade questions and MCTest’s 660 crowdworker-created stories with four questions per story.These datasets supported pattern-matching, rule-based, and logistic-regression baselines.
- Deeper reasoning: Some datasets target deeper reasoning through algebraic equation systems, synthetic tasks stratified by reasoning type, or world knowledge in 4th-grade science exams.Algebra word problems require translating a story into equations, while BAbI separates tasks by reasoning type.
- Open-domain question answering: Open-domain QA answers questions from large document collections, while WikiQA uses Wikipedia passages for sentence selection rather than SQuAD-style answer spans.Answer extraction methods include surface-pattern bootstrapping, dependency trees, and factor graphs over multiple sentences.
- Cloze datasets: Cloze datasets predict a missing word, often a named entity, and can be automatically generated from naturally occurring data at very large scale.The Children’s Book Test predicts a blanked word using the 20 previous sentences.
- Cloze datasets: Unlike cloze queries, SQuAD answers often contain non-entities and much longer phrases, and they are entailed by the passage rather than merely suggested.This contrasts single-word-or-entity cloze answers with SQuAD’s broader answer spans.
3 Dataset Collection
The dataset was collected through passage curation, crowdsourced question-answer creation, and additional answer collection. Curated Wikipedia paragraphs were partitioned for training, development, and testing, while workers produced and validated span-based questions and answers.
- Collection stages: The collection process had three stages: curating passages, crowdsourcing question-answer pairs, and obtaining additional answers.These stages defined the overall dataset-construction workflow.
- Passage curation: 10,000 top-ranked English Wikipedia articles yielded 536 sampled articles and 23,215 extracted paragraphs after filtering.Paragraphs shorter than 500 characters and those containing images, figures, or tables were discarded.
- Passage curation: 80% of the articles formed the training set, while 10% each formed the development and test sets.The articles were partitioned randomly.
- Question-answer collection: Workers created up to 5 questions per paragraph and highlighted each answer span in the paragraph.They were guided by examples of good and bad questions and answers and encouraged to use their own words.
- Additional answers collection: At least 2 additional answers were collected for every development- and test-set question to estimate human performance and strengthen evaluation.Workers selected the shortest answering span after seeing the question and article paragraph.
4 Dataset Analysis
The dataset analysis examines answer-type diversity, reasoning difficulty, and syntactic divergence between questions and answer sentences. It combines manual labeling with an automatic dependency-based measure of divergence to characterize and stratify question difficulty.
- Analysis goals: The development-set analysis targets answer-type diversity, required reasoning, and syntactic divergence between questions and answer sentences.These three properties define the analysis scope.
- Diversity in answers: Answer categorization separates numerical from non-numerical answers and uses constituency parses, POS tags, and NER tags to classify non-numerical answers.Proper noun phrases are further divided into person, location, and other entities.
- Diversity in answers: 19.8% of answers are dates or other numbers, 32.6% are proper nouns, and 31.8% are common nouns.The passage reports these answer-type proportions from Table 2.
- Reasoning required: 192 examples were manually labeled after sampling 4 questions from each of 48 development-set articles, with some examples assigned multiple reasoning categories.All sampled examples exhibited lexical or syntactic divergence between the question and answer.
- Stratification by syntactic divergence: Syntactic divergence is computed by finding shared word-lemma anchors, extracting dependency-tree paths, and taking the minimum edit distance across anchors.Edit distance counts the minimum deletions or insertions needed to transform one unlexicalized path into the other.
5 Methods
The methods compare a logistic regression model with three baselines and restrict candidate answers to constituency-parser spans. Candidate ranking combines lexical overlap, sliding-window selection, and extensive lexical and syntactic features.
- Model comparison: The study compares a logistic regression model against three baseline methods.
- Candidate answer generation: 77.3% of correct development-set answers are constituency-parser constituents, setting an effective accuracy ceiling for the methods.Candidate generation avoids considering all O(L^2) sentence spans by retaining only constituency-parse constituents.
- Baseline ranking: Candidates with maximal unigram/bigram overlap are retained, then ranked using a sliding-window approach.The overlap is computed between the question and the containing sentence excluding the candidate answer.
- Baseline ranking: The implementation also adds a distance-based extension while using only the candidate’s containing sentence as context for efficiency.
- Logistic regression features: The logistic regression model discretizes continuous features into 10 equally sized buckets, producing 180 million features dominated by lexicalized and dependency-path features.Additional features capture word and bigram matches, root matches, lengths, span-word frequencies, constituent labels, POS tags, and lexical and syntactic variation.
6 Experiments
The experiments evaluate SQuAD with exact match and macro-averaged F1, comparing logistic regression, baselines, and humans. Results show a substantial human advantage, with model difficulty concentrated in exact span selection and syntactic divergence, while lexicalized and dependency-path features are especially important.
- Evaluation metrics: Exact match measures predictions exactly matching a ground-truth answer, while macro-averaged F1 averages maximum token-overlap F1 across questions.Both metrics ignore punctuation and articles.
- Human performance: 77.0% exact match and 86.8% F1 are achieved by humans on the test set.Human evaluation treats the second answer as the prediction and the remaining answers as ground truth.
- Model comparison: Logistic regression significantly outperforms baselines but remains below human performance.The experiments report a significant remaining gap between models and humans.
- Error analysis: 79.3% sentence-selection accuracy indicates that much of the difficulty lies in locating the exact answer span within the selected sentence.The model can often identify the sentence containing the answer, but struggles to select the precise span.
- Feature ablations: Lexicalized and dependency tree path features are the most important feature groups in ablation experiments.Dependency tree path features play a larger role here than in the comparison discussed from Chen et al. (2016).
- Stratified performance: Performance is best on dates and other numbers, harder on named entities, and decreases as syntactic divergence increases, unlike stable human performance.Numbers typically have few plausible candidates and single-token answers, whereas named entities offer more candidates; named entities remain relatively easy to identify using POS-tag features.
7 Conclusion
The paper introduces SQuAD, a large Wikipedia-based reading comprehension dataset with crowdsourced question-answer pairs and diverse question and answer types. Its 51.0% F1 logistic regression performance versus 86.8% human F1 indicates substantial room for improvement.
- Conclusion: SQuAD is a large reading comprehension dataset built from Wikipedia articles and crowdsourced question-answer pairs.The dataset targets natural language understanding and is freely available for research exploration.
- Conclusion: SQuAD includes a diverse range of question and answer types.
- Conclusion: 51.0% F1 is achieved by the logistic regression model, compared with 86.8% human F1.The performance gap suggests ample opportunity for improvement.