Source-linked AI summary
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner
TL;DR
Existing reading-comprehension benchmarks have been solved by strong systems, motivating DROP’s harder test of discrete reasoning over paragraph content. The paper constructs and evaluates this benchmark, then adds limited numerical reasoning to a reading-comprehension model. The best baseline reaches 32.7% F1, while the new model reaches 47% F1 and expert humans reach 96%.
Problem
DROP addresses the limited challenge posed by existing reading-comprehension datasets by testing comprehensive paragraph understanding with discrete reasoning operations.
Method
The paper crowdsources 96,567 adversarial questions from Wikipedia and evaluates reading-comprehension and semantic-parsing systems alongside a model with limited numerical reasoning.
Results
47% F1 was achieved by the new model, compared with 32.7% F1 for the best baseline and 96% human performance.
Takeaways & Limitations
DROP provides a challenging benchmark for comprehensive paragraph analysis and for combining distributed representations with symbolic reasoning.
Takeaways & Limitations
Semantic-parsing systems are constrained by incomplete information extraction and spurious logical forms under weak supervision.
Abstract
from arXiv · showhide
Reading comprehension has recently seen rapid progress, with systems matching humans on the most popular datasets for the task. However, a large body of work has highlighted the brittleness of these systems, showing that there is much work left to be done. We introduce a new English reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs. In this crowdsourced, adversarially-created, 96k-question benchmark, a system must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them (such as addition, counting, or sorting). These operations require a much more comprehensive understanding of the content of paragraphs than what was necessary for prior datasets. We apply state-of-the-art methods from both the reading comprehension and semantic parsing literature on this dataset and show that the best systems only achieve 32.7% F1 on our generalized accuracy metric, while expert human performance is 96.0%. We additionally present a new model that combines reading comprehension methods with simple numerical reasoning to achieve 47.0% F1.
1 Introduction
DROP is an English reading-comprehension benchmark designed to require structured reasoning over paragraphs rather than answer extraction alone. It combines adversarial crowdsourcing with discrete operations and shows substantial gaps between current systems and humans.
- Benchmark motivation: DROP requires systems to resolve references in paragraphs and perform discrete operations such as addition, counting, or sorting.These operations target comprehensive paragraph understanding and structured analysis.
- Reasoning demands: The benchmark combines event matching, argument extraction, and numerical operations, such as identifying who threw the longest touchdown pass.This design encourages methods that combine distributed representations with symbolic reasoning.
- Dataset construction: 96,567 questions were crowdsourced from Wikipedia passages, with workers encouraged to write questions that a BiDAF baseline could not answer.The dataset emphasizes sports game summaries and history passages.
- Results: 47% F1 was achieved by a model combining limited numerical reasoning with standard reading-comprehension methods, compared with 32.7% F1 for the best baseline.Expert human performance was 96.4% on the generalized accuracy metric.
2 Related Work
DROP extends prior work on reading comprehension and semantic parsing by combining complex compositional questions with paragraph understanding. Its broader contexts and open-domain passages are intended to encourage comprehensive structural analysis.
- Question answering datasets: DROP focuses on paragraph understanding while other question-answering datasets add conversational state, retrieval, mismatched passages, external knowledge, or multi-document reasoning.The authors position paragraph understanding itself as an unresolved challenge.
- Related datasets: Unlike algebra word-problem datasets, DROP uses longer contexts, a more open domain, and deeper paragraph understanding.The comparison concerns the relationship between numerical phenomena and passage comprehension.
- Semantic parsing: DROP combines complex semantic-parsing-style questions with paragraph understanding rather than assuming a structured tabular representation.The stated goal is to encourage explicit or implicit structural analyses of paragraphs.
- Neural symbolic reasoning: DROP is designed to encourage methods that combine neural representations with discrete symbolic reasoning.The paper presents one model in this direction.
3 DROP Data Collection
DROP was built through a three-phase annotation process: selecting Wikipedia passages, crowdsourcing difficult question-answer pairs, and validating the development and test data. An adversarial QA model helped increase question difficulty.
- Annotation protocol: The annotation protocol extracted suitable Wikipedia passages, crowdsourced question-answer pairs requiring discrete reasoning, and validated the development and test portions.The protocol is explicitly organized into three phases.
- Passage extraction: Wikipedia passages were selected for narrative event sequences and many numbers, yielding about 7,000 passages with emphasis on NFL summaries and history articles.Passages containing at least twenty numbers were additionally sampled.
- Question collection: Workers produced questions across addition/subtraction, minimum/maximum, counting, selection, and comparison categories.They answered questions with spans, dates, or numbers under specified conditions.
- Question collection: A real-time BiDAF model allowed workers to submit only questions it could not solve, increasing the intended difficulty of the collected questions.This adversarial setting was applied during crowdsourcing rather than only filtering generated examples.
- Dataset split: 96,567 question-answer pairs were split by passage into 80% training, 10% development, and 10% test sets.All questions about one passage were assigned to a single split.
- Validation: Cohen’s κ was 0.74 overall during validation, with agreement of 0.81 for numbers, 0.62 for spans, and 0.65 for dates.Invalid questions comprised 0.7% of the data and were filtered out.
4 DROP Data Analysis
DROP questions exhibit diverse language and require integrating multiple, sometimes distant passage spans. Numerical and proper-noun answers are common, reflecting the benchmark’s emphasis on discrete reasoning.
- Question analysis: The training set contains around 30k different words, indicating substantial vocabulary diversity.The dataset statistics summarize properties across its splits.
- Question analysis: The most frequent span-question pattern, “Which team scored”, appears in only 4% of span questions.This supports the reported variety of linguistic constructions.
- Question analysis: The five most frequent number-question patterns all begin with “How many”, indicating the need for counting and other arithmetic operations.The distribution of question-start trigrams is shown in Figure 1.
- Answer analysis: 2.18 passage spans are needed on average to answer a question, the average distance between spans is 26 words, and 20% require at least three spans.A majority of answers are numerical values and proper nouns.
5 Baseline Systems
The baselines evaluate semantic parsers, reading-comprehension models, and heuristic systems on DROP using Exact-Match and numeracy-focused F1. Reading-comprehension methods perform best, while evaluation excludes many questions that cannot be answered as passage spans.
- Baseline Systems: DROP baselines include semantic parsers, SQuAD-style reading-comprehension models, and heuristics targeting annotation artifacts.The evaluation uses Exact-Match and a numeracy-focused macro-averaged F1 score.
- Evaluation Metrics: F1 becomes 0 whenever the predicted and gold answers contain different numbers, regardless of other word overlap.Multiple spans are greedily aligned by bag-of-words overlap before averaging F1.
- Semantic Parsing: Semantic parsers represent paragraphs with dependency, Open IE, or semantic-role structures and use logical forms with predicates, relations, strings, numbers, and dates.The KDG parser maximizes marginal likelihood over executable logical forms and uses beam search at test time.
- Reading Comprehension: The reading-comprehension baselines are adapted to marginalize over matching passage spans, but 45% of training questions are omitted because they lack passage-span answers.This omission limits direct comparison with questions requiring other answer types.
6 NAQANet
NAQANet extends a QANet-style reading-comprehension architecture with answer-specific outputs for passage spans, question spans, counts, and arithmetic expressions. It predicts a partially executed logical form whose numerical operations are completed symbolically, training with weak supervision over executions yielding the correct answer.
- 6 NAQANet: NAQANet adds question-span, count, and addition-or-subtraction answers to a standard reading-comprehension model.The model first predicts the answer type, then predicts the relevant numbers for numerical answers.
- 6.1 Model Description: The architecture uses embedding, encoding, passage-question attention, and output layers, with four output layers corresponding to its answer types.The model uses the QANet architecture through the projected question-aware passage representation.
- Passage span: Passage-span prediction applies three QANet encoder repetitions before computing start and end probabilities over passage positions.The resulting representations are M0, M1, and M2.
- Count: Count prediction treats counting as multiclass classification over ten possible numbers, 0–9, using the passage representation hP.The count distribution is computed from hP.
- Arithmetic expression: Arithmetic prediction extracts passage numbers and assigns each a plus, minus, or zero sign to form an evaluable expression.This supports addition and subtraction over multiple numbers selected from the passage.
- Training: Weak supervision maximizes marginal likelihood over all execution paths, including matching spans, counts, and valid number-sign assignments that produce the correct answer.DROP supplies answer strings rather than the answer type or execution path.
7 Results and Discussion
DROP exposes a large gap between human and model performance, while analysis attributes errors to complex reasoning and limitations in semantic-parsing pipelines. NAQANet performs best, particularly on number-answer questions, but its search is restricted to two-number addition/subtraction.
- Analysis and limitations: The numerical search is limited to addition or subtraction of two numbers because the unrestricted search space is exponential and noisy.Within this restricted space, search and marginalization are exact.
- Model and human performance: More than 50 absolute F1 points separate BERT’s DROP performance from its performance on SQuAD.The comparison indicates that models perform substantially worse on DROP than on prominent prior reading-comprehension datasets.
- Model and human performance: NAQANet obtains the best performance, with gains concentrated on the challenging and frequent number-answer type.Table 6 compares answer-type performance between NAQANet and BERT.
- Analysis and limitations: Semantic-parsing baselines perform poorly because information-extraction pipelines produce logical forms for only part of the data and often miss answer-relevant information.SRL yielded logical forms for 34% of training data, while OpenIE yielded 25%; only 25% of sampled extracted tables contained information needed to answer.
- Analysis and limitations: Only 8 of 60 sampled logical forms were non-spurious, making weakly supervised training difficult to generalize.Spurious logical forms can produce the correct denotation without reflecting question semantics.
8 Conclusion
DROP is a substantially challenging reading-comprehension dataset built around discrete reasoning over paragraphs. The paper’s initial model combines reading comprehension with limited numerical reasoning and improves performance over the best baseline.
- Conclusion: DROP contains complex reading-comprehension questions requiring Discrete Reasoning Over Paragraphs.The dataset targets more comprehensive paragraph analysis than existing benchmarks.
- Conclusion: 32.7% F1 was achieved by the best baseline, while humans achieved 96% on DROP.These results establish the benchmark’s difficulty relative to human performance.
- Conclusion: NAQANet augments QANet with limited numerical reasoning and achieves 47% F1 on DROP.The model is presented as initial work toward combining distributed representations with symbolic reasoning.
- Conclusion: The authors hope DROP will spur research on comprehensive paragraph analysis and methods combining distributed representations with symbolic reasoning.This is the paper’s stated intended research direction.