Source-linked AI summary
Reasoning Over Paragraph Effects in Situations
Kevin Lin, Oyvind Tafjord, Peter Clark, Matt Gardner
TL;DR
ROPES addresses the challenge of applying knowledge from a passage to a new situation, beyond the local structures emphasized by many reading-comprehension datasets. It builds a benchmark of cause-and-effect questions grounded in background passages and novel situations; the RoBERTa baseline performs only slightly above random choice, below expert-human performance.
Problem
Many reading-comprehension models and datasets focus on local predicate-argument structure, while applying passage knowledge to a new situation remains challenging.
Source: §1 · p. 1Method
ROPES pairs expository background passages with novel situations and crowd-authored questions requiring reasoning about their cause-and-effect relations.
Source: §1 · pp. 1–2Results
61.6% F1 for the RoBERTa baseline is only slightly better than random choice, compared with 89.0% F1 for expert humans.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
A key component of successfully reading a passage of text is the ability to apply knowledge gained from the passage to a new situation. In order to facilitate progress on this kind of reading, we present ROPES, a challenging benchmark for reading comprehension targeting Reasoning Over Paragraph Effects in Situations. We target expository language describing causes and effects (e.g., "animal pollinators increase efficiency of fertilization in flowers"), as they have clear implications for new situations. A system is presented a background passage containing at least one of these relations, a novel situation that uses this background, and questions that require reasoning about effects of the relationships in the background passage in the context of the situation. We collect background passages from science textbooks and Wikipedia that contain such phenomena, and ask crowd workers to author situations, questions, and answers, resulting in a 14,322 question dataset. We analyze the challenges of this task and evaluate the performance of state-of-the-art reading comprehension models. The best model performs only slightly better than randomly guessing an answer of the correct type, at 61.6% F1, well below the human performance of 89.0%.
1 Introduction
ROPES tests whether systems can apply cause-and-effect knowledge from an expository passage to a novel situation. Its RoBERTa baseline is only slightly better than random choice, while expert humans perform substantially better.
1 Introduction
- ROPES asks systems to apply causes and effects described in an expository paragraph to a novel situation.Source: §1 · pp. 1–2
- The dataset contains 14,322 questions, built from science-textbook and Wikipedia passages with crowd-authored situations, questions, and answers.Source: §1 · p. 2
- Analysis finds varied cause-and-effect relations, difficulty matching descriptions across background, situation, and question, and multiple reasoning types.Source: §1 · p. 2
- 61.6% F1 for the RoBERTa baseline is only slightly better than randomly choosing an answer; expert humans average 89.0% F1.Source: §1 · p. 2
2 Related Work
ROPES targets implications of a passage rather than mainly local predicate-argument structure, and requires connecting information across a passage. It differs from related datasets in its use of a provided complex passage without retrieval or dialogue question generation.
2 Related Work
- Many reading-comprehension datasets mainly test local predicate-argument structure, whereas ROPES targets passage implications and connects information across a passage.Source: §2 · p. 2
- Unlike ShARC’s dialogue and OpenBookQA’s retrieval setting, ROPES asks questions about a given complex passage without retrieval.Source: §2 · p. 2
- QuaRel pre-specifies causes rather than requiring systems to read them from a background passage, limiting its setting relative to ROPES.Source: §2 · p. 2
- ROPES frames entailment as question answering whose answers require understanding what an expository paragraph entails, rather than classifying sentence pairs.Source: §2 · p. 2
3 Data Collection
ROPES background passages were collected from science textbooks and Wikipedia, then used by crowd workers to create situations, questions, and answers. Workers were encouraged to write minimal pairs so a small question change could reverse the answer.
3 Data Collection
- Background passages were scraped from science textbooks and Wikipedia using causal connectives and qualitative-relation keywords, then filtered for at least one relation.Source: §3 · p. 2
- The collection contains over 1,000 manually filtered background passages, categorized as physical science (49%), life science (45%), economics (5%), and other (1%).Source: §3 · p. 2
- Crowd workers wrote situations involving background relations and questions requiring both the background and situation; they were encouraged to create minimal pairs with opposite answers.Source: §3 · pp. 2–3
- Workers selected 588 background passages for the dataset, according to Table 1.Source: Table 1 · p. 3
4 Dataset Analysis
The analysis examines relation types, grounding between background and situation, and the reasoning questions require. It also reports that situation-based splits can inflate scores through prolific-worker bias, motivating annotator-separated splits.
4 Dataset Analysis
- The analysis randomly samples 100 questions to examine background relation type, grounding in the situation, and required reasoning.Source: §4 · p. 3
- Questions involve causal, qualitative, or combined relations, and grounding may rely on explicit mentions, common-sense facts, or overcoming a large lexical gap.Source: §4 · p. 3
- Question reasoning includes effect and cause comparisons and predictions; most examples compare effects or causes, requiring grounding two occurrences.Source: §4 · pp. 3–4
- Splitting by situation produced high scores due to prolific-worker annotator bias, while separating training and test annotators revealed difficulty generalizing to new workers.Source: §4 · p. 4
5 Baseline performance
The RoBERTa baseline combines background and situation text, and its results are significantly below human performance. The experiments also test performance without the background and after RACE fine-tuning.
5 Baseline performance
- The baseline uses a RoBERTa question-answering model on a passage formed by concatenating the background and situation.A run without the background estimates the presence of annotation artifacts.Source: §5 · p. 4
- Baseline results are significantly lower than human performance on the dev and test sets.Table 5 reports baseline and human performance; the supplied passages do not give the scores.Source: §5 · p. 4, Table 5 · p. 4
- The experiments also fine-tune the model on RACE before fine-tuning on ROPES.Source: §5 · p. 4
- Human performance is estimated by expert annotations on 400 random questions, with no overlap in sampled backgrounds or situations.The non-overlap is intended to prevent humans from benefiting from knowledge of dataset construction.Source: §5 · p. 5
6 Conclusion
ROPES is a 14,322-question reading comprehension benchmark for testing whether systems apply knowledge from text in a new setting. The authors hope it will aid efforts to tie language and reasoning together for more comprehensive text understanding.
6 Conclusion
- ROPES is a reading comprehension benchmark containing 14,322 questions.Source: §6 · p. 5
- The benchmark aims to test whether systems apply knowledge gained from reading text in a new setting.Source: §6 · p. 5
- The authors hope ROPES will aid efforts to tie language and reasoning together for more comprehensive understanding of text.Source: §6 · p. 5