Source-linked AI summary
DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension
Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, Karthik Sankaranarayanan
TL;DR
Existing RC benchmarks do not fully capture low lexical overlap, narrative reasoning, external knowledge, and unanswerability. DuoRC addresses these gaps with cross-version movie-plot QA construction, and state-of-the-art RC systems perform poorly on it despite near-human SQuAD results.
Problem
Existing RC datasets underrepresent narrative reasoning, low lexical overlap, external knowledge, and unanswerability challenges relevant to complex language understanding.
Method
DuoRC builds 186K human-generated QA pairs from 7680 paired Wikipedia and IMDb movie plots, with questions created from one version and answers produced from the other.
Results
State-of-the-art RC models perform poorly on DuoRC despite near-human performance on SQuAD, emphasizing further research avenues.
Takeaways & Limitations
DuoRC complements existing RC datasets by exposing challenges involving external knowledge, common-sense reasoning, deeper language understanding, and multi-sentence inference.
Takeaways & Limitations
The paper omits an MS-MARCO answer-generation model because its authors had not released the code.
Abstract
from arXiv · showhide
We propose DuoRC, a novel dataset for Reading Comprehension (RC) that motivates several new challenges for neural approaches in language understanding beyond those offered by existing RC datasets. DuoRC contains 186,089 unique question-answer pairs created from a collection of 7680 pairs of movie plots where each pair in the collection reflects two versions of the same movie - one from Wikipedia and the other from IMDb - written by two different authors. We asked crowdsourced workers to create questions from one version of the plot and a different set of workers to extract or synthesize answers from the other version. This unique characteristic of DuoRC where questions and answers are created from different versions of a document narrating the same underlying story, ensures by design, that there is very little lexical overlap between the questions created from one version and the segments containing the answer in the other version. Further, since the two versions have different levels of plot detail, narration style, vocabulary, etc., answering questions from the second version requires deeper language understanding and incorporating external background knowledge. Additionally, the narrative style of passages arising from movie plots (as opposed to typical descriptive passages in existing datasets) exhibits the need to perform complex reasoning over events across multiple sentences. Indeed, we observe that state-of-the-art neural RC models which have achieved near human performance on the SQuAD dataset, even when coupled with traditional NLP techniques to address the challenges presented in DuoRC exhibit very poor performance (F1 score of 37.42% on DuoRC v/s 86% on SQuAD dataset). This opens up several interesting research avenues wherein DuoRC could complement other RC datasets to explore novel neural approaches for studying language understanding.
1 Introduction
DuoRC is designed to extend reading comprehension benchmarks toward deeper language understanding by combining low lexical overlap, narrative reasoning, external knowledge, and unanswerability detection. It constructs questions from one movie-plot version and answers from another version of the same story, producing difficult cross-version comprehension conditions.
- Motivation: Existing RC benchmarks often use factual descriptive passages or questions with substantial lexical overlap, limiting the language understanding required.Such overlap can let keyword matching perform well, while narrative passages requiring causality and background knowledge are underrepresented.
- DuoRC Challenges: DuoRC targets four challenges: low question-passage overlap, background and common-sense knowledge, multi-sentence narrative reasoning, and unanswerability detection.These challenges are explicitly presented as extensions beyond existing RC datasets.
- Dataset Construction: DuoRC creates questions from one movie-plot version and answers from a different version of the same movie written by another author.Workers first generate QA pairs from one plot, then a different worker group answers those questions using only the paired plot version.
- Reasoning Requirements: The paired plots differ in detail, narration style, and vocabulary, making corresponding answers require deeper cross-version interpretation.The paper illustrates cases involving inference, alternative valid answers, coreference across sentences, and common-sense reasoning.
- Empirical Motivation: State-of-the-art RC systems perform poorly on DuoRC despite their near-human performance on SQuAD, leaving substantial room for further research.The paper positions DuoRC as complementary to existing RC datasets rather than a replacement.
2 Related Work
Related RC datasets differ in passage type, question-answer construction, and task scope, but many do not cover the combined challenges targeted by DuoRC. The authors argue that additional datasets are useful for preventing progress from stagnating around a single benchmark.
- Existing Dataset Designs: SQuAD, NewsQA, TriviaQA, and MovieQA generally use answers corresponding to document spans, whereas MS-MARCO answers are synthesized from relevant documents.Cloze-style datasets instead generate questions automatically by deleting a word or entity from a sentence.
- NarrativeQA: NarrativeQA includes plot-summary questions and a second task requiring the full book or movie script, making that second task substantially different from DuoRC.The authors characterize NarrativeQA's second task as more futuristic and not comparable to DuoRC.
- Rationale for DuoRC: The authors argue that new RC datasets add distinct challenges, support better QA systems, and reduce overfitting to one benchmark.They present DuoRC as complementary to existing datasets for collectively addressing a wider range of language-understanding challenges.
3 Dataset
DuoRC is built from paired Wikipedia and IMDb movie plots, using cross-version annotation to create challenging reading-comprehension instances. Its combined SelfRC and ParaphraseRC datasets contain 186,089 instances with substantial answer variation and limited overlap.
- 7680 movies with plots longer than 100 words were retained when both Wikipedia and IMDb versions were available.
- SelfRC: 85,773 SelfRC QA pairs were created from the same plot, with workers directly selecting 70% of answers and synthesizing 30%.
- ParaphraseRC: 100,316 ParaphraseRC triplets use questions from one plot version and answers obtained from a different version.
- Answer overlap: 62% of SelfRC and ParaphraseRC questions have partial answer overlap, while 38% have none because of unanswerability, information gaps, or paraphrasing.
- Answer variation: Only 40.7% of questions receive the same answer in both documents; 37.8% show no word overlap and 21% show partial overlap.
- DuoRC1 combines the datasets into 186,089 instances, with question-type distributions and comparisons against other RC datasets reported in Figures 2 and Table 1.
4 Models
The paper establishes DuoRC baselines with span prediction, answer synthesis, and additional NLP preprocessing. These choices address both extractive and synthesized answers while targeting paraphrase, coreference, and long-passage challenges.
- The baseline evaluation uses state-of-the-art RC and language-generation models together with traditional NLP techniques.
- Span prediction: Because many answers are synthesized, the paper evaluates a span model on instances where answers match document spans and uses it to test DuoRC difficulty.
- Answer synthesis: A two-stage model first predicts a span with BiDAF, then generates an answer using a query-based abstractive summarization model.
- The MS MARCO answer-generation model is omitted because its authors had not released the code, preserving the paper’s reproducibility criterion.
- NLP preprocessing: Additional preprocessing targets paraphrase understanding, coreference resolution, and long passages by extracting sentences relevant to the question.
5 Experimental Setup
The experiments establish baselines using span prediction, answer generation, preprocessing, and Accuracy/F-score evaluation for SelfRC and ParaphraseRC.
- Data Splits: The test split contains 70% training, 15% validation, and 15% test QA pairs, with no test movie appearing during training.
- Models and Test Sets: The SpanModel predicts document spans, whereas the GenModel predicts a span and then generates an answer from it.
- Models and Test Sets: The Span-based Test Set includes answers matching document spans, while the Full Test Set also contains answers that do not correspond to spans.
- Answer Generation: The GenModel is trained on span-answer pairs, including approximate spans identified through methods such as Lucene retrieval and longest common subsequence.
- Evaluation Metrics: The evaluation uses Accuracy and F-score, with exact matches required for Accuracy and partial answer overlap credited by F-score.
6 Results and Discussions
Experiments compare preprocessing, span prediction, and generation across SelfRC and ParaphraseRC, showing that paraphrased reading comprehension is substantially harder for existing systems.
- SpanModel vs GenModel: The SpanModel clearly outperforms the GenModel on both SelfRC and ParaphraseRC.
- SelfRC vs ParaphraseRC: Performance drops when models move from SelfRC to ParaphraseRC, supporting the claim that ParaphraseRC is the harder task.
- Preprocessing: Preprocessing improves the ParaphraseRC Span Detection Model, while oracle preprocessing tests whether document length explains poor performance.
- Discussion: The dataset is presented as offering substantial potential for advancing complex language understanding through newer challenges.
7 Conclusion
DuoRC introduces a large-scale paraphrased reading-comprehension dataset designed to test low lexical overlap, external knowledge, common-sense reasoning, and multi-sentence inference. Experiments show that state-of-the-art RC models perform poorly on it despite near-human performance on SQuAD.
- DuoRC contains 186K human-generated QA pairs from 7680 pairs of parallel Wikipedia and IMDb movie plots.
- The dataset is designed to create little or no lexical overlap between questions and answer-containing segments across plot versions.
- DuoRC targets external knowledge, common-sense reasoning, deeper language understanding, and multiple-sentence inferencing in question answering.
- State-of-the-art RC models that perform near human performance on SQuAD perform poorly on DuoRC.
Appendix A Examples
Appendix examples contrast straightforward answering from shorter plots with more challenging answering from longer paraphrased plots.
- SelfRC questions are created from smaller plots, where answering is described as straightforward.
- ParaphraseRC answers are obtained from larger plots and require multi-sentence or sometimes multi-paragraph inferencing.
- In the examples, colors distinguish direct answers from answers spanning multiple sentences for both smaller and larger plots.
A.1 Example 1: Pale Rider (1985)
The examples illustrate how alternative plot descriptions present overlapping events with different wording, detail, and narrative focus. They cover linked actions, character identities, confrontations, and outcomes across multiple sentences.
- In Pale Rider, miners and their families are attacked by Coy LaHood’s men, while Megan’s dog is shot during the raid.
- The Preacher intervenes when Josh attacks Megan, then confronts LaHood’s forces and protects the miners’ community.
- The narrative links events across characters and locations, including Stockburn’s recognition of the Preacher and Megan’s final farewell.
- The Preacher and Hull destroy LaHood’s mining operation with dynamite before the Preacher rides into town alone.
- In Big Jake, a kidnapping for a million-dollar ransom leads to a pursuit, an ambush, and the rescue of Little Jake.
Appendix B Data Analysis
Manual analysis examined 100 cases where SelfRC and ParaphraseRC answers differed or the paraphrase version was unanswerable. The analysis attributes discrepancies to missing information, knowledge requirements, and paraphrasing variation.
- The manual verification covered 100 QA pairs with differing SelfRC and ParaphraseRC answers or a non-answerable ParaphraseRC instance.
- Lack of information was the chief reason for No Answer in the paraphrase plot.
- Some unanswerable questions require educated guesswork, general knowledge, or movie metadata unavailable in the paraphrase passage.
- When answers differed, causes included phrasal paraphrases and spelling variants such as “Rebeca” versus “Rebecca.”
- Figure 3 summarizes the analyzed reasons for answer differences and paraphrase-version non-answerability.
Appendix C Model Architecture
The architecture combines passage preprocessing, span identification, and answer generation to answer questions from the comprehension passage.
- The five-step process optionally preprocesses the input passage before applying BiDAF for span identification and qBAS for answer generation.
Appendix D Performance Analysis
The performance analysis compares SelfRC and ParaphraseRC across subsets, question types, and plot lengths. The supplied passages identify the evaluation dimensions but do not report numerical outcomes.
- The analysis evaluates SelfRC and ParaphraseRC on both the Span Test Subset and the Full Test Set.
- Performance is examined across different question types and plots of different lengths.
- Figure 4 is identified as the model architecture, while Tables 5 and 6 provide QA materials for Pale Rider and Big Jake.