Source-linked AI summary
The NarrativeQA Reading Comprehension Challenge
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, Edward Grefenstette
TL;DR
Existing reading-comprehension datasets often permit shallow answer selection, leaving deeper integration of narrative information insufficiently tested. The paper introduces NarrativeQA, a dataset and task suite built from books and movie scripts, and finds that standard models struggle with book-scale tasks despite producing sensible results on summaries.
Problem
Existing reading-comprehension tasks are often answerable through shallow pattern matching or salience rather than integrating events, entities, and relations across a document.
Method
The paper constructs NarrativeQA from books and movie scripts, using human-written questions and answers based on abstractive summaries and testing access to summaries or full stories.
Results
Standard models give sensible results on summaries but do not get traction on book-scale tasks requiring full-story reading.
Takeaways & Limitations
NarrativeQA exposes challenges involving long documents, distributed evidence, narrative relations, and reasoning beyond surface text.
Takeaways & Limitations
The larger tasks are significantly more difficult because summary questions may require discontiguous sentences or paragraphs from the full story.
Abstract
from arXiv · showhide
Reading comprehension (RC)---in contrast to information retrieval---requires integrating information and reasoning about events, entities, and their relations across a full document. Question answering is conventionally used to assess RC ability, in both artificial agents and children learning to read. However, existing RC datasets and tasks are dominated by questions that can be solved by selecting answers using superficial information (e.g., local context similarity or global term frequency); they thus fail to test for the essential integrative aspect of RC. To encourage progress on deeper comprehension of language, we present a new dataset and set of tasks in which the reader must answer questions about stories by reading entire books or movie scripts. These tasks are designed so that successfully answering their questions requires understanding the underlying narrative rather than relying on shallow pattern matching or salience. We show that although humans solve the tasks easily, standard RC models struggle on the tasks presented here. We provide an analysis of the dataset and the challenges it presents.
1 Introduction
NarrativeQA targets reading comprehension that requires integrating narrative information across entire books or movie scripts, rather than exploiting shallow textual cues.
- Existing reading-comprehension questions can often be answered through shallow pattern matching or global salience instead of understanding narrative elements and their relations.
- Narrative questions require integrating events and relations distributed across several statements, then generating a cogent answer from that combined information.
- NarrativeQA contains human-written questions and answers based solely on abstractive summaries of books and movie scripts.
- The reading-comprehension tasks allow models to answer from either summaries or the full story text.
- Fictional stories provide largely self-contained information about the salient entities and concepts needed to understand their narratives.
2 Review of Reading Comprehension Datasets and Models
The review argues that existing reading-comprehension datasets often constrain answers, documents, or evidence locations in ways that favor shallow local-context methods.
- Cloze-form datasets and answer-option tasks favor models that point to salient tokens rather than synthesize new answers.
- Span-based datasets cannot ask questions whose answers are absent from any contiguous document span, and their answers rarely cross sentence boundaries.
- SearchQA answers are predominantly short, with 80% being bigrams or unigrams and 99% containing five tokens or fewer.
- Many existing datasets use small or unnatural documents, while naturalistic datasets often ask questions answerable from a single sentence.
3 NarrativeQA: A New Dataset
NarrativeQA is a dataset of human-written questions and answers about complex stories, designed to require broader narrative understanding than localized text matching. It supports summary- and story-based answer generation and selection tasks.
- 3.1 Desiderata: NarrativeQA addresses limitations in existing reading-comprehension datasets by using questions that often require integrating information across larger portions of a document.The desiderata emphasize natural, human-generated questions and answers about higher-level relations among entities, places, and events.
- 3.2 Data Collection Method: The dataset uses books and movie scripts paired with human-written summaries, with questions and answers generated solely from those summaries.This annotation setup is intended to reduce questions extracted from localized contexts in the full stories.
- 3.2 Data Collection Method: Annotators wrote diverse questions about characters, events, and their relations, while being instructed to use their own words and avoid copying.Answers were required to be grammatical, complete, and explicit.
- 3.3 Core Statistics: About 30 question–answer pairs were collected per summary, yielding 1,567 stories and 46,765 human-written question–answer pairs.The stories were evenly split between books and movie scripts and divided into non-overlapping training, validation, and test portions.
- 3.4 Tasks: NarrativeQA evaluates both answer generation and answer selection using either summaries or full stories as context.Generated answers are evaluated with text-generation metrics and ranking is evaluated using mean reciprocal rank.
- 3.4 Tasks: The full-story task requires understanding entire books or movie scripts and is currently intractable for existing neural models without adaptation.The paper presents this setting as a more complex task than summary-based reading comprehension.
4 Baselines and Oracles
The paper evaluates information-retrieval and neural baselines for NarrativeQA, including models that generate answers, select spans, or rank candidate answers. Full stories require retrieval first because their length makes direct application of standard architectures computationally infeasible.
- 4 Baselines and Oracles: The experiments compare IR baselines and neural models across answer-generation, span-selection, and answer-ranking approaches.Human performance is also reported for comparison.
- 4.1 Simple IR Baselines: IR baselines retrieve answer spans by comparing candidate spans with either the question or an oracle gold answer.The oracle query provides an upper bound for span-retrieval performance.
- 4.1 Simple IR Baselines: The reported IR metrics include Bleu-1, Rouge-L, and cosine similarity between bag-of-words query and candidate-span embeddings.Candidate spans may be 4-grams, 8-grams, or full sentences, selected according to validation performance.
- 4.2 Neural Benchmarks: The no-context Seq2Seq baseline predicts answers directly from the question without using the summary or story context.This baseline can therefore classify the question and predict an answer with a similar topic or category.
- 4.2 Neural Benchmarks: The AS Reader considers the entire context and is adapted with an LSTM sequence decoder to select tokens sequentially for answer generation.The model predicts from unique word types in the context before decoding an answer sequence.
- 4.3 Neural Benchmarks on Stories: Full-story evaluation requires retrieving a small number of relevant passages before applying neural models because processing entire stories directly is computationally infeasible.The retrieved 200-word chunks are selected by similarity to the question and concatenated in temporal order.
5 Experiments
Experiments evaluate summary- and full-story reading, showing that models perform relatively well on summaries but struggle when retrieval and long-context reasoning are required.
- Experimental setup: The experiments cover summary-reading and full-story tasks, using information-retrieval and neural baselines.Full-story experiments use 200-token chunks and report IR, neural, and human scores.
- Experimental setup: Named entities are replaced with permuted generic markers so models cannot learn entity-specific embeddings.The procedure supports constructing entity representations for stories unseen during training.
- Reading summaries only: On summaries, the neural span prediction model significantly outperforms the other proposed methods but remains below oracle and human scores.The AS Reader and plain sequence-to-sequence model also perform well, while the AS Reader tends to copy subsequent context tokens.
- Reading full stories only: Full-story performance declines for the span-selection oracle because answers were constructed from summaries and longer answer spans are less likely to appear as high-scoring matches.Short answers are easier to retrieve when they name frequently mentioned characters; longer answers are harder to locate in the full story.
- Reading full stories only: Retrieval quality directly affects downstream neural performance, making full-story reading harder than summary reading.The IR-plus-neural approach retrieves relevant story chunks before applying reading-comprehension models.
- Reading full stories only: The AS Reader underperforms the no-context sequence-to-sequence baseline in MRR and gains no distinctive margin from retrieved context.It performs slightly better on other metrics, but increasing the number of retrieved chunks produces no significant differences.
- Reading full stories only: Full-story results leave a large gap to human performance, indicating that the task is difficult for current reading-comprehension models.This gap is presented as evidence that the benchmark meets its design objective of being realistic for humans and challenging for models.
6 Qualitative Analysis and Challenges
Qualitative analysis finds that NarrativeQA questions usually require integrating distant story segments, while retrieval itself remains challenging and dialogue adds difficulty.
- Dataset properties: The dataset meets its design criteria through long, diverse documents and natural human-generated questions and answers.A small manual examination found few shallow paraphrases; many questions require several paragraphs or multiple segments spread across a story.
- Retrieval challenges: A retrieval procedure is a natural response to the computational challenges posed by full-story reading.The paper frames retrieval as a first step before applying reading-comprehension models.
- Retrieval challenges: Retrieval is challenging even for humans unfamiliar with the narrative because questions often require larger story portions and background knowledge about entities.Searching from only a short question is therefore a difficult task in itself.
- Long-range reasoning: Examples show that answering questions can require iterative reasoning across extremely long temporal dependencies.In one example, an injury appears early in the narrative while its lethal consequences are revealed only near the end.
- Long-range reasoning: Movie-script comprehension also requires understanding written dialogue, which is typically non-descriptive even when questions concern descriptive content.The analysis identifies dialogue as an additional challenge specific to movie scripts.
- Future directions: The authors expect complex narrative understanding to require transferring text-understanding capabilities from other supervised learning tasks.
7 Related Work
NarrativeQA is presented as the first large-scale question-answering dataset covering full-length books and movie scripts, alongside related work on narrative understanding.
- NarrativeQA and related work: NarrativeQA is the first large-scale question-answering dataset on full-length books and movie scripts.Related efforts address plot understanding, script summarization, character types, character relationships, and narrative structure.
8 Conclusion
NarrativeQA introduces reading-comprehension tasks built around stories and varying access to summaries, exposing challenges that existing models do not solve at book scale.
- NarrativeQA introduces a dataset and tasks for training and evaluating reading comprehension systems.
- The task range spans answering human-written questions from summaries to answering them from full stories without summaries.
- Book-scale questions may require integrating evidence from discontiguous sentences or paragraphs, making larger tasks significantly more difficult.
- Existing models produce sensible results on summaries but do not get traction on book-scale tasks.
- The authors propose the dataset as a challenge and as a driver for neural models that exceed the complexity supported by existing datasets and tasks.