Source-linked AI summary
Teaching Machines to Read and Comprehend
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, Phil Blunsom
TL;DR
Large-scale supervised data for document reading comprehension was missing, limiting progress in training flexible models. The paper constructs two large corpora and develops attention-based neural readers, which achieve higher accuracy and perform best across both datasets.
Problem
Large-scale document–query–answer training corpora were missing, leaving supervised machine reading largely absent despite the need for flexible models.
Method
The paper converts summary and paraphrase sentences into anonymized context–query–answer triples and trains recurrent attention-based neural readers.
Results
Attentive and Impatient Readers perform best across both datasets and achieve higher accuracy without encoding document or query structure.
Takeaways & Limitations
The methodology provides a scalable supervised reading-comprehension challenge, while attention-based neural networks offer an effective modeling framework for the task.
Takeaways & Limitations
The models cannot yet answer some queries requiring complex inference and long-range reference resolution, and extending them to world knowledge or multi-document queries remains challenging.
Abstract
from arXiv · showhide
Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.
1 Introduction
The paper addresses the lack of large-scale supervised data for real-document reading comprehension by converting summaries and paraphrases into context–query–answer triples. It introduces two news corpora and evaluates attention-based neural models that achieve higher accuracy without explicitly encoding document or query structure.
- Motivation: Supervised machine learning for machine reading has been limited because large-scale natural-language comprehension data are difficult to obtain.Earlier approaches relied mainly on hand-engineered grammars or information-extraction methods that represented documents as queryable predicate-argument triples.
- Related work: Synthetic narratives and queries provide abundant supervised data and let researchers isolate performance on individual simulated phenomena.Results on such data suggest that neural-network models can model reading comprehension.
- Contribution: The authors convert summary and paraphrase sentences with their documents into context–query–answer triples using simple entity detection and anonymisation algorithms.They collect two new corpora containing roughly a million news stories with associated queries.
- Method: The proposed deep-learning models use attention mechanisms within recurrent neural-network architectures to focus on document content relevant to answering a question.Attention also makes the model’s inference process more interpretable by showing which document aspects it uses.
- Results: The neural models achieve higher accuracy than the compared baselines and heuristic benchmarks without specifically encoding document or query structure.This result supports the efficacy of the newly introduced corpora and model designs.
2 Supervised training data for reading comprehension
The paper frames reading comprehension as supervised prediction over document–query–answer triples and introduces large-scale corpora built from online news articles and summaries. Entity anonymization and permutation focus evaluation on comprehension of the provided document rather than world knowledge or co-occurrence.
- Task formulation: Reading comprehension is formulated as estimating p(a|c, q), where c is the context document, q the query, and a the answer.The focused evaluation aims to test a model’s ability to detect and understand linguistic information while excluding additional world knowledge.
- Motivation: Training corpora previously contained only hundreds of document–query–answer examples, limiting their primary use to testing and encouraging unsupervised extraction approaches.These approaches used templates or syntactic/semantic analysers to extract relation tuples and form queryable knowledge graphs.
- Corpus construction: The proposed methodology creates two real-world, large-scale supervised corpora from online newspaper articles and matching summaries.The collection includes 93k CNN articles and 220k Daily Mail articles, whose bullet points summarize aspects of the articles’ information.
- Task focus and anonymization: The task evaluates comprehension of a single document, not world knowledge or co-occurrence, because anonymized queries cannot be answered reliably without their accompanying context.Coreferent entities are replaced with abstract markers, and those markers are randomly permuted whenever a data point is loaded.
- Task focus and anonymization: Anonymization leaves exploiting the presented context as the only remaining strategy for answering questions, while treating entity markers like regular words makes the task harder and models more general.The procedure establishes coreferents, replaces entities with abstract markers, and randomly permutes the markers during training and testing.
3 Models
The section introduces heuristic and NLP-centric baselines alongside three neural reading-comprehension models. The neural models progress from deep sequence encoding to token-level attention and recurrent rereading of the document as each query token is processed.
- Baselines: The authors define majority and exclusive-majority baselines, selecting entities by frequency in the context, with the latter excluding entities observed in the question.The majority baseline chooses the most frequent context entity; exclusive majority chooses the most frequent context entity absent from the question.
- NLP-centric models: NLP-centric benchmarks exploit linguistic annotations, including frame-semantic parses that expose predicate–argument relations for question answering.The frame-semantic benchmark uses parser-generated annotations on unanonymised corpora and does not generalise through a language model beyond parsing.
- Baselines: The word-distance benchmark aligns the question placeholder with candidate entities and scores each by distances between question words and nearby aligned context words.Alignment can match words directly or through coreference information.
- Neural models: The Deep LSTM Reader encodes the concatenated document and query with skip-connected LSTM layers, producing a joint representation for answer prediction.Inputs connect to every hidden layer, and hidden layers connect to the output.
- Neural models: The Attentive Reader separately encodes document and query with bidirectional LSTMs, then forms the document representation as an attention-weighted sum of token outputs.Attention weights represent how strongly the network attends to each document token for answering the query.
- Neural models: The Impatient Reader recomputes document attention for each query token, recurrently accumulating document information before producing the final joint representation.This lets the model reread the document as the query is processed.
4 Empirical Evaluation
The evaluation tests task difficulty, symbolic versus neural approaches, and the contributions of neural components across the CNN and Daily Mail corpora. Attentive and Impatient Readers perform best, supporting attention’s importance, while symbolic benchmarks expose coverage and scaling limitations.
- Experimental aims: The experiments assess task difficulty, compare parse-based and neural methods, and analyze how neural components contribute to performance.Models are evaluated on reading-comprehension corpora with hyperparameters tuned on each corpus’s validation set.
- Neural models: Attentive and Impatient Readers perform best across both datasets, outperforming the other neural models.This result is consistent with the hypothesis that attention helps propagate information over long distances.
- Frame-semantic benchmark: The frame-semantic benchmark highlights the difficulty of the task from a symbolic NLP perspective.Its coverage is poor because the PropBank parser misses many relations, especially in the datasets’ highlight language, and the approach does not trivially scale across multiple sentences and frames.
- Word distance benchmark: The word distance benchmark performs relatively strongly, particularly on Daily Mail, because the dataset’s highlight language benefits word-distance matching.This performance is unexpectedly stronger relative to the frame-semantic benchmark.
- Neural models: The poor Uniform Reader results support attention’s significance because ignoring attention variables is its only difference from the Attentive Reader.Precision@recall statistics also highlight the attentive approach’s strength.
- Attention analysis: Attentive Reader examples require lexical generalisation and co-reference resolution, while bidirectional LSTM encoders integrate long-range contextual information.Illustrated examples include ‘killed’ → ‘deceased’ and ‘ent119 was killed’ → ‘he was identified.’
5 Conclusion
The paper presents supervised document–query–answer data and recurrent, attention-based neural networks as promising tools for machine reading comprehension. It also identifies unresolved challenges involving world knowledge, multi-document queries, complex inference, and long-range reference resolution.
- 5 Conclusion: The supervised paradigm enables large-scale document–query–answer data collection and effective recurrent and attention-based modeling for machine reading comprehension.The authors describe this methodology as a promising avenue toward full natural language understanding systems.
- 5 Conclusion: Attention mechanisms remain broadly extensible, but world knowledge and multi-document queries require mechanisms whose query complexity does not scale linearly with dataset size.The passage frames the employed attention mechanism as one instance of a more general idea.
- 5 Conclusion: The models still cannot answer some queries requiring complex inference and long-range reference resolution.These limitations remain among the challenges for future machine reading systems.
- 5 Conclusion: 763 tokens and 27 entities characterize the average CNN validation document, making most instances substantially harder than the short examples shown.The examples were selected for brevity rather than representativeness of typical document difficulty.
A Model hyperparameters
The attentive models’ precise hyperparameters are reported in Table 6. All models use asynchronous RmsProp training with specified momentum and decay settings.
- Optimization: All models were trained using asynchronous RmsProp.
- Optimization: Momentum was set to 0.9 and decay to 0.95 for all models.
- Model hyperparameters: Table 6 reports the precise hyperparameters for the various attentive models.
B Performance across document length
The attentive models’ performance declines slightly as document length increases, with precision evaluated across document-length deciles and the longest 5% of CNN validation articles.
- Performance across document length: Performance of the attentive models degrades slightly as documents increase in length.Figures 4 and 5 examine performance as a function of document length.
- Performance across document length: Figure 4 reports precision for each document-length decile and for the 5% longest CNN validation articles.The figure plots precision against document length for the attention models.
C Additional Heatmap Analysis
This section extends attention analysis with additional CNN validation queries and reports aggregated precision across document-length deciles.
- Additional Heatmap Analysis: Aggregated precision is reported for documents up to specified lengths, with points marking the ith decile of document lengths across the corpus.The analysis summarizes performance across document-length thresholds.
- Additional Heatmap Analysis: Visualizations cover additional CNN validation queries for both the Attentive Reader and the Impatient Reader.The appendix expands the paper’s attention-mechanism analysis with examples from both models.
C.1 Attentive Reader
The Attentive Reader correctly answers validation queries requiring lexical generalisation and co-reference resolution, but fails on ambiguous queries, difficult entity clustering, and short queries with insufficient information.
- Positive Instances: Some positive examples require identifying the entity referred to by a quoted passage despite strong lexical cues.The quoted entity is non-trivial to identify in the context document.
- Positive Instances: The Attentive Reader correctly answers validation queries requiring significant lexical generalisation and co-reference resolution.Figure 6 identifies the correct answers as ent201 and ent214.
- Negative Instances: The model fails on ambiguous queries where multiple anonymised entities are plausible answers, particularly for geographic-location markers preceded by “in”.The ambiguity remains even under manual evaluation.
- Negative Instances: One failure results from entity-selection clustering: ent15 and ent81 refer to the same person but were not clustered together.The case is difficult because “Kate Middleton” and “The Duchess of Cambridge” refer to the same person.
- Negative Instances: Another failure occurs when the short query provides too little information for the model.The passage describes this as an additional negative case.
C.2 Impatient Reader
The Impatient Reader’s attention is visualized at every query-processing step, showing focus that becomes increasingly accurate as the model parses larger parts of the query. Initially arbitrary attention gradually concentrates on the correct entity ent5 only after the question is sufficiently parsed.
- C.2 Impatient Reader: Attention is highlighted at every time step to show how the Impatient Reader updates its focus while moving through a query.Figures 10–13 visualize attention at time steps 1–12.
- C.2 Impatient Reader: Initially arbitrary attention gradually focuses on the correct entity ent5 once the question has been sufficiently parsed.The model considers larger parts of the query before identifying the correct entity.