Source-linked AI summary
Text Understanding with the Attention Sum Reader Network
Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, Jan Kleindienst
TL;DR
Text comprehension benchmarks based on cloze-style questions require selecting answers from context documents, and their scale supports deep-learning evaluation. The paper presents a simple attention-based model that directly selects an answer from the context, with ensembles achieving new state-of-the-art results on all evaluated datasets.
Problem
Cloze-style text-comprehension tasks need models to select a single answer from a document and question, while large automatically generated datasets make deep-learning approaches applicable.
Method
The model uses attention to directly select the answer from the context, with jointly trained recurrent encoders and word embeddings.
Results
Ensembles of the models set new state-of-the-art results on all evaluated datasets, including 69% accuracy for CBT common-noun prediction.
Takeaways & Limitations
The architecture is simpler than previously published models while achieving new state-of-the-art accuracy across the evaluated datasets.
Takeaways & Limitations
Single-model performance varies considerably with random weight initialization and may be difficult to reproduce; CNN and Daily Mail questions may also be near the datasets’ maximal achievable accuracy because some are ambiguous or too difficult for humans.
Abstract
from arXiv · showhide
Several large cloze-style context-question-answer datasets have been introduced recently: the CNN and Daily Mail news data and the Children's Book Test. Thanks to the size of these datasets, the associated text comprehension task is well suited for deep-learning techniques that currently seem to outperform all alternative approaches. We present a new, simple model that uses attention to directly pick the answer from the context as opposed to computing the answer using a blended representation of words in the document as is usual in similar models. This makes the model particularly suitable for question-answering problems where the answer is a single word from the document. Ensemble of our models sets new state of the art on all evaluated datasets.
1 Introduction
Cloze-style question answering tests whether machines can infer missing text from context, while automatically generated examples provide the scale needed for deep-learning methods.
- 1 Introduction: Text understanding matters because most accumulated human information is stored as plain text and can support question answering from unstructured documents.The introduction cites IBM’s Watson use in the Jeopardy challenge as an example.
- 1 Introduction: Cloze-style questions remove a phrase from a sentence and vary in difficulty with the context and missing phrase.Changing the omitted word type also changes the context understanding required: prepositions are easier than named entities.
- 1 Introduction: Summary sentences that condense document content are particularly suitable for testing text comprehension.
- 1 Introduction: Large numbers of cloze-style questions can be automatically generated from real-world documents, enabling data-hungry deep-learning techniques.This contrasts with smaller datasets such as MCTest, where systems commonly rely on handcrafted features.
- 1 Introduction: The paper introduces the task and datasets, presents a model, compares it with prior architectures, and evaluates performance experimentally.
2 Task and datasets
The task selects a correct answer from candidates using a context document and question, with datasets built from news summaries and children’s books. Dataset construction and anonymization shape the task’s difficulty.
- 2 Task and datasets: The paper introduces the task and relevant large-scale datasets before presenting its model and experiments.
- 2 Task and datasets: Each training example is a tuple (q, d, a, A), where the answer a belongs to candidate set A and appears in document d.Questions and documents are word sequences, and candidate answers are vocabulary words.
- 2 Task and datasets: CNN and Daily Mail use news article bodies as contexts and create cloze questions by replacing named entities in summary sentences.Named entities are anonymized and shuffled per example so models must rely on the context rather than world knowledge.
- 2 Task and datasets: The Children’s Book Test uses 20 consecutive book sentences as context and constructs the question from the subsequent 21st sentence.
- 2 Task and datasets: Task complexity varies by omitted word type; the paper focuses on named entities and common nouns because standard LSTM language models lag on these types.The cited comparison reports human-level performance for verbs and prepositions.
3 Our Model — Attention Sum Reader
The Attention Sum Reader computes query and contextual document embeddings, then uses attention directly to select and score answer words from the document. It is designed for tasks whose answer appears as a document word.
- Design choice: The model directly selects answers from the context instead of blending contextual word representations into a new answer representation.Removing transformations after attention makes the architecture particularly suited to answers that are words in the document.
- Scope: Because the AS Reader selects only document words, it cannot produce an answer absent from the document.This scope boundary is paired with the stated assumption that the ground-truth answer appears in the document.
- Model overview: The AS Reader computes a query embedding and contextual embedding for each document word, then compares them with dot products to identify likely answer occurrences.The document encoder represents each word using the whole document, while the query encoder produces a fixed-length representation.
- Pointer sum attention: Attention weights are normalized over document positions, and the probability of a word is obtained by summing attention across all positions where that word occurs.This mechanism is called pointer sum attention and treats attention as a pointer over discrete context tokens.
- Model instance details: The document and query encoders use bidirectional GRU networks, while the word embedding function is a vocabulary lookup table.The document encoder concatenates forward and backward hidden states; the query embedding concatenates the final states of both directions.
4 Related Work
Related text-comprehension models use attention in different ways, typically blending contextual document representations before scoring candidate answers. The AS Reader instead removes post-attention transformations, while retaining recurrent encoders and attention-based selection.
- Attentive and Impatient Readers: Attentive and Impatient Readers use bidirectional document and query encoders with attention, but their empirical performance on CNN and Daily Mail is almost identical.The Impatient Reader recomputes document attention after each query word, whereas the Attentive Reader uses a simpler single attention computation.
- Attentive Reader: The Attentive Reader forms a fixed-length document representation by weighting contextual word embeddings, then combines it with the query representation to score candidates.Candidate scores are normalized with a softmax over answer candidates.
- Architectural contrast: The AS Reader selects the answer directly from the context using attention rather than computing an answer from a weighted sum of contextual representations.This is the central architectural distinction from the Attentive Reader.
- Other attention models: A related model replaces dot-product attention with a bilinear term but still computes a weighted-sum document embedding before scoring answers.That model performs significantly better than the original Attentive Reader despite simplifying its architecture.
- Memory networks: Memory networks use fixed-length windows and summed word embeddings, limiting their ability to capture dependencies outside the window; the AS Reader uses recurrent contextual embeddings instead.The cited memory-network setup also uses a self-supervision heuristic, which the AS Reader does not require.
- Summary: The model combines recurrent document and query encoders, pointer-style attention, and attention summation, while removing transformations after attention to compute answer probabilities directly.Its pointer-network inspiration differs in output structure and in allowing repeated inputs in the document.
5 Evaluation
The AS Reader is evaluated on CNN, Daily Mail, and CBT as single models and ensembles, with analyses of robustness to context and candidate-answer properties. Ensembles achieve new state-of-the-art results across all evaluated datasets.
- Evaluation setup: The model was evaluated on CNN, Daily Mail, and CBT as both single models and ensemble averages.Answer selection was restricted to candidate answers associated with each question-document pair.
- Evaluation setup: Single-model accuracy varies considerably with random weight initialization, making individual results difficult to reproduce.The standard deviation for identical hyperparameters was 0.6–2.5% in absolute test accuracy.
- Evaluation setup: Ensembles used simple probability averaging, with additional greedy selection based on validation improvement.The ensemble sizes were 14, 16, 84, and 53 models for CNN, Daily Mail, CBT common-noun, and CBT named-entity tasks, respectively.
- Additional analyses: Accuracy decreased as document length increased on CNN and Daily Mail, partly because long documents were rare in training and contained more candidate entities.Accuracy was also negatively correlated with the number of named entities, according to the accompanying analysis.
- CNN and Daily Mail: 69.5% test accuracy was achieved by the best-validation single model on CNN, while the top-20% model average reached 69.9%.Fusing multiple models further increased accuracy on both CNN and Daily Mail.
- Children’s Book Test: On CBT, the named-entity ensemble was 4% better than the best previous result, while the common-noun ensemble reached 69%, 6% above MemNN.The best single model reached 68.6% for named-entity prediction, and single models were 0.4% above MemNN for common-noun prediction.
6 Analysis
The analysis examines how Attention Sum Reader accuracy varies with document length and the number and frequency of candidate answers. Accuracy declines with longer documents and more candidates, while the relationship with document length differs on the Children’s Book Test.
- Document length: Accuracy decreases as document length increases on the CNN and Daily Mail datasets.The authors hypothesize that longer documents are more complex and relatively rare in training data.
- Document length: Longer contexts also contain more named entities, increasing the number of possible answers, which is negatively correlated with accuracy.
- Document length: On the Children’s Book Test, the negative relationship between document length and accuracy appears to disappear.The dataset has a more uniform document-length distribution and exactly 10 candidate answers per example.
- Candidate answers: Accuracy drops as the number of candidate answers increases.Examples with large numbers of candidate answers are relatively uncommon.
- Answer frequency: Accuracy significantly drops when the correct answer is less frequent than other candidate answers in the document.The attention sum inherently favors frequently occurring tokens, although correct answers are usually frequent.
7 Conclusion
The paper presents a new neural network architecture for natural-language text comprehension. Although simpler than previously published models, the model achieves new state-of-the-art accuracy on all evaluated datasets, while some CNN and Daily Mail questions may be near a human-imposed ceiling.
- The paper introduces a new neural network architecture for natural-language text comprehension.
- An external analysis suggests that many CNN and Daily Mail questions are ambiguous or too difficult even for humans.Entity anonymization partly contributes to this difficulty, potentially placing ensemble performance near the maximal achievable accuracy on these datasets.
Appendix A Training Details
The appendix describes training monitoring, convergence, hyperparameter selection, dataset-specific epoch timing, and the software used to implement the model.
- Training procedure: Training was stopped when validation-set error began increasing.
- Training procedure: The models usually converged after two training epochs.
- Training time: Table 5 reports the average duration of one training epoch on each of the four datasets using an Nvidia K40 GPU.
- Hyperparameters: The recurrent hidden-layer and source-embedding dimensions were selected by grid search over expanding parameter ranges.The tested range began at 128 to 384 for both parameters and expanded in increments of 128 until validation accuracy consistently decreased.
- Hyperparameters: Table 6 summarizes the tested ranges and best dimensions, reporting unidirectional GRU hidden units while the bidirectional GRU has twice as many.
- Implementation: The model was implemented with Theano and Blocks.
Appendix B Dependence of accuracy on the frequency of the correct answer
The appendix visualizes how accuracy depends on the correct answer’s frequency relative to other candidates in the Children’s Book Test. It also shows how many test examples place the answer at each frequency rank.
- The Children’s Book Test frequency-dependence plots are similar to those for the news datasets.They are included for completeness.
- Figure 8a shows model accuracy when the correct answer is among the n most frequent named entities, for n ∈[1, 10].
- Figure 8b shows the number of test examples in which the correct answer is the n-th most frequent entity.