Source-linked AI summary
Attention-over-Attention Neural Networks for Reading Comprehension
Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, Guoping Hu
TL;DR
Cloze-style reading comprehension requires filling missing words by reasoning over document and query context, while existing approaches use heuristic combinations or fixed terms. The paper introduces an attention-over-attention reader that combines document-level attentions with query-side attention, plus N-best reranking. On public datasets, the model consistently and significantly improves over state-of-the-art systems, including 2.3% and 2.0% absolute gains over EpiReader on CBTest NE and CN test sets.
Problem
Cloze-style reading comprehension requires models to fill missing words using context from both the document and query, while prior approaches used heuristic merging or predefined non-trainable terms.
Method
The attention-over-attention reader computes document- and query-side attentions, automatically weights individual document attentions, and optionally reranks N-best candidates.
Results
2.3% and 2.0% absolute improvements over EpiReader in CBTest NE and CN test sets were reported, with further 2.0% to 3.7% gains from reranking over AoA Reader.
Takeaways & Limitations
The model delivers consistent and significant improvements over various state-of-the-art systems across several public datasets.
Takeaways & Limitations
Future work targets applying the architecture to other tasks and handling reasoning over several sentences.
Abstract
from arXiv · showhide
Cloze-style queries are representative problems in reading comprehension. Over the past few months, we have seen much progress that utilizing neural network approach to solve Cloze-style questions. In this paper, we present a novel model called attention-over-attention reader for the Cloze-style reading comprehension task. Our model aims to place another attention mechanism over the document-level attention, and induces "attended attention" for final predictions. Unlike the previous works, our neural network model requires less pre-defined hyper-parameters and uses an elegant architecture for modeling. Experimental results show that the proposed attention-over-attention model significantly outperforms various state-of-the-art systems by a large margin in public datasets, such as CNN and Children's Book Test datasets.
1 Introduction
Cloze-style reading comprehension requires models to connect documents and queries while filling missing words. The paper introduces attention-over-attention, a simpler architecture that automatically combines document-level attentions and adds N-best reranking.
- Cloze-style reading comprehension asks machines to fill an appropriate missing word using document and query context.
- Large-scale training data supports neural learning for document–query relationships, motivating datasets such as CNN/Daily Mail and Children’s Book Test.
- The attention-over-attention model places a second attention mechanism over existing document-level attention.
- The model automatically generates attended attention and uses mutual query-to-document and document-to-query information instead of relying on heuristic combinations or many fixed terms.
- The authors also propose N-best reranking to rescore answer candidates and further improve performance.
2 Cloze-style Reading Comprehension
Cloze-style reading comprehension fills a missing word in a query using a document and query context. Public datasets provide large-scale training data, with CNN/Daily Mail and Children’s Book Test differing in how documents and queries are constructed.
- A general cloze-style problem consists of a document D, a query Q, and an answer A, usually a single word appearing in the document.
- Large-scale training data is essential for training neural networks on cloze-style reading comprehension.
- CNN / Daily Mail: CNN/Daily Mail uses news articles as documents and summaries as queries with one entity word replaced by a placeholder.
- Children’s Book Test: Children’s Book Test uses 20 consecutive story sentences as the document and the 21st sentence as the query with one word blanked.
- Children’s Book Test: CBTest subtypes differ in their blank-filling dependence on the document, with related work focusing mainly on named entities and common nouns.
3 Attention-over-Attention Reader
The Attention-over-Attention Reader builds document and query representations, derives bidirectional attentions, and places an additional attention mechanism over document-level attentions for answer prediction. Its architecture adds limited calculations without additional weights, giving it complexity similar to the AS Reader.
- Architecture: The model transforms document and query words into shared embeddings, then uses separate bidirectional RNNs to obtain contextual representations.Sharing the embedding matrix allows both document and query representations to participate in embedding learning.
- Pair-wise Matching Score: A pair-wise matching matrix scores every document–query word pair using the dot product of their contextual representations.The matrix M has dimensions |D|×|Q|, with each entry representing one document–query matching score.
- Attention-over-Attention: Column-wise softmax produces query-to-document attentions, with each column representing document attention conditioned on one query word.The resulting attention α(t) is a distribution over document positions for query word t.
- Attention-over-Attention: Row-wise softmax produces document-to-query attentions, indicating which query words are important for each document word.These query-level attentions are denoted β(t) and complement the document-level attentions.
- Attention-over-Attention: The model averages the document-to-query attentions and takes their dot product with α to form attended document-level attention s.This operation learns the relative contribution of query words and combines individual document-level attentions into the final attention.
- Output and Training: The final output is represented in vocabulary space rather than document-level attention space, while training maximizes the correct answer’s log-likelihood.The architecture is designed to estimate answers directly from document-level attention, and its added calculations do not introduce additional weights.
4 N-best Re-ranking Strategy
The N-best re-ranking strategy mimics double-checking by refilling candidate answers into the query and rescoring alternative completions. It combines language-model features with automatically tuned weights to select the lowest-cost candidate.
- N-best Decoding: The strategy generates an N-best list instead of selecting only the candidate with the highest neural-network probability.Follow-up candidates are retained during decoding for later comparison.
- Candidate Refill: Each candidate is refilled into the query’s blank to form a complete sentence that can be checked in context.This supports checking candidate appropriateness, fluency, and grammar.
- Feature Scoring: Three features score the N-best candidates: global N-gram language modeling, local N-gram language modeling, and word-class language modeling.The global model evaluates fluency from training documents, the local model uses the test-time document, and the word-class model uses clustered word classes.
- Feature Scoring: K-best MIRA automatically tunes the feature weights on the validation set.The tuning procedure is adopted from statistical machine translation practice.
- Re-scoring and Re-ranking: The final answer is the candidate with the lowest weighted feature cost.Weighted feature sums are computed for the N-best sentences before selection.
5 Experiments
Experiments evaluate the model on CNN and CBTest datasets, showing strong gains from the AoA Reader and further improvements from N-best reranking. Ablations indicate that different language-model features benefit named entities and common nouns differently.
- Experimental Setup: The experiments use CNN news and CBTest NE/CN datasets, with results reported in Table 2.
- Overall Results: N-best reranking adds 2.0% to 3.7% improvements over AoA Reader on CBTest NE/CN test sets.
- Overall Results: Learning weights between individual attentions improves CNN validation and test performance by 4.1% and 3.7% over CAS Reader.
- Re-ranking Ablation: Named-entity performance benefits mainly from LMlocal, whereas common-noun performance benefits more from LMglobal and LMwc.LMglobal and LMwc are trained on training data, while LMlocal is trained within each test document part.
- Re-ranking Ablation: Named entities depend more on local features because test-time documents are more likely to contain new named entities than common nouns.
6 Quantitative Analysis
Quantitative analysis examines accuracy by document length and answer frequency rank. AoA Reader gains become larger on documents longer than 700, while both models perform well for frequent and very infrequent answers.
- Document Length: AoA Reader consistently improves over AS Reader across different document lengths.
- Document Length: Figure 2 plots test accuracy against document length and shows the number of samples in each interval.
- Answer Frequency: Figure 3 plots test accuracy against answer frequency rank and shows the number of samples in each rank.
- Answer Frequency: The correct answer is the highest-frequency candidate in over 40% of the test set, or 1071 of 2500 samples.
- Answer Frequency: Both models perform well when the correct answer is more frequent than the other candidates and when its frequency rank exceeds 7.
- Answer Frequency: The analysis suggests that neutral-frequency answers are ambiguous, whereas models tend toward candidates with either very high or very low frequency.
7 Related Work
Related work develops neural approaches and datasets for cloze-style reading comprehension, including attention, memory, pointer, and transfer-based models. The paper differs by learning weights over document-level attentions without heuristic merging.
- CNN/Daily Mail constructs document-query-answer triples from news articles and summaries, with an attention-based neural network for the task.
- The Children’s Book Test generates samples from children’s books and uses a window-based memory network with self-supervised hard-attention learning.
- Pointer-style reading models directly select a single-word answer from the document, requiring the answer to appear there.
- Reading-comprehension models have also been applied to Chinese zero pronoun resolution using automatically generated pseudo-training data.
- Unlike CAS Reader’s heuristic merging functions, this work explicitly learns weights between individual document-level attentions and combines them into final attention.
- The paper presents attention-over-attention as a generally simple mechanism that brings significant improvements over recent systems.
8 Conclusion
The attention-over-attention reader addresses cloze-style reading comprehension by modeling attention on both documents and queries, then producing attended document attention for prediction. It delivers consistent, significant improvements over state-of-the-art systems across several public datasets, while broader task applications and multi-sentence reasoning remain future work.
- The AoA Reader computes attentions for both the document and query to exploit mutual information.
- A weighted sum of attention produces attended attention over the document for final predictions.
- The model achieves consistent and significant improvements over various state-of-the-art systems across several public datasets.
- Applying the architecture to other tasks and investigating reasoning across several sentences are identified as future work.