Source-linked AI summary
Gated-Attention Readers for Text Comprehension
Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W. Cohen, Ruslan Salakhutdinov
TL;DR
The paper addresses answering cloze-style questions over documents, a task used to evaluate text comprehension with objective benchmarks. It introduces a multi-hop Gated-Attention Reader whose multiplicative query–document interactions produce query-specific token representations. The model achieves state-of-the-art performance on several large-scale benchmarks, with more than 4% improvement over competitive baselines.
Problem
Cloze-style document questions provide objective benchmarks for measuring whether systems comprehend contextual text.
Method
The Gated-Attention Reader combines multi-hop recurrent reading with multiplicative, layer-wise interactions between query embeddings and document token representations.
Results
More than 4% improvement over competitive baselines is reported across several large-scale benchmark datasets, with statistically significant gains from Gated Attention.
Takeaways & Limitations
Intermediate-layer analyses show that the reader attends iteratively to different salient aspects of the query before selecting the answer.
Abstract
from arXiv · showhide
In this paper we study the problem of answering cloze-style questions over documents. Our model, the Gated-Attention (GA) Reader, integrates a multi-hop architecture with a novel attention mechanism, which is based on multiplicative interactions between the query embedding and the intermediate states of a recurrent neural network document reader. This enables the reader to build query-specific representations of tokens in the document for accurate answer selection. The GA Reader obtains state-of-the-art results on three benchmarks for this task--the CNN \& Daily Mail news stories and the Who Did What dataset. The effectiveness of multiplicative interaction is demonstrated by an ablation study, and by comparing to alternative compositional operators for implementing the gated-attention. The code is available at https://github.com/bdhingra/ga-reader.
1 Introduction
The paper targets cloze-style document comprehension by combining multi-hop reading with a finer-grained, query-conditioned attention mechanism. Its Gated-Attention Reader improves over strong baselines across three benchmark datasets.
- Motivation: Cloze-style datasets provide automatically constructed, unambiguous benchmarks for measuring document comprehension.They support supervised learning systems that answer questions about context documents.
- Background: Multi-hop architectures scan documents and questions across multiple passes, while attention mechanisms focus on relevant context regions.The paper identifies both as major factors behind recent progress in text comprehension.
- Approach: The Gated-Attention module combines multi-hop reasoning with layer-wise filters that let the query interact with each dimension of token representations.This produces query-specific token representations for answer selection.
- Results: The GA Reader consistently improves over strong baselines on three benchmark datasets.The paper reports that the GA module provides significant improvements for large datasets.
2 Related Work
Prior cloze-comprehension readers use recurrent encoders, token or sentence attention, memory-based multi-hop reasoning, and candidate-focused architectures. These approaches differ in how they represent documents, queries, and candidate answers.
- Task Definition: The task represents a document, a query with a removed phrase, an answer, and a candidate set, requiring selection of the answer from those candidates.Each candidate has at least one token appearing in the document in the datasets considered.
- Recurrent Readers: LSTM-based readers encode document-query pairs and use architectures including single-pass, weighted document attention, and progressively updated reading.
- Attention Readers: The Attention-Sum Reader encodes documents and queries with bidirectional GRUs, then aggregates probabilities for repeated entities using pointer-sum attention.
- Attention Readers: The Attention-over-Attention Reader introduces two-way attention so the query and document attend to each other.
- Multi-hop Architectures: Memory Networks perform multi-hop reasoning by attending over sentence memories and renewing query representations across iterations.Neural Semantic Encoders extend this framework with a write operation that evolves memory during reading.
- Other Architectures: Other systems dynamically represent candidate entities, rerank proposed answers, or use multi-stage hierarchical attention architectures.
3 Gated-Attention Reader
The GA Reader combines multi-hop document reading with gated attention, using query-conditioned multiplicative interactions to refine token representations across layers before selecting an answer.
- Multi-Hop Architecture: The reader performs multiple hops over the document, iteratively refining contextual word representations before mapping the final layer to candidate probabilities.A final attention-sum module converts the last-hop representations into a distribution over candidate answers.
- Model Details: The model uses GRUs to process sequences, with reset and update gates controlling recurrent hidden-state computation and bidirectional outputs concatenating forward and backward states.The GRU maps an input sequence to an output sequence; the BiGRU processes tokens in both directions.
- Multi-Hop Architecture: Each layer uses document and query Bi-GRUs, then applies gated attention to produce the next layer’s document inputs.The document receives contextual embeddings from the previous layer, while a separate query Bi-GRU computes a layer-specific query representation.
- Gated-Attention Module: Gated attention forms a token-specific query representation and multiplies it element-wise with each document-token representation.The multiplicative interaction acts as a fine-grained filter over components of the evolving semantic representation.
- Answer Prediction: At answer prediction, token scores are converted into probabilities, candidate probabilities aggregate matching document-token positions, and the highest-probability candidate is selected.The candidate aggregation is the pointer-sum attention operation used in the AS Reader.
- Further Enhancements: The model can augment word representations with character-level embeddings and append the qe-comm feature to the final document BiGRU inputs.The character composition model uses a Bi-GRU over character embeddings, while qe-comm indicates whether document tokens occur in the query.
4 Experiments and Results
The GA Reader is evaluated across five cloze-style comprehension datasets, with comparisons to prior systems, gating choices, hop counts, and component ablations. Results show strong benchmark performance and identify multiplicative gating, multi-hop processing, pretrained embeddings, token-specific attention, and dataset scale as important factors.
- Evaluation setup: The experiments cover CNN, Daily Mail, two Children’s Book Test subsets, and Who Did What, using cloze queries over documents.CNN and Daily Mail queries remove anonymized entities from article summaries; CBT queries remove common nouns or named entities from a subsequent sentence; WDW queries remove person entities.
- Benchmark comparison: Feature engineering significantly improves WDW and CBT but not CNN or Daily Mail, while fixing embeddings helps the smaller WDW and CBT datasets.The authors relate the dataset difference to anonymization and larger training sizes for CNN and Daily Mail, and describe the fixed-embedding effect as consistent with reduced overfitting.
- Benchmark comparison: GA Reader sets new state-of-the-art results on WDW, CNN, and Daily Mail, while approaching the strongest larger-corpus systems on CBT.It improves over the best previous single models by 3.2% on CNN and 4.3% on Daily Mail; on WDW, qe-comm increases performance by 3.2% and 3.5% on Strict and Relaxed settings.
- GA Reader analysis: Element-wise multiplication outperforms addition and concatenation for gating, supporting the use of query representations as filters on document features.The comparison uses x = d + q, x = d∥q, and x = d⊙q on WDW without qe-comm and with fixed word embeddings.
- GA Reader analysis: Accuracy rises steeply as GA Reader hops increase from K = 1 to K = 3 and then remains constant, indicating the importance of multi-hop processing.At K = 1, the model is equivalent to the AS Reader without GA modules.
- Ablation analysis: Ablations identify pretrained GloVe vectors and token-specific query attention as major contributors, while removing character embeddings reduces performance by about 1%.Token-specific attention gates individual document tokens using query parts relevant to each token rather than only an overall query representation.
- Attention visualization: Intermediate attention visualizations show different layers selecting distinct query clues before the final layer chooses the candidate with the highest match.In one WDW example, the correct answer attends to financial regulatory standards in the first layer and “us president” in the second, whereas an incorrect answer attends to only one aspect.
5 Conclusion
The GA Reader combines multiplicative gated attention with multi-hop processing for cloze-style document questions, achieving strong benchmark performance and supporting evidence from ablations and attention analysis.
- The GA Reader combines a novel multiplicative gating mechanism with a multi-hop architecture for answering cloze-style questions over documents.
- More than 4% improvements over competitive baselines are reported on several large-scale benchmark datasets.
- An ablation study shows statistically significant improvements from using Gated Attention as information filters.
- Multiplicative gating performs better empirically than addition and concatenation for implementing gated attention, although theoretical justification remains future work.
- Intermediate-layer attention analysis indicates that the model iteratively attends to different query aspects before selecting the final answer.
A Implementation Details
The implementation uses shared optimization settings across models and datasets, pretrained word embeddings, and dataset-dependent embedding and character-composition choices.
- The models use ADAM with batch size 32, an initial learning rate of 5 × 10^-4, epoch-based halving, and gradient clipping at 10.
- The number of layers K is set to 3 for all experiments.
- Word lookup tables are initialized with 100d GloVe vectors, while test-time OOV tokens receive unique random vectors.
- Fixing pretrained embeddings improves test performance on the smaller WDW and CBT datasets, possibly by avoiding overfitting.
- Character composition is omitted for CNN and Daily Mail because their entities and candidate answers are anonymized to generic tokens.
B Attention Plots
The supplied figures are layer-wise attention visualizations of the GA Reader trained on WDW-Strict.
- Figure 4 is a layer-wise attention visualization of the GA Reader trained on WDW-Strict.
- Figure 5 is a layer-wise attention visualization of the GA Reader trained on WDW-Strict.
- Figure 6 is a layer-wise attention visualization of the GA Reader trained on WDW-Strict.
- Figure 7 is a layer-wise attention visualization of the GA Reader trained on WDW-Strict.