Source-linked AI summary
Bidirectional Attention Flow for Machine Comprehension
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, Hannaneh Hajishirzi
TL;DR
Machine comprehension requires modeling interactions between a query and its context, but prior attention methods often summarize information early or attend in one direction. BIDAF uses a hierarchical, multi-granular architecture with bidirectional attention flow that preserves query-aware context information, and it achieves state-of-the-art results on SQuAD and CNN/DailyMail.
Problem
Prior machine-comprehension attention methods often summarize context into fixed-size vectors, couple attention temporally, or use only one attention direction.
Method
BIDAF is a hierarchical multi-stage network using character-, word-, and contextual embeddings with bidirectional attention flow and no early summarization.
Results
BIDAF achieves state-of-the-art results on SQuAD and CNN/DailyMail cloze tests, including SQuAD ensemble scores of 73.3 EM and 81.1 F1.
Takeaways & Limitations
The experiments and ablations support BIDAF as a suitable representation for machine comprehension and show the importance of its model components.
Abstract
from arXiv · showhide
Machine comprehension (MC), answering a query about a given context paragraph, requires modeling complex interactions between the context and the query. Recently, attention mechanisms have been successfully extended to MC. Typically these methods use attention to focus on a small portion of the context and summarize it with a fixed-size vector, couple attentions temporally, and/or often form a uni-directional attention. In this paper we introduce the Bi-Directional Attention Flow (BIDAF) network, a multi-stage hierarchical process that represents the context at different levels of granularity and uses bi-directional attention flow mechanism to obtain a query-aware context representation without early summarization. Our experimental evaluations show that our model achieves the state-of-the-art results in Stanford Question Answering Dataset (SQuAD) and CNN/DailyMail cloze test.
1 INTRODUCTION
BIDAF addresses limitations in prior machine-comprehension attention mechanisms by preserving context information and combining attention in both directions. It achieves leading results on SQuAD and CNN/DailyMail.
- Introduction: BIDAF is a hierarchical multi-stage network that represents context at multiple granularities and produces query-aware context representations.It incorporates character-level, word-level, and contextual representations.
- Introduction: Its attention layer computes attention at every context time step instead of summarizing the context into a fixed-size vector.The attended vectors and previous-layer representations flow into subsequent modeling layers, reducing information loss from early summarization.
- Introduction: BIDAF uses memory-less attention whose computation at each time step depends only on the current query and context representations.The authors report that memory-less attention gives a clear advantage over dynamic attention.
- Introduction: The model applies attention in both query-to-context and context-to-query directions, providing complementary information.This contrasts with prior approaches that were usually uni-directional.
- Introduction: BIDAF outperforms previous approaches on the SQuAD test-set leaderboard and achieves state-of-the-art results on the CNN/DailyMail cloze test.The CNN/DailyMail result uses a modification only to the output layer.
2 MODEL
BIDAF builds query-aware context representations through six hierarchical layers, combining multi-granular embeddings, bidirectional attention, contextual modeling, and task-specific prediction. Its attention flow preserves token-level information rather than early-summarizing the inputs.
- Architecture: BIDAF comprises six layers: character embedding, word embedding, contextual embedding, attention flow, modeling, and output.The output layer is application-specific and can be modified for different comprehension tasks.
- Embedding layers: Character-level CNNs map each word’s characters to a fixed-size vector, while pre-trained word embeddings represent whole words.The character CNN outputs are max-pooled over the word’s character sequence.
- Contextual embedding: Bidirectional LSTMs create contextual representations for both the context and query from the embedding-layer outputs.The forward and backward outputs are concatenated into matrices H and U.
- Attention flow: The attention flow layer links context and query representations without summarizing either modality into a single feature vector.Attention vectors and earlier embeddings flow into the subsequent modeling layer, reducing information loss from early summarization.
- Attention flow: A shared context-query similarity matrix supports both context-to-query and query-to-context attention.The resulting fused vectors form query-aware representations for each context word.
- Modeling and output: The modeling layer uses two bidirectional LSTM layers to capture query-conditioned interactions among context words.For extractive QA, the output layer predicts the answer phrase’s start and end indices.
3 RELATED WORK
Related machine-comprehension work differs in how it computes and uses attention, while BIDAF retains attention vectors instead of early-summarizing the query and context. Bidirectional attention also connects BIDAF to improvements reported in visual question answering.
- Machine comprehension: Large datasets such as CNN/DailyMail, Children’s Book Test, and SQuAD enabled the development of end-to-end neural machine-comprehension models.The authors evaluate BIDAF on SQuAD and CNN/DailyMail.
- Machine comprehension: Prior machine-comprehension attention methods include dynamic attention, one-time attention, and multi-hop attention variants.These groups differ in whether attention is updated over time, computed once, or repeatedly recomputed across layers.
- Machine comprehension: Unlike models that summarize the query and context in the attention layer, BIDAF lets attention vectors flow into later modeling layers.This preserves the attention information for subsequent processing.
- Visual question answering: In visual question answering, attending from the image to question words in addition to question-to-image attention was reported to improve performance.The authors identify this as consistent with BIDAF’s bidirectional attention finding in language.
4 QUESTION ANSWERING EXPERIMENTS
On SQuAD, BIDAF is evaluated against prior systems, ablations, and qualitative analyses, achieving strong test performance while showing contributions from its components and contextual representations.
- Results: BIDAF’s SQuAD evaluation compares its performance with competing approaches on the test set and with ablations on the development set.The table caption specifies that ablation results are single-run results.
- Results: 73.3 EM and 81.1 F1: BIDAF ensemble outperformed all previous approaches on the hidden SQuAD test set.
- Ablations: Character-level and word-level embeddings both contribute to BIDAF’s performance on the SQuAD development set.The authors associate word-level embeddings with whole-word semantics and character-level embeddings with rare or out-of-vocabulary words.
- Visualizations: Contextual embeddings separate the two usages of “May” that are not separated in the word-embedding space.The visualization uses t-SNE plots of month-name representations from the two feature spaces.
- Discussion: BIDAF answered more than 86% of development questions and outperformed the traditional baseline across the ten most frequent question-initial categories.The comparison is presented using a Venn diagram and a breakdown by question first word.
- Error Analysis: Among 50 sampled incorrect questions, 50% involved imprecise answer boundaries, while syntactic complications and ambiguities accounted for 28%.The remaining errors involved paraphrase, external knowledge, multiple sentences, or tokenization.
5 CLOZE TEST EXPERIMENTS
BIDAF is adapted for CNN/DailyMail cloze-style comprehension by predicting a single entity answer, and it outperforms prior single-run models on both datasets.
- Dataset: CNN/DailyMail cloze examples pair a news article with an incomplete summary sentence whose missing word is an anonymized named entity.The datasets contain 300k/4k/3k CNN and 879k/65k/53k train/dev/test examples, respectively.
- Model Details: For cloze comprehension, BIDAF predicts only the answer’s start index, masks non-entity words, and sums probabilities across entity instances matching the answer.The end-index prediction is omitted because each answer is a single word.
- Results: BIDAF outperformed previous single-run models on both CNN and DailyMail validation and test data.The results table also includes previous ensemble methods for comparison.
6 CONCLUSION
BIDAF combines hierarchical representations with bidirectional attention flow to produce query-aware context representations without early summarization. Experiments report state-of-the-art results on SQuAD and CNN/DailyMail, while ablations support the importance of its components.
- BIDAF represents context at multiple granularities and uses bidirectional attention flow to form query-aware context representations without early summarization.
- The model achieves state-of-the-art results on both the SQuAD and CNN/DailyMail cloze-test benchmarks.
- Ablation analyses demonstrate the importance of each component in the model.
- Visualizations and discussions indicate that BIDAF learns representations suitable for machine comprehension and attends to correct paragraph locations for complex questions.
- Future work proposes extending the approach with multiple attention-layer hops.
A ERROR ANALYSIS
The error analysis categorizes BIDAF’s incorrect SQuAD answers into six error types and illustrates the categories with selected examples. One reported example involves an answer span that includes an incorrect year range rather than the requested death year.
- The table reports each error type with its ratio and an illustrative context-question-prediction example.
- Incorrect preprocessing is listed as one error type with a reported ratio of 2%.
- Table 4 categorizes BIDAF’s EM-incorrect SQuAD answers into six different error categories.The analysis randomly selected EM-incorrect answers and classified them by error type.
B VARIATIONS OF SIMILARITY AND FUSION FUNCTIONS
The appendix evaluates alternative similarity and fusion functions on SQuAD development data. It notes non-trivial differences among similarity definitions and finds no benefit from adding an MLP to the fusion function.
- Table 5 compares variations of the similarity function α and fusion function β on SQuAD development data.
- The appendix defines alternative similarity and fusion functions to test how these design choices affect model performance.
- Dot-product similarity computes similarity between two vectors, with matrix transpose denoted by ⊤.
- A linear similarity variant uses a trainable weight matrix and simplifies Equation 1 by dropping the h ◦u term from the concatenation.
- Adding an MLP to β does not help and yields a slightly worse result than β without an MLP.