Source-linked AI summary
Dynamic Coattention Networks For Question Answering
Caiming Xiong, Victor Zhong, Richard Socher
TL;DR
Single-pass QA models lack a way to recover from incorrect local maxima in answer prediction. DCN combines co-dependent question-document representations with an iterative span decoder, achieving state-of-the-art SQuAD results while sometimes failing to resolve competing answer maxima.
Problem
Single-pass answer-span estimation does not provide the iterative updating used to recover from incorrect local maxima.
Method
DCN uses a coattentive encoder for question-document interactions and a dynamic decoder that iteratively estimates answer-span boundaries.
Results
75.9% F1 for a single model and 80.4% F1 for an ensemble on SQuAD were reported as state-of-the-art results.
Takeaways & Limitations
Iterative decoding enables the model to explore multiple plausible answer local maxima and can adjust initially incorrect boundaries toward the correct span.
Takeaways & Limitations
With a maximum of four iterations, the decoder sometimes alternates indefinitely between competing local maxima; it converges in 2.7 iterations on average.
Abstract
from arXiv · showhide
Several deep learning models have been proposed for question answering. However, due to their single-pass nature, they have no way to recover from local maxima corresponding to incorrect answers. To address this problem, we introduce the Dynamic Coattention Network (DCN) for question answering. The DCN first fuses co-dependent representations of the question and the document in order to focus on relevant parts of both. Then a dynamic pointing decoder iterates over potential answer spans. This iterative procedure enables the model to recover from initial local maxima corresponding to incorrect answers. On the Stanford question answering dataset, a single DCN model improves the previous state of the art from 71.0% F1 to 75.9%, while a DCN ensemble obtains 80.4% F1.
1 INTRODUCTION
Question answering requires both language understanding and world knowledge, but earlier hand-annotated datasets were too small for data-intensive deep models. SQuAD expanded the dataset scale while preserving span-based answers and diverse reasoning, and DCN improved reported performance.
- Motivation: Earlier human-annotated QA datasets were high quality but small, limiting training for expressive deep neural networks.Large-scale training requires more data than these datasets provided.
- Background: Semi-automated QA datasets enabled training more expressive models but differed from natural human-annotated datasets in required reasoning types.The trade-off was greater scale alongside less natural reasoning demands.
- Background: SQuAD was orders of magnitude larger than prior hand-annotated datasets, with document-span answers, diverse answers, and multi-sentence reasoning.Constraining answers to reference-document spans limits the candidate space while retaining varied reasoning requirements.
- Contribution: The DCN combines coattention between questions and documents with a dynamic decoder that estimates answer-span boundaries iteratively.Its reported results were 75.9% F1 for a single model and 80.4% F1 for an ensemble, compared with earlier leaderboard results.
2 DYNAMIC COATTENTION NETWORKS
The Dynamic Coattention Network combines question–document coattention with an iterative decoder that repeatedly estimates answer-span boundaries. Its coattention encoder fuses context from both inputs, while the decoder updates predictions using prior boundary estimates and can recover from incorrect local maxima.
- 2.2 COATTENTION ENCODER: The coattention encoder captures interactions between the question and document and fuses attention contexts into a co-dependent representation.It computes document summaries in light of question words, question summaries in light of document words, and a representation of prior attention contexts before adding temporal information with a bidirectional LSTM.
- 2.2 COATTENTION ENCODER: The affinity matrix L = D⊤Q scores all document–question word pairs, then row- and column-wise normalization produces attention weights AQ and AD.AQ attends across the document for each question word, while AD attends across the question for each document word.
- 2.3 DYNAMIC POINTING DECODER: The dynamic decoder alternates between predicting answer-span start and end positions while maintaining an LSTM-based state.At each iteration, it uses the coattention encoding associated with current boundary estimates to produce new estimates through a multilayer network.
- 2.3 DYNAMIC POINTING DECODER: The Highway Maxout Network computes start scores, while a separate network with the same architecture computes end scores.The design uses maxout pooling across model variations and includes a highway connection between the first and final maxout layers.
- 2.3 DYNAMIC POINTING DECODER: The decoder trains on start and end cross-entropy losses across iterations and halts when both estimates stabilize or a maximum iteration count is reached.The start and end scores are computed by separate neural networks with the same architecture but different parameters.
3 RELATED WORK
The DCN builds on neural question-answering and co-attention work by combining a coattentive encoder with an iterative dynamic decoder. Unlike single-pass span prediction, it updates start and end positions iteratively.
- 3 RELATED WORK: Earlier neural QA systems included attentive readers, memory networks, pointer-style attention, and Match-LSTM-based models.These approaches span cloze-style and extractive question-answering settings.
- 3 RELATED WORK: Co-attention had also been applied to visual question answering by conditioning image representations on questions and question representations on images.The DCN is inspired by this bidirectional co-attention formulation.
- 3 RELATED WORK: The DCN combines a novel coattentive encoder with a dynamic decoder for question answering.The encoder and decoder are presented as the model’s central components.
- 3 RELATED WORK: The DCN iteratively updates answer start and end positions instead of estimating both in a single pass.This design is described as analogous to Iterative Conditional Modes.
4 EXPERIMENTS
On SQuAD, the DCN achieves strong test performance while its iterative decoder explores answer-span alternatives. Experiments also examine decoder ablations, input-length effects, question types, and error distributions.
- 71.6% exact match and 80.4% F1 place the ensemble DCN first overall on SQuAD test data.
- The dynamic decoder repeatedly estimates answer-span endpoints conditioned on previous estimates, allowing exploration of multiple plausible local maxima.
- Question 1 corrects an initially incorrect start point by iteration 3, while Question 2 reaches the correct answer after both endpoints initially fail.
- The iterative HMN decoder with pool size 16 performs best on development-set ablations, with little improvement after four maximum iterations.
- Longer documents and questions show no notable performance degradation, whereas longer answers reduce performance because identifying the correct span becomes harder.
- DCN mean F1 exceeds previous systems across question types, performing best on “when” questions and struggling with complex “why” questions.
5 CONCLUSION
The DCN combines coattention with an iterative decoder to estimate answer spans and recover from incorrect initial predictions. On SQuAD, it achieves state-of-the-art performance with both single and ensemble models.
- The DCN uses a coattention encoder to learn co-dependent question and document representations.
- Its dynamic decoder iteratively estimates answer spans, enabling recovery from initial local maxima corresponding to incorrect predictions.
- 75.9% F1 is achieved by a single DCN model on SQuAD.
- 80.4% F1 is achieved by the DCN ensemble on SQuAD.
- The DCN significantly outperforms all other models.
A.1 PERFORMANCE WITHOUT ATTENTION
A model without attention uses an LSTM encoder followed by a decoder over document-word hidden states. Its performance is substantially worse than models with attention.
- The no-attention model uses an LSTM encoder that ingests the question before the document.
- The decoder receives hidden states corresponding to words in the document.
- 33.3% exact match and 41.9% F1 are achieved without attention.
- The no-attention model performs significantly worse than models with attention.
A.2 SAMPLES REQUIRING DIFFERENT TYPES OF REASONING
The DCN is evaluated on examples requiring different reasoning types, but the limited sample set prevents conclusive effectiveness claims. The examples nonetheless indicate promise for challenging multi-sentence reasoning tasks.
- The evaluation generates DCN predictions for examples requiring different types of reasoning.
- Because the example set is very limited, it does not conclusively demonstrate effectiveness across reasoning tasks.
- The examples show promise for challenging question answering tasks involving reasoning over multiple sentences.
WHAT IS THE RANKINE CYCLE SOMETIMES CALLED?
The Rankine cycle is sometimes called a practical Carnot cycle. The supplied examples concern lexical variation and show the ground truth and prediction as “practical Carnot cycle.”
- The Rankine cycle is sometimes referred to as a practical Carnot cycle.
- The reasoning type listed for the example is lexical variation, specifically synonymy.
- The ground-truth answer is “practical Carnot cycle.”
- The prediction is “practical Carnot cycle.”
WHICH TWO GOVERNING BODIES HAVE LEGISLATIVE VETO POWER?
The European Parliament and the Council of the European Union can amend and veto legislation during the legislative process.
- The European Parliament and the Council of the European Union have powers of amendment and veto during legislative progress.
WHAT COLLECTION DOES THE V&A THEATRE & PERFORMANCE GALLERIES HOLD?
The V&A Theatre & Performance galleries opened in March 2009 and hold the UK’s biggest national collection of material about live performance since Shakespeare’s day.
- The V&A Theatre & Performance galleries opened in March 2009 and were formerly known as the Theatre Museum.
- They hold the UK’s biggest national collection of material about live performance in the UK since Shakespeare’s day.
- The collection covers drama, dance, musical theatre, circus, music hall, rock and pop, and most other forms of live entertainment.