Source-linked AI summary
SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering
Chenguang Zhu, Michael Zeng, Xuedong Huang
TL;DR
Conversational QA requires models to understand a passage together with dialogue history, a setting traditional single-turn MRC models cannot directly address. SDNet combines attention over passage and conversation context with BERT representations. On CoQA, it improves overall F1 over the previous state of the art, with a further gain from ensembling.
Problem
Conversational QA requires understanding a passage and previous questions and answers, whereas traditional single-turn MRC does not directly address that dialogue context.
Source: §1 · p. 1Method
SDNet combines inter-attention and self-attention over passage and dialogue context, prepends prior question-answer rounds, and uses a weighted sum of fixed BERT layer outputs.
Source: §1 · p. 1, §2.3 · p. 4Results
1.6% overall F1 improvement over the previous state-of-the-art model on CoQA; the ensemble further improves overall F1 by 2.7%.
Source: §3 · p. 6Takeaways & Limitations
On CoQA, where many questions require both passage and prior-turn understanding, SDNet reports superior results over previous approaches.
Source: §4 · p. 7
Abstract
from arXiv · showhide
Conversational question answering (CQA) is a novel QA task that requires understanding of dialogue context. Different from traditional single-turn machine reading comprehension (MRC) tasks, CQA includes passage comprehension, coreference resolution, and contextual understanding. In this paper, we propose an innovated contextualized attention-based deep neural network, SDNet, to fuse context into traditional MRC models. Our model leverages both inter-attention and self-attention to comprehend conversation context and extract relevant information from passage. Furthermore, we demonstrated a novel method to integrate the latest BERT contextual model. Empirical results show the effectiveness of our model, which sets the new state of the art result in CoQA leaderboard, outperforming the previous best model by 1.6% F1. Our ensemble model further improves the result by 2.7% F1.
1 INTRODUCTION
Conversational QA requires models to understand both a passage and the context carried across dialogue turns, which traditional single-turn MRC models are not designed to handle directly. SDNet combines attention over passage and dialogue context with BERT embeddings, improving CoQA overall F1 over the previous state of the art.
1 INTRODUCTION
- Conversational QA requires understanding both the passage and previous question-answer context, unlike traditional single-turn MRC.Traditional MRC questions about a passage usually have no connection to one another.Source: §1 · p. 1
- SDNet combines inter-attention and self-attention with a weighted sum of BERT layer outputs and prepended prior question-answer rounds.The paper reports substantial prediction-accuracy gains from each component.Source: §1 · p. 1
- SDNet improves CoQA overall F1 by 1.6% over the previous state-of-the-art model, from 75.0% to 76.6%.The ensemble reaches 79.3% F1.Source: §1 · p. 1
2 APPROACH
SDNet turns conversational QA into reading comprehension by adding recent dialogue history to the current question. It combines contextual encoders, attention, and answer prediction to use both passage and conversation context.
2 APPROACH
- Prepending the latest N question-answer rounds to the current question converts conversational response generation into a machine-reading-comprehension task.Question and answer markers distinguish the utterance types.Source: §2 · p. 2
2.1 MODEL OVERVIEW
- The model combines recurrent contextualization with question-to-passage attention before and after the RNNs, then predicts answer-span boundaries.Self-attention also relates words at different positions in the question and passage.Source: §2.1 · pp. 2–3
2.2 ENCODING LAYER
- SDNet uses frozen BERT parameters and a weighted combination of BERT layer outputs to form contextualized word embeddings.BERT's subword-token hidden states are combined into word representations.Source: §2.1 · p. 2, §2.2 · p. 3
2.3 INTEGRATION LAYER
- Separate BiLSTMs encode passage and question, while an added question RNN and question self-attention support higher-level understanding and concept carry-over.The paper identifies concept carry-over and coreference resolution as motivations for question self-attention.Source: §2.3 · pp. 3–4
- Multilevel question-to-context attention uses representations from all RNN layers, while history-of-word combination limits output dimensions and context self-attention links word pairs.An additional RNN then generates the final context representation.Source: §2.3 · p. 4
2.4 OUTPUT LAYER
- The output layer predicts answer-span start and end positions and handles span, yes, no, and unknown answers; inference selects the largest probability with spans capped at 15 words.Training maximizes the ground-truth answer probability across these answer types.Source: §2.4 · pp. 4–5
3 EXPERIMENTS
On CoQA, SDNet outperforms the previous state-of-the-art model, and its ensemble improves further. The experiments also examine dataset setup, training progress, model components, and how much dialogue history to prepend.
Evaluation setup
- CoQA spans multiple domains, averages more than 15 question-answer turns per passage, and includes all out-of-domain passages in the test set.The evaluation uses word-level F1 and compares SDNet with PGNet, DrQA, DrQA+PGNet, BiDAF++, and FlowQA.Source: §3 · pp. 5–6
Results
- 1.6% overall F1 improvement over FlowQA; the SDNet ensemble adds 2.7% overall F1 and reaches 80.7% F1 on in-domain datasets.The paper reports that this is the first model to exceed 80% F1 on in-domain datasets.Source: Table 2 · p. 6, §3 · p. 6
Ablation Studies
- Removing BERT reduces development-set F1 by 7.15%, while weighted BERT-layer outputs improve it by 1.75% over using only the last layer.Using BERT-base instead of BERT-large reduces F1 by 2.61%; variational dropout and self-attention improve it by 0.24% and 0.75%, respectively.Source: Table 3 · p. 6, §3 · p. 6
Results
- SDNet surpasses all but one baseline after epoch two and reaches state-of-the-art results after eight epochs.Figure 2 tracks CoQA development-set F1 over training epochs.Source: §3 · p. 6, Figure 2 · p. 7
Contextual history
- Excluding dialogue history reduces F1 by as much as 8.56%, and performance peaks with N = 2 history rounds.The final SDNet model uses N = 2.Source: §3 · pp. 6–7, Table 4 · p. 7
4 CONCLUSIONS
The conclusion presents SDNet as a contextual-attention network for conversational question answering, combining inter-attention, self-attention, and BERT. On CoQA it outperforms the previous state-of-the-art model by 1.6% overall F1; future work targets open-domain multiturn QA.
Conclusion
- SDNet combines inter-attention and self-attention over passage and conversation history with an innovative use of BERT, outperforming the previous state-of-the-art model by 1.6% overall F1 on CoQA.The proposed next setting is open-domain multiturn QA with a large corpus or knowledge base, where the target passage is not directly available.Source: §4 · p. 7
A IMPLEMENTATION DETAILS
Implementation uses fixed BERT contextual embeddings alongside GloVe, with tokenization, regularization, optimization, and layer dimensions specified for training.
Embeddings and tokenization
- Each spaCy token receives the average BERT embedding of its BPE sub-tokens, and BERT-Large-Uncased weights remain fixed.The model uses 300-dimensional GloVe embeddings alongside BERT contextual embeddings.Source: A IMPLEMENTATION DETAILS · p. 8
Training
- Training uses dropout rates of 0.4 on BERT outputs and 0.3 on other layers, with variational dropout sharing masks across RNN timesteps and passage-level batches.All questions and answers associated with one passage are batched together.Source: A IMPLEMENTATION DETAILS · p. 8
- Adamax training uses learning rate 0.002, β = (0.9, 0.999), ϵ = 10^-8, 30 epochs, and gradient clipping at length 10.Source: A IMPLEMENTATION DETAILS · p. 8
Layer dimensions
- Attention modules use hidden sizes of 250 or 300, while the two-layer question and context RNNs use 125-dimensional layers.Word-level attention, the flow module, and question self-attention are 300-dimensional; multilevel question-to-context and context self-attention are 250-dimensional.Source: A IMPLEMENTATION DETAILS · p. 8