Source-linked AI summary
BERT with History Answer Embedding for Conversational Question Answering
Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang, Mohit Iyyer
TL;DR
Multi-turn conversational search needs models that use conversation history to answer current questions, while existing approaches either prepend turns or use complicated attention. The paper introduces history answer embedding within a BERT-based ConvQA framework, achieving 62.4 F1 on QuAC and better robustness and efficiency than the compared approaches.
Problem
ConvQA must model conversation history to understand current questions, but existing methods either prepend history turns or use complicated attention mechanisms.
Method
The paper defines separate history-selection and history-modeling components, then uses history answer embeddings to introduce selected history into BERT.
Results
62.4 F1 was achieved on the QuAC leaderboard, while HAE outperformed simple history prepending and the models were at least 5 times faster than FlowQA.
Takeaways & Limitations
HAE provides a simple BERT-based approach that remains robust with longer conversation history and supports efficient ConvQA modeling.
Abstract
from arXiv · showhide
Conversational search is an emerging topic in the information retrieval community. One of the major challenges to multi-turn conversational search is to model the conversation history to answer the current question. Existing methods either prepend history turns to the current question or use complicated attention mechanisms to model the history. We propose a conceptually simple yet highly effective approach referred to as history answer embedding. It enables seamless integration of conversation history into a conversational question answering (ConvQA) model built on BERT (Bidirectional Encoder Representations from Transformers). We first explain our view that ConvQA is a simplified but concrete setting of conversational search, and then we provide a general framework to solve ConvQA. We further demonstrate the effectiveness of our approach under this framework. Finally, we analyze the impact of different numbers of history turns under different settings to provide new insights into conversation history modeling in ConvQA.
1 INTRODUCTION
The paper frames ConvQA as a concrete conversational-search setting where modeling history is essential, then proposes history answer embedding for BERT-based models. It reports improved QuAC performance and robustness to longer histories.
- Motivation: Conversational search requires systems to track changing information needs across multiple interaction turns.ConvQA omits proactive questioning but provides a concrete setting for modeling history between conversation cycles.
- Problem formulation: ConvQA incorporates conversation history into machine comprehension through history selection and history modeling.This work focuses on history modeling and uses a rule-based history-selection method.
- Motivation: A QuAC example shows that the answer to an earlier turn can be needed to interpret a later follow-up such as “that way.”This illustrates why previous history turns can be essential to understanding the current information need.
- Motivation: Previous methods prepend history turns, mark answers in passages, or use complicated attention mechanisms, limiting long-history handling or increasing overhead.These limitations motivate a simpler history answer embedding approach.
- Approach: History answer embedding is tailored to BERT-based architectures and is described as simple, robust, effective, and more training-efficient than previous approaches.The method is introduced as an early attempt to model information-seeking conversation history in a BERT-based model.
- Results: 62.4 F1 on the QuAC leaderboard was achieved with significantly shorter training time than the state-of-the-art method.The paper also analyzes how different amounts of conversation history affect performance.
2 RELATED WORK
The related work positions ConvQA near machine comprehension while emphasizing its conversational organization and information-seeking setting. The paper focuses on modeling history rather than proactive question asking or other conversational-search tasks.
- ConvQA and machine comprehension: ConvQA differs from machine comprehension because its questions are organized in conversations, requiring conversation-history modeling.The paper treats ConvQA as closely related to machine comprehension while focusing on the additional conversational structure.
- History modeling: Existing ConvQA approaches prepend history turns or mark history answers in the passage, while the proposed method is presented as more robust for longer histories.The comparison motivates alternative history modeling within ConvQA.
- Datasets: CoQA is often factoid and entity-based, whereas QuAC mainly contains non-factoid questions in an information-seeking setting.Because QuAC better matches the paper’s information-seeking interest, the experiments focus on QuAC.
- Conversational search: Related conversational-search work includes proactive question asking, user-intent prediction, next-question prediction, external-knowledge ranking, and observational studies.This paper instead concentrates on conversation-history handling as one part of building conversational-search systems.
3 OUR APPROACH
The approach frames ConvQA as machine comprehension augmented by modular history selection and history modeling. It introduces history answer embedding (HAE) to incorporate selected answer history into BERT without prepending turns.
- 3.1 Task Definition: ConvQA predicts an answer span within passage p for question q_k using preceding conversation history H_k.
- 3.2 A ConvQA Framework: The framework separates history selection, which chooses useful prior turns, from history modeling, which incorporates them into the ConvQA model.
- 3.2 A ConvQA Framework: The implemented selection rule always chooses the immediate previous j turns, based on their expected relevance to the current question.
- 3.3.1 Machine Comprehension: The BERT-based machine comprehension model packs question and passage tokens, then predicts start and end probabilities for the answer span.
- 3.3.2 History Answer Embedding: HAE adds two embeddings indicating whether each token belongs to a history answer, influencing contextual BERT representations and converting MC into ConvQA.
- 3.3.3 Model Training: Training creates one-history-turn variations, merges the selected immediate turns, uses the merged answers for HAE, and applies sliding windows to long passages.
4 EXPERIMENTS
Experiments on QuAC evaluate BERT-based ConvQA models and history-modeling methods using answer-overlap and human-equivalence metrics. HAE handles longer histories more robustly than prepending turns, while maintaining higher training efficiency than FlowQA.
- Experimental Setup: QuAC provides interactive information-seeking dialogs in which many questions contain coreferences to conversation history.The dataset includes over 11K training dialogs and 83K training questions, with validation sets of over 1K dialogs and 7K questions.
- Experimental Setup: The evaluation uses word-level F1, HEQ-Q, and HEQ-D to measure answer overlap and human-equivalence at question and dialog levels.Table 2 reports validation and leaderboard test scores; statistical significance is tested only for F1.
- Main Evaluation Results: Incorporating conversation history significantly boosts ConvQA performance, and BERT-based models outperform BiDAF++.The improvement from adding history is observed for both BiDAF++ and BERT-based models.
- Main Evaluation Results: HAE outperforms simply prepending history turns and achieves comparable performance to FlowQA with substantially higher training efficiency.The authors report that their models are at least 5 times faster than FlowQA in training; HAE is slightly slower than PH(Q)A but improves performance considerably.
- Impact of Conversation History: HAE with 5 or 6 history turns gives the best performance and handles more history than simply prepending turns.By contrast, PHQA and PHA degrade dramatically as the number of history turns grows, with PH(Q)A performing best at 1 or 2 turns.
- Impact of Conversation History: Long prepended questions shrink BERT’s passage input, helping explain why prepending-based methods degrade as history grows.The authors note that HAE is more length-efficient for incorporating long history under BERT’s limited maximum sequence length.
5 CONCLUSIONS AND FUTURE WORK
The paper introduces a general framework for handling conversation history in ConvQA and proposes history answer embedding for BERT-based models. Experiments and history-length analysis support the method’s effectiveness, while future work will combine it with learned history selection.
- Conclusions and Future Work: The framework separates conversation-history handling into history selection and history modeling.The proposed work focuses on history modeling and leaves learned history selection for future work.
- Conclusions and Future Work: The proposed history answer embedding method models conversation history in ConvQA and is evaluated through extensive experiments and history-length analysis.The analysis examines different amounts of conversation history under different settings.
- Conclusions and Future Work: Future work will integrate the history modeling method with a learned history selection strategy for ConvQA.