Source-linked AI summary
Open-Retrieval Conversational Question Answering
Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, Mohit Iyyer
TL;DR
Existing conversational QA settings often assume a given passage or candidate set, neglecting retrieval from large collections. This paper introduces ORConvQA with the OR-QuAC dataset and an end-to-end Transformer system, finding that learnable retrieval is crucial and that modeling conversation history across components substantially improves performance.
Problem
Existing conversational QA settings select or extract answers from given candidates or passages, neglecting retrieval from large collections in conversational search.
Method
The paper introduces ORConvQA, constructs OR-QuAC by combining QuAC, CANARD, and Wikipedia, and builds an end-to-end Transformer system with a retriever, reranker, and reader.
Results
A learnable retriever is crucial for ORConvQA, while enabling history modeling in all system components yields substantial improvement and the reranker contributes a regularization effect.
Takeaways & Limitations
ORConvQA evaluation supports treating retrieval and conversation history as central components of conversational question answering.
Takeaways & Limitations
Retriever pretraining uses CANARD rewrites to train one uniform retriever across history window sizes, rather than separately training for each window size.
Abstract
from arXiv · showhide
Conversational search is one of the ultimate goals of information retrieval. Recent research approaches conversational search by simplified settings of response ranking and conversational question answering, where an answer is either selected from a given candidate set or extracted from a given passage. These simplifications neglect the fundamental role of retrieval in conversational search. To address this limitation, we introduce an open-retrieval conversational question answering (ORConvQA) setting, where we learn to retrieve evidence from a large collection before extracting answers, as a further step towards building functional conversational search systems. We create a dataset, OR-QuAC, to facilitate research on ORConvQA. We build an end-to-end system for ORConvQA, featuring a retriever, a reranker, and a reader that are all based on Transformers. Our extensive experiments on OR-QuAC demonstrate that a learnable retriever is crucial for ORConvQA. We further show that our system can make a substantial improvement when we enable history modeling in all system components. Moreover, we show that the reranker component contributes to the model performance by providing a regularization effect. Finally, further in-depth analyses are performed to provide new insights into ORConvQA.
1 INTRODUCTION
ORConvQA addresses a limitation of conversational QA by requiring systems to retrieve evidence from a large collection before extracting answers. The paper introduces OR-QuAC and an end-to-end Transformer system, showing benefits from learnable retrieval and history modeling.
- Existing conversational QA settings select or extract answers from provided candidates or passages, neglecting retrieval in conversational search.
- ORConvQA requires retrieving evidence from a large collection before extracting answers.
- OR-QuAC combines information-seeking QuAC dialogs, CANARD question rewrites, and over 11 million Wikipedia passages.
- The end-to-end system contains Transformer-based retriever, reranker, and reader components, with history questions concatenated to the current question.
- A system without history information performs comparably to conversational BERTserini, while enabling history modeling across components yields substantial improvement.
- The experiments identify a learnable retriever as crucial and show that the reranker contributes through a regularization effect.
2 RELATED WORK
Related work spans question answering, open-domain QA, conversational QA, and conversational search. The paper distinguishes ORConvQA by restoring retrieval and modeling history for each dialog question.
- Question Answering: Traditional QA and open-domain QA use retrieval to find relevant passages before answer extraction, whereas many newer approaches emphasize extraction or generation.
- Conversational Question Answering: Existing conversational QA largely focuses on response ranking or conversational machine comprehension, so retrieval is often neglected.
- Conversational Question Answering: Open-retrieval matters because answers from questions in one dialog may come from different passages, requiring retrieval for each question.
- Conversational Question Answering: Conversational QA must address history modeling in both retrievers and readers, but suitable datasets for ORConvQA were unavailable.
- Conversational Search: Conversational search research also includes recommendation, product search, intent prediction, and question retrieval alongside conversational QA.
3 THE OR-QUAC DATASET
OR-QuAC adapts QuAC’s genuine information-seeking dialogs to open retrieval by using CANARD rewrites and a large Wikipedia passage collection. Its construction also has a synthetic-data limitation.
- Dataset construction: OR-QuAC aggregates QuAC, CANARD, and Wikipedia to create an open-retrieval conversational QA dataset.
- Self-Contained Information-seeking Dialogs: QuAC provides human information-seeking dialogs in which seekers know only an article title and summary while asking about hidden Wikipedia text.
- Self-Contained Information-seeking Dialogs: Incomplete initial questions make some QuAC dialogs difficult to interpret for open retrieval.
- Self-Contained Information-seeking Dialogs: The dataset replaces only initial QuAC questions with context-independent CANARD rewrites, making dialogs self-contained while preserving later history dependencies.
- Collection: The Wikipedia collection is cleaned, split into chunks of at most 384 wordpieces, and expanded to over 11 million retrieval passages.
- Collection: Because the dataset is synthetic, answers within a dialog are distributed in the same text section, unlike potentially dispersed real-world answers.
4 AN END-TO-END ORCONVQA SYSTEM
The end-to-end ORConvQA system retrieves passages, reranks them, and extracts answer spans while modeling conversational history across its components.
- The system contains a learnable passage retriever, reranker, and reader, with the reranker and reader based on a shared encoder.
- 4.3 Passage Retriever: The dual-encoder retriever separately encodes history-enriched questions and collection passages, then scores them using their representations.The passage encoder is pretrained and frozen for offline encoding of the collection.
- 4.3 Passage Retriever: The retriever uses a history window and prepends the conversation’s initial question when it is not already included.The resulting input sequence contains the initial question, selected history questions, and the current question.
- 4.4 Passage Reader/Reranker: Given each retrieved passage, the shared BERT encoder represents the concatenated history, current question, and passage for reranking and answer extraction.The reader predicts answer spans from token-level start and end scores.
- 4.4 Passage Reader/Reranker: The reranker assigns passage scores from the sequence representation, while the reader scores candidate start and end tokens to produce answer spans.At inference, the system selects the answer span with the highest overall score across retrieved passages.
- 4.5 Training: Training has retriever pretraining followed by concurrent learning of the retriever question encoder, reranker, and reader.Retriever pretraining uses reformatted questions from CANARD to support a uniform retriever across history-window sizes, followed by question-encoder fine-tuning.
5 EXPERIMENTAL SETUPS
The experiments compare ORConvQA with established open-retrieval baselines and evaluate answer quality, retrieval quality, and human-level equivalence. They also specify Transformer-based implementation settings and report statistically significant comparisons in Table 3.
- Competing Methods: DrQA uses a TF-IDF retriever with an RNN reader, while BERTserini uses BM25 retrieval with a BERT reader.
- Competing Methods: The study compares paragraph-level BERTserini retrieval using the top 5 passages and an ORConvQA variant without history.The no-history variant sets the history window size to w = 0 while retaining the first dialog question for the retriever.
- Evaluation Metrics: Answer quality is measured with word-level F1 and human equivalence scores at question and dialog levels.HEQ-Q and HEQ-D measure whether system F1 meets or exceeds human F1 for questions and dialogs, respectively.
- Evaluation Metrics: Retrieval quality is measured with MRR for retrievers and rerankers and Recall for retrievers, using the top 5 passages.MRR evaluates the rank of the first positive passage, while Recall measures the fraction of relevant passages retrieved.
- Implementation Details: The system uses ALBERT retriever encoders, a BERT reranker-reader, half-precision training, and a 10% learning-rate warm-up.The implementation uses PyTorch and Hugging Face models; Table 3 marks statistically significant improvements over the strongest baseline with p < 0.05.
6 EVALUATION RESULTS
Experiments on OR-QuAC show that the full system benefits from learned retrieval, conversation history, and reranking, with performance depending on history-window and retriever-update settings.
- 6.1 Main Evaluation Results: The model without history performs on par with history-aware BERTserini on the test set, while its learned retriever achieves higher retrieval metrics.The authors attribute most overall performance gain to the learned retriever because the reader is similar to BERTserini.
- 6.1 Main Evaluation Results: The model with history significantly improves over the strongest baseline, with p < 0.05 under a Student’s paired t-test.The result supports the effectiveness of the model and the importance of incorporating conversation history.
- 6.1 Main Evaluation Results: The reranker consistently outperforms the retriever despite jointly modeling questions and passages at greater computational expense.The separate retriever models questions and passages independently, whereas reranking uses joint modeling.
- 6.2 Ablation Studies: Removing the reranker degrades overall performance because the reader can overfit before the retriever receives enough fine-tuning.The additional reranking objective provides a regularization effect for the reader/reranker encoder.
- 6.2 Ablation Studies: Replacing the learned retriever with TF-IDF causes a dramatic performance drop, further supporting the importance of learned retrieval for ORConvQA.This ablation directly contrasts the learned retriever with DrQA’s TF-IDF retriever.
- 6.3 Additional Analyses: Any history improves overall performance over no history, peaking at w = 6, while retrieval metrics generally increase as more history is incorporated.Retriever recall has a trimodal distribution, and its valleys align with those of the F1 curve.
7 CONCLUSIONS AND FUTURE WORK
The work introduces ORConvQA and the OR-QuAC dataset, and develops an end-to-end Transformer system for open-retrieval conversational QA. Experiments identify learnable retrieval, comprehensive history modeling, reranking, and initial-dialog questions as important, while future work targets retriever tuning, weak supervision, and improved history modeling.
- The paper introduces ORConvQA as a step toward conversational search and creates OR-QuAC to support research on the task.
- The proposed end-to-end system combines Transformer-based retrieval, reranking, and reading components.
- Experiments show that a learnable retriever is crucial, while modeling history in every system component substantially improves performance.
- The reranker contributes to performance through a regularization effect, and the initial question is essential for understanding the user’s information need.
- Future work will study weak supervision, downstream-task-tunable retrieval, and more effective history modeling methods.