Source-linked AI summary
Question Rewriting for Conversational Question Answering
Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, Raviteja Anantha
TL;DR
Conversational QA must interpret follow-up questions whose meaning depends on prior dialogue, while existing approaches leave contextual and end-to-end performance gaps. The paper decomposes the task into question rewriting and question answering, using QR to produce standalone questions for existing QA models. The resulting system achieves state-of-the-art retrieval on TREC CAsT and improves answer extraction on QuAC, with remaining end-to-end gaps attributed mostly to QA errors.
Problem
Conversational QA requires resolving anaphora, ellipsis, and other contextual dependencies in follow-up questions, while existing approaches leave room for improvement.
Method
The paper adds a question-rewriting component that converts ambiguous follow-up questions into unambiguous standalone questions before existing QA models process them.
Results
The approach sets new state-of-the-art results on TREC CAsT passage retrieval and improves answer span extraction on QuAC with the same QR model.
Takeaways & Limitations
QR performance is sufficiently high on CANARD and TREC CAsT, while QA performance lags even with human-rewritten questions, pointing to standard QA models as the major improvement target.
Takeaways & Limitations
The error attribution analysis assumes that humans always produce correct question rewrites.
Abstract
from arXiv · showhide
Conversational question answering (QA) requires the ability to correctly interpret a question in the context of previous conversation turns. We address the conversational QA task by decomposing it into question rewriting and question answering subtasks. The question rewriting (QR) subtask is specifically designed to reformulate ambiguous questions, which depend on the conversational context, into unambiguous questions that can be correctly interpreted outside of the conversational context. We introduce a conversational QA architecture that sets the new state of the art on the TREC CAsT 2019 passage retrieval dataset. Moreover, we show that the same QR model improves QA performance on the QuAC dataset with respect to answer span extraction, which is the next step in QA after passage retrieval. Our evaluation results indicate that the QR model we proposed achieves near human-level performance on both datasets and the gap in performance on the end-to-end conversational QA task is attributed mostly to the errors in QA.
1 INTRODUCTION
The paper frames conversational QA as requiring contextual interpretation of follow-up questions and proposes question rewriting (QR) to make them standalone. The approach improves conversational QA while offering traceability, reuse of existing QA systems, and modular communication across heterogeneous components.
- Motivation: Follow-up questions can involve anaphora and ellipsis, requiring mechanisms that resolve contextual dependencies beyond existing co-reference tools.The paper notes that previous approaches incorporating conversation history still leave substantial room for improvement, especially beyond a single input document.
- Approach: QR rewrites ambiguous follow-up questions into unambiguous questions that existing QA models can process outside the conversation context.The QR component is added before the QA component in the proposed architecture.
- Practical advantages: QR provides traceability by exposing the question answered by the QA model, enabling separation of interpretation errors from QA errors.The paper uses this property in its error analysis to investigate error sources and their correlation.
- Practical advantages: QR supports reuse by reducing conversational QA to standard QA and allowing existing models and datasets to be leveraged across alternative QA architectures.The authors state that new non-conversational QA models can be ported into conversational settings using QR.
- Practical advantages: QR can communicate concise, relevant questions between heterogeneous systems and help determine which distributed system should answer them.This avoids sending the entire conversation to a third-party API.
- Results: The approach sets new state-of-the-art passage-retrieval results on TREC CAsT and improves answer extraction on QuAC using the same QR model.The work evaluates the approach in both retrieval and extractive settings.
2 RELATED WORK
Related work extends QA with conversation history or rewriting, but retrieval settings require concise queries and prior rewriting work did not fully assess end-to-end impact. This paper addresses that gap across retrieval and extractive QA.
- Conversational QA: Conversational QA introduces dependencies between questions and previous dialogue turns, and appending conversation history can fall short for retrieval QA.Retrieval systems require concise queries for candidate selection, while co-reference models may not resolve missing context in follow-up questions.
- Question rewriting: Query rewriting has been applied to data integration, query optimization, sponsored-search expansion, and conversation-conditioned dialogue systems.These applications establish rewriting as a technique used across several task settings.
- Question rewriting: Earlier CANARD work released human question rewrites from QuAC but evaluated rewriting quality without measuring its effect on end-to-end conversational QA.The present study extends evaluation to passage retrieval.
- Concurrent work: Concurrent approaches to conversational QA included heuristic rewriting and CANARD-trained term-selection models evaluated on TREC CAsT.The heuristic approach used part-of-speech tags, dependency parses, and co-reference resolution.
3 QUESTION REWRITING TASK
Question rewriting converts context-dependent follow-up questions into semantically equivalent standalone questions before QA processing. The rewritten question preserves the correct answer under the conversation context.
- Question rewriting task: QR translates each ambiguous follow-up question into a semantically equivalent, unambiguous question that can be processed independently.For example, “What is its GDP?” becomes “What is the GDP of Xi’an?”
- Question rewriting task: Formally, given conversation context C and potentially implicit question Q, a QR model generates explicit question Q′ with the same correct answer A.The paper presents QR as a component that can be combined with non-conversational QA models.
4 APPROACH
The approach separates conversational question interpretation from question answering by rewriting context-dependent questions into explicit questions, then applying standard retrieval or extractive QA models.
- Overall Architecture: The architecture uses a question rewriting component to convert conversational questions into explicit questions before standard QA processing.The same QR framework supports both passage retrieval and answer extraction architectures.
- Question Rewriting Model: The QR model encodes the question and previous conversation turns with a unidirectional Transformer Decoder and generates rewrite tokens autoregressively.Experiments use five previous turns separated by a special [SEP] token, with teacher forcing during training.
- Question Rewriting Model: The rewriting model combines intermediate token distributions into a final distribution using learned weighting coefficients.Intermediate distributions are produced from the decoder’s last hidden state, while weighting uses input embeddings and the first attention head output.
- Retrieval QA Model: Retrieval QA first selects candidate passages with BM25 and then re-ranks question-passage pairs using a Transformer-based binary relevance model.The re-ranker predicts whether each passage answers the rewritten question and outputs a relevance score in [0, 1].
- Extractive QA Model: Extractive QA concatenates the rewritten question and passage, encodes them with BERT, and predicts the answer span’s start and end positions.At inference, all valid spans are scored by adding start and end probabilities, and the highest-scoring span is returned; a special [CLS] token represents No Answer.
5 EXPERIMENTAL SETUP
The experiments evaluate one QR model across retrieval and extractive conversational QA, using established datasets, metrics, baselines, alternative rewriting models, and non-conversational pre-training.
- Experimental Goals: The evaluation examines performance, reuse, and traceability when introducing question rewriting into conversational QA.The research questions compare competitive systems, assess non-conversational pre-training, and measure component-level error contributions.
- Datasets: The study evaluates CANARD-derived QuAC extractive QA and TREC CAsT retrieval QA, training QR only on CANARD and testing the same model on both datasets.TREC CAsT contains 173 judged questions across 20 dialogues, with passages drawn from MS MARCO and TREC CAR.
- Metrics: Retrieval quality is measured with MAP, MRR, NDCG@3, and P@1, while extractive QA uses F1, Exact Match, and No Answer accuracy.Retrieval considers the top 1,000 documents per query with relevance-grade cutoff 2.
- Metrics: Question rewriting quality is evaluated with ROUGE-1 recall and Universal Sentence Encoder similarity, whose correlations with human judgments are 0.69 and 0.71.ROUGE-1 recall is adopted for the experiments because it correlates with human rewriting judgments.
- Baselines and Models: Baselines include original questions, context-augmented questions, human rewrites, coreference resolution, PointerGenerator, CopyTransformer, and Transformer++.Human rewrites provide an upper-bound estimate and support downstream QA ablations; extractive QA also reuses MultiQA pre-training before CANARD fine-tuning.
6 RESULTS
The question-rewriting approach consistently outperforms dialogue-history baselines in retrieval and extractive QA, while error analysis attributes most failures to QA rather than QR. Pre-training improves extractive QA, especially with rewritten questions, and QR performance closely tracks end-to-end QA performance.
- Performance: Question rewriting consistently outperforms baselines using previous dialogue turns in both retrieval and extractive QA, with further improvement over recent TREC CAsT results.The comparison covers Tables 3–5 and the latest reported TREC CAsT results.
- QR evaluation: QR performance is close to manual rewriting, and its metric can predict which model performs consistently better across both QA tasks.The precision-recall curve evaluates Anserini + BERT on the TREC CAsT test set.
- Retrieval QA: BERT passage re-ranking produces almost a two-fold increase in MAP, while keyword-based baselines outperform co-reference and PointerGenerator models on all three metrics.Using more keyword turns increases MRR and NDCG@3 but slightly decreases MAP.
- Extractive QA: The extractive QA models achieve 71% accuracy on human rewrites when distinguishing passages without answers, and dialogue-history baselines tend to answer more questions as unanswerable.Controlling for this conservative behavior, the QR model performs better by answering questions that have answers.
- Reuse: Non-conversational MultiQA pre-training improves all extractive QA models, with human-rewritten questions gaining 7% in EM and 6% in F1.QR-based models require no additional fine-tuning because their QA input remains non-conversational; a small portion of training data adapts them to CANARD.
- Traceability: The error analysis estimates that QA causes 29% of retrieval errors and 55% of extractive errors, compared with 11% and 5% attributed to QR, respectively.These estimates are rough because cases failing both QA and QR cannot be automatically distinguished.
7 CONCLUSION
The evaluation shows that question rewriting effectively extends standard QA approaches to conversational settings and achieves state-of-the-art retrieval performance. The analysis indicates that future gains should primarily come from improving QA models, while joint retrieval and additional context modeling remain future directions.
- Conclusion: Question rewriting effectively extends standard QA approaches to conversational QA, achieving state-of-the-art results on TREC CAsT and superior answer-span extraction on CANARD/QuAC.The conclusion treats QR as challenging but promising for conversational QA.
- Conclusion: The QR metric is a reliable indicator for selecting models with better end-to-end QA performance, potentially avoiding more costly end-to-end evaluation.Future work includes evaluating QR in a joint passage-retrieval and answer-span-extraction model.
- Conclusion: QR performance is sufficiently high on CANARD and TREC CAsT, while QA performance lags even with human-rewritten questions.The authors therefore identify standard QA models as the main source of potential improvement.
- Future work: The generic QR-QA architecture could incorporate user models or multimodal environmental context as auxiliary inputs.The authors present evaluation with such additional context as a promising future direction.