Source-linked AI summary
Few-Shot Conversational Dense Retrieval
Shi Yu, Zhenghao Liu, Chenyan Xiong, Tao Feng, Zhiyuan Liu
TL;DR
Conversational dense retrieval must handle context-dependent queries despite dense retrievers' substantial supervision needs and conversational search's long-tail data. ConvDR uses a teacher-student query encoder with inherited document representations, achieving strong few-shot and fully supervised results while improving efficiency and context handling. Its OR-QuAC gains require caution because that dataset contains synthetic artifacts that simplify retrieval.
Problem
Conversational dense retrieval needs contextualized query representations, but dense retrievers require substantial supervision and conversational search has complex, personalized, long-tail information needs.
Method
ConvDR uses dense dot-product retrieval with inherited ad hoc document encodings and trains a student conversational query encoder to mimic oracle rewrite embeddings from a dense-retrieval teacher.
Results
ConvDR achieves state-of-the-art first-stage retrieval on TREC CAsT and OR-QuAC, outperforming sparse query-rewriting systems and nearly doubling previously published QR-QuAC accuracy.
Takeaways & Limitations
ConvDR captures salient context while filtering unrelated prior turns, maintaining retrieval quality as conversations evolve and improving online latency by removing question generation.
Takeaways & Limitations
OR-QuAC gains should be interpreted cautiously because its synthetic artifacts place all relevant passages for a dialog on the first-round entity's Wikipedia page.
Abstract
from arXiv · showhide
Dense retrieval (DR) has the potential to resolve the query understanding challenge in conversational search by matching in the learned embedding space. However, this adaptation is challenging due to DR models' extra needs for supervision signals and the long-tail nature of conversational search. In this paper, we present a Conversational Dense Retrieval system, ConvDR, that learns contextualized embeddings for multi-turn conversational queries and retrieves documents solely using embedding dot products. In addition, we grant ConvDR few-shot ability using a teacher-student framework, where we employ an ad hoc dense retriever as the teacher, inherit its document encodings, and learn a student query encoder to mimic the teacher embeddings on oracle reformulated queries. Our experiments on TREC CAsT and OR-QuAC demonstrate ConvDR's effectiveness in both few-shot and fully-supervised settings. It outperforms previous systems that operate in the sparse word space, matches the retrieval accuracy of oracle query reformulations, and is also more efficient thanks to its simplicity. Our analyses reveal that the advantages of ConvDR come from its ability to capture informative context while ignoring the unrelated context in previous conversation rounds. This makes ConvDR more effective as conversations evolve while previous systems may get confused by the increased noise from previous turns. Our code is publicly available at https://github.com/thunlp/ConvDR.
1 INTRODUCTION
Conversational search must resolve context-dependent queries with limited supervision, motivating ConvDR's dense, teacher-student approach. ConvDR improves retrieval accuracy, efficiency, and robustness by learning contextualized embeddings that focus on informative prior turns.
- Conversational queries contain omissions, references, and ambiguities, making underlying information needs harder to understand than in ad hoc retrieval.
- Dense retrieval addresses query understanding and vocabulary mismatch by matching conversational-query embeddings directly with document embeddings.
- Dense retrieval requires substantial relevance supervision, while conversational search has complex, personalized, long-tail information needs and limited training signals.
- ConvDR trains a student conversational query encoder to mimic oracle query-rewrite embeddings from an ad hoc dense-retrieval teacher, combining knowledge distillation with ranking loss.
- 9% and 48% higher retrieval accuracy than a previous query-rewriting model on TREC CAsT, with accuracy exceeding manual reformulations on CAsT-19; knowledge distillation nearly doubles prior accuracy on QR-QuAC.
- ConvDR simplifies the pipeline and improves online latency by eliminating costly question generation while producing embeddings aligned with relevant document encodings.
- ConvDR captures informative context while ignoring unrelated prior turns, maintaining retrieval quality as conversations become longer and noisier.
2 RELATED WORK
Related work frames conversational retrieval as a query-side problem: concise, context-dependent questions require reformulation, while dense retrieval offers an alternative representation space. CAsT and OR-QuAC provide complementary evaluation settings, though OR-QuAC contains synthetic artifacts.
- Conversational search supports multi-turn interactions for complex information needs and has become increasingly relevant with smartphones and speakers.
- The CAsT benchmark evaluates retrieval of relevant passages for each turn in manually curated conversational voice-search dialogues.
- CAsT research identifies query understanding as the main challenge because concise conversational queries depend on context and create vocabulary mismatch.
- Query reformulation methods expand context terms or generate de-contextualized queries before applying sparse retrieval pipelines.
- Dense retrieval encodes queries and documents as embeddings and retrieves them in a dense representation space, with strong performance in ad hoc search and open-domain QA.
- OR-QuAC converts multi-turn QuAC questions and evidence passages into a conversational retrieval task, but its relevant passages typically come from one Wikipedia section.
3 METHODOLOGY
ConvDR replaces reformulation-based conversational retrieval with dense query-document matching and a teacher-student strategy that transfers ad hoc retrieval knowledge for few-shot learning.
- Conversational search: Conversational queries require context resolution because they are concise, under-specified, and dependent on previous turns.
- Reformulation-based retrieval: Standard systems reformulate conversational queries into fully grown ad hoc queries before applying BM25 retrieval and BERT reranking.Reformulation may use query expansion or generation models and can rely on manual oracle queries for supervision.
- Conversational Dense Retrieval: ConvDR maps conversational queries and documents into one embedding space and retrieves documents using the query-document dot product.It uses a BERT Siamese/Dual-Encoder architecture, with the current and previous queries concatenated for contextualized query encoding.
- Conversational Dense Retrieval: ConvDR can optimize retrieval-oriented representations with a negative log likelihood ranking loss instead of directly reformulating the query.The model applies BERT to the concatenation of all conversation queries and the document during reranking.
- Few-shot learning: Dense retrieval needs more relevance supervision, while conversational search often lacks sufficient labels for its contextualized queries.This supervision gap is especially important because ConvDR must learn context dependence across previous turns.
- Few-shot learning: A teacher-student framework distills oracle-query embeddings from an ANCE ad hoc retriever, inherits its document embeddings, and optionally combines distillation with ranking loss.The framework assumes oracle and conversational queries share information needs, while document information is shared across ad hoc and conversational search.
4 EXPERIMENTAL METHODOLOGIES
The experiments evaluate ConvDR across few-shot TREC CAsT and supervised OR-QuAC settings, using established retrieval, rewriting, and reranking baselines. Performance is measured with task-specific retrieval metrics, while additional comparisons examine statistical significance and online efficiency.
- Datasets: The evaluation uses TREC CAsT 2019 and 2020 for few-shot conversational search and OR-QuAC as a supervised benchmark.CAsT contains manually constructed multi-turn dialogues, whereas OR-QuAC is a synthetic conversational retrieval dataset with many labels.
- Metrics: CAsT is evaluated with MRR1, NDCG@3, and hole rate, while OR-QuAC uses Recall@5, MRR@5, and MAP@10.The CAsT evaluation treats relevance scale ≥2 as positive for MRR on CAsT-20.
- Baselines: Baselines combine BM25 or ANCE retrieval with raw, automatically rewritten, or manually rewritten queries and BERT reranking.Published systems include query-reformulation pipelines such as pgbert, h2oloo_RUN2, CFDA_CLIP_RUN7, RRF, and Auto-Rewriter.
- Training and implementation: ConvDR and BERT reranker variants use knowledge distillation, ranking loss, or their Multi-Task combination, alongside zero-shot variants trained only on ad hoc retrieval.The implementation also compares fixed ANCE document embeddings with fine-tuned ConvDR query encoders and uses RRF to combine ConvDR with the best BERT ranker.
- Efficiency and setup: The study compares ConvDR with sparse solutions for online inference efficiency, measuring ANCE retrieval on GPUs with FAISS.The experimental setup uses ANCE checkpoints as teachers, with separate checkpoint choices for CAsT and OR-QuAC.
5 EVALUATION RESULTS
ConvDR achieves strong first-stage and reranking performance across few-shot CAsT and supervised OR-QuAC settings, while its dense representations approximate manual rewrites and preserve relevant conversational context. Its gains also simplify retrieval by eliminating query rewriting, though OR-QuAC results require caution because synthetic artifacts may oversimplify the task.
- Overall Performance: ConvDR improves BM25-Query Rewriter by 68% and 113% on CAsT and performs on par with ANCE-Manual on CAsT-19.It also largely eliminates the automatic–manual gap on CAsT-20.
- Overall Performance: Solely using embedding dot products, ConvDR outperforms the best participating CAsT-19 system, CFDA_CLIP_RUN7.The comparison is against a system using sparse retrieval and neural information-retrieval techniques.
- Overall Performance: ConvDR→BERT transfers first-stage gains to reranking and outperforms query-rewriting baselines on CAsT-19, but reranking remains farther from manual performance.The authors identify reranking atop dense-retrieval candidates as requiring further research.
- Overall Performance: ConvDR nearly doubles OR-QuAC first-stage accuracy over ORConvQA reranking, while ConvDR→BERT surpasses ANCE-Manual→BERT-Manual.With many synthetic relevance labels, fully supervised ConvDR outperforms previous methods by large margins.
- Overall Performance: OR-QuAC gains should be interpreted cautiously because its synthetic data contains artifacts that oversimplify retrieval, including shared Wikipedia-page structure across a dialogue.The dataset nevertheless provides evidence about ConvDR with large-scale feedback signals.
- Learned Query Representations: ConvDR (KD) approximates ANCE-Manual embeddings, achieves the highest query–relevant-document similarity, and maintains accuracy across conversation turns.Unlike automatic rewrites, it remains close to the current query while balancing information from previous turns.
6 CONCLUSION
ConvDR performs conversational first-stage retrieval entirely in dense embedding space and uses oracle-query imitation for few-shot training. Experiments on TREC CAsT and OR-QuAC show state-of-the-art retrieval, better context handling, and improved online latency.
- Its few-shot strategy trains the query encoder to mimic manual oracle-query embeddings from a well-trained ad hoc dense retriever.
- Experiments on TREC CAsT and OR-QuAC demonstrate state-of-the-art first-stage retrieval performance.
- Analyses show that ConvDR captures salient context from previous turns while improving online latency.