PaperlayerView exact revision on arXiv

Source-linked AI summary

Contextualized Query Embeddings for Conversational Search

Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin

arXiv:2104.08707v2cs.IR

TL;DR

Conversational search must resolve context-dependent queries, yet training data is limited and multi-stage systems can be slow. This paper integrates query reformulation into dense retrieval and trains with pseudo-relevance labels; it reports effectiveness on par with state-of-the-art pipelines at much lower latency, as well as strong open-domain conversational QA retrieval.

  • Problem

    Conversational queries can be ambiguous without dialogue context, and limited training data complicates training retrieval models for this setting.

    Source: §1 · p. 1

  • Method

    The model incorporates conversational query reformulation into a bi-encoder’s query encoder and uses pseudo-relevance labels for training.

    Source: §1 · p. 2, §8 · p. 9

  • Results

    CQE’s conversational-search effectiveness is on par with state-of-the-art multi-stage pipelines, with at least an order of magnitude lower query latency, and it serves as a strong open-domain conversational QA retriever.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    CQE embeddings can be transformed into text for failure analysis and hybrid sparse–dense retrieval, which can further improve retrieval effectiveness.

    Source: §1 · p. 2

    The work considers only historical queries as context, leaving system responses and multi-speaker conversations outside its evaluated setting.

    Source: §8 · p. 9

Abstract

from arXiv · show

This paper describes a compact and effective model for low-latency passage retrieval in conversational search based on learned dense representations. Prior to our work, the state-of-the-art approach uses a multi-stage pipeline comprising conversational query reformulation and information retrieval modules. Despite its effectiveness, such a pipeline often includes multiple neural models that require long inference times. In addition, independently optimizing each module ignores dependencies among them. To address these shortcomings, we propose to integrate conversational query reformulation directly into a dense retrieval model. To aid in this goal, we create a dataset with pseudo-relevance labels for conversational search to overcome the lack of training data and to explore different training strategies. We demonstrate that our model effectively rewrites conversational queries as dense representations in conversational search and open-domain question answering datasets. Finally, after observing that our model learns to adjust the $L_2$ norm of query token embeddings, we leverage this property for hybrid retrieval and to support error analysis.

1 Introduction

Conversational queries can be ambiguous without dialogue context, while limited training data and slow multi-stage pipelines complicate conversational search. The paper integrates query reformulation into dense retrieval and reports competitive retrieval effectiveness with substantially lower latency.

1 Introduction

  • Conversational utterances can be ambiguous in isolation, and limited training data complicates learning to retrieve relevant passages from dialogue.The cited passages identify omission and coreference as common sources of ambiguity.Source: §1 · p. 1
  • Existing multi-stage pipelines can exceed 10 seconds per query on GPUs and assume reformulated queries are independent of downstream retrieval.The paper questions that independence assumption.Source: §1 · pp. 1–2
  • The proposed bi-encoder incorporates conversational query reformulation into its query encoder and uses pseudo-relevance labels to train dense representations.The dataset addresses limited end-to-end conversational-search training data.Source: §1 · p. 2
  • Contextualized query embeddings can be transformed into text for failure analysis and can support hybrid sparse and dense retrieval.The paper also reports that the hybrid approach can improve CQE retrieval effectiveness.Source: §1 · p. 2
  • CQE matches state-of-the-art multi-stage pipelines in conversational-search effectiveness with at least an order of magnitude lower query latency, and serves as a strong open-domain conversational QA retriever.The latency comparison excludes re-ranking.Source: §1 · p. 2
  • The paper reports zero-shot effectiveness on open-domain conversational question answering.Source: §1 · p. 2

2 Preliminaries

Conversational search retrieves relevant passages for each turn using the current query and prior conversational context. The preliminaries contrast a multi-stage decomposition into reformulation and retrieval with the challenges of that approach.

2 Preliminaries

  • Conversational search aims to retrieve relevant passages for each user query using the earlier turns of the conversation as context.Source: §2 · p. 2
  • The task can be represented by a relevance-scoring function that evaluates a contextualized query against a passage.The function is parameterized by θ.Source: §2 · p. 2
  • A multi-stage pipeline factorizes conversational search into information retrieval and conversational query reformulation components, which can be tuned with data from different stages.Source: §2 · p. 2
  • Common pipelines reformulate queries with a fine-tuned language model, then use BM25 retrieval and BERT re-ranking; the combined pretrained models can be slow.They also assume that human-rewritten queries are optimal, which may not hold for downstream retrieval.Source: §2 · p. 2

3 Our Approach

The approach incorporates conversational query reformulation into a bi-encoder, representing queries with contextualized token embeddings for retrieval. CQE can also expose token importance through embedding norms, enabling selected context terms to form decontextualized queries for sparse retrieval.

3.1 Bi-encoder Model

  • Bi-encoder retrieval scores a conversational query and passage by their embedding dot product, with passage embeddings precomputed and query texts encoded at search time.The model uses average pooling over token embeddings and trains with passages from the training batch.Source: §3.1 · p. 3

3.2 Contextualized Query Embeddings

  • CQE uses BERT’s contextualized token embeddings for the conversation context and current query, then average-pools query token embeddings into a vector for nearest-neighbor retrieval.The resulting vector is used to search for top-k passages with Faiss.Source: §3.2 · p. 3

3.3 Interpreting CQE

  • CQE can adjust token embedding L2 norms, which act as term importance: norms increase for relevant query–passage pairs and decrease otherwise.In the example, “neolithic” and “revolution” increase in normalized norm after fine-tuning, while “start” and “end” decrease.Source: §3.3 · p. 4
  • Selecting context tokens whose embedding L2 norms meet a threshold and appending them to the user query yields a decontextualized bag-of-words query for sparse retrieval.The paper also uses these decontextualized queries for failure analysis.Source: Figure 2 · p. 3, §3.3 · p. 4

4 Training Data and Strategies

The paper builds weakly supervised conversational-search training data by pairing human-rewritten queries with pseudo-labeled passages, then explores strategies for fine-tuning CQE.

Weakly supervised training data

  • CANARD queries are paired with CAsT passages labeled using BM25 retrieval and ColBERT re-ranking, treating the top three passages as relevant and the rest as non-relevant.The training data combines CANARD, ColBERT, and the TREC CAsT passage collection.Source: §4 · p. 4

Bi-encoder warm-up

  • The bi-encoder is first warmed up on MS MARCO before CQE fine-tuning.The warm-up follows prior conversational-search work and uses Lin et al. (2020)’s training procedure.Source: §4 · pp. 4–5

CQE fine-tuning

  • CQE fine-tuning freezes the passage encoder and trains only the query encoder on conversational-query, positive-passage, and negative-passage triplets.Positive passages come from ColBERT labels; negatives are sampled from BM25’s top 200 passages.Source: §4 · p. 5

Fine-tuning strategies

  • The training strategies test ColBERT-ranked hard negatives and soft-label distillation to address limitations of ordinary negative sampling and one-hot pseudo-labels.Soft labels use ColBERT logits and minimize KL divergence over in-batch query–passage pairs.Source: §4 · p. 5

5 Experimental Setup

Experiments evaluate CQE on CAsT retrieval and ORConvQA transfer, comparing it with query-reformulation and retrieval baselines under shared retrieval and evaluation settings.

Datasets

  • Experiments use CAsT19 and CAsT20 retrieval datasets, with ORConvQA included to test zero-shot generalization.The CAsT corpus contains 38 million passages; ORConvQA evaluation is detailed separately.Source: §5 · p. 5

Query reformulation baselines

  • CQE is compared with three query-reformulation systems and human queries, with reformulated queries encoded by the same pretrained bi-encoder used for stand-alone queries.The compared systems are Few-Shot Rewriter, QuReTeC, and NTR (T5).Source: §5 · p. 5
  • CQE-sparse applies CQE’s decontextualized text to sparse retrieval, using an L2 threshold of 10.5 tuned on CAsT19 training data.CQE and the other methods share passage embeddings because the passage encoder is frozen.Source: §5 · p. 5

Model details and evaluation

  • The main CQE experiments use BERT-base, combine hard-negative mining with soft labeling, and report retrieval effectiveness, latency, model size, and significance tests.Latency is measured on a single 12GB NVIDIA Tesla K80 GPU; retrieval metrics include nDCG and recall at 1000 and nDCG@3.Source: §5 · p. 6

6 Results

CQE offers competitive retrieval with lower latency than multi-stage pipelines, performs strongly in zero-shot ORConvQA transfer, and benefits from combining hard negatives with soft labels.

6.1 Results on CAsT

  • CQE beats other query-reformulation models in dense retrieval, while NTR (T5) has comparable top-ranking accuracy; CQE-hybrid significantly improves over CQE dense-only retrieval.Unlike CQE, the other reformulation modules are built independently, increasing combined memory and latency; NTR grows from 220M to 330M parameters.Source: §6.1 · p. 6
  • Single-stage CQE competes with BERT-base multi-stage pipelines at much lower latency, except for MVR, while BERT-large re-ranking improves effectiveness but is slower.MVR fuses three re-ranked lists from separate neural query-reformulation models.Source: Table 4 · p. 7, §6.1 · p. 7
  • Case studies show CQE can outperform human reformulations by retaining contextual terms or adding useful terms, but related additions can also hurt retrieval.The examples motivate better negative sampling to select key terms under challenging contexts.Source: §6.1 · p. 7, Figure 4 · p. 7

6.2 Zero-shot Transfer to ORConvQA

  • On ORConvQA, CQE beats the retrieval baselines and its retrieval gains benefit answer-span F1; hybrid retrieval further improves retrieval and the downstream task.CQE is fine-tuned only on weakly supervised data, whereas the baselines are fine-tuned on ORConvQA with answer-containing passages as positives.Source: §6.2 · p. 8, Table 5 · p. 8

6.3 Fine-Tuning Ablation

  • Combining hard-negative sampling with soft labeling is the best tested fine-tuning strategy; soft labels bring substantial gains, while hard negatives alone do not directly improve effectiveness.The comparison also reports a modest gain from adding hard negatives when soft labels are used.Source: Table 6 · p. 8, §6.3 · p. 8

7 Related Work

Related work spans conversational query reformulation and conversational question answering; this paper positions CQE as an end-to-end alternative to multi-stage retrieval pipelines.

Conversational search

  • Conversational search includes both multi-turn clarification for a single item and searching for multiple topic-related items; this work addresses the latter scenario.The paper’s example is finding destinations, hotels, and transportation while planning a vacation.Source: §7 · p. 8

Query reformulation

  • Prior CAsT query-reformulation work includes rule-based expansion and language models trained to mimic human rewrites.Some prior systems fuse ranked lists from different reformulators but remain multi-stage pipelines.Source: §7 · pp. 8–9

Conversational question answering

  • Prior conversational question-answering research often targets answer-span extraction, while open-domain ConvQA uses a retriever-reader pipeline.CQE is presented as a weakly supervised retriever that also supports dense–sparse hybrid retrieval.Source: §7 · p. 9

8 Conclusions

The conclusion presents CQE as a way to integrate conversational query reformulation and dense retrieval, with effectiveness comparable to multi-stage pipelines at lower query latency. It also identifies limits in the context types and reranker settings considered.

8 Conclusions

  • The work integrates conversational query reformulation and dense passage retrieval in a bi-encoder framework, using pseudo-relevance labels to address limited training data.Source: §8 · p. 9
  • CQE matches state-of-the-art multi-stage pipelines with much lower query latency and serves as a strong dense retriever for open-domain conversational question answering.Source: §8 · p. 9
  • The study uses only historical queries as context and leaves cross-encoder rerankers and contexts such as system responses or multiple speakers for future work.Source: §8 · p. 9

A Dense–Sparse Hybrid Settings

The hybrid setting combines sparse and dense passage retrieval scores over their retrieved candidates. It approximates a linear score combination and tunes its parameters on CAsT19 training data.

A Dense–Sparse Hybrid Settings

B Model Fusion Study

The fusion study combines ranked lists from CQE and other query formulations. CQE fuses better with the tested CQR models, including in sparse retrieval, and beats human queries in dense retrieval fusion.

B Model Fusion Study

Related papers

Loading 2104.08707v2…