PaperlayerView exact revision on arXiv

Source-linked AI summary

CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning

Zeqiu Wu, Yi Luan, Hannah Rashkin, David Reitter, Hannaneh Hajishirzi, Mari Ostendorf, Gaurav Singh Tomar

arXiv:2112.08558v3cs.CL

TL;DR

Conversational retrieval must interpret questions in dialogue context, and retraining established retrievers can be expensive or infeasible. CONQRR uses reinforcement learning to optimize contextual rewrites for off-the-shelf retrieval, outperforming existing QR models on QReCC and showing effectiveness without human rewrite supervision.

  • Problem

    Conversational passage retrieval must interpret each question within dialogue context, while retraining established retrievers can be expensive or infeasible.

    Source: §1 · p. 1

  • Method

    CONQRR uses reinforcement learning with a novel reward based on retrieval performance to train contextual query rewrites from weak retrieval supervision for off-the-shelf retrievers.

    Source: §1 · pp. 1–2

  • Results

    CONQRR outperforms existing QR models by over 12% for BM25 and 14% for a neural dual encoder, averaging over three retrieval metrics, and achieves state-of-the-art results on QReCC.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CONQRR achieves better retrieval results without query-rewriting supervision than strong fully supervised baselines and is robust to out-of-domain dialogues.

    Source: §1 · p. 2, §5 · p. 9

    When the downstream retriever can be fine-tuned, CONQRR (mix) underperforms dialogue context in both topic-concentrated and topic-shifted scenarios.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Compared to standard retrieval tasks, passage retrieval for conversational question answering (CQA) poses new challenges in understanding the current user question, as each question needs to be interpreted within the dialogue context. Moreover, it can be expensive to re-train well-established retrievers such as search engines that are originally developed for non-conversational queries. To facilitate their use, we develop a query rewriting model CONQRR that rewrites a conversational question in the context into a standalone question. It is trained with a novel reward function to directly optimize towards retrieval using reinforcement learning and can be adapted to any off-the-shelf retriever. CONQRR achieves state-of-the-art results on a recent open-domain CQA dataset containing conversations from three different sources, and is effective for two different off-the-shelf retrievers. Our extensive analysis also shows the robustness of CONQRR to out-of-domain dialogues as well as to zero query rewriting supervision.

1 Introduction

Conversational retrieval must interpret questions using dialogue context, while retraining established retrievers can be expensive or infeasible. CONQRR uses reinforcement learning to optimize rewrites for retrieval and reports gains over existing query-rewriting models.

1 Introduction

  • The model rewrites context-dependent conversational questions for black-box, off-the-shelf retrievers, avoiding the need to retrain them.Retraining established retrievers can be expensive or infeasible because of their complicated system designs.Source: §1 · p. 1
  • Human rewrites can omit dialogue information useful to retrieval, while prior supervised QR models are trained separately from downstream retrievers.Human rewrites are often instructed to be unambiguous to readers outside the dialogue, which may not align with retrieval optimization.Source: §1 · pp. 1–2
  • CONQRR uses reinforcement learning and a novel reward based on retrieval performance to train rewrites using weak retrieval supervision.The reward approximates retrieval performance on in-batch passages and does not assume a specific retriever design.Source: §1 · p. 2
  • CONQRR outperforms existing QR models by over 12% for BM25 and 14% for a neural dual encoder, averaging over three retrieval metrics.The gains appear across all three QReCC subsets, including one present only in the test set.Source: §1 · p. 2
  • CONQRR achieves state-of-the-art results on QReCC and provides better retrieval results without human rewrite supervision than strong fully supervised baselines.The analysis also reports robustness to out-of-domain dialogues, topic shifts, and long dialogue contexts.Source: §1 · p. 2

2 Related Work

The related work distinguishes open-domain CQA retrieval from document-based conversational QA and prior conversational search settings. It places CONQRR alongside retriever-specific and rewrite-based approaches, and contrasts its use of reinforcement learning with earlier work.

2.1 Conversational Question Answering

  • QReCC is an open-domain CQA dataset in which an agent retrieves relevant passages before generating an answer, unlike document-based conversational QA that does not require retrieval.The section describes CQA dialogue context as including both user and agent turns.Source: §2.1 · p. 2, §2.2 · p. 2

2.2 Conversational Retrieval

  • Prior conversational retrieval models often fine-tune a specific retriever, while QR approaches use human rewrites; CQA context includes agent turns as well as user questions.The paper contrasts these approaches with QR designed for off-the-shelf retrieval and reports limitations of human-rewrite supervision.Source: §2.2 · pp. 2–3

2.3 RL for Text Generation

  • Earlier RL retrieval work reformulates non-conversational queries using gold passage labels, whereas CONQRR rewrites conversational queries with weak retrieval supervision and an approximate metric.The cited earlier models use multiple retrieval rounds, while CONQRR rewrites from dialogue context.Source: §2.3 · p. 3
  • RL has also been applied to optimize text-generation metrics, and to rewrite non-conversational queries for question answering.The paper frames its conversational retrieval setting as distinct from these applications.Source: §2.3 · p. 3

3 Approach

The approach rewrites dialogue context into a query for a fixed, black-box retriever. It contrasts supervised T5 rewriting with CONQRR’s reinforcement-learning training optimized for retrieval.

3 Approach

  • The task rewrites dialogue context and the current question into a query for a fixed retriever that returns ranked passages but cannot be fine-tuned.Source: §3 · p. 3

3.1 T5QR

  • T5QR fine-tunes T5 to generate a human rewrite from dialogue utterances concatenated in reverse order, placing the current question first and truncating distant context first.Cross-entropy training maximizes the likelihood of the human rewrite, a self-contained version of the current question.Source: §3.1 · p. 3

3.2 CONQRR

  • Weak labels select the passage whose span has the highest token-overlap F1 with the following answer, searching BM25’s top 100 human-rewrite results.Source: §3.2 · p. 4
  • Sampled rewrites receive a self-critical reward equal to their approximate in-batch retrieval score minus greedy decoding’s score, with success defined by ranking the positive passage first.The score uses in-batch passages rather than retrieval over the full corpus.Source: §3.2 · p. 4
  • CONQRR can use pure RL or a tunable RL/CE mixture, and inference greedily rewrites dialogue context before passing the query to the retriever.Source: §3.2 · p. 5

3.3 Retriever Models

  • Experiments use two fixed general-domain retrievers: BM25 with Pyserini defaults (k1=0.82, b=0.68) and a T5-base dual encoder fine-tuned on MS MARCO.Source: §3.3 · p. 5

4 Experiment

The experiments evaluate CONQRR with BM25 and a dual-encoder retriever on QReCC, then analyze its retrieval performance across subsets, supervision levels, and dialogue conditions. CONQRR improves on supervised rewriting baselines, while results vary with loss choice and topic shift.

4 Experiment

  • QReCC contains 14k conversations and 80k question-answer pairs from three sources; experiments use BM25 and DE with multiple supervised and CONQRR variants.Evaluation uses MRR, Recall@10, and Recall@100; CONQRR is initialized from T5QR and trained with mixed or pure RL loss.Source: §4 · p. 5, §4.1 · p. 6

4.2 Quantitative Results

  • CONQRR (RL) outperforms Transformer++ by 18% on average across three metrics using BM25 on QReCC, establishing a new state of the art.T5QR exceeds Transformer++ by 5% in the same comparison.Source: Table 2 · p. 5, §4.2 · p. 6
  • On overall QReCC, CONQRR beats T5QR on all three metrics; MRR and Recall@10 gains are roughly 15% with RL and 9–14% with mixed loss for both retrievers.Recall@100 gains range from 4–12%; DE leads on MRR and Recall@10 across subsets, while BM25 leads on Recall@100.Source: Table 3 · p. 6, §4.2 · p. 6

4.3 Analysis

  • With DE, CONQRR significantly outperforms T5QR even with 0% or 1% of QR supervision; mixed and pure RL perform similarly when supervision is scarce.T5QR’s performance curve flattens after 1% supervision, while mixed-loss performance degrades at 100%, possibly because cross-entropy does not directly optimize retrieval.Source: §4.3 · p. 7
  • On topic-concentrated examples, dialogue context beats human rewrites by 22% with BM25 and 17% with DE; CONQRR (RL) also surpasses human rewrites on specified metrics.Topic-concentrated examples reuse a document from an earlier turn; this subset contains about 4.7k examples.Source: Table 4 · p. 7, §4.3 · p. 7
  • On topic-shifted examples, human rewrites beat dialogue context by 52% with BM25 and 61% with DE, while CONQRR (mix) beats dialogue context by 30% and 44%, respectively.This subset contains about 1.1k examples whose gold passage comes from a document not used in earlier turns.Source: Table 4 · p. 7, §4.3 · p. 7
  • For topic-shifted conversations, QR models and human rewrites show little performance drop or may improve with longer contexts, while dialogue-context robustness worsens.With topic-concentrated conversations, compared models are similarly robust to context length, with CONQRR (mix) slightly more robust than T5QR.Source: Figure 4 · p. 7, §4.3 · pp. 7–8
  • CONQRR rewrites are generally longer and overlap more with gold passages; sampled successes often add relevant context, while failures commonly omit context or misinterpret requests.With DE, CONQRR (RL) rewrites are more than twice as long as T5QR, CONQRR (mix), and human rewrites. In sampled wins, 70% added correct context; in sampled failures, 56% had less context and 34% added misinterpreted information.Source: §4.3 · p. 8, Table 6 · p. 8

5 Conclusion and Future Work

CONQRR uses reinforcement learning and a retrieval-oriented reward to train query rewriting for off-the-shelf retrievers, and achieves state-of-the-art retrieval on QReCC. The paper also reports effectiveness with no query-rewriting supervision and identifies future tasks and dialogue formats to explore.

5 Conclusion and Future Work

  • CONQRR uses reinforcement learning with a novel reward to optimize rewrites directly toward retrieval for off-the-shelf retrievers.The reward is based on approximate retrieval performance on in-batch passages and does not assume a specific retriever design.Source: §5 · p. 9
  • CONQRR achieves state-of-the-art retrieval on QReCC conversations from three sources and outperforms strong supervised baselines with zero query-rewriting supervision.The conclusion also reports effectiveness with off-the-shelf BM25 and a dual encoder.Source: §5 · p. 9
  • Future work could apply query rewriting to question answering, response generation, and sentence rewriting, and investigate dialogue with clarification questions.Source: §5 · p. 9

Limitations

The reported advantages and costs of query rewriting depend on the retrieval setting and conversation type. The paper notes additional training and annotation costs, and limits evaluation to standard alternating-question-and-answer dialogue.

Limitations

  • With a fixed off-the-shelf retriever, rewriting is useful for topic shifts, while dialogue context alone may work well when conversations stay on one topic.When the retriever can be fine-tuned, CONQRR (mix) underperforms dialogue context in both topic-concentrated and topic-shifted scenarios.Source: Limitations · p. 9
  • Combining dialogue context with a rewritten query may further improve overall retrieval performance.Source: Limitations · p. 9
  • CONQRR training takes 9 versus 2 hours for fine-tuning a similar-size dual encoder, while retriever fine-tuning requires about 24 hours for re-indexing.CONQRR uses autoregressive decoding during training; re-indexing is also memory-consuming.Source: Limitations · p. 9
  • CONQRR (RL) can work on overall QReCC without rewrite labels, but is less robust than CONQRR (mix) on out-of-domain and topic-shifted examples.The paper says additional human rewrites may be needed for robust performance in those scenarios.Source: Limitations · p. 9
  • Evaluation covers only the standard CQA format of alternating questions and answers, leaving other dialogue acts such as clarification questions for future investigation.Source: Limitations · p. 9

Ethical Considerations

The paper frames query rewriting as an intermediate aid for open-domain conversational passage retrieval, while noting risks in interpreting rewrites and retrieved content.

Ethical Considerations

A.1 Additional Implementation Details

The implementation uses T5-base models and specifies training, decoding, and retrieval settings, including a BM25 approximation for reward computation during training.

A.1 Additional Implementation Details

  • Training used batch size 64, 1k warm-up steps, 10k total steps, and learning rates e−3 for T5QR and e−4 for CONQRR.Adafactor with default parameters and linear decay after 10% of training steps were used; supervised models were selected by dev Rouge-1 F1 and RL models by dev in-batch gold-passage prediction accuracy.Source: §A.1 · p. 12
  • For BM25 experiments with the pure RL loss, CONQRR was initialized from a T5QR model trained on 10% of rewrite labels because initialization from 100% labels was unstable.The paper notes a similar stability observation in prior work.Source: §A.1 · p. 12
  • T5QR and CONQRR use T5-base with about 220M parameters; Transformer++ uses GPT2-medium with about 345M parameters.The dialogue-context and rewrite limits are 384 and 64 tokens, respectively; training used top-k sampling with k=20 and five sampled rewrites per example.Source: §A.1 · p. 12
  • BM25-light supplied training rewards, using T5 subword tokenization and no special operations such as stemming, while reported inference used Pyserini BM25.The 128-token query and 2000-token passage limits for BM25-light require truncation in fewer than 0.1% of cases.Source: §A.1 · p. 13
  • The dual encoder used a maximum query or passage length of 384 tokens, and increasing that limit further reduced performance.Average passage length was 378 tokens.Source: §A.1 · p. 13

A.2 Additional Data Details

The section describes QReCC’s three conversation subsets and the preparation steps used for the data and validation split.

A.2 Additional Data Details

  • QReCC has 63k QuAC-Conv, 16k NQ-Conv, and 748 TREC-Conv question-answer pairs, with TREC-Conv appearing only in the test set.For NQ-Conv, one randomly chosen NQ question starts each conversation and the remaining conversation is annotated.Source: §A.2 · p. 13
  • The authors use QReCC’s original task setup and randomly reserve 5% of the training set for validation.QReCC reuses questions from QuAC and TREC conversations and re-annotates answers; its original split is train/test.Source: §A.2 · p. 13
  • To address ambiguous first queries in some QuAC-Conv conversations, the authors replace all QReCC first user queries with their corresponding human rewrites.They follow Anantha et al. (2021) for this replacement.Source: §A.2 · p. 13

A.3 Additional Evaluation Details

The section explains how missing gold passage labels affect evaluation and how the updated script handles those examples.

A.3 Additional Evaluation Details

  • Missing gold passage labels do not affect CONQRR training because its weak labels are induced without using the provided labels.The original evaluation script assigns a score of 0 to examples without valid gold passage labels.Source: §A.3 · p. 13
  • The updated evaluation script removes examples without valid gold passage labels—roughly 50% of the evaluation set—leaving 6,396 QuAC-Conv, 1,442 NQ-Conv, and 371 TREC-Conv test instances.That is 8,209 test instances in total.Source: §A.3 · p. 13
  • The authors use the updated script for most experiments and the original script for Table 2 baseline comparison; the scripts differ only by a scaling factor and lead to the same model-comparison conclusions.Source: §A.3 · p. 13

A.4 Additional Analysis

The additional analyses compare retriever behavior, test whether rewrite length explains performance, and examine fine-tuning, label efficiency, and rewrite examples. They also identify cases where CONQRR rewrites rank gold passages worse than human rewrites.

Lower Recall@100 with DE

  • BM25 has better Recall@100 than the off-the-shelf DE retriever across all subsets, while DE consistently leads on MRR and Recall@10.The authors suggest DE may be more vulnerable to domain shift without fine-tuning and note that longer passages can reduce DE performance; QReCC passages average 378 words.Source: §A.4 · p. 13

Analysis of Longer Rewrites

  • CONQRR (mix) outperforms length-matched T5QR on all three QReCC evaluation metrics, even with a brevity penalty applied during T5QR decoding.Figure 5 compares T5QR with and without the penalty against CONQRR (mix), using DE as the retriever.Source: Figure 5 · p. 13, §A.4 · pp. 13–14

Additional Data Efficiency

  • Figure 6 plots QReCC Recall@100 against the percentage of QR labels used for training with DE; similar trends are reported for Recall@10 and BM25.Source: Figure 6 · p. 14, §A.4 · p. 14

Fine-tuned Retriever

  • Fine-tuning DE improves results for all tested inputs, and dialogue context outperforms CONQRR in topic-shifted cases; human rewrites still help with topic shifts.The tested inputs are dialogue context, CONQRR (mix) rewrites, and human rewrites.Source: §A.4 · p. 14, Table 7 · p. 14

Additional Rewrite Examples

  • Additional tables provide CONQRR (mix) rewrite examples and gold-passage ranks for DE, alongside rewrites trained toward BM25 and their BM25 rankings.Source: §A.4 · p. 14, Table 9 · p. 15

Error Analysis

  • Two error-analysis examples show CONQRR (mix) rewrites ranking the gold passage worse than human rewrites when they omit context or misinterpret the request.Table 8 also notes examples where alternative or less-contextual CONQRR rewrites lead to a lower gold-passage rank.Source: §A.4 · p. 14, Table 10 · p. 15, Table 11 · p. 15
Loading 2112.08558v3…