PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Multi-Step Query Rewriting via Corpus Feedback for Conversational Search

João Coelho, Hong Wang, Jie Yuan, Zhuoer Wang, Samson Koelle, Wei Niu

arXiv:2610.10955v1cs.IR

TL;DR

Most conversational query rewriters choose a single rewrite before retrieval, without the chance to adapt it to returned corpus evidence. This paper trains an agent to rewrite and retrieve iteratively, then reports strong results across datasets and retrieval settings, with later pseudo-documents grounded in earlier passages.

  • Problem

    Most CQR methods issue a single rewrite before retrieval, and reformulating queries after observing retrieved evidence remains comparatively unexplored.

    Source: §1 · pp. 1–2

  • Method

    The agent iteratively applies typed rewriting operations and retrieves, conditioning later rewrites on returned passages, with supervised fine-tuning followed by retrieval-reward reinforcement learning and no human rewrite annotations.

    Source: §1 · p. 2

  • Results

    The agent achieves the strongest ranking precision among compared methods on TopiOCQA and QReCC, transfers zero-shot to TREC CAsT, and remains effective under dense retrievers.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    Retrieval-reward optimization alone yields an action order that resolves intent first and generates pseudo-documents grounded in passages retrieved earlier.

    Source: §4.3 · p. 10, §5 · p. 12

    The reward uses RIRS without step penalties or latency costs, so cost-aware early termination remains a proposed extension.

    Source: §5 · p. 12

Abstract

from arXiv · show

Conversational Query Rewriting (CQR) turns a context dependent user turn into a standalone query for a retriever, and most methods do this in a single step from the dialogue history before retrieving once. The rewrite is therefore fixed before any corpus evidence is available to correct its reference resolution or its vocabulary. We recast CQR as a sequential retrieval problem: an agent rewrites the current turn, retrieves, and conditions its next rewrite on the returned passages. The agent acts in a typed space of three rewriting operations, resolving conversational intent into a standalone query, generating lexical reformulations, or synthesizing pseudo-documents for document-to-document matching, together with a stop action that ends the episode. We train the policy with supervised fine-tuning followed by reinforcement learning against a single retrieval-quality reward, using no human rewrite annotations. Across TopiOCQA and QReCC, the agent outperforms several retrieval-aligned baselines, while remaining effective across retrieval backends and generalizing to the CAsT benchmarks without additional training. Further analysis shows that, through retrieval-reward optimization alone, the learned policy develops a behavior of grounding pseudo-documents in passages retrieved by earlier steps, substantially improving retrieval.

1 Introduction

Most conversational query rewriters decide before retrieval, leaving the rewrite unable to adapt to corpus evidence; how to refine conversational queries from retrieved passages remains comparatively unexplored. This work makes rewriting sequential and reports gains across several retrieval settings.

1 Introduction

  • Most CQR methods issue one rewrite before retrieval, so they cannot adapt it to the target corpus’s language and entities.Existing feedback methods typically use predefined mechanisms rather than learning how to incorporate retrieved information from retrieval outcomes.Source: §1 · pp. 1–2
  • How conversational queries should be reformulated after observing retrieved evidence remains comparatively unexplored in agentic search.Agentic search systems commonly alternate search and reasoning for end-to-end question answering, where retrieval is an intermediate step.Source: §1 · p. 2
  • The agent iteratively rewrites and retrieves using three typed operations—intent resolution, lexical reformulation, and pseudo-document generation—plus a stop action.It is trained with supervised fine-tuning followed by reinforcement learning, using retrieval quality alone and no human rewrite annotations.Source: §1 · p. 2
  • The multi-step agent improves over multiple baselines in in-domain, cross-backend, and zero-shot transfer settings, while grounding later reformulations in previously retrieved corpus evidence.The evaluation trains on TopiOCQA and QReCC with BM25, tests dense retrieval at inference, and transfers to TREC CAsT without further training.Source: §1 · p. 2

2 Related Work

Prior conversational rewriting methods use annotations, retrieval-aligned objectives, or iterative refinement, but generally do not condition inference-time rewrites on retrieved passages. This work learns reformulation choices and stopping from retrieval reward, with each later rewrite informed by earlier retrievals.

Conversational Query Rewriting

  • Earlier conversational rewriters used human rewrites or retrieval-aligned training, while dense retrievers encode dialogue history directly and must themselves be trained.Rewriters instead produce standalone queries usable by off-the-shelf retrievers.Source: §2 · p. 3
  • ConvSearch-R1 uses retrieval feedback during training but issues inference-time rewrites without corpus evidence, while ICR refines queries without observing retrieved passages.ICR alternates clarification questions and rewrites and fuses rankings from its intermediate rewrites.Source: §2 · p. 3

Pseudo-Relevance Feedback

  • Pseudo-relevance feedback uses top-ranked documents to refine queries, while generated pseudo-documents can diverge from the target corpus’s facts and vocabulary.Conditioning generation on retrieved passages is one response to that divergence.Source: §2 · p. 3
  • Conversational feedback methods use hand-designed extraction and merging rules, whereas this agent learns which reformulation to apply and when to stop from retrieval reward.Its pseudo-documents are grounded in passages retrieved at earlier steps rather than generated zero-shot.Source: §2 · p. 3

Reinforcement Learning for Search Agents

  • Conversational search agents such as ChatR1 and ConvAgent primarily optimize answer quality and require reference answers, unlike this policy’s final-ranking reward and typed reformulation actions.The policy requires passage-relevance labels, not answers or rewrite annotations.Source: §2 · p. 4

3 Methodology

The method treats each conversational turn as an episode of iterative rewriting and retrieval, with later decisions conditioned on earlier results. It trains a typed-action agent through supervised fine-tuning and retrieval-reward reinforcement learning.

3.1 Task Formulation

  • Each turn becomes a multi-step episode in which the agent issues intermediate rewrites, observes retrieval results, and commits to a final retrieval state.This episode is internal to resolving a turn and is distinct from the conversation’s user turns.Source: §3.1 · p. 4

3.2 Agent Design

  • The agent can resolve intent, rephrase lexically, generate a pseudo-document for document-to-document matching, or stop; subsequent actions can use passages retrieved by earlier rewrites.Intent resolution targets references and ellipsis, while rephrasing targets vocabulary mismatch; the action prompt imposes no order.Source: §3.2 · pp. 4–5

3.3 Training

  • Supervised fine-tuning distills format-valid trajectories from a larger model so the small base model can emit typed actions before retrieval-reward alignment.The SFT checkpoint establishes action format but is not yet aligned with the retriever.Source: §3.3 · p. 5
  • GRPO trains on groups of full agent trajectories, assigning group-normalized advantages to valid tokens while masking retrieved-passage tokens from the loss.The clipped objective uses per-token importance ratios between the current and behavior policies.Source: §3.3 · pp. 5–6
  • RIRS provides a dense reward from the final rewrite’s gold-passage rank: ranks 1–10 map to, ranks 11–100 to [0, 1], and lower ranks receive zero.Reward uses the retriever’s top-100 ranking, including when the agent observes only top-k passages; invalid action-format trajectories receive zero.Source: §3.3 · p. 6

4 Experiments

The experiments evaluate the agent in-domain, then analyze its trajectories, transfer, and inference cost. The full multi-step, multi-action agent leads the compared methods on TopiOCQA and QReCC, with further gains linked to sequential interaction and corpus feedback.

4.1 Experimental Setup

  • Experiments use QReCC and TopiOCQA for training and in-domain evaluation, with TREC CAsT reserved for zero-shot transfer; retrievers are not trained or adapted.The study compares trained rewriter baselines and system configurations that vary whether typed actions and iteration are allowed.Source: §4.1 · pp. 6–7

4.2 Results

  • The full multi-step, multi-action agent achieves the highest MRR and NDCG@3 among compared methods on TopiOCQA and QReCC, with gains concentrated at the top of the ranking.The comparison is over BM25; the table marks best results per column and significant improvements over the ConvSearch-R1 rerun.Source: §4.2 · p. 7, Table 1 · p. 8
  • Typed actions alone reduce performance, while the full agent is strongest when it can sequence them using retrieved evidence; iteration’s advantage emerges under reinforcement learning, not SFT.The single-step, single-action configuration matches the ConvSearch-R1 formulation, and QReCC shows the same overall pattern with smaller but significant gains.Source: §4.2 · pp. 7–8

4.3 Analysis and Ablations

  • Retrieval quality rises steeply through the first three inference steps and then flattens; the one-step agent scores below reinforcement-learned baselines.The inference-time truncation comparison uses the same 5-step policy without retraining, isolating the allowed interaction budget.Source: §4.3 · p. 9, Figure 1 · p. 9
  • RL yields a consistent action order: intent resolution opens every trajectory, pseudo-document generation dominates the last step, and average search length reaches 4.9 steps on TopiOCQA and 2.9 on QReCC.SFT rollouts average 1.7 search steps and only 60% begin with intent resolution; the reward scores final ranking without specifying action order or length.Source: Figure 2 · p. 9, §4.3 · p. 10
  • Pseudo-documents incorporate retrieved-passage vocabulary without simply copying passages; replacing the contextual passages with unrelated-query passages lowers TopiOCQA nDCG@3 from 40.6 to 25.3.Median 8-gram containment is 0, while mean unigram Jaccard is 0.137 against the agent’s context versus 0.014 for unrelated-query context.Source: §4.3 · p. 10

4.4 Generalization Across Domains and Retrieval Backends

  • Both trained policies transfer to TREC CAsT without further training, and the TopiOCQA-trained policy is stronger and exceeds ConvSearch-R1 on NDCG@3 in both years.Across retriever backends, dense retrievers improve every metric over BM25 on TopiOCQA, but neither surpasses BM25 on QReCC; overall performance depends on dataset-retriever fit.Source: §4.4 · pp. 10–11, Table 2 · p. 11, Table 3 · p. 11

4.5 Cost of Sequential Interaction

  • Sequential interaction raises cumulative generation cost as contexts expand: 93% of TopiOCQA trajectories issue a fifth rewrite, compared with 4% on QReCC.Individual actions are shorter than ConvSearch-R1’s approximately 400-token generation, but later interactions drive additional cost.Source: §4.5 · pp. 11–12, Figure 3 · p. 12

5 Conclusions and Future Work

The paper frames conversational query rewriting as sequential retrieval, with intermediate passages guiding successive rewrites. The policy learns an action order from retrieval-quality rewards, while cost-aware stopping and parallel reformulations remain future directions.

5 Conclusions and Future Work

  • Rewarded solely on retrieval quality, the policy resolves conversational intent, optionally refines wording, then generates pseudo-documents grounded in earlier retrieved passages.The formulation uses reinforcement learning without human rewrite annotations and achieves the strongest ranking precision among compared methods on TopiOCQA and QReCC, transfers zero-shot to TREC CAsT, and remains effective under dense retrievers.Source: §5 · p. 12
  • Adding step penalties or latency costs could support cost-aware policies that terminate early when conversational ambiguity is low.The paper used the RIRS reward directly to facilitate comparison with prior studies.Source: §5 · p. 12
  • Issuing multiple reformulations in parallel and rewarding their fused ranking could encourage complementary rewrites, broaden recall, and shorten the sequential loop.Source: §5 · p. 12

A Reproducibility Details

The appendix describes the agent’s training setup and prompt workflow: it processes the current query with dialogue history, selects one retrieval action at each step, and can stop when further rewriting is unlikely to help.

A Reproducibility Details

  • The agent receives the current query followed by dialogue-history snippets ordered from most recent to oldest, and rewrites only the current query using history for disambiguation.Source: A Reproducibility Details · p. 13
  • At each step, the agent assesses the remaining result gaps, chooses one action, and submits a single query for retrieval.Source: A Reproducibility Details · p. 13
  • The prompt defines intent queries as focused standalone rewrites and rephrase queries as alternative terminology or wording for the same information need.Source: A Reproducibility Details · p. 13
  • HyDE queries are hypothetical answer passages intended to add answer-side vocabulary for matching, while stop submits current results when they look relevant or more rewriting is unlikely to help.Source: A Reproducibility Details · p. 13
  • Prompt guidelines call for resolving coreferences, ellipsis, topic shifts, and gaps after retrieval; they also recommend adopting retrieved terminology and stopping early when results look relevant.Source: A Reproducibility Details · p. 13
Loading 2610.10955v1…