CompCQR: Compositional Query Generation for Training-Free Conversational Search

1 Introduction

• Conversational utterances are often short, ambiguous, and dependent on prior dialogue, making them difficult to use directly for retrieval.CQR reformulates the current utterance into a stand-alone query grounded in dialogue history.Source: §1 · p. 1
• CompCQR generates many compositional rewrites with minimal LLM use, avoiding expensive training or excessive inference.It aggregates documents retrieved by the candidate queries into a ranked list, then uses LLM reasoning to select documents based on answerability and balance recall with precision.Source: §1 · p. 2
• CompCQR outperforms the previous best-performing baseline by up to 22.5% on dense retrievers and 14.7% on sparse retrievers across four conversational search benchmarks.It achieves competitive or superior performance over training-based query-rewriting baselines with a more LLM-efficient pipeline.Source: §1 · p. 2
• CompCQR performs well across multiple datasets, retrievers, and long conversational contexts.The introduction describes the approach as robust across diverse retrieval settings.Source: §1 · p. 2

2 Potential of Compositional Queries

• Five components—the original query and two LLM-generated query–answer pairs—are permuted to produce 325 candidate queries using two LLM calls.The components are the original query, a rewritten query–answer pair, and a follow-up query–answer pair.Source: §2 · p. 2

2.2 Effectiveness of Compositional Queries

• Compositional rewriting achieves the highest average and upper-bound MRR across 325 candidates with two LLM calls, versus 325 IID calls and 32.1× higher FLOPs for sequential generation.The comparison uses 100 randomly sampled TopiOCQA instances.Source: Figure 2 · p. 3, §2.2 · p. 3

2.1 Retrieval Coverage of Compositional Queries

• Reordering identical components yields top-document Jaccard overlaps of 0.72 at top-10 and 0.71 at top-100, corresponding to a 28% top-10 Jaccard distance.The passage attributes this retrieval diversity to retriever sensitivity to query order.Source: §2.1 · p. 3

3 Methodology

• The framework combines compositional query generation and document ranking for recall with LLM-based document selection for precision.Source: §3 · p. 3

3.1 Task Formulation

• At turn t, history consists of prior query-answer pairs; an LLM rewrites the current query using that history, and a retriever returns ranked top-k documents.Source: §3.1 · p. 3

3.2 Compositional Query Reformulation

• Two LLM prompts produce rewritten and follow-up query-answer pairs; with the original query, these five components yield ordered, nonrepeating combinations.Source: §3.2 · pp. 3–4

3.3 Document Sorting by Retrieval Scores

• RRF aggregates documents across candidates using their top-100 ranks, then breaks ties by candidate-query frequency, favoring documents ranked highly and retrieved frequently.Source: §3.3 · p. 4

3.4 Best Document Selection

• An LLM selects an answer-supporting document from the top-k prefix of the 100 RRF-ranked documents, moves it to rank one, and preserves the remaining order.Source: §3.4 · p. 5

4.1 Experimental Setup

• Evaluation covers QReCC, TopiOCQA, CAsT19, and CAsT20, using only test instances with gold annotations.Reported metrics are MRR, NDCG@3, Recall@10, and Recall@100.Source: §4.1 · p. 5
• Experiments use Llama-3.1-8B-Instruct and GPT-4.1-mini, alongside BM25 and ANCE retrieval.RRF settings are tuned on the validation set, with s=0 and top-K values of 10 for Llama and 40 for GPT-4.1-mini.Source: §4.1 · pp. 5–6

4.2 Main Results

• The CAsT-19 and CAsT-20 comparisons cover prior baselines under dense retrieval.The table distinguishes Llama-3.1-8B-Instruct results from GPT-4.1-mini results.Source: Table 3 · p. 6, §4.2 · p. 6
• Up to 22.5% (+11.3 MRR) improvement over training-based baselines is reported in some settings, without training labels.CompCQR also outperforms prior training-free LLM-based query-rewriting methods across all settings and prompting-based methods on CAsT-19 and CAsT-20.Source: §4.2 · p. 6
• CompCQR is evaluated on TopiOCQA and QReCC with ANCE dense and BM25 sparse retrievers.The method is compared with training-based, hybrid, and training-free LLM-based baselines.Source: §4.2 · p. 6, §4.1 · p. 6

5.1 Ablation Studies

• Removing LLM-based selection causes a 4.5-point drop, and both selection and RRF-based document aggregation contribute substantially to retrieval quality.Applying the same selection step to LLM4CS yields a much smaller improvement; full reranking of the top 10 documents also degrades performance versus top-1 selection.Source: §5.1 · p. 7

5.2 Aggregation Method under Scaling

• RRF consistently outperforms mean aggregation; scaling RRF from 5 to 325 candidates gains 3 NDCG@3 points, versus 1.4 for mean aggregation.At 325 candidates, adding LLM selection yields the largest gain over RRF aggregation alone: 5.0 NDCG@3 points.Source: §5.2 · p. 7

5.3 Computational Efficiency

• With 5 candidates, CompCQR reaches 40.9 MRR versus LLM4CS’s 40.5 with 325 candidates, while running approximately 1.9× faster.With 325 candidates, CompCQR reaches 42.3 MRR; retrieval and sorting account for 10.5% of total runtime, and the method requires three LLM calls per query.Source: §5.3 · p. 7, Table 6 · p. 7, Table 7 · p. 7

5.4 Analysis on Different Retrievers

• CompCQR improves over LLM4CS by 4.6 MRR points with GTR and 2.8 with Contriever, extending the gains beyond ANCE.These additional dense-retriever experiments use TopiOCQA and GPT-4.1-mini.Source: Table 8 · p. 7, §5.4 · p. 8

5.5 Long-conversation Performance

• The performance gap over LLM4CS widens in later TopiOCQA conversation turns, indicating greater robustness in long conversations.This comparison uses GPT-4.1-mini.Source: §5.5 · p. 8, Figure 4 · p. 8

Component contributions

• The rewritten answer ã has the highest Shapley value for both ANCE (16.2) and BM25 (14.54).Shapley values are scaled by 100 in the component analysis.Source: §6 · p. 8, Table 9 · p. 8

Follow-up components

• Removing the follow-up query and answer lowers overall MRR by 6.4 points.The comparison is between the full model and a variant without qf and af.Source: §6 · p. 8, Table 10 · p. 8

Retriever-specific preferences

• ANCE ranks rewritten query ˜q second in influence (ϕ = 9.98), while BM25 favors follow-up answer af slightly over ˜q (ϕ = 6.32 vs. ϕ = 6.27).The analysis suggests dense and sparse retrievers benefit from different supporting components and composition strategies.Source: §6 · p. 8

Component ordering

• For ANCE, performance is generally higher when rewritten answer ã appears earlier in the query sequence.Figure 5 reports MRR change from the mean for queries sharing the same component set, grouped by ã’s position.Source: §6 · pp. 8–9, Figure 5 · p. 9

Conversational retrieval approaches

• Conversational dense retrieval encodes dialogue context directly, while CQR reformulates the current query into a standalone form for existing retrieval models.Examples include contextualized dense embeddings, compact conversational retrievers, LLM-based augmentation, and conversational sparse retrieval.Source: §7 · p. 9

LLM-based CQR

• Recent CQR methods include LLM-generated rewrites for initialization in IterCQR [Jang et al., 2024] and direct prompting for multiple rewrites and hypothetical responses in LLM4CS [Mao et al., 2023b].LLM-aided informative query rewriting [Ye et al., 2023] uses a rewrite-then-edit framework.Source: §7 · p. 9
• ConvSearch-R1 [Zhu et al., 2025] explores training reformulation models through supervised fine-tuning and reinforcement learning.The paper positions its work as sharing the goal of leveraging LLMs for CQR while improving efficiency through compositional generation.Source: §7 · p. 9

• CompCQR outperforms training- and prompting-based methods without explicit training labels across dense and sparse retrievers.Source: §8 · p. 9
• Document-level RRF increasingly outperforms query-level aggregation as the number of query candidates grows.Source: §8 · p. 9
• In long conversations, CompCQR’s performance gap over existing query reformulation methods widens as the conversation progresses.Source: §8 · p. 9

• Evaluation is limited to English benchmarks, leaving transfer to other languages unclear.Source: Limitations · p. 9
• The study does not identify the most effective query-composition subset for a particular dataset or retriever.Source: Limitations · p. 9
• The framework still requires multiple retriever calls, while the study uses permutations without repetition and compact open-source or API-based LLMs.The authors suggest reducing retrieval calls, exploring repeated components or additional future queries, and testing larger open-source models.Source: Limitations · pp. 9–10

A.1 Implementation Details

• Experiments used A6000 GPUs, Faiss and Pyserini, and dense retrievers including ANCE, GTR, and Contriever; main results use a single selection run.For sparse retrieval, LLM4CS queries were concatenated because its embedding-averaging approach does not apply to BM25.Source: §A.1 · p. 13

A.2.1 RRF Softening Hyperparameter

• RRF softening values from 0 to 100 did not significantly affect MRR, and s = 0 achieved the best performance.Source: §A.2.1 · p. 13, Figure 6 · p. 13

A.2.2 LLM Document Selection

• As the candidate set grows, both models’ best-document accuracy decreases; GPT-4.1-mini MRR peaks at k = 40, while Llama-3.1-8B becomes unstable above 20 candidates.The selected settings are top 40 for GPT-4.1-mini and top 10 for Llama-3.1-8B.Source: §A.2.2 · pp. 13–14, Table 11 · p. 13
• Replacing GPT-4.1-mini with Llama-3.1-8B for selection drops MRR by 5.0 points, while using GPT-4.1-mini in the Llama-based setting raises it by 4.1 points.The analysis attributes final performance to both query-candidate quality and selection-model effectiveness.Source: Table 12 · p. 13, §A.2.2 · p. 14

A.3 Dataset

• The four test datasets are QReCC, TopiOCQA, CAsT-19, and CAsT-20; hyperparameter tuning uses 700 sampled TopiOCQA training turns.Source: §A.3 · p. 14, Table 13 · p. 14

B.1 Document Selection on Different Method

• On TopiOCQA, LLM-based document selection improves LLM4CS by up to 1.2 MRR points, while CompCQR benefits by up to 4.5 points.Source: §B.1 · p. 14

B.2 Multi-run Statistics for Best-document Selection

• Across three runs of top-10 selection, MRR is 42.2 ± 0.9 and nDCG is 41.3 ± 0.6.The authors describe performance as consistent across runs.Source: §B.2 · p. 14

B.3 Comparison with Re-ranking

• Selecting one top document performs better than reranking multiple documents, which the authors describe as a more complex task that introduces noise.Source: §B.3 · p. 14, Table 15 · p. 14

C.1 Number of Queries and Components

• On TopiOCQA with ANCE, upper-bound MRR improves as more queries are applied, with gains beginning to saturate around 250 queries.Source: §C.1 · p. 15, Figure 7 · p. 15

D.1 Dense Retrievers

• GTR and Contriever also show substantial top-10 document changes under component reordering, though their change rates are lower than ANCE’s.The analysis presents order sensitivity as a property of dense retrievers, not an ANCE-specific artifact.Source: §D.1 · p. 15, Table 16 · p. 15

D.2 Large Open-source Models

• With Qwen2.5-72B, CompCQR raises MRR from 43.4 to 47.4 (9.2% relative), and with Llama-3.3-70B from 45.3 to 48.6 (+7.3% relative).Both comparisons use TopiOCQA with the ANCE retriever.Source: Table 17 · p. 15, §D.2 · p. 15

E Topic Shift Performance

• Under topic shifts, GPT-4.1-mini maintains or improves performance, while LLaMA-3.1-8B shows a slight degradation.The authors connect the difference to model capacity and say interpreting conversational context matters for handling topic transitions.Source: Figure 8 · p. 15, E Topic Shift Performance · p. 15

F LLM Calls and FLOPs

• CompCQR achieves the best MRR with three LLM calls; without LLM selection, it surpasses LLM4CS (k=16) with two calls and fewer FLOPs.The authors attribute the gain to compositional generation rather than additional inference; the selection stage’s extra input-token cost comes from longer inputs, not repeated generation.Source: Table 18 · p. 16, F LLM Calls and FLOPs · p. 16

G.1 Mitigating Hard Negatives

• LLM selection chooses a more complete answer over a topically relevant but partial hard negative ranked first by RRF.The selected gold document was originally ranked third.Source: §G.1 · p. 16, Table 19 · p. 17

G.2 Failure Analysis

• In a failure case, auxiliary components drift from the rewritten query’s topic and dilute its signal, moving the gold passage from rank 1 to rank 8 under RRF.The authors caution that plausible but mismatched evidence can make component diversification hurt.Source: §G.2 · p. 16, Table 20 · p. 17

H Comparison with CDR

• CompCQR outperforms the second-best CDR method by 7.6 NDCG@3 points on TopiOCQA.The comparison uses the same dense retriever as the Conv-ANCE (2020) baseline.Source: H Comparison with CDR · p. 16, Table 21 · p. 17

J.1–J.4 Implementation Prompts

• The prompts generate rewrite and follow-up query-answer pairs, then analyze query context and passage relevance to select or rank documents.The full-reranking prompt ranks the top-k passages from most to least relevant.Source: §J.1 · p. 18, §J.2 · p. 18, §J.3 · p. 18

K Usage of GenAI

• AI-assisted coding supported data analysis and figure/table scripts; the authors reviewed and validated the code and results.Source: K Usage of GenAI · p. 19