Source-linked AI summary
Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search
Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, Hongjin Qian
TL;DR
Conversational search needs robust interpretation of contextual intent because multi-turn queries can be linguistically incomplete and existing limited-data methods struggle with diverse sessions. LLM4CS uses multiple prompted rewrites and hypothetical responses, aggregates them into an intent representation, and achieves strong results across three CAsT benchmarks, outperforming established methods and human rewrites. The approach also incurs latency and lacks downstream retrieval awareness.
Problem
Conversational search must recover contextual intent from queries containing omissions and coreference, while diverse and long-tailed sessions challenge methods trained on limited data.
Method
LLM4CS prompts LLMs to generate multiple query rewrites and hypothetical responses, then aggregates them into an integrated representation of contextual search intent.
Results
LLM4CS shows remarkable performance in automatic and human evaluations on CAsT-19, CAsT-20, and CAsT-21, surpassing CQR and CDR baselines and human rewrites.
Takeaways & Limitations
Generating hypothetical responses and properly aggregating multiple generated results are crucial for improving conversational-search performance and robustness.
Takeaways & Limitations
The framework requires multiple LLM invocations, increasing retrieval time, and its generation process lacks awareness of downstream retrieval signals.
Abstract
from arXiv · showhide
Precisely understanding users' contextual search intent has been an important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on limited data still show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. Recently, large language models (LLMs) have demonstrated amazing capabilities for text generation and conversation understanding. In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLMs as a text-based search intent interpreter to help conversational search. Under this framework, we explore three prompting methods to generate multiple query rewrites and hypothetical responses, and propose to aggregate them into an integrated representation that can robustly represent the user's real contextual search intent. Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks, including CAsT-19, CAsT-20, and CAsT-21, demonstrate the remarkable performance of our simple LLM4CS framework compared with existing methods and even using human rewrites. Our findings provide important evidence to better understand and leverage LLMs for conversational search.
1 Introduction
Conversational search must recover users’ contextual intent from linguistically incomplete, increasingly diverse and long-tailed multi-turn sessions. LLM4CS addresses this challenge by generating and aggregating multiple rewrites and hypothetical responses, with evaluations showing performance improvements over established baselines and human rewrites.
- 1 Introduction: Conversational search is difficult because current-turn queries can contain omissions and coreference that obscure their intended meaning in multi-turn context.The challenge is especially important as conversations become longer and existing methods degrade on complex sessions.
- 1 Introduction: Existing CQR methods rewrite queries into standalone forms, whereas CDR methods encode conversational sessions and passages into latent representations for dense retrieval.CQR offers practical reuse of ad-hoc search models, while CDR can learn directly from session-passage relevance signals.
- 1 Introduction: Because conversational sessions are diverse and long-tailed, methods trained on limited data remain unsatisfactory, particularly for complex or longer conversations.Human rewrites have shown advantages on sessions with complex response dependency, while public CAsT-21 results report degradation as conversations lengthen.
- 1 Introduction: LLM4CS investigates whether LLMs’ text-generation and conversation-understanding abilities can better capture contextual search intent for conversational search.Its design is motivated by prior successes using LLMs for query generation, query expansion, and document prediction in information retrieval.
- 1 Introduction: LLM4CS generates multiple query rewrites and hypothetical responses, then aggregates them into an integrated representation of contextual search intent.The framework uses three tailored prompting methods to leverage LLMs as text-based search intent interpreters.
- 1 Introduction: The best LLM4CS method improves search performance over state-of-the-art CQR and CDR baselines and surpasses human rewrites in automatic and human evaluations.Experiments cover the CAsT-19, CAsT-20, and CAsT-21 conversational search benchmarks.
2 Related Work
Related conversational-search research centers on query rewriting and dense retrieval, while broader IR work uses LLMs for relevance modeling. LLM4CS differs by focusing on LLM-based interpretation of contextual search intent for conversational search.
- Conversational Search: Conversational query rewriting reformulates a session into a standalone query, while conversational dense retrieval directly encodes the whole session for end-to-end retrieval.CQR research includes token selection, generative rewriting, reinforcement learning, and ranking-signal optimization; CDR research emphasizes context denoising, data augmentation, and hard negatives.
- IR with LLMs: Information-retrieval studies use LLMs for query generation, query expansion, document prediction, retrieval augmentation, URL-based document retrieval, and reranking.These approaches enhance relevance modeling through generated or predicted search content.
- IR with LLMs: LLM4CS focuses specifically on using LLMs to capture users’ contextual search intent and facilitate conversational search.This focus distinguishes it from prior LLM-for-IR studies that target broader relevance-modeling functions.
3 LLM4CS: Prompting Large Language Models for Conversational Search
LLM4CS treats an LLM as a text-based search-intent interpreter, generating rewrites and hypothetical responses before aggregating them for conversational passage retrieval. It explores multiple prompting strategies, aggregation rules, and optional chain-of-thought reasoning within this framework.
- 3.1 Task Formulation: LLM4CS formulates conversational passage retrieval as finding passages relevant to the current query’s real intent given its multi-turn conversation context.The task targets the first retrieval step that supplies evidence for conversational search.
- 3.2 Prompting Methods: The framework prompts an LLM with the current query and conversation context, using REW, RTR, and RAR to generate rewrites and hypothetical responses.REW generates rewrites directly; RTR generates responses after rewriting, whereas RAR generates both together in one stage.
- 3.2.4 Incorporating Chain-of-Thought: LLM4CS also tests contextual-search chain-of-thought prompting, instructing the LLM to reason about intent before producing rewrites and responses.Demonstrations include manually written reasoning that reflects how humans infer current-turn intent from conversation history.
- 3.3 Content Aggregation: The framework encodes generated rewrites and hypothetical responses with a shared ad-hoc retriever and aggregates their vectors into one final search-intent representation.At retrieval time, candidate passages are encoded with the same retriever, and passages most similar to the final vector are returned.
- 3.3.1 MaxProb: MaxProb selects the rewrite and hypothetical response with the highest generation probabilities, making it efficient because it avoids requiring multiple generations.For RTR and RAR, the method mixes the selected rewrite and response vectors.
- 3.3.2 Self-Consistency (SC): Self-Consistency selects the intent vector most similar to the cluster center, treating it as the most popular intent among multiple generations.RTR first selects a rewrite and then selects a response generated from that rewrite; RAR needs no response selection because each rewrite has one response.
- 3.3.3 Mean: Mean averages rewrite and corresponding hypothetical-response vectors to incorporate diverse intent information from all generated sources.The method is intended to reinforce popular intents while supplementing plausible intents absent from a single rewrite or response.
4 Experiments
Experiments on three TREC CAsT benchmarks evaluate LLM4CS against conversational search baselines, human rewrites, prompting and aggregation variants, and chain-of-thought ablations. LLM4CS achieves the strongest reported retrieval performance, while hypothetical responses, aggregation choices, and tailored chain-of-thought each affect results.
- 4.1 Datasets and Metrics: The evaluation uses CAsT-19, CAsT-20, and CAsT-21, which provide human rewrites and expert relevance judgments, with CAsT-20 and CAsT-21 having more complex session structures than CAsT-19.Metrics are MRR, NDCG@3, and Recall@100; CAsT-21 documents are scored using their highest-scored passage.
- 4.2 Implementation details: LLM4CS combines multiple query rewrites and hypothetical responses generated by few-shot prompting, with RAR plus Mean aggregation and tailored CoT forming the strongest reported configuration.The baselines include query rewriting, dense retrieval, zero-shot retrieval, and combined rewriting-expansion systems, with shared ANCE retrieval used for fair comparison where applicable.
- 4.4 Main Results: LLM4CS outperforms all compared baselines, improving over the second-best results by more than 18% across all metrics on CAsT-20 and CAsT-21, while surpassing human rewrites on most metrics.Exceptions are Recall@100 on CAsT-19 and NDCG@3 on CAsT-21.
- 4.5 Effects of Different Prompting Methods and Aggregation Methods: RAR and RTR outperform REW, showing that hypothetical responses supplement short rewrites, while REW remains competitive on the more challenging CAsT-20 and CAsT-21 datasets.REW reaches 0.380 versus 0.350 on CAsT-20 and 0.465 versus 0.385 on CAsT-21 in the reported comparison.
- 4.5 Effects of Different Prompting Methods and Aggregation Methods: Mean and SC consistently outperform MaxProb, but Mean does not always outperform SC, indicating that selecting among generations can be preferable to using either only the top prediction or all generations.The reported CAsT-20 example shows that fusing all generated contents is not consistently better than fusing one rewrite and one response.
- 4.6 Effects of Chain-of-Thought: Adding tailored chain-of-thought generally improves retrieval, with especially notable gains for REW; multiple hypothetical responses appear to reduce its incremental impact.The authors attribute this pattern to the improved quality and robustness already provided by multiple hypothetical responses.
5 Human Evaluation
Human evaluation compares LLM4CS rewrites with human rewrites and T5QR across the three CAsT datasets. LLM4CS more often conveys the real search intent, especially on the more complex datasets, and produces fewer coreference errors.
- 5 Human Evaluation: LLM4CS rewrites fully convey users’ real search intent in 85.5%, 89.4%, and 84.8% of cases on CAsT-19, CAsT-20, and CAsT-21, respectively, versus 75.1%, 62.0%, and 58.6% for T5QR.Rewrites are labeled Good-H, Good-A, Bad-C, or Bad-O by comparing each model’s rewrite with the corresponding human rewrite and assessing contextual intent.
- 5 Human Evaluation: Good-A rewrites are more common on CAsT-20 and CAsT-21 than on CAsT-19, reflecting that complex sessions permit different wording while preserving the same search intent.On CAsT-19, most good rewrites more closely resemble the human rewrites.
- 5 Human Evaluation: LLM4CS produces coreference errors in fewer than 3% of cases, compared with approximately 10% for T5QR.This comparison directly targets a major challenge in conversational search: resolving context-dependent references.
6 Conclusion
LLM4CS is a prompting framework that uses multiple query rewrites and hypothetical responses, aggregated to represent contextual search intent for conversational search.
- LLM4CS generates multiple query rewrites and hypothetical responses, then aggregates them to represent users’ contextual search intent.
Limitations
The framework improves search performance through multiple generations and aggregation, but this design introduces latency and lacks awareness of downstream retrieval signals.
- Multiple LLM generations increase retrieval time, making query latency potentially intolerable compared with conventional search engines.The authors suggest obtaining all informative content in one generation as a possible remedy.
- LLM generation lacks awareness of downstream retrieval, leaving ranking-signal integration as an open research direction.
Appendix
Figure 4 illustrates LLM4CS prompting as a structured combination of instructions, demonstrations, inputs, method-specific prompts, and chain-of-thought.
- A Prompt of LLM4CS: The prompt comprises Instruction, Demonstration, and Input components, with color-coded regions for REW, RTR, RAR, and chain-of-thought prompting.
B.1 Examples of Chain-of-Thought
Table 4 demonstrates how chain-of-thought supports the rewriting process by making the rationale behind generated rewrites explicit.
- B.1 Examples of Chain-of-Thought: The generated chain-of-thought effectively explains the rationale behind the rewrite produced by LLM4CS.
B.2 Examples of Human Evaluation
The human-evaluation examples cover four categories, while the accompanying examples illustrate LLM4CS prompts and chain-of-thought rewriting.
- Figure 4 illustrates LLM4CS prompts through Instruction, Demonstration, and Input components, with color-coded regions for prompting variants and chain-of-thought.
- Table 4 provides an example of the chain-of-thought effect using CoT and Our Rewrite fields generated by LLM4CS.
- Table 5 presents examples spanning four categories used in the human evaluation.