Source-linked AI summary
Few-Shot Generative Conversational Query Rewriting
Shi Yu, Jiahua Liu, Jingqin Yang, Chenyan Xiong, Paul Bennett, Jianfeng Gao, Zhiyuan Liu
TL;DR
Conversational query rewriting must recover omitted context so concise queries can be handled by retrieval systems. The paper generates weak supervision from ad hoc search sessions and fine-tunes GPT-2 with few or zero manual rewrites. The resulting rewriter improves CAsT ranking accuracy by 12% in the few-shot setting and remains comparable to prior automatic systems without manual rewrites.
Problem
Conversational queries omit information and depend on previous turns, while manual rewrites are expensive and may be unavailable.
Method
The paper uses rule-based and self-supervised methods to generate weak supervision from ad hoc search sessions for fine-tuning a GPT-2 query rewriter.
Results
12% NDCG@3 improvement over the best automatic CAsT 2019 runs is achieved with manual rewrites from 50 sessions, while zero-shot performance remains comparable to the previous best automatic run.
Takeaways & Limitations
GPT-2 learns task syntax and context dependencies from limited supervision, including long-term and multiple coreferences.
Takeaways & Limitations
The approach addresses the challenge that large quantities of manual query-rewrite labels are expensive and not always available across domains.
Abstract
from arXiv · showhide
Conversational query rewriting aims to reformulate a concise conversational query to a fully specified, context-independent query that can be effectively handled by existing information retrieval systems. This paper presents a few-shot generative approach to conversational query rewriting. We develop two methods, based on rules and self-supervised learning, to generate weak supervision data using large amounts of ad hoc search sessions, and to fine-tune GPT-2 to rewrite conversational queries. On the TREC Conversational Assistance Track, our weakly supervised GPT-2 rewriter improves the state-of-the-art ranking accuracy by 12%, only using very limited amounts of manual query rewrites. In the zero-shot learning setting, the rewriter still gives a comparable result to previous state-of-the-art systems. Our analyses reveal that GPT-2 effectively picks up the task syntax and learns to capture context dependencies, even for hard cases that involve group references and long-turn dependencies.
1 INTRODUCTION
Conversational queries are concise and context dependent, making them difficult for existing retrieval systems. This work uses weak supervision to train GPT-2 for rewriting with few or zero manual labels, achieving strong benchmark results and handling difficult dependencies.
- Motivation: Conversational IR supports multi-round information access, but concise queries often omit concepts and refer to previous discussion.These properties distinguish conversational sessions from typical keyword-based search.
- Motivation: De-contextualized rewrites include necessary information and can be better handled by existing ad hoc ranking systems.Automatic rewriting nevertheless showed a 30%+ NDCG drop relative to systems using manual rewrites.
- Research gap: Large amounts of manual query-rewrite labels are expensive to collect and may be unavailable across domains.This motivates learning with few or even zero manual rewriting labels.
- Approach: The paper generates weak supervision from ad hoc search sessions using rule-based and self-supervised approaches, then fine-tunes GPT-2 to rewrite conversational queries.The methods target context dependencies in concise conversational queries.
- Results: 12% NDCG@3 improvement over the best automatic runs in CAsT 2019 is achieved using manual rewrites from only 50 conversational sessions.With no manual rewrites, the weakly supervised GPT-2 remains comparable to the previous best automatic run.
- Analysis: GPT-2 learns task syntax and resolves context dependencies with as few as three conversational sessions, including long-term and multiple coreferences.The analysis reports that the model generates questions rather than stories and handles hard reference cases.
2 PRELIMINARIES
The task rewrites a context-dependent conversational query into a fully de-contextualized query using previous turns. GPT-2 generates the rewritten query token by token, trained with either ground-truth or weak-supervision labels and used before standard ranking.
- Task definition: Conversational search queries are concise, with information needs often presented in previous queries of the same session.The session is represented as a sequence of queries Q1 through QN.
- Task definition: The rewriting task converts a context-dependent query Qk into a fully de-contextualized query Q′k using previous queries Q<k.The resulting query better reflects user intent and is easier for ad hoc search.
- Generation model: GPT-2 directly generates rewritten query words one by one with a transformer decoder conditioned on the conversation history and current query.The input concatenates prior and current queries with separators, followed by a beginning-of-sequence marker.
- Training: Training uses either ground-truth query rewrites or weak-supervision labels as the target sequence.Both training and inference use standard GPT-2 adapted to generate queries instead of plain text.
- Ranking: After rewriting, standard ad hoc ranking retrieves 100 documents with BM25 and reranks them using a BERT ranker.The rewritten query is the input to the downstream ranking pipeline.
3 WEAK SUPERVISION
The paper constructs weak supervision for conversational query rewriting by transforming abundant ad hoc search sessions into conversation-like sessions paired with fully specified queries. It uses rule-based omission and coreference operations alongside a self-supervised GPT-2 query simplifier.
- Weak supervision construction: Ad hoc search sessions provide fully specified queries that can serve as pseudo-target rewrites for weak supervision.These sessions are converted into conversation-like sessions, creating pairs that approximate real conversational queries and manual rewrites.
- Weak supervision construction: The conversion creates weakly supervised session pairs by transforming fully specified sessions into contextual sessions.The paper proposes two approaches for this conversion: rules and self-learning.
- Rule-Based: The rule-based method mimics omission by removing repeated noun phrases and coreference by replacing prior noun phrases with pronouns.Singular phrases become “it,” “he,” or “she,” while plural phrases become “they” or “them.”
- Self-Learn: The self-learning method trains a GPT-2 query simplifier to generate contextual queries from fully specified queries.Unlike rewriting, which restores context, simplification removes information already presented in earlier turns.
- Self-Learn: The query simplifier is trained with a handful of manual rewrites and then applied to MS MARCO ad hoc sessions to generate conversation-like data.It reverses the source and target direction used for query rewriting while retaining the same GPT-2 setup.
4 EXPERIMENTAL METHODOLOGIES
The experiments evaluate weakly and few-shot query rewriting on TREC CAsT using MS MARCO sessions for supervision, common rewriting baselines, and shared ranking models. They measure ranking, rewrite similarity, and answer quality across zero-shot and few-shot settings.
- Evaluation: The main ranking metric is NDCG@3, supplemented by BLEU-2 for rewrite similarity and ROUGE-L for question answering.NDCG@3 is averaged across all turns, and BLEU-2 compares automatic rewrites with Oracle queries.
- Baselines: The benchmark compares original queries, coreference-based reformulations, GPT-2 baselines, and established automatic CAsT runs.The baselines include AllenNLP Coref, GPT-2 Raw, MARCO Raw, clacBase, pgbert, and CFDA_CLIP_RUN7.
- Implementation: All experiments use a pretrained GPT-2 medium rewriter and a shared BM25 ad hoc ranker, with a BERT ranker trained only on MS MARCO passage labels.CAsT relevance labels are reserved for testing in the BERT ranking setup.
- Training settings: Zero-shot experiments fine-tune GPT-2 only on converted MS MARCO sessions, whereas few-shot experiments additionally use manual rewrites with five-fold cross-validation.The cross-validation split is by session, and no testing fold is exposed during training.
5 EVALUATION RESULTS
GPT-2 narrows the accuracy gap in conversational search with few-shot and zero-shot rewriting, while learning useful rewriting behavior from very limited supervision. Its effectiveness emerges quickly, though errors more often omit complete context than add false information.
- Conversational Search Accuracy: GPT-2 substantially narrows the gap between concise conversational queries and manual oracle queries in TREC CAsT.The comparison includes Original and AllenNLP Coref baselines.
- Conversational Search Accuracy: 10+% improvement over the state of the art comes from Rule-Based + CV or Self-Learn + CV with weak supervision data.The authors attribute the improvement mainly to better query rewriting.
- Conversational Search Accuracy: GPT-2 (CV) is effective with 50 CAsT sessions, while Rule-Based zero-shot performance is on par with the best TREC CAsT automatic runs.GPT-2 Raw and MARCO Raw produce sub-par results by comparison.
- Few-Shot Study: GPT-2 produces reasonable rewrites with three conversational sessions or 30 manual labels and matches the best automatic runs with 10 sessions.Figure 2a examines performance as the number of fine-tuning sessions changes.
- Few-Shot Study: GPT-2 quickly learns rewriting syntax, including question generation and copying concepts from previous turns, during fine-tuning.Figure 2b tracks question and copied-word fractions across fine-tuning steps.
- Case Study: GPT-2 resolves some long-range group references, but its errors more often result from missing complete contexts than adding false information.One example resolves “their” to two cancer types, including one mentioned three turns earlier.
6 CONCLUSION
This work demonstrates that GPT-2 can perform conversational query rewriting using weak supervision from rules or a handful of manual rewriting labels. The resulting rewriter establishes a new state of the art on the TREC CAsT benchmark.
- Conclusion: GPT-2 creates new state-of-the-art results on TREC CAsT using rule-generated weak supervision or a handful of manual rewriting labels.It outperforms prior query expansion, contextual ranking, and coreference resolution methods.