PaperlayerView exact revision on arXiv

Source-linked AI summary

Query2doc: Query Expansion with Large Language Models

Liang Wang, Nan Yang, Furu Wei

arXiv:2303.07678v2cs.IRcs.CL

TL;DR

Existing query expansion methods have had limited success, while dense retrievers generally do not use them. Query2doc prompts LLMs to generate pseudo-documents and concatenates them with queries for sparse or dense retrieval. It consistently improves retrieval across models and datasets, including over 15% BM25 gains on TREC DL, but adds inference and search costs.

  • Problem

    Traditional query expansion has shown limited success, and most state-of-the-art dense retrievers do not adopt it.

  • Method

    Query2doc few-shot prompts LLMs to generate pseudo-documents, then concatenates each pseudo-document with the original query for sparse or dense retrieval.

  • Results

    Query2doc produces consistent improvements across retrieval models and datasets, including over 15% improvements over BM25 on TREC DL 2019 and 2020.

  • Takeaways & Limitations

    LLM prompting can improve existing sparse and dense retrievers without model fine-tuning, although benefits diminish for strong reranked systems.

  • Takeaways & Limitations

    Query2doc is less efficient because LLM generation is slow and query expansion increases inverted-index search time.

Abstract

from arXiv · show

This paper introduces a simple yet effective query expansion approach, denoted as query2doc, to improve both sparse and dense retrieval systems. The proposed method first generates pseudo-documents by few-shot prompting large language models (LLMs), and then expands the query with generated pseudo-documents. LLMs are trained on web-scale text corpora and are adept at knowledge memorization. The pseudo-documents from LLMs often contain highly relevant information that can aid in query disambiguation and guide the retrievers. Experimental results demonstrate that query2doc boosts the performance of BM25 by 3% to 15% on ad-hoc IR datasets, such as MS-MARCO and TREC DL, without any model fine-tuning. Furthermore, our method also benefits state-of-the-art dense retrievers in terms of both in-domain and out-of-domain results.

1 Introduction

Query2doc uses LLM-generated pseudo-documents to expand short or ambiguous queries, improving sparse and dense retrieval without changing training pipelines or model architectures. Experiments report strong gains for BM25 and benefits for dense retrievers, especially on challenging queries.

  • 1 Introduction: Query expansion rewrites queries using pseudo-relevance feedback or external knowledge, but traditional methods have had limited success and are rarely used by state-of-the-art dense retrievers.Document expansion has been effective for sparse retrieval, whereas most leading dense retrievers do not adopt query expansion.
  • 1 Introduction: LLMs can generate relevant pseudo-documents that supply background information for short, ambiguous, or underspecified queries.Their pretraining on trillions of tokens gives them broad memorized knowledge and language patterns that can guide retrieval.
  • 1 Introduction: Query2doc concatenates an LLM-generated pseudo-document with the original query, providing a simple expansion method that requires no training-pipeline or architecture changes.The resulting framework is designed to remain orthogonal to progress in LLMs and information retrieval.
  • 1 Introduction: Query2doc also benefits strong dense retrievers, while gains diminish when the retrievers already use cross-encoder-based reranking.Zero-shot out-of-domain experiments report improvements on most datasets, and larger language models perform best while smaller models provide marginal gains.

2 Method

query2doc generates pseudo-documents with few-shot prompts and integrates them into sparse and dense retrieval through query concatenation. It also distinguishes the concatenation strategies and training settings used for these retrievers.

  • Query2doc generation: Few-shot prompting generates a pseudo-document d′ from query q using labeled training examples.The prompt includes the instruction “Write a passage that answers the given query:” and k = 4 randomly sampled labeled pairs.
  • Sparse Retrieval: For sparse retrieval, query2doc repeats q n times before concatenating it with d′ to balance their relative weights.The resulting q+ is used for BM25 retrieval.
  • Sparse Retrieval: BM25 uses n = 5 as a generally good value without dataset-specific tuning.The paper treats this as a fixed general setting rather than tuning n separately for each dataset.
  • Dense Retrieval: For dense retrieval, q+ concatenates q and d′ with a [SEP] separator.This differs from sparse retrieval, which repeats the original query to adjust its relative weight.
  • Dense Retrieval: Dense retrievers are studied both with DPR initialized from BERTbase and BM25 hard negatives and with state-of-the-art retrievers distilled from a cross-encoder.The paper investigates contrasting training settings to assess query2doc more comprehensively.
  • Comparison with Pseudo-relevance Feedback: Unlike conventional pseudo-relevance feedback, query2doc obtains expansion text from LLM-generated pseudo-documents rather than initial retrieval results.The method therefore does not rely on the quality of potentially noisy or irrelevant initial results.

3 Experiments

Experiments evaluate query2doc on in-domain and zero-shot out-of-domain retrieval, showing strong gains for BM25 and dense retrievers in-domain but mixed transfer across domains.

  • In-domain results: BM25 + query2doc improves over BM25 by over 15% on TREC DL 2019 and 2020.The evaluation includes MS-MARCO passage ranking and TREC DL datasets, using MRR@10, R@k, and nDCG@10.
  • In-domain results: BM25 + query2doc requires no model fine-tuning, unlike docT5query, which trains a T5-based query generator with labeled data.docT5query achieves better numbers on the MS-MARCO dev set but has a higher training requirement.
  • In-domain results: Dense retrievers combined with query2doc outperform their corresponding baselines on all metrics.The reported dense systems include DPR, SimLM, and E5.
  • In-domain results: Query2doc gains diminish for dense retrievers using intermediate pre-training or knowledge distillation from cross-encoder re-rankers.This pattern is shown for the SimLM + query2doc and E5 + query2doc variants.
  • Out-of-domain results: Zero-shot out-of-domain results are mixed: entity-centric datasets such as DBpedia improve most, while NFCorpus and Scifact show minor ranking-quality decreases.The decreases are attributed to distribution mismatch between training and evaluation.

4 Analysis

The analysis examines how query2doc behaves across LLM scales, supervision levels, query-construction choices, and generated-text reliability. Larger models and combining original queries with pseudo-documents improve retrieval, while factual errors remain a trustworthiness concern.

  • Scaling up LLMs is Critical: Performance steadily improves as the query-expansion model scales from 1.3B to 175B parameters, with GPT-4 achieving the best results.Smaller models generate shorter texts with more factual errors; GPT-4 could not be used in the main experiments because of quota limits.
  • Performance Gains are Consistent across Data Scales: DPR + query2doc consistently outperforms the DPR baseline by approximately 1%, regardless of the amount of data used for fine-tuning.Figure 2 compares the variants on MRR on the MS-MARCO dev set across labeled-data percentages.
  • How to Use Pseudo-documents: Concatenating the original query with generated pseudo-documents performs substantially better than using pseudo-documents alone in sparse retrieval.The original query and pseudo-documents are complementary.
  • Case Analysis: LLM-generated pseudo-documents can provide detailed and mostly accurate information, but subtle factual errors pose a significant challenge for trustworthy retrieval systems.The case analysis gives a season-year error in a theme-song fact as an example.

5 Related Work

Prior work includes query and document expansion methods for reducing lexical mismatch, while LLMs enable few-shot generation of pseudo-documents for query expansion.

  • Query Expansion: Query expansion rewrites queries using pseudo-relevance feedback or external resources such as WordNet to bridge the lexical gap with documents.The related-work discussion identifies Rocchio and RM3 as classical query-expansion approaches.
  • Document Expansion: Document expansion enriches document representations by appending generated relevant terms, as in doc2query's predicted pseudo-queries.Doc2query trains a seq2seq model to predict pseudo-queries from documents and adds them to the index.
  • Large Language Models: LLMs trained on trillions of tokens support zero-shot and few-shot prompting, enabling pseudo-document generation from a query; HyDE instead uses zero-shot prompting and pseudo-document embeddings.The comparison distinguishes query2doc's few-shot generation from HyDE's zero-shot embedding-based approach.

6 Conclusion

The paper presents query2doc as a simple way to use LLMs for query expansion across sparse and dense retrievers. Empirical evaluations report consistent improvements across retrieval models and datasets.

  • Conclusion: Query2doc prompts LLMs with few-shot examples to generate pseudo-documents, then augments queries for existing sparse or dense retrievers.The approach is motivated as distilling LLMs through prompting.

Limitations

query2doc introduces efficiency costs from both LLM inference and longer expanded queries. These costs should be considered in real-world deployment.

  • Table 6 benchmarks retrieval latency on MS-MARCO dev queries using top-100 retrieval with a single thread and averaging across queries.LLM API-call latency is difficult to measure precisely because it depends on server load.
  • LLM inference can be considerably slower because pseudo-documents require token-by-token autoregressive decoding.
  • Expanded queries also slow inverted-index searching as the number of query terms increases.

A Implementation Details

The implementation uses standard retrieval and training configurations while generating pseudo-documents through few-shot prompting. Additional experiments examine GPT-4 self-rewriting and several prompting and evaluation choices.

  • Dense-retriever training uses hyperparameters listed in Table 7 and released checkpoints or teacher scores where specified.SimLM uses the [CLS] vector, whereas E5 uses mean pooling for text embeddings.
  • Prompts contain 4 randomly selected MS-MARCO training examples for each API call to increase prompt diversity.The complete prompt is shown in Table 11.
  • GPT-4 prompting is instructed to write an answering passage directly rather than ask for clarification.This instruction is intended to mitigate GPT-4’s tendency to request clarification.
  • Ablation experiments fine-tune for 40 epochs or 18k steps, whichever occurs first.
  • Iterative prompting asks GPT-4 to rewrite generated pseudo-documents while correcting factual errors and avoiding unnecessary changes.The rewrite prompt template is shown in the supplied implementation materials.

# End of passage

GPT-4 makes very few changes when iteratively rewriting generated pseudo-documents. The paper presents this experiment alongside additional generation examples and multi-run sparse-retrieval results.

  • GPT-4 makes very few changes to generated pseudo-documents during iterative self-rewriting.This suggests either that the pseudo-documents are already high quality or that GPT-4 cannot correct its own errors.
  • Table 8 reports the results of GPT-4 iterative prompting for self-verification of generated pseudo-documents.
  • Table 9 provides additional examples of LLM generations in the same format as Table 5.
  • Table 10 reports sparse-retrieval results across three random runs.The runs vary through few-shot-example selection and autoregressive sampling.

C Results Across Multiple Runs

The method’s randomness comes from few-shot-example selection and autoregressive top-p sampling. Sparse-retrieval results are therefore averaged across three random runs with standard deviations reported.

  • Two randomness sources are few-shot-example selection and autoregressive top-p sampling by the LLM.
  • Sparse-retrieval results are reported as averages and standard deviations across 3 random runs.The results are presented in Table 10.
  • Selecting few-shot examples by semantic similarity to the query is proposed as a possible future improvement.
Loading 2303.07678v2…