PaperlayerView exact revision on arXiv

Source-linked AI summary

Promptagator: Few-shot Dense Retrieval From 8 Examples

Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, Ming-Wei Chang

arXiv:2209.11755v1cs.CLcs.IR

TL;DR

Neural retrievers struggle to transfer across diverse tasks whose query distributions and search intents differ, despite abundant supervision in source QA datasets. PROMPTAGATOR uses few-shot prompting to generate synthetic task-specific queries and trains end-to-end retrievers from them. With no more than 8 examples, it outperforms heavily engineered baselines across 11 retrieval tasks, while reranking provides an additional reported gain.

  • Problem

    Retrievers trained on one or two QA datasets may not perform well across tasks with different search intents and query distributions.

  • Method

    PROMPTAGATOR prompts an LLM to generate and consistency-filter synthetic task-specific queries, then trains retrievers and rerankers on the generated data.

  • Results

    PROMPTAGATOR with two-to-eight examples produced significantly better retrievers than models trained on MS MARCO or NQ, and reranking adds another 5 points on the standard retrieval metric.

  • Takeaways & Limitations

    A few task-specific examples, amplified through LLM query generation, can produce efficient end-to-end retrievers with promising retrieval performance gains.

  • Takeaways & Limitations

    The paper leaves the efficiency of generated data and the amount of query-document pairs needed for each task insufficiently explored.

Abstract

from arXiv · show

Much recent research on information retrieval has focused on how to transfer from one task (typically with abundant supervised data) to various other tasks where supervision is limited, with the implicit assumption that it is possible to generalize from one task to all the rest. However, this overlooks the fact that there are many diverse and unique retrieval tasks, each targeting different search intents, queries, and search domains. In this paper, we suggest to work on Few-shot Dense Retrieval, a setting where each task comes with a short description and a few examples. To amplify the power of a few examples, we propose Prompt-base Query Generation for Retriever (Promptagator), which leverages large language models (LLM) as a few-shot query generator, and creates task-specific retrievers based on the generated data. Powered by LLM's generalization ability, Promptagator makes it possible to create task-specific end-to-end retrievers solely based on a few examples {without} using Natural Questions or MS MARCO to train %question generators or dual encoders. Surprisingly, LLM prompting with no more than 8 examples allows dual encoders to outperform heavily engineered models trained on MS MARCO like ColBERT v2 by more than 1.2 nDCG on average on 11 retrieval sets. Further training standard-size re-rankers using the same generated data yields another 5.0 point nDCG improvement. Our studies determine that query generation can be far more effective than previously observed, especially when a small amount of task-specific knowledge is given.

1 INTRODUCTION

Retrieval tasks differ in search intents and query distributions, making transfer from a small number of QA datasets difficult. PROMPTAGATOR uses few-shot LLM query generation to build task-specific retrievers and reports strong results across 11 tasks.

  • Motivation: BEIR tasks differ in search intent, query distribution, and retrieval corpus, challenging transfer from QA-trained retrievers.Dbpedia-Entity retrieves mentioned entities, whereas FEVER retrieves evidence supporting or refuting statements; HotpotQA and FiQA also have distinct query forms.
  • Few-shot Retrieval: PROMPTAGATOR defines Few-shot Retrieval using a task description and a few annotated examples to express task-specific search intents.The approach targets adaptation when new-task supervision is limited.
  • Method: PROMPTAGATOR prompts an LLM to generate synthetic task-specific queries, filters them using round-trip consistency, and trains retrievers on the resulting data.The method avoids using Natural Questions or MS MARCO query-document pairs to train the retriever directly.
  • Results: PROMPTAGATOR with two-to-eight examples produced significantly better retrievers than recent models trained on MS MARCO or NQ with over 500K human-annotated examples.This result is reported in Figure 1(c).
  • Results: PROMPTAGATOR outperforms ColBERT v2 and SPLADE v2 on 11 retrieval tasks, while reranking adds another 5 points on the standard retrieval metric.The paper also presents LLM-generated retrievers as efficient to serve with high accuracy.

2 FEW-SHOT RETRIEVAL TASK

The paper formalizes retrieval tasks by their corpus, query distribution, and search intent, then extends BEIR to evaluate learning from only a few in-domain examples. This setting emphasizes adapting to task-specific queries and intents, not only document domains.

  • Retrieval Task: A retrieval task consists of a document corpus, a query distribution, and an underlying search intent.The same query-document pair can have different relevance under different intents.
  • Retrieval Task: Retrieval tasks vary in corpus, query form, and relevance definition, including answer retrieval, supporting arguments, and counterarguments.Query forms can include short keywords, questions, and arguments.
  • Few-shot Retrieval: The target setting assumes a new retrieval corpus is available but annotated query-document pairs are limited, while query and intent divergence remains under-explored.The paper focuses on expressing search intent through a short description and very few examples.
  • Few-shot Retrieval: Few-shot Retrieval asks whether machines can learn a task-specific retriever from 8 or fewer examples and defines an evaluation setting built on BEIR.The setting is motivated by the ability to understand a task from a short instruction and a few examples.
  • BEIR Setting: BEIR contains 18 datasets across 9 domains and covers intents including QA retrieval, duplicate-question discovery, and fact checking.The original BEIR evaluation was zero-shot, disallowing evaluation queries and relevant pairs for training.
  • BEIR Setting: The few-shot BEIR setting samples 2 to 8 in-domain relevant query-document examples as task-specific supervision.Examples come from development data when available, or test data for tasks with only a test set, with sampled test examples treated as failed retrievals during evaluation.

3 PROMPTAGATOR

PROMPTAGATOR amplifies a few task examples into synthetic query-document data, filters that data for retrieval consistency, and trains task-specific retrievers and rerankers. The method supports few-shot and zero-shot variants while retaining relatively simple model architectures.

  • 3 PROMPTAGATOR: PROMPTAGATOR transforms a few annotated examples into many synthetic query-document examples by prompting a large language model.The generated data is intended to preserve the target task’s query distribution and search intent.
  • 3.1 PROMPT-BASE QUERY GENERATION: Task-specific prompts combine document and query descriptions with annotated query-document examples to guide generation for a target retrieval task.For ArguAna, the prompt labels documents as arguments and queries as counterarguments.
  • 3.1 PROMPT-BASE QUERY GENERATION: The generation process accepts a synthetic pair only when the language model produces a correctly formatted query description; otherwise, it drops the output.Applying the prompt across the document collection yields a large synthetic dataset.
  • 3.2 CONSISTENCY FILTERING USING ONLY GENERATED DATA: Consistency filtering trains an initial retriever on generated pairs and retains a query only when its source document appears among the retriever’s Top-K results.This uses generated data alone to enforce round-trip consistency across different search intents.
  • 3.3 FEW-SHOT PROMPTAGATOR RETRIEVER: The filtered synthetic data trains a standard dual encoder using a pretrain/fine-tune recipe for task-specific retrieval under data scarcity.The approach uses a standard dual-encoder architecture rather than requiring a more elaborate retrieval design.
  • 3.5 DISCUSSION: The framework uses 110M-size dual encoder and reranker models, relying on generated-data quality rather than hard negative mining, distillation, or larger models.The simplified recipe is presented as a way to highlight the power of few-shot data.
  • 3.3 FEW-SHOT PROMPTAGATOR RETRIEVER: PROMPTAGATOR++ trains a slower but more accurate cross-attention reranker on the same synthetic data to refine candidates retrieved by PROMPTAGATOR.It reranks the top 200 retrieved passages using 31 sampled negatives.
  • 3.4 ZERO-SHOT PROMPTAGATOR RETRIEVER: PROMPTAGATOR also supports zero-shot query generation through a universal prompt applied regardless of the target task, followed by retriever and reranker training.The zero-shot variants are called zero-shot PROMPTAGATOR and zero-shot PROMPTAGATOR++.

4 EXPERIMENTS

Experiments show that few-shot PROMPTAGATOR improves retrieval across diverse BEIR tasks, with additional gains from reranking and task-adapted query generation. Ablations identify consistency filtering, example quality, and query-distribution matching as important factors.

  • 4.2 MAIN RESULTS: Few-shot PROMPTAGATOR increases averaged nDCG@10 by over 2 points over its zero-shot version and outperforms strong retrieval baselines.It surpasses GenQ, GPL, ColBERT v2, and SPLADE v2 despite using a relatively simple training procedure and architecture.
  • 4.2 MAIN RESULTS: PROMPTAGATOR++ adds another 5 points in nDCG@10 and outperforms larger rerankers including UPR and monoT5-3B.PROMPTAGATOR++ uses a standard 110M reranker, whereas many competing rerankers use 3B-parameter models.
  • 4.2 MAIN RESULTS: The largest few-shot gains occur on Touche-2020 and ArguAna, whose controversial-topic and counter-argument intents differ from traditional QA retrieval.Few-shot prompting adapts to task-specific search intents that are poorly represented by factoid-question training data.
  • 4.3 ABLATION STUDY: Round-trip filtering improves performance on 8 of 11 datasets and raises average nDCG@10 by 2.5 points.Filtering can hurt on the smallest datasets, NFCorpus and SciFact, where the authors conjecture that further training may overfit filtered data.
  • 4.3 ABLATION STUDY: The filter usually removes overly general queries or hallucinated query-document pairs, but can also discard relevant pairs when another document ranks higher.The authors suggest query-specific K values as a possible way to retain such pairs.
  • 4.3 ABLATION STUDY: Eight examples plus an LLM can replace a significant portion of supervised examples when training dual encoders on MS MARCO.Figure 2(b) compares generated-data training with different amounts of supervised data.
  • 4.3 ABLATION STUDY: Prompted LLM query generation, rather than the shared filtering or training recipe, is identified as the main contributor to PROMPTAGATOR’s advantage over GenQ and NQ-QGen.NQ-QGen uses the same filtering, dual-encoder training, batch sizes, and training steps for an apple-to-apple comparison.
  • 4.4 QUALITATIVE ANALYSIS: Few-shot PROMPTAGATOR generates ArguAna queries with a first-word distribution closer to real queries, whereas NQ-QGen mostly generates questions.The few-shot model can generate diverse queries even with only four examples in the prompt.

5 RELATED WORK

Related work spans efficient representation-based retrieval, expressive interaction-based models, few-shot learning, and prompted query generation. PROMPTAGATOR builds on these lines while using prompted LLMs to generate synthetic data for end-to-end retriever training.

  • Neural retrieval models: Representation-based models independently encode queries and passages into a shared dense space and score relevance using vector similarity.Their single-vector representations support fast approximate nearest-neighbor retrieval.
  • Neural retrieval models: Interaction-based models explicitly model query-document term interactions to increase representational expressivity beyond a single dot product.This added interaction structure generally trades some retrieval efficiency for greater modeling capacity.
  • Few-shot Learning: Few-shot learning commonly provides an LLM with task instructions and examples without parameter updates, or fine-tunes the model alongside those inputs.The paper adopts the no-parameter-update approach.
  • Prompt-based Query Generation: Prior retrieval work has used prompted LLMs for reranking or for generating synthetic data to train rerankers over BM25, rather than end-to-end dense retrievers.UPR reranks passages directly, while InPars generates data for a T5 reranker.
  • Retrievers with late interactions: ColBERT and SPLADE increase query-document interaction through token-level representations, whereas dual encoders remain more efficient but less expressive.The paper positions PROMPTAGATOR’s dense retrievers within this efficiency-expressivity trade-off.

6 CONCLUSION AND DISCUSSIONS

PROMPTAGATOR creates task-specific, end-to-end retrievers from only a few annotated examples by amplifying them with prompt-based LLM query generation. The authors identify generated-data efficiency, prompt sensitivity, and knowledge transfer from LLMs to retrievers as important directions for future work.

  • Conclusion: PROMPTAGATOR creates task-specific, end-to-end retrievers using only a few annotated examples.Prompt-based LLM query generation amplifies the few-shot examples and simplifies training neural retrievers for new tasks.
  • Discussion: Generated-data efficiency remains unresolved, including how many query-document pairs each task needs and how to use them more efficiently.
  • Discussion: Final retriever performance may be sensitive to the prompt, a relationship the authors identify for further investigation.
  • Discussion: The authors propose connecting PROMPTAGATOR to distillation to better understand knowledge transfer from the LLM to dual encoders.

7 COMPUTE USAGE AND ENVIRONMENTAL IMPACT

PROMPTAGATOR uses a 137B-parameter FLAN model to generate synthetic queries, creating substantial computational and environmental costs. The resulting distillation reduces inference cost despite the expensive generation process.

  • Compute usage: 137B FLAN generated 58.46M queries totaling 610M words for PROMPTAGATOR.The generation used 29.23M queries with two prompts.
  • Environmental impact: FLAN is based on the pretrained model used for LaMDA, whose pretraining consumed 451 MWh and produced 25.2 tCO2e.
  • Compute usage: The distillation process is computationally expensive but significantly reduces inference cost for standard-sized dual encoders.

A ANALYSIS ON PROMPTS

The prompt analysis compares dataset-specific templates and generated-query characteristics across BEIR tasks. Few-shot prompts produce queries that are generally closer to original queries than zero-shot prompts, with task-dependent forms such as arguments or questions.

  • Prompt comparison: Few-shot prompts generally generate queries closer to original queries than zero-shot prompts.Zero-shot outputs are mostly questions, whereas few-shot outputs better reflect task-specific query forms.
  • Prompt comparison: ArguAna few-shot queries are generally longer and more claim-like than its short, question-like zero-shot queries.
  • Prompt templates: Dataset-specific prompt templates pair passages with task-relevant query descriptions, including arguments, evidence, entities, findings, and questions.Examples include ArguAna counterarguments, FEVER truth judgments, and DBPedia-Entity entities.
  • Prompt templates: Table 4 lists prompt templates for different BEIR datasets.
  • Query analysis: Table 6 reports average query length.
  • Analysis materials: The prompt analysis includes sampled examples from ArguAna, FiQA, and HotpotQA and reports contributions related to prompt design and query-generation experiments.
Loading 2209.11755v1…