Source-linked AI summary
Generative Retrieval with Few-shot Indexing
Arian Askari, Chuan Meng, Mohammad Aliannejadi, Zhaochun Ren, Evangelos Kanoulas, Suzan Verberne
TL;DR
Training-based generative retrieval fine-tunes models to associate queries with document identifiers, with high training costs and limited use of pre-trained knowledge. Few-Shot GR instead prompts an LLM to build a docid bank and retrieve through it, using multiple identifiers per document. It outperforms all baselines except for two metric-specific comparisons.
Problem
Training-based generative retrieval relies on fine-tuning query–docid associations, which requires substantial training resources and may underuse pre-trained LLM knowledge.
Source: §1 · p. 2Method
Results
Few-Shot GR outperforms all baselines except GenRET on NQ320K Recall@10 and NOVO on MS300K MRR@10.
Source: §4 · p. 5Takeaways & Limitations
Few-shot indexing is significantly more efficient than training-based indexing, while achieving strong retrieval performance.
Source: §1 · p. 2, §5 · p. 6Evaluation uses corpora of 100K and 320K documents; effectiveness on million-document corpora and other datasets remains to be tested.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Existing generative retrieval (GR) methods rely on training-based indexing, which fine-tunes a model to memorise associations between queries and the document identifiers (docids) of relevant documents. Training-based indexing suffers from high training costs, under-utilisation of pre-trained knowledge in large language models (LLMs), and limited adaptability to dynamic document corpora. To address the issues, we propose a few-shot indexing-based GR framework (Few-Shot GR). It has a few-shot indexing process without any training, where we prompt an LLM to generate docids for all documents in a corpus, ultimately creating a docid bank for the entire corpus. During retrieval, we feed a query to the same LLM and constrain it to generate a docid within the docid bank created during indexing, and then map the generated docid back to its corresponding document. Moreover, we devise few-shot indexing with one-to-many mapping to further enhance Few-Shot GR. Experiments show that Few-Shot GR achieves superior performance to state-of-the-art GR methods requiring heavy training.
1 Introduction
Few-Shot GR replaces training-based indexing with prompting an LLM to create a docid bank, then retrieves through that bank. The introduction also motivates one-to-many docids to address the limits of assigning a document only one identifier.
1 Introduction
- Training-based indexing has high training overhead and may underuse pre-trained LLM knowledge because fine-tuning requires large query–docid datasets and substantial computing resources.The passage says the mismatch between pre-training and GR fine-tuning objectives may cause an LLM to forget pre-trained knowledge.Source: §1 · pp. 1–2
- Few-Shot GR prompts an LLM, without training, to generate free-text docids for every corpus document and creates a docid bank for retrieval.Retrieval uses the same LLM and constrained beam search to generate a docid valid in the bank.Source: §1 · p. 2
- One-to-many indexing generates multiple docids per document, allowing different queries relevant to that document to retrieve it through different identifiers.The authors report that generating only one docid per document limits retrieval quality.Source: §1 · p. 2
- Few-Shot GR achieves superior performance to state-of-the-art GR methods requiring heavy training.The authors also report that it outperforms or performs comparably to state-of-the-art methods.Source: §1 · p. 2
2 Methodology
Few-Shot GR builds a corpus-wide bank by generating multiple docids for each document from pseudo queries. At retrieval, the same LLM generates a bank-valid docid that is mapped back to its document.
2 Methodology
- For each document, indexing generates n pseudo queries, uses an LLM to produce n corresponding docids, then deduplicates the bank.The one-to-many mapping is intended to capture diverse relevance signals.Source: §2 · p. 3
- The indexing stage prompts an LLM in a few-shot manner to generate docids for the corpus.The resulting docid bank contains docids for all corpus documents.Source: §2 · p. 3
- At retrieval, the same prompt and LLM generate a docid constrained to the bank, which is mapped to its corresponding document.The bank is deduplicated so each docid uniquely corresponds to one document.Source: §2 · p. 3
3 Experimental setup
The setup evaluates Few-Shot GR on NQ320K and MS300K against non-GR and training-based GR baselines. It uses Llama-3-8B-Instruct and generates 10 docids per document during indexing.
Datasets
- NQ320K and MS300K provide 7,830 and 5,187 test queries, respectively, with corpora of 100K and 320K documents.NQ320K has 320k query–document pairs; MS300K has 300k.Source: §3 · p. 4
Implementation details
- Few-Shot GR uses Llama-3-8B-Instruct and generates 10 docids per document; pseudo queries are generated with InPars.The maximum and minimum docid lengths are 15 and 3 tokens, respectively.Source: §3 · pp. 4–5
Baselines and evaluation metrics
- Retrieval quality is assessed against BM25, dense-retrieval baselines, and training-based GR systems using Recall@1, Recall@10, and dataset-specific MRR metrics.MRR@100 is reported on NQ320K and MRR@10 on MS300K.Source: §3 · p. 4
Evaluation scope
- Figure 2 tracks NQ320K retrieval quality against the number of generated docids per document, while Table 2 compares LLM choices.Table 2 covers Few-Shot GR with different LLMs on NQ320K.Source: Figure 2 · p. 5, Table 2 · p. 5
4 Result and analysis
Few-Shot GR outperforms the baselines across nearly all reported metrics, with two stated exceptions. Its indexing is substantially more efficient than training-based GR, while retrieval latency is similar.
Comparison with baselines
- Few-Shot GR beats all baselines except GenRET on Recall@10 for NQ320K and NOVO on MRR@10 for MS300K.The authors contrast its results with those two exceptions, which require large training corpora and heavy model-specific training.Source: §4 · p. 5
Impact of number of docids
- 27.2%: Llama-3 Recall@10 improves when generated docids per document increase from 1 to 10, with performance saturating at 10.The reported trend is similar on MS300K; the paper shows NQ320K in Figure 2.Source: §4 · p. 5
Efficiency of indexing and retrieval
- Few-Shot GR indexes significantly more efficiently than DSI-QG and GenRET, while its retrieval query latency is similar to theirs.Table 3 reports indexing time in hours and query latency in milliseconds on NQ320K; GenRET’s training time was reported as 7 days on 100 A100 GPUs.Source: §4 · p. 6, Table 3 · p. 6
Impact of LLM choices
- Llama-3-8B-Instruct performs best across most metrics, followed by Zephyr-7B-β; both markedly outperform T5-base.The authors identify LLM selection as another factor in Few-Shot GR performance.Source: §4 · p. 6
5 Conclusions & Future Work
Few-Shot GR uses prompting for indexing without training, and the paper reports superior performance to training-intensive state-of-the-art GR methods. The authors identify evaluation at larger corpus sizes and on other datasets as future work.
Conclusion
- Few-Shot GR records query–docid associations by prompting an LLM, avoiding training, and uses one-to-many mapping to enhance indexing.The conclusion reports superior performance to training-intensive state-of-the-art GR methods.Source: §5 · p. 6
Future work
- The evaluated corpora contain 100K and 320K documents, leaving million-document generalization and evaluation on datasets such as BEIR for future work.The authors also identify conversational search and automatic retrieval-quality prediction as future directions.Source: §5 · p. 6
Appendix
The appendix presents an example of pseudo queries and corresponding docids for one NQ320K document. The example contains 10 distinct, varied docids.
Case study of docids generated by Few-Shot GR
- Table 4 pairs pseudo queries with generated docids for one NQ320K document and illustrates 10 distinct, varied identifiers.The example uses Llama-3-8B-Instruct.Source: Appendix · p. 9, Table 4 · p. 9