PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models are Built-in Autoregressive Search Engines

Noah Ziems, Wenhao Yu, Zhihan Zhang, Meng Jiang

arXiv:2305.09612v1cs.CLcs.IR

TL;DR

Dense retrievers allow only shallow question–document interaction, while autoregressive retrievers can be costly to train at scale. LLM-URL prompts an LLM to generate URLs and retrieve their documents. It reports better retrieval performance than existing methods across three QA benchmarks, with a limitation in updating knowledge or adapting to new domains.

  • Problem

    Dual-encoder retrievers encode questions and documents independently, limiting their interaction, while training autoregressive search engines becomes costly as candidate collections grow.

    Source: §1 · pp. 1–2

  • Method

    LLM-URL prompts an LLM to generate Web URLs for a question, fetches and ranks passages from the linked documents, and gives selected passages to a reader.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    LLM-URL significantly outperforms existing retrieval methods on Recall@K, and its retrieved documents improve downstream QA performance measured by EM against baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A few query–URL demonstrations can prompt an LLM to generate URLs whose corresponding documents contain open-domain answers, without explicit training to map questions to document identifiers.

    Source: §1 · p. 2

    The approach is limited in updating its knowledge or adapting to new domains because adding knowledge would likely require retraining, and generated URLs can be invalid.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Document retrieval is a key stage of standard Web search engines. Existing dual-encoder dense retrievers obtain representations for questions and documents independently, allowing for only shallow interactions between them. To overcome this limitation, recent autoregressive search engines replace the dual-encoder architecture by directly generating identifiers for relevant documents in the candidate pool. However, the training cost of such autoregressive search engines rises sharply as the number of candidate documents increases. In this paper, we find that large language models (LLMs) can follow human instructions to directly generate URLs for document retrieval. Surprisingly, when providing a few {Query-URL} pairs as in-context demonstrations, LLMs can generate Web URLs where nearly 90\% of the corresponding documents contain correct answers to open-domain questions. In this way, LLMs can be thought of as built-in search engines, since they have not been explicitly trained to map questions to document identifiers. Experiments demonstrate that our method can consistently achieve better retrieval performance than existing retrieval approaches by a significant margin on three open-domain question answering benchmarks, under both zero and few-shot settings. The code for this work can be found at \url{https://github.com/Ziems/llm-url}.

1 Introduction

The introduction frames shallow question–document interaction and the training cost of autoregressive retrieval as gaps. It presents LLM-URL, which prompts LLMs to generate URLs and reports stronger retrieval and downstream QA results.

1 Introduction

  • Dual-encoder retrievers encode questions and documents independently, enabling only shallow interactions and potentially missing fine-grained information in a single similarity score.DPR is given as an example of independently encoding each side and comparing the representations.Source: §1 · p. 1
  • Autoregressive search engines generate document identifiers, but training costs rise at LLM scale, and smaller models performed significantly worse in the authors’ pilot study.The introduction also notes that retrieval tasks can have distinct requirements and motivates generalization from a few examples.Source: §1 · pp. 1–2
  • LLM-URL prompts LLMs with a few query–URL demonstrations to generate Web URLs, nearly 90% of whose corresponding documents contain answers to open-domain questions.The approach uses existing URLs rather than identifiers that require explicit training to map questions or documents to them.Source: §1 · p. 2
  • LLM-URL retrieval significantly outperforms existing methods on Recall@K, while passage ranking reduces supporting passages and maintains high recall.The introduction also reports improved downstream QA performance measured by EM against baseline methods.Source: §1 · p. 2

2 Related Work

Traditional retrievers use sparse matching or dense representations, while autoregressive approaches generate document identifiers. LLM-URL relates to the latter but prompts pretrained LLMs to generate URLs.

2.1 Traditional Document Retrievers

  • Dense retrievers such as DPR encode questions and documents independently, allowing only shallow interactions; sparse methods match overlapping question and passage content.DPR is trained on query–document pairs with hard negatives, and later work improves training and reranking.Source: §2.1 · p. 2

2.2 Autoregressive Search Engines

  • Autoregressive retrieval generates document identifiers, including page titles, hierarchical paths, or distinctive n-grams, as an intermediate target for retrieval.These approaches are also called differential search index or neural document indexers.Source: §2.2 · p. 2
  • LLM-URL’s pipeline generates URLs, retrieves the linked documents, splits them into passages, ranks and filters them, then supplies selected passages to a reader.The figure caption describes this pipeline from question input to final answer.Source: Figure 2 · p. 3
  • Prior autoregressive methods generate document indexes, whereas this work reports that prompting pretrained LLMs can outperform traditional dense retrieval models.The cited related-work passage describes generation with cross-attention as an advantage of autoregressive methods.Source: §2.2 · p. 3

3 Proposed Method

LLM-URL uses an LLM to generate URLs for a question, fetches their documents, and ranks document passages before sending selected evidence to a reader. It uses existing URLs rather than trained identifier-to-document mappings.

3 Proposed Method

  • The method prompts an LLM to generate m Wikipedia URLs, using the instruction “Which m Wikipedia URLs would have the answer?” and optionally appending demonstrations.Few-shot prompts use d=10 demonstrations in this work; zero-shot prompts omit them.Source: §3 · p. 3
  • Generated text is filtered to extract URLs, which are fetched to create a document set; malformed URLs or links to nonexistent pages can leave fewer than m documents.The generated text may also include extra information that is discarded.Source: §3 · pp. 3–4
  • Recall improves substantially as the number of generated documents first increases, but the marginal benefit decreases at larger m.The figure caption describes recall gains as significant when m is small and diminishing as it increases.Source: Figure 3 · p. 4
  • LLM-URL splits fetched documents into passages, scores their relevance to the question, and sends the top n passages to a reader to generate an answer.BM25 or DPR can provide the passage scoring function.Source: §3 · p. 4
  • URLs are pre-existing identifiers that can contain topic-related words, and HTTP requests retrieve their documents without training a mapping or building an index.The authors report that prompting for Wikipedia titles instead of URLs significantly reduces Recall@1.Source: §3 · p. 4

4 Experiments

The experiments evaluate LLM-URL as a retriever and its downstream open-domain QA performance on three datasets. They use GPT-3 text-davinci-003 and compare zero-shot and few-shot prompting.

4 Experiments

  • URL generation uses GPT-3 text-davinci-003 with greedy decoding and temperature 0.Several prompts were tested, with little performance difference observed; the best-performing prompt was used.Source: §4 · p. 4
  • The experiments assess document or passage retrieval and open-domain QA on Web Questions, Natural Questions, and TriviaQA.The section frames retrieval results as direct evidence and QA performance as indirect evidence of competitiveness.Source: §4 · p. 4

4.1 Retrieval

LLM-URL is evaluated for document and passage retrieval against established baselines. Its strongest gains are in document retrieval, while passage-retrieval performance varies by cutoff and comparator.

4.1 Retrieval

  • Evaluation restricts retrieval to Wikipedia; Recall@k measures documents or passages containing a gold answer, passage results report Recall@1, Recall@10, and Recall@100, and Google API document Recall@10 is omitted for high cost.Contriever and BM25 are usually used for passage retrieval; Google API is a document-retrieval baseline restricted to Wikipedia.Source: §4.1 · p. 5, §4.1.1 · p. 5, Table 1 · p. 5

4.1.1 Document Retrieval

  • Document Recall@1 improves over the strongest baseline by 20.4%, 11.2%, and 13.2% in zero-shot WebQ, NQ, and TriviaQA, and by 24.9%, 12.8%, and 16.7% in few-shot, respectively.The comparison is reported across all three datasets; Table 1 reports document retrieval by Recall@k.Source: §4.1.1 · p. 5, Table 1 · p. 5
  • As the number of generated URLs m rises from 1 to 10, valid generations remain above 68%, but their rate falls, indicating diminishing returns.Recall scores converge as m increases, and experiments do not test values above 10.Source: §4.1.1 · p. 6, Figure 4 · p. 6

4.1.2 Passage Retrieval

  • For passage Recall@1, zero-shot LLM-URL improves over the strongest baseline by 16.2%, 5.3%, and 1.1% on WebQ, NQ, and TriviaQA; few-shot gains are 16.8%, 11.8%, and 6.3%, respectively.For Recall@100, zero-shot improves by 5.0% on WebQ but trails the best baseline by 1.7% on NQ and 0.4% on TriviaQA; few-shot is slightly better on WebQ and TriviaQA but worse on NQ.Source: Table 2 · p. 5, §4.1.2 · pp. 6–7
  • Despite being limited to only the passages from 10 documents, LLM-URL performs better than baseline methods for smaller k and performs as well as baseline methods for higher values of k.In the zero-shot setting, LLM-URL improves relative Recall@100 by 5.0% for WebQ and performs worse than the best baseline on NQ and TriviaQA by 1.7% and 0.4%, respectively.Source: §4.1.2 · pp. 6–7
  • Against DSI and SEAL, LLM-URL trails the best baseline on passage Recall@1 by 8.8% in zero-shot and 3.1% in few-shot, and on Recall@10 by 18.7% and 18.4%, respectively.Table 3 reports Recall@1 and Recall@10, with BM25 used to rank LLM-URL passages.Source: Table 3 · p. 6, §4.1.2 · p. 7

4.2 Open-Domain Question Answering

The section reports open-domain QA exact-match results for LLM-URL paired with InstructGPT. Performance is stronger than the best baseline on WebQ, near parity on TriviaQA, and lower on NQ, while retrieval is more effective for common entities.

4.2 Open-Domain Question Answering

  • In zero-shot QA, LLM-URL+InstructGPT beats the best baseline by 13.3% on WebQ and 1.3% on TriviaQA, but trails it by 39.5% on NQ.The reported metric is exact match (EM), based on normalized generated answers matching an acceptable answer.Source: §4.2 · p. 7
  • In few-shot QA, LLM-URL+InstructGPT beats the best baseline by 16.9% on WebQ and 2.3% on TriviaQA, but trails it by 37.4% on NQ.The authors characterize WebQ performance as significantly better, TriviaQA as on par, and NQ as slightly worse than existing methods.Source: §4.2 · p. 7
  • LLM-URL has much higher Recall@1 for questions about common entities than for uncommon entities; common means an entity among Wikipedia’s top one million most common.The common-versus-uncommon comparison is shown in Figure 5.Source: Figure 5 · p. 7
  • The authors describe LLM-URL as a promising retrieval solution for knowledge-intensive tasks requiring little to no training data.EM measures exact string match with a correct answer.Source: §4.2 · p. 7

4.3 Discussions

The discussion highlights LLM-URL’s real-time retrieval from changing sources, its weaker recall for questions about uncommon entities, and a case study where it retrieves answer-bearing documents missed by BM25 and Contriever.

4.3.1 Time Sensitive Queries

  • LLM-URL retrieves documents from their live sources, allowing time-sensitive answers when the source stays current and the URL remains unchanged.The discussion contrasts this with methods that need re-encoding or retraining to reflect changed document content.Source: §4.3.1 · p. 7

4.3.2 Frequent Entities Analysis

  • Across Web Questions, Natural Questions, and TriviaQA, recall for common-entity questions is many times greater than for the rest of the dataset.The analysis divides questions according to whether the answer entity appears among Wikipedia’s top one million common entities.Source: §4.3.2 · pp. 7–8
  • Retrieved Wikipedia documents average 10k words versus 644 words for an average Wikipedia article, which the authors believe reflects greater detail on common entities.The authors relate this discrepancy to LLM-URL’s stronger recall on common-entity questions.Source: §4.3.2 · p. 8

4.3.3 Case Study

  • LLM-URL retrieves seven answer-bearing documents in the jellyfish case, while BM25 and Contriever retrieve none containing the answer.The top retrieved document is titled “jellyfish,” and the ranked passages let InstructGPT produce the gold answer.Source: §4.3.3 · p. 8
  • Generating each URL conditioned on previously generated URLs yields complementary documents and a more informative evidence context for open-domain question answering.LLM-URL locates documents related to both the question and its answer, unlike the baselines in this example.Source: §4.3.3 · pp. 8–9

5 Conclusion and Future Work

The conclusion reports that LLMs can generate URLs whose corresponding documents often answer open-domain questions, and that passage ranking can reduce unnecessary evidence while retaining high recall. Future work includes tuning demonstrations and prompts for particular retrieval tasks.

5 Conclusion and Future Work

  • With a few query-URL demonstrations, LLMs generated URLs whose corresponding documents contained answers to nearly 90% of WebQ questions.The conclusion also reports that ranking passages after splitting retrieved documents filtered unnecessary passages while retaining high recall and outperforming baselines by a significant margin.Source: §5 · p. 9
  • Whether few-shot demonstrations can be tuned for specific retrieval tasks remains an open question.The authors point to prompt tuning in Promptagator as related work showing significant performance improvements.Source: §5 · p. 9
  • The authors identify per-question prompt tuning as a potential retrieval improvement, alongside selecting diverse demonstrations through clustering.Prior work cited in the passage reports improved retrieval and downstream QA performance from clustering-based demonstration selection.Source: §5 · p. 9

Limitations

The approach has limitations in updating its knowledge and adapting to new domains, and practical use may be cumbersome because it requires large models, slow web requests, and document processing.

Limitations

  • Adding newly learned information or adapting to a new domain may require retraining because the approach relies on the language model to contain that knowledge.This limits the approach’s ability to swap in recent documents or quickly adapt to a new downstream task.Source: Limitations · p. 9
  • When tasked with generating 10 URLs, LLM-URL may generate only 6 or 7 that link to valid documents; its very large language models, slow web requests, and document processing may make it cumbersome in practice.Source: Limitations · p. 9

Related papers

Loading 2305.09612v1…