PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Retrieval: End-to-End Information Retrieval with One Large Language Model

Qiaoyu Tang, Jiawei Chen, Zhuoqun Li, Bowen Yu, Yaojie Lu, Cheng Fu, Haiyang Yu, Hongyu Lin, Fei Huang, Ben He, Xianpei Han, Le Sun, Yongbin Li

arXiv:2403.00801v2cs.IRcs.AIcs.CL

TL;DR

Existing IR systems often use LLMs only in separate components, limiting integration across the retrieval process. Self-Retrieval unifies corpus internalization, passage generation, and reranking in one LLM; it outperforms existing retrieval methods across evaluated tasks and improves RAG performance.

  • Problem

    IR systems typically use isolated components, limiting knowledge sharing and deep integration of LLM capabilities across the system.

    Source: §1 · pp. 1–2

  • Method

    Self-Retrieval internalizes the corpus through self-supervised learning, generates passages using constrained decoding, and reranks them through self-assessment in one LLM.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 4

  • Results

    Self-Retrieval outperforms sparse, dense, and generative methods across document- and passage-level retrieval tasks and improves RAG performance; on NQ passage retrieval, its MRR@5 is 69.45.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    Experiments on RAG demonstrate the system’s compatibility with LLMs and superior downstream performance in that setting.

    Source: §1 · p. 2

    Evaluation is limited to 200K Wikipedia documents and 3M passages, and retrieval is less efficient than sparse or dense methods.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The rise of large language models (LLMs) has significantly transformed both the construction and application of information retrieval (IR) systems. However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being constructed independently of LLMs. This separated architecture restricts knowledge sharing and deep collaboration between them. In this paper, we introduce Self-Retrieval, a novel end-to-end LLM-driven information retrieval architecture. Self-Retrieval unifies all essential IR functions within a single LLM, leveraging the inherent capabilities of LLMs throughout the IR process. Specifically, Self-Retrieval internalizes the retrieval corpus through self-supervised learning, transforms the retrieval process into sequential passage generation, and performs relevance assessment for reranking. Experimental results demonstrate that Self-Retrieval not only outperforms existing retrieval approaches by a significant margin, but also substantially enhances the performance of LLM-driven downstream applications like retrieval-augmented generation.

1 Introduction

Self-Retrieval addresses the fragmentation of IR systems and LLMs by integrating indexing, retrieval, and reranking within one LLM. The paper reports strong results across retrieval benchmarks and improved downstream RAG performance.

1 Introduction

  • Pipeline-based IR systems isolate components, limiting knowledge sharing and deep integration of LLM capabilities across the system.The paper notes that this fragmentation can also make implementations complex and potentially suboptimal.Source: §1 · pp. 1–2
  • A single LLM internalizes the corpus through self-supervised learning, generates relevant passages with constrained decoding, and reranks them through self-assessment.For RAG, the model can also integrate the reader and generate answers after retrieval.Source: §1 · p. 2
  • Self-Retrieval substantially outperforms sparse, dense, and generative retrieval methods on document- and passage-level tasks, while also improving retrieval-augmented generation performance.The introduction reports evaluation on NQ, TriviaQA, and MS MARCO, and progressively better performance with larger LLMs.Source: §1 · p. 2
  • The authors present corpus internalization as a way to memorize, organize, and retrieve learned documents during pre-training, with potential for more transparent and trustworthy text generation.They also point to compatibility with LLMs’ internal knowledge as a potential advantage for LLM-based applications.Source: §1 · p. 2

2 Related Work

Prior work uses LLMs to improve individual IR components, while Self-Retrieval combines retrieval and answer generation in one end-to-end LLM-driven framework. The paper distinguishes its approach from dense and generative retrieval through its use of generated passage content rather than vector or identifier proxies.

LLM for IR

  • LLMs have been used for query rewriting, retrieval, and reranking, but Self-Retrieval integrates the full IR procedure in one LLM without external components.Examples include pseudo-document generation [46], conversational query rewriting [15], pseudo-query generation [6] [17], and direct generative retrieval [42] [5].Source: §2 · p. 3

Dense retrieval

  • Dense retrieval matches query and document vectors, a representation that the paper says constrains LLM language understanding and semantic inference.Prior work explores improved loss functions [45], multi-vector methods [38], synthetic queries [47], and large query-document collections [30].Source: §2 · p. 3

Generative retrieval

  • Generative retrieval commonly generates document identifiers, with methods spanning identifier design, query generation, architecture, and continual learning.GENRE introduced constrained generation of entity names [7], and DSI extended the approach to document-specific identifiers [42].Source: §2 · p. 3
  • Unlike generate-then-read [52] and UniGen [23], Self-Retrieval uses constrained decoding for accurate document generation and can retrieve and answer in one turn.The paper also distinguishes its generated document content from identifier-based retrieval and notes that it integrates reranking and answer generation.Source: §2 · p. 3

3 Self-Retrieval

Self-Retrieval places indexing, passage generation, and relevance assessment within one LLM. Its training and inference procedures combine these functions and can extend to answer generation for RAG.

3 Self-Retrieval

  • Self-Retrieval integrates indexing, retrieval, and reranking into one LLM, which internalizes the corpus, generates relevant passages, and assesses their relevance.The model’s logits support reranking generated passages.Source: §3 · pp. 3–4

3.1 Indexing: Internalize the Corpus

  • For indexing, each sentence in a passage is used to train the LLM to generate that source passage autoregressively, internalizing corpus information in its parameters.The paper frames this sentence-to-passage objective as retrieval-like and aligned with continued pre-training.Source: §3.1 · p. 4

3.2 Retrieval: Generate Relevant Passage through Constrained Decoding

  • For retrieval, the model generates a document title and relevant passage from the query, while trie-based constrained decoding restricts output to corpus continuations.Generation stops once enough tokens uniquely identify a document, after which the full document is appended to the context.Source: §3.2 · p. 4

3.3 Reranking: Assess the Relevance

  • For reranking, the model self-assesses whether each passage can answer the query, and combines title-generation and assessment scores to order passages.Training uses gold passages as positives and samples negatives from the same and different documents.Source: §3.3 · pp. 4–5

3.4 Training & Inference

  • Training combines self-supervised corpus instances with query-passage and relevance-assessment examples, while inference generates titles, passages, then reranks them.Appending a gold answer after assessment lets the model also generate answers as part of a RAG system.Source: §3.4 · p. 5

4 Experimental Results

Across passage- and document-retrieval benchmarks, Self-Retrieval outperforms existing methods, and it also improves retrieval-augmented generation performance. Its results further show benefits from larger models, comparable corpus-size degradation to BGE-FT, and strong reranking performance.

4.2 Main Results

  • Self-Retrieval 3B improves MRR@5 over fine-tuned BGE by 5.46 on NQ and 5.07 on TriviaQA.On NQ, Self-Retrieval's MRR@5 is 69.45, versus 57.03 for GritLM and 50.20 for DSI-XXL.Source: Table 1 · p. 6, §4.2 · p. 7
  • Against GenRet on NQ320K, Self-Retrieval improves Hits@1 by 5.2, Hits@10 by 3.8, and MRR@100 by 4.8 points.It achieves these results using only the original training set, while other methods commonly use query generation to augment training data.Source: §4.2 · p. 7, Table 2 · p. 7
  • On MS MARCO, Self-Retrieval performs comparably to GenRet and significantly outperforms other baselines.The experiments address the non-Wikipedia setting and the lack of document titles by using automatically generated titles.Source: §4.2 · p. 7, Table 3 · p. 7
  • Removing indexing, document titles, or self-assessment substantially degrades performance on NQ and TriviaQA.Indexing supports performance on unseen passages; titles guide relevant content generation, and self-assessment refines retrieved passages for ranking.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Performance on Retrieval-Augmented Generation

  • On NQ and TriviaQA subsets with 10K and 40K documents, Self-Retrieval significantly outperforms a BGE-FT retrieval and fine-tuned-reader baseline in Exact Match.The comparison uses the top-1 retrieved passage as context and evaluates across different model scales.Source: §4.3 · p. 8, Table 5 · p. 8

4.4 Detailed Analysis

  • Within each model series, increasing model size consistently improves Hits@1 and Hits@5 on NQ.The evaluated backbones include StableLM, Llama2, and Qwen-1.5.Source: §4.4 · p. 8, Figure 2 · p. 9
  • As corpus size grows from 10K to 200K documents, Self-Retrieval's performance declines at a rate comparable to BGE-FT's, with the rate of decline gradually diminishing.The analysis scales from 290K to 3M passages on NQ and TriviaQA.Source: §4.4 · p. 9, Figure 4 · p. 9
  • On TriviaQA top-100 passage sets retrieved by DPR-FT, SEAL, and GritLM, reranking improves ranking performance, and Self-Retrieval's reranker beats fine-tuned BGE-Reranker in most scenarios.Reranking performance is measured using MRR@5.Source: §4.4 · p. 9, Figure 3 · p. 9

5 Conclusion

The paper presents Self-Retrieval as an end-to-end LLM architecture that unifies indexing, retrieval, and reranking. It reports strong performance across retrieval benchmarks and application scenarios, while identifying reliability and trustworthiness as future-work goals.

5 Conclusion

  • Self-Retrieval unifies indexing, retrieval, and reranking within a single LLM.The approach internalizes the corpus, generates relevant content, and performs self-assessment within the unified framework.Source: §5 · p. 10
  • The unified framework facilitates knowledge sharing and deep collaboration among IR components.Source: §5 · p. 10
  • The paper reports strong performance across various retrieval benchmarks and application scenarios.Source: §5 · p. 10
  • Future work aims to further enhance the reliability and trustworthiness of LLM generation.Source: §5 · p. 10

Limitations

The current evaluation uses a bounded corpus and Self-Retrieval is less efficient than sparse or dense retrieval methods. Larger, noisier collections and incremental corpus expansion remain open directions.

Limitations

  • Evaluation is limited to 200K Wikipedia documents and 3M passages; larger, noisier collections still need testing.Source: Limitations · p. 10
  • Self-Retrieval has lower retrieval efficiency than sparse or dense methods, potentially limiting use in specialized knowledge systems.Source: Limitations · p. 10
  • Incremental learning and dynamic corpus expansion remain future research directions.Source: Limitations · p. 10

A Dataset Statistics

The dataset-statistics material describes NQ and TriviaQA and defines the dataset-count abbreviations used in Table 6.

A Dataset Statistics

B Baselines

The baselines span sparse, dense, and generative retrieval approaches, including fine-tuned and reranker-enhanced variants.

B Baselines

  • Sparse baselines include BM25, based on probabilistic relevance and term-frequency statistics, and DocT5Query, which expands documents with generated queries.Source: B Baselines · p. 15
  • Dense baselines include DPR [19], BGE, and GritLM, alongside fine-tuned DPR and BGE variants.Source: B Baselines · p. 15
  • Other dense baselines include Sentence-T5 [31], GTR-XL [32], and OpenAI’s text-embedding-ada-002.Source: B Baselines · p. 15
  • Generative baselines include DSI [42], DSI-QG [59], and SEAL [5], which use document identifiers or constrained identifier generation.Source: B Baselines · p. 15
  • Additional generative baselines are NCI [49], Ultron [55], DynamicRetriever [54], and GenRet [39].Source: B Baselines · pp. 15–16

C Ablation on Chunk Size

On NQ, Self-Retrieval was compared with strong baselines at a 100-word chunk size, complementing the main experiments’ 200-word setting. It significantly outperformed the baselines with both chunk sizes.

C Ablation on Chunk Size

  • Self-Retrieval significantly outperformed the baselines with both 100- and 200-word chunk sizes on NQ.The 100-word setting complements the main experiments’ 200-word setting.Source: C Ablation on Chunk Size · p. 16, Table 7 · p. 16
  • The additional comparison used 100-word chunks and strong baselines on NQ.The main experiments used a 200-word chunk size.Source: C Ablation on Chunk Size · p. 16
  • Table 7 presents retrieval performance with a chunk length of 100 words.Source: Table 7 · p. 16

D Full Comparison with Retriever-Reranker Pipeline

Self-Retrieval was evaluated against two-stage retriever-reranker pipelines built from state-of-the-art retrievers and three reranking approaches. It achieved superior performance over most combinations.

D Full Comparison with Retriever-Reranker Pipeline

  • Table 8 compares Self-Retrieval with traditional two-stage retriever-reranker pipelines.Source: Table 8 · p. 16
  • Self-Retrieval achieved superior performance compared with most two-stage retriever-reranker combinations.The comparison included BGE, GTR, GritLM, and DSI-XL retrievers paired with three reranking approaches.Source: D Full Comparison with… · p. 16, Table 8 · p. 16
  • The evaluated pipelines paired BGE, GTR, GritLM, or DSI-XL with a BGE reranker, fine-tuned BGE reranker, or RankGPT.Source: D Full Comparison with… · p. 16

E Efficiency Analysis

On NQ using an NVIDIA A100-80G GPU, Self-Retrieval required slightly more computational resources than DSI but offered performance benefits. Its beam-size and memory comparisons illustrate different quality-efficiency trade-offs.

E Efficiency Analysis

  • Beam size 10 for Self-Retrieval achieved significantly higher H@5 scores than DSI-XL with beam size 100.This illustrates a retrieval-quality and computational-efficiency trade-off.Source: E Efficiency Analysis · p. 17, Table 9 · p. 17
  • Self-Retrieval used 30MB of memory versus 444MB for SEAL, using a lightweight trie rather than SEAL’s FM-Index post-processing mechanism.Source: E Efficiency Analysis · p. 17
  • Self-Retrieval required slightly higher computational resources than DSI while providing notable performance benefits.Source: E Efficiency Analysis · p. 17
  • The efficiency analysis used the NQ dataset and an NVIDIA A100-80G GPU.Source: E Efficiency Analysis · p. 17
  • The authors state that optimization techniques and hardware advancements could further benefit Self-Retrieval’s efficiency.Examples include quantization and attention acceleration.Source: E Efficiency Analysis · p. 17
Loading 2403.00801v2…