PaperlayerView exact revision on arXiv

Source-linked AI summary

RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation

Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yongkang Wu, Zhonghua Li, Qi Ye, Zhicheng Dou

arXiv:2412.11919v1cs.CLcs.AIcs.IR

TL;DR

LLMs can hallucinate when internal memory leaves them struggling with long-tail or newly updated knowledge. RetroLLM unifies retrieval and generation through constrained corpus-evidence generation, and outperforms traditional and complex RAG strategies across five open-domain QA datasets.

  • Problem

    LLMs can hallucinate because reliance on internal memory leaves them struggling with long-tail or newly updated knowledge.

    Source: §1 · p. 1

  • Method

    RetroLLM unifies retrieval and answer generation in autoregressive decoding, using hierarchical FM-Index constraints and forward-looking constrained decoding to generate relevant corpus evidence.

    Source: §1 · pp. 1–2

  • Results

    RetroLLM demonstrates superior performance to traditional RAG and complex RAG strategies across five open-domain QA datasets, including in-domain and out-of-domain tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework avoids a separate embedding model and reduces redundant input while allowing the model to decide how much evidence to retrieve and when to answer.

    Source: §1 · p. 2, §6 · p. 9

    Although faster than most complex RAG methods in query latency, RetroLLM is slightly slower than Naive RAG because generated evidence adds output tokens.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) exhibit remarkable generative capabilities but often suffer from hallucinations. Retrieval-augmented generation (RAG) offers an effective solution by incorporating external knowledge, but existing methods still face several limitations: additional deployment costs of separate retrievers, redundant input tokens from retrieved text chunks, and the lack of joint optimization of retrieval and generation. To address these issues, we propose \textbf{RetroLLM}, a unified framework that integrates retrieval and generation into a single, cohesive process, enabling LLMs to directly generate fine-grained evidence from the corpus with constrained decoding. Moreover, to mitigate false pruning in the process of constrained evidence generation, we introduce (1) hierarchical FM-Index constraints, which generate corpus-constrained clues to identify a subset of relevant documents before evidence generation, reducing irrelevant decoding space; and (2) a forward-looking constrained decoding strategy, which considers the relevance of future sequences to improve evidence accuracy. Extensive experiments on five open-domain QA datasets demonstrate RetroLLM's superior performance across both in-domain and out-of-domain tasks. The code is available at \url{https://github.com/sunnynexus/RetroLLM}.

1 Introduction

LLMs can struggle with long-tail or newly updated knowledge, while existing RAG methods have deployment, input-token, flexibility, and joint-optimization limitations. RetroLLM unifies retrieval and generation, and the paper reports superior performance across in-domain and out-of-domain QA tasks.

1 Introduction

  • Existing RAG methods face retriever deployment costs, redundant input tokens, inflexible chunk granularity, and limited joint optimization with generators.Generative retrieval still requires mapping DocIDs back to document content before answer generation.Source: §1 · p. 1
  • RetroLLM generates evidence and the final answer in one autoregressive process, eliminating the need for a separate retriever.It uses FM-Index constraints to retrieve fine-grained evidence.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Constrained evidence generation can falsely prune relevant sequences when early prefixes mislead decoding, a challenge linked to large corpora and uncertain continuation relevance.The proposed strategies target both the large early prefix search space and difficulty predicting relevance from short prefixes.Source: §1 · p. 2
  • Hierarchical FM-Index constraints generate corpus-level clues to identify candidate documents, reducing irrelevant decoding paths before evidence generation.Forward-looking constrained decoding uses future-window relevance to guide evidence generation.Source: §1 · p. 2
  • Across five open-domain QA datasets, RetroLLM outperforms traditional and complex RAG strategies on in-domain and out-of-domain tasks.The experiments test different base LLMs and parameter sizes.Source: §1 · p. 2

2 Preliminary

The preliminary section formulates RAG as constrained evidence generation followed by answer generation. It also reports that constraining search to relevant documents mitigates false pruning and improves evidence generation accuracy.

2.1 Task Formulation

  • RAG is formulated as two stages: generating corpus-constrained evidence from a query, then generating an answer based on that evidence.The evidence sequence is produced under an FM-Index built on the corpus; the answer generation stage follows the retrieved evidence.Source: §2.1 · pp. 2–3

2.2 Empirical Study

  • Corpus-level FM-Index constraints can falsely prune relevant sequences during beam search, as initially relevant prefixes may later lead to irrelevant documents.This problem motivates restricting constraints to relevant documents.Source: §2.2 · p. 3
  • Restricting FM-Index constraints to relevant documents substantially reduces the early decline in prefix relevance and improves evidence accuracy across beam sizes.The corpus-level approach shows a particularly severe relevance decline within the first 13 tokens.Source: §2.2 · p. 3, Figure 2 · p. 3

3 RetroLLM: Retrieval in Generation

RetroLLM folds retrieval into autoregressive generation, using corpus-level clues to select candidate documents before generating constrained evidence and an answer. It guides evidence decoding with query-relevant future windows.

3.1 Hierarchical FM-Index Constraints

  • Corpus-level FM-Indexes constrain clue generation, while document-level FM-Indexes constrain evidence generation to selected documents.The framework builds a global index over the corpus and an index for each document.Source: §3.1 · p. 3

3.2 Clue Generation and Document Scoring

  • Generated clues are weighted toward phrases that are rare across the corpus and documents, then used to rank candidate documents.The scoring uses clue weights and term frequency; generated clues typically contain only 1–3 key phrases.Source: §3.2 · p. 4
  • Auxiliary lexical clues supplement generated clues, and weighted reciprocal rank fusion combines their document rankings into the candidate set.The auxiliary clues are selected from a sparse lexical model’s vocabulary.Source: §3.2 · p. 4

3.3 Forward-Looking Constrained Evidence Generation

  • The evidence decoder finds and merges clue-containing windows in candidate documents, scores their query relevance, and adjusts token logits toward relevant future sequences.Merged windows are capped at length lmax, and λ controls the weight of future relevance in the logits.Source: §3.3 · p. 5

3.4 Answer Generation

  • After sufficient evidence is generated, the model transitions to free generation to produce the answer to the original query.Source: §3.3 · p. 5, §3.4 · p. 5

3.5 Training of RetroLLM

  • Training simulates inference to construct clue, evidence, and answer targets for joint retrieval-generation learning, using next-token loss while masking 80% of middle evidence tokens.The target data selects relevant evidence, including evidence that contains the answer and is judged to answer the query.Source: §3.5 · p. 5

4 Experimental Settings

Experiments evaluate RetroLLM on five open-domain QA datasets using accuracy, F1, and token-count metrics, with comparisons across RAG baselines and implementation variants. Results report lower token use, retrieval and ablation patterns, and effects of backbone size and evidence quantity.

4.1 Datasets and Evaluation Metrics

  • The evaluation covers NQ, TriviaQA, PopQA, HotpotQA, and 2WikiMultiHopQA, using accuracy, F1, and token count to assess answers and token consumption.The first three are single-hop datasets and the last two are multi-hop datasets.Source: §4.1 · p. 6

4.2 Baselines

  • Baselines include direct-generation models and Naive RAG plus complex RAG methods, with RAG baselines using E5-baseen and LLMs sized 7B or 8B.The direct-generation set includes Llama3-8B, Mistral-7B, Qwen2.5-7B, and ChatGPT.Source: §4.2 · p. 6

4.3 Implementation Details

  • Implementation uses a December 20, 2018 Wikipedia dump and Mistral-7B-Instruct, with maximum evidence of five for single-hop and ten for multi-hop QA.Training uses LoRA for three epochs; SPLADE-v3 expands clues and BGE-reranker-base scores window sequences.Source: §4.3 · p. 6

4.4 Experimental Results

  • RetroLLM uses approximately 2.1x fewer tokens than Naive RAG and 6x fewer than Iter-RetGen on average, with an improved cost-efficiency balance.Input tokens are lower, while output tokens are slightly higher because RetroLLM generates fine-grained evidence.Source: §4.4 · pp. 7–8
  • For multi-hop QA, RetroLLM outperforms other retrieval methods on R@1 and R@5 using 4.24 passages on average; on single-hop QA, its R@5 trails E5 with 3.29 versus 5 passages.Single-hop R@1 is superior; the lower R@5 is attributed to flexible retrieval and fewer passages.Source: §4.4 · p. 7, Table 2 · p. 7
  • Removing future windows, clue generation, or clue expansion reduces performance; basic constrained generation performs worst, while removing constraints also lowers performance.The basic constrained method falls below the unconstrained variant, illustrating the false-pruning trade-off.Source: §4.4 · p. 7, Table 3 · p. 7
  • Performance steadily improves with parameter size across Mistral, Llama3, and Qwen2.5 models ranging from 1B to 14B; Mistral generally leads Llama3, followed by Qwen2.5.The reported average is across five datasets.Source: §4.4 · p. 7, Figure 4 · p. 7
  • For multi-hop QA, performance stabilizes around six evidence pieces because additional evidence can add both useful and distracting information.For retrieval up to one to five pieces, performance continues to improve as evidence quantity increases.Source: §4.4 · p. 8, Figure 5 · p. 8

5 Related Work

Related work covers retrieval-augmented generation and generative retrieval, including their training approaches and integration boundaries. Existing generative retrieval systems may still require mapping document identifiers back to documents before generation.

Retrieval-augmented Generation

  • RAG commonly uses separate dense retrieval, while approaches train the retriever and generator jointly or separately, or prompt generators with retrieved context.Joint training faces architectural differences and document-index update challenges; GritLM uses separate attention mechanisms and OneGen still takes passage chunks as input.Source: §5 · p. 8

Generative Retrieval

  • Prior unified approaches retain integration boundaries: GritLM has separate attention mechanisms, OneGen uses retrieved passage chunks, and some generative retrieval systems require DocID-to-document mapping.These examples illustrate differing ways retrieval and downstream generation remain separated.Source: §5 · p. 8
  • Generative retrieval generates document identifiers without traditional indices, but many methods focus on retrieval rather than downstream tasks, and UniGen and CorpusLM still map identifiers to documents before generation.RICHES streamlines this step but does not solve false pruning, which leads to suboptimal downstream performance.Source: §5 · pp. 8–9

6 Conclusion

The conclusion presents RetroLLM as a unified retrieval-and-generation framework that generates corpus evidence under FM-Index constraints. It reports improved evidence accuracy strategies and superior performance on open-domain QA tasks.

6 Conclusion

  • RetroLLM unifies retrieval and generation, allowing language models to generate evidence directly from a corpus under FM-Index constraints.The framework eliminates separate retrievers and reduces redundant input.Source: §6 · p. 9
  • Hierarchical FM-Index constraints and forward-looking decoding are proposed to improve evidence accuracy by focusing generation on relevant information.Source: §6 · p. 9
  • Experiments report that RetroLLM outperforms existing methods on open-domain QA tasks.Source: §6 · p. 9

7 Limitations

RetroLLM still requires clue expansion, is slightly slower than Naive RAG, and currently unifies evidence retrieval with answer generation; the appendix describes FM-Index as a compressed substring-search structure.

7 Limitations

  • Clue expansion remains necessary for robust generated clues, preventing fully end-to-end optimization of RAG.Future work could remove this requirement.Source: §7 · p. 9
  • RetroLLM is slightly slower than Naive RAG because generated evidence adds output tokens, despite outperforming most complex RAG methods in query latency.The paper suggests using a smaller model for constrained evidence generation and a larger model for answer generation.Source: §7 · p. 9
  • RetroLLM currently unifies evidence retrieval and final-answer generation, leaving other reasoning processes for future integration.Proposed additions include query intent analysis, question decomposition, retrieval-necessity assessment, evidence relevance judgment, and source citations.Source: §7 · p. 9

A The FM-Index

  • The FM-Index is a space-efficient structure for indexing large text corpora and enabling fast substring search.It combines the Burrows-Wheeler Transform with run-length encoding.Source: A The FM-Index · p. 15

A.1 Data Structure

This section explains how the FM-Index supports substring search and how RetroLLM addresses false pruning in constrained evidence decoding by narrowing prefix choices and considering future relevance.

A.1 Data Structure

  • The FM-Index stores the Burrows-Wheeler Transform’s first and last columns, with auxiliary structures supporting efficient rank and select operations.The BWT is formed by sorting cyclic rotations and taking their last characters.Source: §A.1 · p. 15

A.2 Supporting Functions

  • Backward search locates substring occurrences by iteratively refining an interval; count returns its size, locate maps it to text positions, and extract reconstructs the substring.Backward search, count, and locate each run in O(k log |V|); extract runs in O(k).Source: §A.2 · pp. 15–16

A.3 Examples

  • For S = "banana$", backward search for P = "ana" ends at interval [3, 4]; its L-column characters ’n’ and ’b’ map to following characters ’n’ and ’$’ in S.Source: §A.3 · p. 16

B False Pruning in Constrained Decoding

B.2 What Causes False Pruning?

  • Two causes are abundant initial prefix choices in large corpora and the model’s inability to anticipate content beyond its current token decision.Source: §B.2 · p. 17

B.3 How to Mitigate False Pruning?

  • Clue generation narrows decoding to a candidate-document subset, while future windows and relevance scores guide evidence generation toward relevant text.Source: §B.3 · p. 17
  • Set-based retrieval methods generate term sets or document identifiers, but are not applicable when the task is to generate meaningful evidence directly.Source: §B.3 · p. 17

C.1 Details of Datasets

The appendices describe the QA datasets and corpus, baseline and constrained-generation configurations, and RetroLLM’s FM-Index implementation.

C.1 Details of Datasets

  • Evaluation uses NQ, TriviaQA, and PopQA for single-hop QA, and HotpotQA and 2WIKI for multi-hop QA.Source: §C.1 · p. 17

C.2 Statistics

  • The Wikipedia retrieval corpus contains 21,015,324 passages and 3,232,907 documents.The passage also reports dataset training and test-set sizes.Source: §C.2 · p. 17

D.1 Implementation Details for Baselines

  • RAG baselines use FlashRAG defaults; Self-RAG uses its trained Llama2-7B checkpoint, while other baselines use Mistral-7B-Instruct.Source: §D.1 · p. 18

D.2 Implementation Details for Naive Constrained Generation

  • Naive constrained beam generation uses five beams, five beam groups, and a diversity penalty of 1.0; answers are freely generated from the top-ranked sequence.Beam groups and the penalty encourage diversity among generated sequences.Source: §D.2 · p. 18

D.3 Implementation Details for RetroLLM

  • RetroLLM’s implementation comprises FM-Index building, training, and inference, run on eight NVIDIA A800-80GB GPUs and a 64-core Intel Xeon CPU.Source: §D.3 · p. 18

D.3.1 FM-Index Building

  • The FM-Index is implemented with SDSL-lite and C++ functions for prefix locating, allowed-token lookup, counting, and sequence extraction, wrapped for Python with SWIG.Source: §D.3.1 · p. 18

D.3.2 Training

Training data construction simulates inference, while optimization emphasizes key evidence tokens; inference proceeds through clue generation, constrained evidence generation, and unconstrained answer generation.

D.3.2 Training

  • Training data simulates inference by retrieving clues and documents, selecting evidence judged answer-relevant, and extracting target clues from queries and evidence.Evidence must contain the answer and be labeled relevant by an LLM.Source: §D.3.2 · p. 18
  • Training uses sequence-to-sequence loss and LoRA, excluding the middle 80% of evidence tokens from training to focus on evidence beginnings and endings.Special-token embeddings are also trained.Source: §D.3.2 · pp. 18–19

D.3.3 Inference

  • Inference generates corpus-constrained clues, expands them with SPLADE-v3, generates evidence using document-level FM-Index constraints and future-window relevance, then generates answers without constraints.The maximum evidence count is five for single-hop QA and ten for multi-hop QA.Source: §D.3.3 · p. 19

E Detailed Experimental Results

  • Detailed experiments cover retrieval performance, performance across base models, and performance with different numbers of generated evidence.The supplied captions identify tables comparing sparse, dense, and generative retrieval; base models; and evidence counts, without giving their results.Source: E Detailed Experimental Results · p. 19, Table 6 · p. 19, Table 7 · p. 19

E.1 Analysis of Retrieval Performance

RetroLLM retrieves with strong single- and multi-hop accuracy, and its performance varies with backbone size and evidence quantity. Its case examples show clue generation followed by evidence generation.

E.1 Analysis of Retrieval Performance

  • On PopQA, RetroLLM achieves 57.0% R@1 versus E5’s 51.7%; it averages 2.80 passages on TriviaQA and 3.20 on NQ, versus 5 for baselines.Source: §E.1 · p. 20
  • On HotpotQA, RetroLLM reaches 35.6% R@1 versus E5’s 32.3% and SPLADEv3’s 32.9%; on 2WIKI it reaches 23.0% versus E5’s 21.6%.On 2WIKI it uses an average of 4.40 passages versus the baseline’s 5.Source: §E.1 · p. 20
  • Naive constrained beam search reaches only 13.1% R@1 on NQ versus RetroLLM’s 51.6%, and 23.0% on TriviaQA versus 61.1%.Source: §E.1 · p. 20

E.2 Impact of Different Base Models

  • Larger backbones generally improve RetroLLM’s performance; Llama3 NQ accuracy rises from 54.4% with Llama3.2-1B to 59.2% with Llama3-8B.Mistral generally outperforms Llama3, which generally outperforms Qwen2.5; Mistral-7B reaches 61.6% accuracy on NQ and 74.3% on TriviaQA.Source: §E.2 · pp. 20–21

E.3 Impact of Generated Evidence Quantity

  • On NQ, accuracy rises from 42.2% with one evidence piece to 61.5% with five; on TriviaQA, it rises from 59.3% to 74.6%.Source: §E.3 · p. 21
  • On HotpotQA, accuracy rises from 50.6% with one evidence piece to 67.4% with six, then yields diminishing returns at 68.5% with ten.The passage attributes the plateau to useful and distracting information in additional evidence.Source: §E.3 · p. 21

F.1 Examples from RetroLLM

  • RetroLLM examples show a two-stage retrieval process: clues identify relevant terms, then evidence generation retrieves snippets that support the answer.For multi-hop QA, the examples show snippets forming a coherent information chain.Source: §F.1 · p. 21, F Case Study · p. 21

F.2 Comparing RetroLLM with Naive Beam Search Method

The comparison illustrates how corpus-level constrained beam search can produce irrelevant evidence, while RetroLLM narrows the document subset with clues before forward-looking evidence generation.

F.2 Comparing RetroLLM with Naive Beam Search Method

  • Tables 11–12 compare NQ outputs from both methods; Table 12 marks correct answers in green and incorrect answers in red.Source: §F.2 · p. 21, Table 12 · p. 25
  • Naive constrained beam search produces irrelevant or incoherent evidence and incorrect answers, while RetroLLM’s clue-first, forward-looking approach reduces false pruning and improves evidence retrieval accuracy.Source: §F.2 · p. 21
  • RetroLLM first generates clues to narrow the relevant document subset, then performs forward-looking constrained evidence generation within that subset.Source: §F.2 · p. 21
Loading 2412.11919v1…