PaperlayerView exact revision on arXiv

Source-linked AI summary

LLatrieval: LLM-Verified Retrieval for Verifiable Generation

Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, Xipeng Qiu

arXiv:2311.07838v3cs.CLcs.AIcs.IR

TL;DR

Verifiable generation relies on supporting documents, but widely used retrievers can bottleneck the pipeline and limit its performance. LLatrieval has the LLM iteratively verify and update retrieval results; experiments report better performance than baselines and state-of-the-art results. The approach relies on real-time LLM inference, which may not suit low-latency or high-throughput tasks.

  • Problem

    Widely used retrievers can bottleneck verifiable generation, and their capabilities are described as inferior to LLMs, limiting overall performance.

    Source: §1 · p. 2

  • Method

    LLatrieval has the LLM verify whether retrieved documents support answering and update them iteratively until they do.

    Source: §1 · p. 2

  • Results

    LLatrieval significantly outperforms baselines on correctness and verifiability across all three datasets, beating the used retriever by 3.4 points on correctness and 5.9 points on citation-F1.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The verify-update process refines retrieval until documents support answering, facilitating generation of both correct and verifiable answers.

    Source: §6 · p. 9

    Because LLatrieval relies on real-time LLM inference, it may not suit tasks requiring low latency or high throughput.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Verifiable generation aims to let the large language model (LLM) generate text with supporting documents, which enables the user to flexibly verify the answer and makes the LLM's output more reliable. Retrieval plays a crucial role in verifiable generation. Specifically, the retrieved documents not only supplement knowledge to help the LLM generate correct answers, but also serve as supporting evidence for the user to verify the LLM's output. However, the widely used retrievers become the bottleneck of the entire pipeline and limit the overall performance. Their capabilities are usually inferior to LLMs since they often have much fewer parameters than the large language model and have not been demonstrated to scale well to the size of LLMs. If the retriever does not correctly find the supporting documents, the LLM can not generate the correct and verifiable answer, which overshadows the LLM's remarkable abilities. To address these limitations, we propose \LLatrieval (Large Language Model Verified Retrieval), where the LLM updates the retrieval result until it verifies that the retrieved documents can sufficiently support answering the question. Thus, the LLM can iteratively provide feedback to retrieval and facilitate the retrieval result to fully support verifiable generation. Experiments show that LLatrieval significantly outperforms extensive baselines and achieves state-of-the-art results.

1 Introduction

Verifiable generation pairs LLM answers with supporting documents so users can check them, but its reliability depends on retrieval quality. LLatrieval addresses this by having the LLM iteratively verify and update retrieval results.

1 Introduction

  • LLMs can produce factual errors and fabricated content, often referred to as hallucination.Source: §1 · p. 1
  • Verifiable generation pairs answers with supporting documents, letting users verify answers and improving their reliability.Source: §1 · p. 1
  • Retrieval quality matters because retrieved documents both supplement knowledge for answer generation and provide evidence for verification.Source: §1 · p. 2
  • LLatrieval lets the LLM verify whether retrieved documents support answering, then update them through verify-update iterations.Progressive Selection and Missing-Info Query help scan document candidates and supplement missing information.Source: §1 · p. 2
  • LLatrieval significantly outperforms extensive baselines and achieves new state-of-the-art results on verifiable generation.Source: §1 · p. 2
  • Further analyses report that each component contributes critically to improvements and that the method shows potential to scale retrieval by scaling the LLM.Source: §1 · p. 2

2 Background: Verifiable Generation

Verifiable generation asks an LLM to answer with citations to supporting documents, enabling users to check answers. In the retrieval-read pipeline, a retriever selects documents and the LLM uses them to generate an answer and citations.

2 Background: Verifiable Generation

  • Verifiable generation requires answers with citations to supporting documents, which allow users to verify them more conveniently.Source: §2 · p. 2

2.1 Pipeline

  • Retrieval-read has two stages: retrieve relevant documents for a question, then generate an answer from those documents.Source: §2.1 · p. 2
  • The retriever ranks documents using similarity scores, and the LLM generates an answer from the retrieved documents.Source: §2.1 · p. 2
  • The pipeline uses an instruction and demonstrations to prompt the LLM to generate an answer with citations to the retrieved documents.Source: §2.1 · p. 3

3 LLatrieval: LLM-Verified Retrieval

LLatrieval feeds the LLM back into retrieval through verify-update iterations: it checks whether documents support answering the question, updates them when they do not, and stops when support is sufficient.

3.1 Retrieval Verification

  • The LLM checks whether retrieved documents support answering the question, using binary classification or support scores filtered at threshold τ.Source: §3.1 · p. 3

3.2 Retrieval Update

  • Progressive Selection scans sliding-window candidate batches, chooses k supporting documents from old and new pools, removes irrelevant items, and integrates critical information without enlarging D.Source: §3.2 · p. 4
  • Missing-info queries identify absent information as questions or pseudo-passages, with pseudo-passages intended to neighbor relevant documents for retrieval.Source: §3.2 · p. 4

3.3 Verify-Update Iteration

  • Each iteration retrieves candidates for missing information, selects from current and new documents, then verifies D; iterations end when support passes, varying with retrieval demands and LLM cost.Source: §3.3 · pp. 4–5

4 Experiments

On ALCE, LLatrieval outperforms the baselines across correctness and verifiability. Analyses examine its components, threshold and iteration behavior, feedback source, and performance across LLMs.

4.1 Experimental Settings

  • Experiments use ALCE’s three long-form QA datasets and compare retrieval, reranking, and LLM-augmented retrieval methods under the same retrieval-read pipeline.Source: §4.1 · p. 5

4.2 Main Results

  • LLatrieval outperforms baselines on all three datasets’ correctness and verifiability; against the used retriever, it gains 3.4 correctness and 5.9 citation-F1 points.Traditional retrievers also trail LLM-augmented methods, while neural rerankers do not fully mitigate the retriever bottleneck.Source: Table 1 · p. 6, §4.2 · pp. 6–7

4.3 Analyses

  • Examples filtered at higher first-iteration thresholds have better generation quality, supporting the verification score’s ability to evaluate retrieval results.Source: Figure 5 · p. 7, §4.3 · p. 7
  • On ASQA and QAMPARI, progressive selection improves performance; verification identifies weaker documents, and retrieval updates improve generation quality for failed cases.The reported ablations indicate that each component contributes to overall performance.Source: §4.3 · p. 7, Table 2 · p. 7
  • Increasing the threshold brings better performance at higher LLM cost, but a too-strict threshold does not necessarily improve performance, including ASQA correctness at threshold 8.The reported failure cases involve absent relevant information in the document pool or failure to follow the missing-info query instruction.Source: Figure 6 · p. 7, §4.3 · pp. 7–8
  • Across closed- and open-source LLMs, LLatrieval consistently improves on the original retriever, and stronger LLMs lead to better performance.The authors describe this as potential to scale retrieval performance by scaling the LLM.Source: Table 3 · p. 8, §4.3 · p. 8
  • Performance improves with more verify-update iterations, while LLatrieval can outperform RankGPT with fewer average document candidates by stopping when support is sufficient.Source: Figure 7 · p. 8, §4.3 · p. 8
  • Internal LLM feedback achieves performance comparable to gold-answer feedback while using less LLM cost.Source: Table 4 · p. 8, §4.3 · p. 8

5 Related Work

Related work covers verifiable-generation methods and evaluation, as well as LLM-enhanced retrieval approaches. LLatrieval is presented as iteratively verifying and updating retrieval results for verifiable generation.

5 Related Work

  • Prior verifiable-generation work explores LLMs that browse web pages and answer long-form questions with supporting evidence, alongside methods for retrieving evidence and revising unsupported content.Source: §5 · pp. 8–9
  • Verifiability research includes human assessment of source support and automated approaches for evaluating answer attribution and citation recall or precision.Source: §5 · p. 9
  • LLM-enhanced retrieval work covers query rewriting, reranking, and data augmentation, while LLatrieval iteratively updates retrieval until documents sufficiently support verifiable generation.Source: §5 · p. 9

6 Conclusion

LLatrieval iteratively refines retrieval results until the LLM verifies that the documents support answering the question. The authors report that it outperforms extensive baselines and achieves new state-of-the-art results.

6 Conclusion

  • LLatrieval iteratively refines retrieval results until the LLM verifies that the documents support answering the question.Source: §6 · p. 9
  • LLatrieval outperforms extensive baselines and achieves new state-of-the-art results, according to the reported experiments.Source: §6 · p. 9
  • The authors state that this approach facilitates retrieval for generating answers that are both correct and verifiable.Source: §6 · p. 9

Limitations

The authors identify real-time LLM inference and possible LLM bias in final retrieval results as limitations of LLatrieval.

Limitations

  • Real-time LLM inference may make LLatrieval unsuitable for tasks requiring low latency or high throughput.Source: Limitations · p. 9
  • Because LLatrieval relies on LLMs, it may introduce bias into the final retrieval result.Source: Limitations · p. 9

A.1 Method Details

The appendix specifies implementation settings and compared methods, and identifies tables covering iteration details, candidate quantities, and retrieval–generation quality. It also points to instruction tables.

A.1 Method Details

  • The setup uses the gpt-3.5-turbo-0301 API at temperature 0, with a window size of 20, 50 document candidates per query, five supporting documents, and at most four iterations.Source: §A.1 · p. 14
  • Wikipedia supplies the ASQA and QAMPARI document pools with BGE-large retrieval, while ELI5 uses Sphere with BM25 retrieval.Source: §A.1 · p. 14
  • Progressive selection places current documents before new candidates and uses LLM-generated summaries; HYDE generates ELI5’s first-iteration query.Source: §A.1 · p. 14
  • Compared methods include original-instruction HYDE, LAMER, and RankGPT; several retriever and reranker variants are also specified.The listed variants include DPR, Contriever, GTR, instructor, BGE-Embedding, monoBERT, monoT5, and BGE-Reranker.Source: §A.1 · p. 14
  • Tables 5 and 6 are identified as covering iteration details and average candidate quantities for passage-style and question-style missing-information queries.Source: Table 5 · p. 14, Table 6 · p. 14
  • Table 7 is identified as covering the correlation between retrieval quality and generation quality.Source: Table 7 · p. 14

A.3 Average Document Candidates

The section reports per-iteration candidate and verification counts, and an exploratory positive correlation between retrieval quality and generation quality.

A.3 Average Document Candidates

  • Tables 5 and 6 report average document-candidate quantities and examples passing retrieval verification per iteration for the Table 1 result.Source: §A.3 · p. 14

B The Correlation between Retrieval Quality and Generation Quality

C Performance across Various Hyper-Parameters

LLatrieval's performance is reported across hyper-parameters and retrievers, with improvements over original retrievers across the tested settings.

C Performance across Various Hyper-Parameters

Related papers

Loading 2311.07838v3…