PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Verifiable Text Generation with Evolving Memory and Self-Reflection

Hao Sun, Hengyi Cai, Bo Wang, Yingyan Hou, Xiaochi Wei, Shuaiqiang Wang, Yan Zhang, Dawei Yin

arXiv:2312.09075v3cs.CL

TL;DR

LLMs can produce hallucinated facts that undermine reliability. VTG combines evolving memory, citation verification, and retrieval to support verifiable generation. Across five datasets and three knowledge-intensive tasks, it significantly outperforms baselines on citation quality and answer correctness.

  • Problem

    LLMs often generate hallucinated facts and inaccurate information, undermining their reliability.

    Source: §1 · p. 1

  • Method

    VTG combines evolving long- and short-term memory with two-tier claim–citation verification and active retrieval using diverse queries.

    Source: §1 · p. 2

  • Results

    Across five datasets and three knowledge-intensive tasks, VTG significantly outperforms baselines on citation quality and answer correctness, with 22% relative citation-quality improvement over RERANK using Text-Davinci-003.

    Source: §1 · p. 2, §3.4 · p. 6

  • Takeaways & Limitations

    Experiments across five datasets and three knowledge-intensive tasks verify VTG’s effectiveness for verifiable text generation.

    Source: §5 · p. 8

    Multiple API calls and frequent verification raise computational cost, potentially restricting use in resource-limited settings.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Despite the remarkable ability of large language models (LLMs) in language comprehension and generation, they often suffer from producing factually incorrect information, also known as hallucination. A promising solution to this issue is verifiable text generation, which prompts LLMs to generate content with citations for accuracy verification. However, verifiable text generation is non-trivial due to the focus-shifting phenomenon, the intricate reasoning needed to align the claim with correct citations, and the dilemma between the precision and breadth of retrieved documents. In this paper, we present VTG, an innovative framework for Verifiable Text Generation with evolving memory and self-reflection. VTG introduces evolving long short-term memory to retain both valuable documents and recent documents. A two-tier verifier equipped with an evidence finder is proposed to rethink and reflect on the relationship between the claim and citations. Furthermore, active retrieval and diverse query generation are utilized to enhance both the precision and breadth of the retrieved documents. We conduct extensive experiments on five datasets across three knowledge-intensive tasks and the results reveal that VTG significantly outperforms baselines.

1 Introduction

LLM hallucinations undermine reliability, while verifiable text generation must handle shifting content, claim–evidence alignment, and retrieval trade-offs. VTG combines evolving memory, verification, and retrieval mechanisms, and is reported to outperform baselines across several tasks.

  • Verifiable text generation must adapt to changing content focus, determine whether evidence logically supports each claim, and balance retrieval precision against breadth.Source: §1 · pp. 1–2
  • VTG uses iterative generation and verification, with evolving long- and short-term memory and a two-tier verifier to assess claims against evidence.Source: §1 · p. 2
  • Active retrieval begins when memory does not support a claim, while diverse queries broaden the evidence gathered for it.Source: §1 · p. 2
  • VTG significantly outperforms baselines on citation quality and answer correctness across five datasets and three knowledge-intensive tasks.Source: §1 · p. 2

2 Methodology

VTG generates claims and citations from evolving memories, checks whether citations support claims, and uses retrieval when memory cannot validate a claim. Verified evidence can be simplified and retained to support later generation.

2.1 Overall Framework

  • VTG generates claims from long- and short-term memories, verifies cited documents and the combined memory, and routes supported evidence to simplification or unsupported claims to retrieval.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 Generation with Evolving Memory

  • The long-term memory retains documents relevant to the original question, while short-term memory is updated as the generation focus shifts; claim and citation generation use both.Source: §2.2 · pp. 3–4

2.3 Citation Construction

  • A generation verifier checks the cited set, then a memory verifier checks the full memory; successful support leads to citation simplification, while failed support triggers claim regeneration with retrieved evidence.Source: §2.3 · p. 4

2.4 Evidence Finder

  • When memory fails to support a claim, the evidence finder generates context-aware queries from the original question, claim, and unfinished content, then retrieves documents to update short-term memory.Source: §2.4 · pp. 4–5

3 Experiment

The experiments compare VTG with generation and postprocessing baselines across five datasets and three QA task types, reporting gains in citation quality and answer correctness. Analyses examine component contributions, retrieval breadth, trial budgets, and token costs.

3.1 Baselines

  • The main comparison uses four best-performing ALCE baselines—VANILLA, SUMM, SNIPPET, and RERANK—while fuller experiments also compare postprocessing baselines.The listed postprocessing methods are POSTCITE, REFINECITE, VERICITE, and VERIREFINE.Source: §3.1 · p. 5

3.2 Datasets and Evaluation; 3.3 Implementation Details

  • Evaluation spans five datasets across multihop, long-form, and open-domain QA, measuring answer correctness and citation quality; experiments use Vicuna-13B and Text-Davinci-003, Wikipedia from 2018, and DPR.The retrieval corpus is the Wikipedia dump from December 20, 2018.Source: §3.2 · p. 5, §3.3 · p. 6

3.4 Main Results

  • VTG beats baselines across datasets, metrics, and LLM sizes; citation quality improves 22% over RERANK with Text-Davinci-003 and 9% with Vicuna-13B, with approximately 5% overall answer-correctness gains.The citation-quality improvements are relative to RERANK; the correctness improvement is relative to leading baselines across both LLMs.Source: Table 1 · p. 6, Table 2 · p. 6, §3.4 · p. 6
  • RERANK is strongest among baselines for citation quality but fluctuates in correctness; SUMM and SNIPPET's correctness advantage on Vicuna diminishes on Text-Davinci-003, and both struggle with citation quality.The reported comparison suggests simplified documents can add noise for Text-Davinci-003 and make correct citations harder to generate.Source: §3.4 · pp. 6–7

3.5 Analysis

  • Every component removal lowers performance; verifier removal drops most, simplifier removal reduces citation precision, and memory or diverse-query removal harms both quality dimensions.The ablation study is on 2WikiMultihopQA.Source: Table 3 · p. 7, §3.5 · p. 7
  • On ASQA, increasing generated queries M and documents per query N initially improves correctness and citation quality, but excessively large pools introduce noise and reduce performance.The analysis covers retrieval parameters M and N.Source: Figure 3 · p. 7, §3.5 · pp. 7–8
  • Increasing max trials T correlates with better correctness and citation quality but increases token use, trading effectiveness against computational cost; “Trials” is average iterations per claim.The analysis uses 2WikiMultihopQA.Source: §3.5 · p. 7, Table 4 · p. 7
  • On NQ with Text-Davinci-003, VTG costs less tokens than RERANK while significantly improving citation quality; users can tune cost through max-trial T.VANILLA has the lowest token cost, while RERANK has the highest among the compared methods.Source: Figure 4 · p. 8, §3.5 · p. 8

4 Related Work

Related work covers retrieval-augmented generation strategies and verifiable text generation. The paper identifies focus-shifting and complex claim-citation relationships as gaps in prior verifiable-generation methods.

4.1 Retrieval-augmented LLMs.

4.2 Verifiable Text Generation

  • Prior verifiable-generation work includes evidence-based browsing, RARR, knowledge graphs, and citation-quality evaluation, but the paper says these methods do not effectively address focus-shifting or complex claim-citation relationships.Source: §4.2 · p. 8

5 Conclusion

VTG combines evolving memory, two-tier claim-citation verification, active retrieval, and diverse query generation; experiments across five datasets and three knowledge-intensive tasks verify its effectiveness.

  • Experiments across five datasets and three knowledge-intensive tasks verify VTG’s effectiveness.Source: §5 · p. 8
  • VTG’s evolving long short-term memory adaptively retains valuable and up-to-date documents.Source: §5 · p. 8
  • A two-tier verifier with an evidence finder supports deeper analysis and reflection on claim-citation relationships.Source: §5 · p. 8
  • Active retrieval and diverse query generation are used to enhance retrieval precision and breadth.Source: §5 · p. 8

Limitations

VTG has relatively high computational cost from multiple API calls and frequent verification, and its verification effectiveness depends on NLI-model precision.

  • Multiple API calls and frequent verification make VTG relatively costly, potentially restricting use in resource-intensive or computationally limited settings.Source: Limitations · p. 9
  • Suboptimal NLI-model accuracy risks incorporating erroneous information and compromising generated-text verifiability.Source: Limitations · p. 9

Ethics Statement

The authors report compliance with the ACL Ethics Policy and do not foresee negative ethical impacts from the work.

  • The work reports ACL Ethics Policy compliance, public evaluation datasets and LLMs, and no foreseen negative ethical impact.Source: Ethics Statement · p. 9

A Baselines

The paper compares VTG with four ALCE baselines and four citation-focused methods, which use different strategies for selecting or refining citations.

A Baselines

  • The ALCE comparison uses VANILLA, SUMM, SNIPPET, and RERANK, with k = 5 and K = 10.The methods use multiple demonstrations in their initial prompts.Source: A Baselines · p. 12
  • VANILLA generates cited responses from the top-k ranked documents, while SUMM first condenses information from the top-K documents.SUMM integrates the summary into the prompt before generation.Source: A Baselines · p. 12
  • SNIPPET extracts relevant spans from top-K documents for cited generation, whereas RERANK selects among four high-temperature responses using citation recall.Source: A Baselines · p. 12
  • POSTCITE adds citations from the top-100 retrieved passages after answer generation, while REFINECITE removes unnecessary citations using NLI.Source: A Baselines · p. 12
  • VERICITE checks whether citations support generated sentences and can replace them with a better-matching passage; VERIREFINE combines this verification with citation refinement.Source: A Baselines · p. 12

B Datasets and Settings

This section summarizes the datasets and experimental settings and defines citation recall and precision, which are also assessed with Qwen-MAX.

B Datasets and Settings

  • Dataset statistics and experimental settings are summarized in Table 5.Source: B Datasets and Settings · p. 12, Table 5 · p. 13
  • Citation recall averages a binary score across claims, counting a claim as supported when citations entail it according to an NLI model.Source: B Datasets and Settings · p. 12
  • Citation precision averages a binary score across citations, requiring the claim to have supported citations and the individual citation to be non-irrelevant.The LLM evaluation asks Qwen-MAX whether cited passages fully support a sentence and whether each citation does so individually.Source: B Datasets and Settings · pp. 12–13

C Focus Shifting Phenomenon

Sentence similarity declines with distance during generation, illustrating the paper’s focus-shifting phenomenon. The analysis averages embedding-based similarity matrices across questions.

C Focus Shifting Phenomenon

  • The analysis represents output sentences with BGE embeddings, computes pairwise cosine similarities, and averages the resulting matrices across questions.Source: C Focus Shifting Phenomenon · p. 13
  • Sentence similarity decreases as sentence distance grows, indicating that content focus diverges as generation continues.Source: C Focus Shifting Phenomenon · p. 13

D Use of Different NLI models

With different NLI models for verification and evaluation, VTG outperforms all tested baselines on correctness and citation quality. The appendices provide implementation settings, the algorithm reference, and experimental prompts.

D Use of Different NLI models

  • On 300 sampled ASQA and NQ questions, VTG outperformed all baselines in correctness and citation quality with separate NLI models for verification and evaluation.The verifier was t5_11b_trueteacher_and_anli; evaluation used t5_xxl_true_nli_mixture, with Vicuna-13B-v1.5-16K as the base LLM.Source: D Use of Different NLI models · p. 13, Table 10 · p. 15

E Hyper-parameters

F Algorithm

G Prompts

  • The prompts include generators for answers, snippets, summaries, citations, and diverse verification queries, plus citation-evaluation instructions.VANILLA and RERANK share a prompt; the query-generation prompt requests diverse questions focused on different claim aspects.Source: G Prompts · pp. 14, 16, 17
Loading 2312.09075v3…