PaperlayerView exact revision on arXiv

Source-linked AI summary

Citation-Enhanced Generation for LLM-based Chatbots

Weitao Li, Junkai Li, Weizhi Ma, Yang Liu

arXiv:2402.16063v4cs.CLcs.AI

TL;DR

LLM chatbots can produce hallucinated responses, a challenge that limits their applicability. CEG retrieves supporting documents and uses NLI for post-hoc citations, with regeneration when needed. The paper reports state-of-the-art performance in hallucination detection and regeneration across three benchmarks.

  • Problem

    Hallucinated chatbot responses remain a challenge that limits LLM chatbots’ applicability.

    Source: §1 · p. 1

  • Method

    CEG retrieves documents relevant to generated content, uses NLI for citations, and regenerates responses when needed.

    Source: §1 · p. 2

  • Results

    CEG achieves state-of-the-art performance in hallucination detection and regeneration across three benchmarks.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The framework is presented as flexible across LLMs and usable without additional training or annotations.

    Source: §6 · p. 9

    The experiments use only Wikipedia and no fine-tuned retriever, limiting demonstrated applicability to general-knowledge question answering.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) exhibit powerful general intelligence across diverse scenarios, including their integration into chatbots. However, a vital challenge of LLM-based chatbots is that they may produce hallucinated content in responses, which significantly limits their applicability. Various efforts have been made to alleviate hallucination, such as retrieval augmented generation and reinforcement learning with human feedback, but most of them require additional training and data annotation. In this paper, we propose a novel post-hoc Citation-Enhanced Generation (CEG) approach combined with retrieval argumentation. Unlike previous studies that focus on preventing hallucinations during generation, our method addresses this issue in a post-hoc way. It incorporates a retrieval module to search for supporting documents relevant to the generated content, and employs a natural language inference-based citation generation module. Once the statements in the generated content lack of reference, our model can regenerate responses until all statements are supported by citations. Note that our method is a training-free plug-and-play plugin that is capable of various LLMs. Experiments on various hallucination-related datasets show our framework outperforms state-of-the-art methods in both hallucination detection and response regeneration on three benchmarks. Our codes and dataset will be publicly available.

1 Introduction

Hallucination remains an unresolved challenge for LLM chatbots, while many existing mitigation approaches require additional training or annotation. The paper proposes post-hoc Citation-Enhanced Generation (CEG), combining retrieval and NLI with response regeneration, and reports the best performance on three hallucination-related benchmarks.

  • Hallucinations are an unsolved chatbot challenge that constrains deployment in sensitive settings such as healthcare and education.Source: §1 · p. 1
  • Earlier mitigation approaches include retrieval augmentation and RLHF, but often require additional training and extensive data annotation.Source: §1 · pp. 1–2
  • CEG applies retrieval augmentation and NLI after generation, prompting response regeneration when necessary.The framework adds citations to generated content.Source: Figure 1 · p. 1, §1 · p. 2
  • The authors present post-hoc citation with regeneration as a novel approach and describe the framework as flexible for existing LLMs.Source: §1 · p. 2
  • CEG achieves the best performance on three hallucination-related benchmarks.Source: §1 · p. 2

2 Related Work

Related work covers training- and inference-time hallucination controls, then distinguishes citation-augmented methods by whether they draw on trained knowledge or external retrieval. The paper positions CEG as a post-hoc alternative to pre-hoc citation annotation.

2.1 Hallucination Control in LLMs

  • Training-time approaches use high-fidelity pretraining data, instruction fine-tuning, or RLHF penalties, but require extra training and annotations.Source: §2.1 · p. 2
  • Inference-time methods include activation intervention and retrieval before generation, but may leave hallucinations unverified and provide no citations.Source: §2.1 · p. 2

2.2 Citation Augmented LLMs

  • Citation-augmented methods are parametric, drawing on training-internalized information, or non-parametric, retrieving evidence from external corpora.Source: §2.2 · p. 2
  • Most prior non-parametric citation methods are pre-hoc, requiring response generation alongside annotation; CEG instead generates citations post-hoc.Source: §2.2 · pp. 2–3

3 Approach

CEG verifies generated responses after generation by retrieving documents for response segments, classifying their support with NLI, and regenerating responses containing potential hallucinations. The framework is adaptable across LLMs without additional training or fine-tuning.

3.1 Overview

  • CEG combines retrieval, NLI-based citation generation, and response regeneration to verify generated claims and produce a more reliable response when needed.The framework does not require additional training or fine-tuning and is intended to work across different LLMs.Source: §3.1 · p. 3

3.2 Retrieval Augmentation Module

  • Generated responses are split into claims when needed, which serve as queries for post-hoc retrieval of relevant documents from a candidate corpus.The described setup uses dense retrieval over Wikipedia documents and selects top-k results, filtering low-similarity documents with a threshold.Source: §3.2 · p. 3

3.3 Citation Generation Module

  • An NLI module assesses whether retrieved documents substantiate each response segment and assigns factual or nonfactual labels that guide citation use.Figure 2 depicts response segments, retrieved documents for each segment, and the NLI-generated factual/nonfactual labels.Source: §3.3 · pp. 3–4, Figure 2 · p. 4

3.4 Response Regeneration Module

  • For regeneration, the prompt combines the original query with retrieved documents and annotated nonfactual segments, and the framework repeats verification until the response is accepted or the attempt limit is reached.The parameter T can constrain the maximum number of regeneration attempts.Source: §3.4 · p. 5

4 Experimental Settings

The experiments use four hallucination-related datasets, with different tasks and baselines across datasets. The settings specify the GPT backbones, dataset construction, evaluation strategies, and metrics.

4.1 Overview

  • Experiments use WikiBio GPT-3 and FELM for hallucination detection, HaluEval for hallucination generation, and WikiRetr for retrieval and citation annotation evaluation.GPT model versions vary by dataset for comparison with existing baselines; ChatGPT denotes GPT-3.5-Turbo-1106 and GPT-4 denotes GPT-4-0613 unless specified otherwise.Source: §4.1 · p. 5

4.2 WikiBio GPT-3 Dataset

  • WikiBio GPT-3 contains 1,908 manually annotated sentences from generated passages, labeled as factual or nonfactual across three original accuracy categories.The passages were generated from 238 randomly selected WikiBio biographies; 1,392 segments are categorized as nonfactual and 516 as factual.Source: §4.2 · p. 5, §4.1 · p. 5

4.3 FELM Dataset

  • FELM experiments use the WorldKnowledge subset and compare prompt strategies with ChatGPT, GPT-4, and Vicuna-33B using nonfactual, factual, and balanced accuracy.The strategies include vanilla prompts, CoT reasoning, hyperlinks to reference documents, and human-annotated reference documents.Source: §4.3 · p. 5

4.4 HaluEval Dataset

  • The HaluEval QA setting samples 2,000 Wikipedia-based instances, each comprising a question, a correct answer, and a hallucinated answer.Accuracy is the evaluation metric, with vanilla, CoT-augmented, and Pre-RAG prompts among the baselines.Source: §4.4 · p. 6

4.5 WikiRetr Datasets

  • WikiRetr comprises two datasets of rewritten claims paired with original passages and supports retrieval and citation-generation analyses.The study compares SimCSE BERT, Sentence BERT, and All-mpnet-base-v2 using Recall@k and Precision@k; NLI evaluation samples 100 instances from each dataset and compares model annotations with manual annotations.Source: §4.5 · p. 6

5 Experimental Results and Analyses

The experiments report stronger hallucination detection and regeneration performance for CEG, then examine its modules, retrieval and NLI choices, and the effect of retrieval depth. Results also show that more retrieved documents do not always improve performance.

5.1 Performance on Hallucination Detection

  • CEG leads WikiBio GPT-3 in Balanced_ACC and factual-segment AUC-PR, and achieves the best balanced accuracy with GPT-4 on FELM.On WikiBio GPT-3, it ranks second in AUC-PR of nonfactual segments; on FELM, it also beats other ChatGPT baselines and all pre-retrieval baselines.Source: §5.1 · pp. 6–7, Table 2 · p. 7

5.1 Performance on Hallucination Detection; 5.2 Results on Hallucination Regeneration

  • CEG outperforms various state-of-the-art baselines across two benchmarks with distinct LLM backbones; with GPT-3.5-Turbo-Instruct, it achieves 69.45% accuracy, an 8.10% improvement over pre-hoc retrieval.Source: §5.1 · p. 7, §5.2 · p. 7

5.2 Results on Hallucination Regeneration

  • On HaluEval, CEG with GPT-3.5-Turbo-Instruct achieves 69.45% accuracy, an 8.10% improvement over pre-hoc retrieval.CEG with ChatGPT also outperforms the pre-retrieval strategy.Source: §5.2 · p. 7, Table 3 · p. 7

5.3.1 Ablation Study

  • With k = 4 and threshold = 0.5 on FELM Worldknowledge, ablations indicate that retrieval augmentation improves results across ChatGPT and GPT-4, while the threshold filters irrelevant documents.The authors conclude that all designed modules contribute to CEG improvements.Source: Table 4 · p. 7, §5.3.1 · p. 8

5.3.2 Retrieval Models

  • SimCSE BERT performs best in post-hoc retrieval, reaching 76.8% with the top 10 documents among 64 million candidates.The authors set k between 4 and 6 to balance efficiency and effectiveness, noting that larger k requires more computation.Source: §5.3.2 · p. 8, Figure 3 · p. 8

5.3.3 NLI Models in Citation Generation

  • GPT-3.5-Turbo-Instruct and GPT-4 achieve higher agreement with human annotators than task-specific True-9B in the citation-generation NLI task.The authors also report that LLMs perform better on data they generate.Source: §5.3.3 · p. 8, Table 5 · p. 8
  • The citation-generation module reaches precision scores of 78.2 and 75.7 on the two WikiRetr datasets; WikiRetr-GPT4 has lower recall than WikiRetr-GPT3.The authors describe WikiRetr-GPT4 as more challenging and higher-quality, with precision exceeding recall.Source: §5.3.3 · p. 8, Table 6 · p. 8

5.3.4 Hyper-parameter Analysis

  • Across FELM and WikiBio GPT-3, more retrieved documents do not always improve performance; the best results occur near k = 5, at Top-4 for FELM and Top-6 for WikiBio GPT-3.The authors attribute the decline at larger k to longer inputs, dispersed attention, and lower relevance of retrieved documents; too few or no retrieved documents also worsen results.Source: §5.3.4 · pp. 8–9, Figure 4 · p. 8

6 Conclusion

The conclusion presents CEG as a post-hoc framework combining retrieval augmentation and natural language inference to reduce hallucinations. It reports state-of-the-art performance on detection and regeneration, while identifying corpus expansion as future work.

6 Conclusion

  • CEG combines post-hoc retrieval augmentation and natural language inference to reduce hallucinations in LLMs.The framework is presented as distinct from previous hallucination studies because it operates post hoc.Source: §6 · p. 9
  • The authors report state-of-the-art performance in hallucination detection and regeneration across three hallucination-related benchmarks and their datasets.Further analyses are reported to show the effectiveness of the proposed modules and adopted models.Source: §6 · p. 9
  • The authors plan to expand the corpus to support more scenarios.Source: §6 · p. 9

Limitations

The experiments are limited by the retriever and corpus, the evaluation data, the NLI method, and API costs.

Limitations

  • The experiments use no fine-tuned retriever and only Wikipedia, limiting applicability to general-knowledge question answering.The authors state that these choices demonstrate the model's effectiveness but constrain its applicability.Source: Limitations · p. 9
  • The evaluation uses existing benchmarks and manual annotations, without new QA datasets to verify regeneration performance.Source: Limitations · p. 9
  • The citation-generation NLI method inherently relies on the LLM's world knowledge; more powerful NLI methods may improve performance.Source: Limitations · p. 9
  • Regeneration prompting and NLI-based citation generation incur API costs.The paper reports costs for the method and for creating WikiRetr datasets in Table 11.Source: Limitations · p. 9

A Prompts Used in Our Experiments

The appendix documents prompts used across the experiments, including response-verification and regeneration prompts, and notes which tables contain them.

A Prompts Used in Our Experiments

  • FELM prompts cover CEG and its variants, while HaluEval prompts cover baselines and response regeneration.FELM prompts are in Tables 12–13; HaluEval baseline prompts are in Tables 14–16 and its regeneration prompt is in Table 17.Source: A Prompts Used in Our Experiments · p. 11
  • The sentence-verification prompt asks whether a sentence is supported by retrieved context and requests a yes-or-no answer.Source: A Prompts Used in Our Experiments · p. 11
  • The WikiBio NLI prompt pairs a retrieved passage as premise with a hypothesis to verify.The prompt used for WikiBio is presented in the appendix.Source: A Prompts Used in Our Experiments · p. 11
  • WikiRetr prompts for GPT models and True-9B support reported agreement and precision results, and the True-9B prompt matches Gao et al. (2023a).The GPT prompt is associated with Table 5 agreement rates and Table 6 precision; the True-9B prompt uses the same prompt as Gao et al. (2023a).Source: A Prompts Used in Our Experiments · p. 11
  • The appendix identifies the original prompt source as ALCE's evaluation script.Source: A Prompts Used in Our Experiments · p. 11

B Detailed Information About Adopted Datasets

The appendix points readers to dataset statistics, an example, and a table of module use across the experimental datasets.

B Detailed Information About Adopted Datasets

C Case Studies on HaluEval Dataset

The appendix presents two case studies from the HaluEval evaluation in Tables 18 and 19.

C Case Studies on HaluEval Dataset

D Discussion about the reliability of WikiRetr datasets

Manual annotation found that 90% and 94% of generated claims were supported by their original documents. The paper presents this consistency as exceptionally high, alongside comparison figures from FELM and Pinocchio.

Related papers

Loading 2402.16063v4…