Source-linked AI summary
The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models
Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, Ji-Rong Wen
TL;DR
Factual hallucinations hinder trustworthy LLM deployment, while existing work lacks a systematic account of how to detect, explain, and mitigate them. This paper builds HaluEval 2.0, develops an automatic detector, and studies hallucination sources and mitigation across training and inference stages. It reports empirical findings on pre-training, fine-tuning, prompting, inference, and mitigation techniques, while noting limits from restricted training details, insufficient experiments, and the absence of new mitigation strategies.
Problem
Factual hallucinations hinder trustworthy LLM deployment, and existing empirical work lacks a systematic study of detection, sources, and mitigation.
Method
The paper constructs HaluEval 2.0, develops an automatic hallucination detector, and evaluates hallucination sources and mitigation techniques across LLM training and inference stages.
Results
The study reports empirical findings on hallucination sources and mitigation across pre-training, fine-tuning, prompt design, inference, RLHF, retrieval augmentation, self-reflexion, and advanced decoding.
Takeaways & Limitations
The benchmark can be reused for further research, and the findings support understanding the sources and mitigation of LLM hallucinations.
Takeaways & Limitations
Analysis of pre-training and SFT is limited by undisclosed training details and computational constraints, experiments remain insufficient, and no new mitigation strategies are proposed.
Abstract
from arXiv · showhide
In the era of large language models (LLMs), hallucination (i.e., the tendency to generate factually incorrect content) poses great challenge to trustworthy and reliable deployment of LLMs in real-world applications. To tackle the LLM hallucination, three key questions should be well studied: how to detect hallucinations (detection), why do LLMs hallucinate (source), and what can be done to mitigate them (mitigation). To address these challenges, this work presents a systematic empirical study on LLM hallucination, focused on the the three aspects of hallucination detection, source and mitigation. Specially, we construct a new hallucination benchmark HaluEval 2.0, and designs a simple yet effective detection method for LLM hallucination. Furthermore, we zoom into the different training or utilization stages of LLMs and extensively analyze the potential factors that lead to the LLM hallucination. Finally, we implement and examine a series of widely used techniques to mitigate the hallucinations in LLMs. Our work has led to several important findings to understand the hallucination origin and mitigate the hallucinations in LLMs. Our code and data can be accessed at https://github.com/RUCAIBox/HaluEval-2.0.
1 Introduction
LLM hallucinations remain a major obstacle to trustworthy deployment, motivating a systematic study of their detection, sources, and mitigation. The study introduces HaluEval 2.0, analyzes factors across model-development stages, and reports findings on training, prompts, and inference.
- Hallucinations are plausible but factually incorrect outputs that restrict trustworthy LLM deployment in real-world applications.
- Existing empirical work largely addresses detection, sources, or mitigation separately, leaving a need for systematic analysis across all three challenges.
- HaluEval 2.0 evaluates factuality hallucination, while the proposed detector extracts factual statements and then determines whether they contain hallucinations.
- Pre-training: Lower-frequency pre-training knowledge produces more hallucinations, whereas specialized pre-training data can substantially alleviate hallucinations in specific domains.
- Fine-tuning and prompt design: Improved instructions and in-context learning can reduce hallucinations, but overly complex instructions increase them.
- Inference and prompt design: Inference effects depend on method and domain: diversity-oriented decoding increases professional-domain hallucinations, while greedy search worsens open-ended-domain hallucinations.
2 Hallucination in LLMs
The paper focuses on factuality hallucination, categorizing common ways LLM outputs can conflict with, exceed, omit, or become outdated relative to factual knowledge. Table 1 illustrates six defined hallucination types using incorrect and unverifiable content.
- The paper focuses on factuality hallucination because it is a primary source of erroneous LLM responses.
- Entity-error hallucination contains erroneous people, dates, locations, or objects that contradict world knowledge.
- Relation-error hallucination contains incorrect quantitative or chronological relations between entities.
- Incompleteness hallucination occurs when outputs fail to preserve the factual completeness of requested aggregated information.
- Outdatedness hallucination occurs when content was once correct but is no longer current because models use time-limited training corpora.
- The taxonomy also includes overclaim and unverifiability hallucinations, illustrated by statements exceeding factual knowledge or lacking verification.
- The taxonomy showcases frequent hallucination types rather than encompassing every possible form.
3 Experimental Setup
The experimental setup constructs HaluEval 2.0 from fact-intensive questions spanning five domains and evaluates representative open-source and closed-source LLMs. The benchmark contains 8,770 questions and supports hallucination analysis across diverse models.
- HaluEval 2.0 is an upgraded benchmark containing large-scale fact-intensive questions from biomedicine, finance, science, education, and open-domain settings.
- The questions are drawn from six established datasets, using different dataset splits to obtain sufficient high-quality fact-intensive questions.
- 8,770 questions comprise the benchmark: 1,535 biomedicine, 1,125 finance, 1,409 science, 1,701 education, and 3,000 open-domain questions.
- Models: The experiments include six representative instruction-tuned open-source models, including Alpaca, Vicuna, YuLan-Chat, and Llama 2-Chat.
- Models: The experiments also include five representative API-accessed closed-source models: text-davinci-002/003, ChatGPT, Claude, and Claude 2.
4 Hallucination Detection
The paper presents an automatic hallucination detector that extracts factual statements and judges whether they are hallucinated, then evaluates its reliability and LLM behavior across domains and models.
- Detection Approach: The detection framework decomposes hallucination detection into extracting factual statements and judging each statement for hallucinations.GPT-4 is used for both extraction and judgment, leveraging instruction-following and world knowledge.
- Reliability: 92.6%, 94.7%, 92.7%, 91.5%, and 93.9% matching rates were obtained across biomedicine, finance, science, education, and open domain, respectively.The comparison used 1,000 questions, with two labelers and one checker per sample.
- Evaluation Metrics: MiHR measures the proportion of hallucinatory statements within responses, whereas MaHR measures the proportion of responses containing hallucinatory statements.For both metrics, smaller values indicate better performance.
- Results and Analysis: Open-source and closed-source models show a significant performance gap in their tendency to generate hallucinations.Among open-source models, larger models generally show lower hallucination rates, but MaHR and MiHR are not strongly positively correlated.
- Results and Analysis: Hallucination rates vary by domain, with higher rates in education and open domain; Llama 2-Chat reaches around 80% in open domain.ChatGPT and Claude show rates from 40% to 50% in open domain, while the evaluation questions were selected for difficulty.
- Results and Analysis: LLMs also have limited ability to recognize hallucinations, with ChatGPT achieving 58.53% accuracy on summarization.The reported generation and recognition results indicate an implicit correlation, while avoiding generation is more challenging.
5 Hallucination Source
The study examines hallucination sources across pre-training, supervised fine-tuning, prompt design, and inference, identifying factors associated with factuality errors.
- Pre-training: Pre-training token scale produces oscillating hallucination rates, while specialized data can alleviate hallucinations in specific domains.Baichuan 2 was evaluated across checkpoints trained on approximately 0.2 to 2.4 trillion tokens; general task performance improved, but hallucination rates varied across checkpoints and domains.
- Pre-training: Lower-frequency pre-training knowledge is associated with more hallucinations, especially for entities in the long tail of occurrence frequency.ChatGPT and Llama 2-Chat showed the lowest hallucination tendency for the most frequent entities, while most long-tail entities had relatively high hallucination rates.
- Supervised Fine-Tuning: Balancing instruction complexity reduces hallucinations, whereas overly complex and diverse instructions increase them.Improved instructions can alleviate hallucinations, but adding constraints, reasoning steps, topic diversity, or excessive difficulty can raise hallucination levels.
- Prompt Design: Detailed task descriptions and in-context learning can reduce hallucinations, especially in professional domains, but rewriting questions or moving task descriptions can increase them.The reported effects vary by domain, with more details particularly helpful in biomedicine and science.
- Inference Methods: Decoding strategies affect hallucination rates by domain: diversity-oriented methods increase professional-domain hallucinations, while greedy search worsens open-ended-domain hallucinations.Beam search balances diversity and factuality, whereas token-by-token generation can preserve earlier mistakes and quantization generally exacerbates hallucinations, especially at 4-bit precision.
- Inference Methods: 8-bit quantization usually has minimal hallucination impact, whereas 4-bit quantization significantly increases hallucinations.Quantized models also reduce memory footprint and accelerate inference compared with the original 16-bit model.
6 Hallucination Mitigation
The paper examines widely used hallucination-mitigation methods across RLHF, retrieval augmentation, self-reflexion, decoding, and prompt improvement, finding that effectiveness depends on domains, model scale, and method settings.
- RLHF: RLHF can alleviate hallucinations, but its effectiveness varies by domain and is milder in highly professional domains such as science.The paper reports more pronounced reductions in biomedicine and open-domain settings.
- Overall Findings: The mitigation findings support domain-aware and model-aware choices rather than uniformly effective hallucination-reduction techniques.The reported effects differ across domains, model sizes, reasoning abilities, retrieval settings, and decoding hyper-parameters.
- Retrieval Augmentation: Retrieval augmentation significantly mitigates hallucinations, with stronger effects for smaller models and lower document-question relevance increasing hallucination risk.ChatGPT and Llama 2-Chat mostly perform best with top-2 retrieval, while ChatGPT prefers top-5 in science.
- Self-Reflexion: Self-reflexion becomes effective around 70B in these experiments, whereas it can increase erroneous responses for 7B and 13B models.Smaller models may suspect correct original answers and generate errors instead.
- Advanced Decoding: Balancing generation diversity and factuality can reduce hallucinations while retaining text quality, but the proposed decoding methods are sensitive to hyper-parameters.Greedy-nucleus sampling performs comparably or better in biomedicine and science, while factual-nucleus performs well in science, education, and open domains.
- Prompt Improvement: Prompt improvements have an oscillatory effect, while chain-of-thought reasoning benefits larger models such as ChatGPT but exacerbates hallucinations for Llama 2-Chat 7B.Prompt engineering should consider model size, domain characteristics, and task difficulty.
7 Related Work
Related work studies hallucination through two linked strands: understanding and detecting hallucinated content, and mitigating it across LLM development and use. Because source analysis can inform detection, prior research often discusses those topics together.
- Hallucination Source and Detection: Hallucination source and detection research examines model internals, prompts, and generated responses to understand or identify factual errors.Internal-state studies analyze representations such as hidden activations and attention, while API-accessible models are studied through input-output relationships.
- Hallucination Source and Detection: Source analysis and detection are jointly discussed because understanding hallucination origins can support the design of detection methods.
- Hallucination Mitigation: Mitigation spans pre-training, generation, and postprocessing, using dataset curation, decoding strategies, external knowledge, prompts, and fact-checking.Postprocessing can use the LLM itself or external knowledge to verify generated text and correct hallucinations.
8 Conclusion and Limitation
The paper presents a comprehensive empirical analysis of hallucination detection, sources, and mitigation, supported by a reusable benchmark and an automatic detection approach. Its analyses remain constrained by limited training-stage information, insufficient experiments, and reliance on GPT-4 for detection.
- Conclusion: The study constructs HaluEval 2.0, develops an LLM-based automatic detector, and evaluates hallucination sources and mitigation strategies across multiple LLM stages.The examined stages include pre-training, SFT, RLHF, and inference.
- Conclusion: The reusable benchmark and empirical findings on hallucination sources and mitigation are presented as the paper’s major contributions.
- Limitations: The pre-training and SFT analyses are limited by undisclosed training details and insufficient computational resources.
- Limitations: The experiments do not fully analyze hallucination mechanisms, and the paper proposes no new mitigation strategy.
- Limitations: The GPT-4-based detection approach may introduce minor errors.
Contributions
The paper lists student-author contributions spanning project leadership, writing, hallucination evaluation, dataset and code construction, experimental design, and implementation across multiple mitigation techniques.
- Junyi Li led the project, wrote and revised the paper, defined the hallucination taxonomy, and designed detection, evaluation, experiments, figures, and tables.
- Jie Chen constructed the factual-hallucination evaluation dataset and implemented GPT-4-based evaluation, effective decoding, and RLHF data-generation code.
- Ruiyang Ren contributed to the outline, benchmark, and evaluation metrics, and designed experiments on retrieval augmentation and pre-training knowledge frequency.
- Xiaoxue Cheng designed and implemented experiments on prompt design, self-reflexion, prompt improvement, and part of supervised fine-tuning.