Source-linked AI summary
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, Ji-Rong Wen
TL;DR
LLMs can produce unverifiable or source-conflicting hallucinations, yet their prevalence and recognizability remain unclear. HaluEval addresses this with a large benchmark and finds that ChatGPT hallucinates in about 19.5% of responses while LLMs struggle to detect such content.
Problem
The prevalence and types of LLM hallucinations, and the ability of LLMs to recognize them, remain unclear despite deployment risks.
Method
HaluEval combines sampling-then-filtering to generate difficult hallucinated samples with human annotation of ChatGPT responses.
Results
About 19.5% of ChatGPT responses contain fabricated unverifiable information, while existing LLMs struggle to recognize hallucinations.
Takeaways & Limitations
HaluEval supports systematic analysis of hallucination recognition and indicates that explicit knowledge and intermediate reasoning can improve LLM detection.
Takeaways & Limitations
The benchmark’s hallucinated-sample quality is limited by ChatGPT’s ability to follow complex generation instructions, and it does not investigate hallucination causes.
Abstract
from arXiv · showhide
Large language models (LLMs), such as ChatGPT, are prone to generate hallucinations, i.e., content that conflicts with the source or cannot be verified by the factual knowledge. To understand what types of content and to which extent LLMs are apt to hallucinate, we introduce the Hallucination Evaluation benchmark for Large Language Models (HaluEval), a large collection of generated and human-annotated hallucinated samples for evaluating the performance of LLMs in recognizing hallucination. To generate these samples, we propose a ChatGPT-based two-step framework, i.e., sampling-then-filtering. Besides, we also hire some human labelers to annotate the hallucinations in ChatGPT responses. The empirical results suggest that ChatGPT is likely to generate hallucinated content in specific topics by fabricating unverifiable information (i.e., about $19.5\%$ responses). Moreover, existing LLMs face great challenges in recognizing the hallucinations in texts. However, our experiments also prove that providing external knowledge or adding reasoning steps can help LLMs recognize hallucinations. Our benchmark can be accessed at https://github.com/RUCAIBox/HaluEval.
1 Introduction
HaluEval addresses the risks and unclear scope of LLM hallucinations with a 35,000-sample benchmark spanning general user queries and three task-specific settings. Its findings show that ChatGPT generates unverifiable content, LLMs struggle to detect hallucinations, and explicit knowledge or reasoning improves recognition while ground-truth comparisons can hurt performance.
- HaluEval provides 35,000 hallucinated or normal samples: 5,000 general user queries with ChatGPT responses and 30,000 examples covering question answering, knowledge-grounded dialogue, and text summarization.
- The benchmark combines human annotation of general ChatGPT responses with a two-stage automatic pipeline that generates and filters task-specific hallucinated samples.The task-specific generation uses one-pass and conversational instructions to create diverse samples before selecting plausible, difficult hallucinations.
- ChatGPT fabricates unverifiable information in about 19.5% of responses, across topics including language, climate, and technology.
- Existing LLMs struggle to identify hallucinations, with ChatGPT achieving only 62.59% accuracy on question answering.
- Explicit knowledge and intermediate reasoning improve hallucination recognition, whereas contrasting hallucinated samples with ground truth worsens performance.
2 The HaluEval Benchmark
HaluEval combines automatically generated hallucinated samples with human-annotated ChatGPT responses to study hallucination types, prevalence, and recognition. Its two-step generation process yields 30,000 samples across question answering, knowledge-grounded dialogue, and summarization, while annotation identifies hallucinations in 19.5% of 5,000 responses.
- Automatic generation: The sampling instructions specify an intention description, hallucination pattern, and demonstration, enabling controlled generation across multiple hallucination patterns and tasks.Few-shot demonstrations help ChatGPT understand the intended hallucination pattern for the seed task.
- Automatic generation: The automatic pipeline uses ChatGPT for diverse hallucination sampling followed by high-quality filtering to retain plausible, difficult hallucinated samples.Sampling uses one-pass or conversational instruction schemas, while filtering selects between hallucinated candidates using ground-truth-enhanced instructions.
- Benchmark composition: 30,000 generated samples cover question answering, knowledge-grounded dialogue, and text summarization, with 10,000 examples per task.The benchmark also includes 5,000 manually annotated ChatGPT responses for general user queries.
- Human annotation: 977 of 5,000 annotated ChatGPT responses contain hallucinations, corresponding to 19.5%.Queries were preselected using low average BERTScore similarity among three sampled responses, and three labelers judged responses as unverifiable, non-factual, or irrelevant using majority voting.
- Benchmark uses: The benchmark supports analyzing hallucination-prone content and evaluating LLMs’ ability to recognize hallucinations in generated and annotated samples.Researchers can also use the provided instructions on their own datasets for hallucination evaluation and analysis.
3 Experiments
Experiments show that LLMs struggle to recognize implicit hallucinations, while retrieved knowledge substantially improves ChatGPT’s recognition accuracy. Failures are unevenly distributed across hallucination patterns and topics, and sample contrast performs worst because generated hallucinations resemble factual outputs.
- Experimental Setup: The evaluation covers five closed-source and five open-source LLMs without fine-tuning or hyper-parameter tuning, using deterministic temperature-zero evaluation.The evaluated models include GPT-3, InstructGPT, ChatGPT, Claude, Claude 2, Alpaca, Vicuna, ChatGLM, Falcon, and Llama 2-Chat.
- Hallucination Recognition: ChatGPT achieves only 58.53% accuracy on hallucination recognition in text summarization, barely above chance, demonstrating difficulty identifying implicit factual errors.Table 5 reports classification accuracy for evaluated LLMs; the generated hallucinated samples differ from factual outputs mainly in key factual spans.
- Failure Analysis: ChatGPT’s recognition failures are unevenly distributed: over half in QA, dialogue, and summarization arise from the first hallucination pattern.These patterns are comprehension, extrinsic-soft, and factual for QA, dialogue, and summarization, respectively.
- Failure Analysis: Hallucination recognition is topic-sensitive, with ChatGPT frequently failing on film, company, band, technology, climate, and language topics across task settings.Frequent QA topics include film, school, and company; user queries and responses most often involve story, health, language, technology, and computer.
- Improving Recognition: Retrieving Wikipedia knowledge raises ChatGPT’s QA recognition accuracy from 62.59 to 76.83, whereas CoT reasoning mildly helps overall but harms QA and dialogue.Summarization does not receive external knowledge because its source document already provides the needed information.
- Improving Recognition: Providing ground-truth examples for sample contrast yields the worst results, indicating that highly similar factual and hallucinated samples confuse LLMs.This result also indicates that HaluEval is challenging for hallucination evaluation.
4 Related Work
Prior work addresses LLM hallucination through verification, external evidence, fact critics, and analyses of its causes, while hallucination benchmarks and metrics often remain task- or model-specific. HaluEval responds by providing a benchmark centered on general user queries and ChatGPT responses to expose LLM blind spots.
- Hallucination in LLMs: Prior studies mitigate hallucination using verification systems, structured-data evidence, and token-level fact critics, while analyzing causes across tasks remains an active research direction.These approaches include verification for non-factual entities in summarization (Zhao et al., 2020), structured interfaces such as knowledge graphs and databases (Lan et al., 2022), and cross-task analyses (Das et al., 2023; Cao et al., 2022).
- Hallucination Evaluation: HaluEval differs by building a benchmark with general user queries and ChatGPT responses that can serve as a public platform for exposing LLM blind spots.The work positions its dataset as a benchmark rather than another task-specific evaluation resource.
- Hallucination Evaluation: Existing hallucination evaluations span NLP tasks, but examples such as BEGIN classify dialogue outputs into only three categories and can be limited to single tasks and small models.The cited evaluation line includes work by Dziri et al. (2022b), Gupta et al. (2022), Dziri et al. (2022a), Rashkin et al. (2021), and Li et al. (2023b).
- Hallucination Evaluation: Existing metrics quantify hallucination in specialized settings, including PARENT for table-to-text n-gram lexical entailment and TRUE for example-level area under the ROC curve.PARENT is attributed to Dhingra et al. (2019), while TRUE is attributed to Honovich et al. (2022).
5 Conclusion
HaluEval is a large-scale benchmark of generated and human-annotated hallucination samples for evaluating LLMs’ recognition abilities. It uses sampling-then-filtering to create diverse, difficult samples and shows that LLMs often fail to recognize or avoid hallucinations, motivating strategies to improve recognition.
- HaluEval provides generated and human-annotated hallucination samples for evaluating LLMs’ ability to recognize hallucinations.
- Sampling-then-filtering generates diverse samples through two sampling methods before filtering and selecting difficult cases.
- Existing LLMs mostly fail to recognize hallucinations and tend to generate hallucinated content, motivating strategies to improve recognition.
6 Limitations
The benchmark’s generated hallucination quality depends on ChatGPT’s instruction-following capacity despite filtering, and its samples may be misused because they resemble ground-truth samples.
- Limitations: ChatGPT-generated hallucinated samples remain limited by its ability to follow complex sampling instructions, so quality control is still necessary despite filtering.The approach uses ChatGPT for automatic sample generation, making generation quality dependent on the model’s capacity.
- Limitations: Because hallucinated samples closely resemble ground-truth samples, the benchmark could be misused, requiring monitoring and regulation of its spread and use.
Appendix
The appendix provides supplementary benchmark information, covering hallucination sampling, filtering, evaluation, and benchmark details.
- Appendix A: Appendix A presents instructions for hallucination sampling.
- Appendix B: Appendix B presents instructions for hallucination filtering.
- Appendix C: Appendix C presents instructions for evaluation.
- Appendix D: Appendix D presents details of the benchmark.
A Hallucination Sampling
The hallucination sampling instructions for dialogue and summarization are presented in Table 9 and Table 10, respectively.
- A Hallucination Sampling: The sampling instructions are provided separately for dialogue in Table 9 and summarization in Table 10.
B Hallucination Filtering
This section presents the hallucination-sampling instructions for dialogue and summarization, provided in separate tables.
- The dialogue hallucination-sampling instructions are presented in Table 11.
- The summarization hallucination-sampling instructions are presented in Table 12.
- The sampling instructions are organized separately for dialogue and summarization.
C Hallucination Recognition
The hallucination-recognition instructions are specified separately for question answering, dialogue, and summarization. These instructions appear in Tables 13, 14, and 15, respectively.
- Table 13 presents the hallucination-recognition instructions for question answering.
- Table 14 presents the hallucination-recognition instructions for dialogue.
- Table 15 presents the hallucination-recognition instructions for summarization.
D Details of HaluEval
This section documents the instructions used to sample, filter, and recognize hallucinations across knowledge-grounded dialogue, text summarization, and question answering, and reports generated-sample counts by hallucination pattern.
- Table 16 reports the number of generated samples for each hallucination pattern, P-I through P-IV.For dialogue and summarization, the table considers three patterns; “Summa.” abbreviates summarization.
- Tables 9–12 specify hallucination sampling and filtering instructions for knowledge-grounded dialogue and text summarization.
- Tables 13–15 specify hallucination-recognition instructions for question answering, knowledge-grounded dialogue, and text summarization.