Source-linked AI summary
Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models
Yuyan Chen, Qiang Fu, Yichen Yuan, Zhihao Wen, Ge Fan, Dayiheng Liu, Dongmei Zhang, Zhixu Li, Yanghua Xiao
TL;DR
LLM hallucinations create unfaithful or inconsistent answers, motivating reliable detection. The paper introduces RelD, trained on the bilingual RelQA dataset and comprehensive metrics, and finds effective detection across diverse LLMs and both in-distribution and out-of-distribution data.
Problem
LLM hallucinations produce unfaithful or inconsistent content, creating a critical reliability problem for generated answers.
Method
RelD is trained on RelQA, a bilingual question-answering dialogue dataset containing LLM-generated answers evaluated with LLM, human, machine, and composite metrics.
Results
RelD effectively detects hallucinations across answers from different LLMs and performs well on both in-distribution and out-of-distribution datasets.
Takeaways & Limitations
The findings contribute to detecting reliable LLM-generated answers and provide implications for future hallucination mitigation.
Takeaways & Limitations
Multi-class classification can introduce overly complex information that diverges from human intuitive perception of reliability.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks, including question answering and dialogue systems. However, a major drawback of LLMs is the issue of hallucination, where they generate unfaithful or inconsistent content that deviates from the input source, leading to severe consequences. In this paper, we propose a robust discriminator named RelD to effectively detect hallucination in LLMs' generated answers. RelD is trained on the constructed RelQA, a bilingual question-answering dialogue dataset along with answers generated by LLMs and a comprehensive set of metrics. Our experimental results demonstrate that the proposed RelD successfully detects hallucination in the answers generated by diverse LLMs. Moreover, it performs well in distinguishing hallucination in LLMs' generated answers from both in-distribution and out-of-distribution datasets. Additionally, we also conduct a thorough analysis of the types of hallucinations that occur and present valuable insights. This research significantly contributes to the detection of reliable answers generated by LLMs and holds noteworthy implications for mitigating hallucination in the future work.
1 INTRODUCTION
LLM hallucinations produce factual errors that threaten reliable use, especially in high-stakes domains. The paper addresses this challenge with RelD, trained on RelQA and comprehensive evaluation metrics, and reports robust detection across models and dataset settings.
- Factual errors in generated text create hallucinations that hinder LLM adoption, particularly where accuracy is crucial.The paper identifies medicine and finance as examples of domains where erroneous information can cause economic losses or jeopardize safety.
- Hallucinations arise from complex factors including erroneous training information and models’ tendency to prioritize parameterized knowledge during generation.
- Existing hallucination detection uses statistical, model-based, and human-based evaluations, but lacks a comprehensive balance across evaluation criteria and increases research costs.The paper motivates combining automatic metrics with model-based evaluation to better align with human evaluation trends.
- RelD combines a bilingual generated-answer dataset with comprehensive metrics to detect hallucinations in LLM answers.RelQA includes answers generated by LLMs and metrics spanning LLM assessment, human, machine, and composite evaluations.
- RelD effectively detects hallucinations across answers generated by different LLMs and performs well on both in-distribution and out-of-distribution datasets.The paper also analyzes hallucination types and their underlying causes.
2 DATA CONSTRUCTION
RelQA combines nine heterogeneous question-answering datasets with LLM-generated answers and multiple evaluation metrics to train the RelD hallucination discriminator. Its exploratory analysis shows substantial variation across datasets and metrics, including reliability differences between SQuAD and QUAC.
- Dataset construction: RelQA integrates nine datasets spanning extractive reading comprehension, multiple-choice, and multi-turn dialogue, then standardizes and preprocesses them for answer generation.The sources include Wikipedia, Baidu Zhidao, Bing search, education, news, web articles, and general knowledge; question-adaptive prompts address dataset-specific formats.
- Dataset construction: Answers are generated with LLaMA, BLOOM, GPT-J, GPT-3, and GPT-3.5, with repeated generation used to improve answer stability.Long texts are processed with sliding windows, and outputs are retained for later filtering and selection.
- Metric selection: RelQA evaluates answers using LLM-assessment, human, machine, and composite metrics that capture quality, alignment, correctness, overlap, similarity, diversity, and aggregate reliability.The human metric uses answer-reference matching and goodness scores, while the composite metric combines multiple measures into a final score and binary tag.
- Metric selection: The composite final tag labels answers reliable when the weighted average score exceeds 0.5, with Recall and ROUGE sometimes receiving twice the weight.Weights can be dynamically optimized, and the final tag summarizes whether an answer is considered reliable.
- Data exploratory analysis: SQuAD has the highest goodness percentage at 99.47% and final reliability proportion at 78.57%, whereas QUAC has 82.72% goodness and 16.44% final reliability.MSMARCO has the highest similarity percentage at 74.89%, while QUAC has the lowest at 60.28%.
- Data exploratory analysis: Human evaluations vary sharply across datasets, with reliable proportions from 0.42% for DuReader-master to 32.79% for SQuAD and ambiguous evaluations reaching 96.38% for NewsQA.Unreliable proportions range from 0.49% for SQuAD to 17.16% for QUAC.
3 DISCRIMINATOR
RelD is a discriminator trained to assess the reliability of LLM-generated answers by fitting human-aligned reliability scores through multi-class and binary classification. It uses a pretrained language-model backbone, weighted probability conversion, and cross-entropy training.
- 3.1 REGRESSION TO MULTI-CLASS CLASSIFICATION: The method converts reliability-score regression into multi-class classification, ultimately selecting ten categories after comparing alternative category counts.
- 3.1 REGRESSION TO MULTI-CLASS CLASSIFICATION: Cross-entropy trains RelD by comparing true category labels with predicted class probabilities produced through softmax.
- 3.2 MULTI-CLASS TO BINARY-CLASS CLASSIFICATION: RelD converts multi-class predictions into binary reliability decisions using normalization, discrete values, or weighted average probability.
- 3.2 MULTI-CLASS TO BINARY-CLASS CLASSIFICATION: Weighted average probability uses class probabilities and weights, with a 0.5 threshold, to provide a refined fit for regression-oriented reliability decisions.
- RelD takes questions, contexts, and LLM-generated answers as input and produces a reliability classification label.
- ELECTRA is selected as RelD’s backbone because experiments show it outperforms BERT, RoBERTa, and DeBERTa on the reported metrics.
4 EXPERIMENTS
Experiments evaluate RelD across multiple LLMs, distribution settings, design choices, metric weights, backbones, and prediction categories. RelD exceeds 0.8 across automatic and human-in-the-loop evaluations for all selected LLMs and remains above 0.7 on IID and OOD data when the IID ratio is at least five.
- 4.2 MAIN RESULTS: Above 0.8 on both automatic and human-in-the-loop evaluations across all selected LLMs, RelD shows robust reliability detection with minimal model variation.The evaluations are significantly correlated, suggesting automatic RelQA scoring can largely replace human scoring.
- 4.2 MAIN RESULTS: Above 0.7 on both IID and OOD datasets when the IID ratio is at least five, RelD demonstrates strong generalization beyond its training distribution.OOD evaluation uses the entire OOD dataset, while IID evaluation uses 30% of the IID data.
- 4.3 ABLATION STUDY: Weighted average probability outperforms normalization and discrete values on human-in-the-loop metrics, despite slightly underperforming normalization on automatic metrics.
- 4.3 ABLATION STUDY: Ten categories improve human-in-the-loop performance relative to four, six, and eight categories, leading RelD to use ten-class classification.
- 4.3 ABLATION STUDY: Optimizing metric weights improves RelD on both automatic and human-in-the-loop metrics compared with the original weights.
- 4.4 EXPLORATORY ANALYSIS: RelD performs best for one prediction category, while other categories are more variable because contexts or questions may be complex and ambiguous.
5 RELATED WORK
Related work frames hallucination detection through statistical, model-based, and human-based evaluations, alongside separate efforts to mitigate hallucination. These approaches span automatic metrics, learned evaluators, and human judgments.
- Hallucination detection research includes statistical metrics, model-based metrics, and human-based evaluations.
- Hallucination mitigation studies address machine translation, summarization, prompting, generation quality, and iterative self-evaluation.
6 CONCLUSIONS AND FUTURE WORK
The paper proposes RelD, trained on the bilingual RelQA dataset, to detect hallucinations in LLM-generated answers. RelD performs well on both in-distribution and out-of-distribution datasets.
- RelD effectively detects hallucinations in LLM-generated answers and performs well on both in-distribution and out-of-distribution datasets.