Source-linked AI summary
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, Tong Zhang
TL;DR
RAG systems can still generate claims unsupported by retrieved evidence, but high-quality datasets for measuring this problem are scarce, especially in RAG settings. The paper introduces RAGTruth, a large corpus of naturally generated responses with fine-grained annotations, and benchmarks detection methods. Fine-tuning Llama with RAGTruth achieves competitive performance, while span-level hallucination detection remains challenging.
Problem
High-quality, large-scale datasets for hallucination detection are scarce in RAG settings, where naturally occurring hallucinations are relatively infrequent and existing resources are often synthetic, small, or not RAG-focused.
Method
The paper introduces RAGTruth, a large-scale RAG corpus of nearly 18,000 naturally generated responses with word-level annotations, and uses it to benchmark LLM hallucinations and detection methods.
Results
Fine-tuning Llama-2-13B on RAGTruth achieves performance competitive with prompt-based hallucination detection approaches using GPT-4, and the resulting detector reduces hallucinations in generated responses.
Takeaways & Limitations
RAGTruth supports developing specialized hallucination detection models for retrieval-augmented generation using fine-grained naturally generated-response annotations.
Takeaways & Limitations
The benchmark may not address situations that are significant for some practical applications, because hallucination scenarios and detection techniques continue to evolve.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) has become a main technique for alleviating hallucinations in large language models (LLMs). Despite the integration of RAG, LLMs may still present unsupported or contradictory claims to the retrieved contents. In order to develop effective hallucination prevention strategies under RAG, it is important to create benchmark datasets that can measure the extent of hallucination. This paper presents RAGTruth, a corpus tailored for analyzing word-level hallucinations in various domains and tasks within the standard RAG frameworks for LLM applications. RAGTruth comprises nearly 18,000 naturally generated responses from diverse LLMs using RAG. These responses have undergone meticulous manual annotations at both the individual cases and word levels, incorporating evaluations of hallucination intensity. We not only benchmark hallucination frequencies across different LLMs, but also critically assess the effectiveness of several existing hallucination detection methodologies. Furthermore, we show that using a high-quality dataset such as RAGTruth, it is possible to finetune a relatively small LLM and achieve a competitive level of performance in hallucination detection when compared to the existing prompt-based approaches using state-of-the-art large language models such as GPT-4.
1 Introduction
RAGTruth addresses the lack of large, high-quality hallucination datasets for RAG by providing fine-grained annotations of naturally generated LLM responses. The paper benchmarks hallucination and detection methods, showing competitive detection after fine-tuning and reduced hallucinations using the resulting detector.
- Motivation: RAG settings remain vulnerable to unsupported or contradictory claims despite retrieved context, while existing hallucination datasets are often synthetic, small, or not RAG-focused.This limits understanding and evaluation of hallucinations in retrieval-augmented applications.
- Dataset: RAGTruth is a large-scale dataset for word-level hallucination evaluation across several common RAG tasks.It contains nearly 18,000 fully annotated natural responses generated by open-source and closed-source LLMs.
- Evaluation: The paper compares hallucination detection methods at both passage and word levels using the RAGTruth corpus.The contribution includes evaluating different approaches to identifying hallucinated content in RAG responses.
- Detection results: Fine-tuning Llama-2-13B on RAGTruth achieves performance competitive with prompt-based approaches using GPT-4.The result supports developing specialized hallucination detectors from high-quality RAG data.
- Detection results: The fine-tuned hallucination detector significantly reduces hallucinations in LLM responses, including responses from models with inherently low hallucination rates such as GPT-4.The reported improvement extends beyond models with high baseline hallucination rates.
2 Related Work
Prior hallucination research spans conventional NLG benchmarks, synthetic and natural-response datasets, uncertainty measures, and proxy-model checking. RAGTruth differs by focusing on hallucination evaluation under retrieval-augmented generation, where responses must remain consistent with retrieved context.
- Hallucination benchmarks: Hallucination benchmarks have been widely studied in conventional natural language generation, while LLM evaluation has increased demand for high-quality datasets.Existing resources include HaluEval and FELM, which contain synthetic or naturally generated responses with factual hallucination annotations.
- RAG-focused evaluation: Prior datasets often annotate factual hallucinations in LLM responses, whereas RAGTruth centers evaluation on RAG settings.The distinction is the explicit focus on consistency between retrieved information and generated responses.
- Detection methods: Hallucination detection methods use intrinsic uncertainty metrics, accessible proxy models, or other model-based checking strategies.These approaches address settings where direct access to a model’s output uncertainty may be unavailable.
3 Construction Process of RAGTruth
RAGTruth was constructed through response generation with multiple LLMs followed by human span-level annotation, covering three RAG tasks and four hallucination categories. Its annotations also preserve context-dependent cases and special labels for implicit truths and null-value errors.
- Hallucination Taxonomy: The taxonomy distinguishes evident and subtle conflicts from evident and subtle introductions of baseless information.Evident conflicts directly oppose references, while subtle conflicts alter contextual meaning; baseless-information categories cover unsupported or inferred additions.
- Tasks and Data Sources: RAGTruth covers question answering, data-to-text writing, and news summarization in RAG settings.Question answering uses MS MARCO passages, data-to-text writing uses Yelp business records, and news summarization uses CNN/Daily Mail and recent news articles.
- The data pipeline generates responses with multiple LLMs and natural prompts, then has human labelers annotate hallucinated spans.
- Models: Six instruction-following models each produce one response per input sample, yielding six responses for every sample.The models include GPT-3.5-turbo, GPT-4, Mistral-7B-Instruct, and three Llama-2 chat models.
- Human Annotation: Each response is independently labeled by two annotators, with a third review when their annotations differ considerably.The reported consistency rates are 91.8% at the response level and 78.8% at the span level.
- Adaptive Evaluation: The corpus labels information absent from references as hallucination even when it may be truthful, and separately marks implicit_true and due_to_null spans.These labels accommodate strict RAG evaluation and cases where null values are interpreted as false or negative information.
4 Hallucination Benchmark Analysis
The benchmark analysis characterizes hallucination patterns across tasks, models, lengths, and response positions. It finds that unsupported information is more common than conflicting information, while hallucinations vary with task, model scale, length, and location.
- Dataset Statistics: 2,965 instances span question answering, data-to-text writing, and news summarization, with six model responses per instance.The dataset includes 989 question-answering, 1,033 data-to-text, and 943 news-summarization instances.
- Hallucination Types: Baseless information is significantly more prevalent than information conflicting with context, especially in question answering.Evident baseless information and evident conflict constitute significant portions of their respective categories.
- Hallucination vs Tasks: Data-to-text writing has the highest response-level hallucination frequency across the three tasks.Inconsistent handling of JSON fields, particularly time and attributes, contributes substantially to hallucinations in this task.
- Hallucination vs Models: GPT-4-0613 exhibits the lowest hallucination frequency among the collected models.The two OpenAI models show notably lower hallucination rates than the other models.
- Hallucination vs Models: Hallucination density decreases with Llama-2 model scale except in data-to-text writing, while Mistral-7B-Instruct produces the most hallucinated responses.Hallucination density is the average number of hallucination spans per hundred response words.
- Hallucination vs Length: The average number of hallucinations per response generally increases with response length, while context length significantly increases hallucinations only in news summarization.The authors attribute the task difference to greater structure in the other two contexts.
- Location of Hallucinations: Hallucinations are more likely near response ends in question answering and news summarization, whereas data-to-text hallucinations occur relatively more often in the first half.Business-attribute hallucinations frequently occur in the brighter region of the data-to-text heatmap.
5 Experimental Setup
The experiments compare prompt-based, sampling-based, cross-examination, and fine-tuned hallucination detectors on a common RAGTruth test set. Evaluation covers both response-level identification and the more demanding task of locating hallucinated spans.
- Hallucination Detection Algorithms: Four detection approaches are evaluated: hallucination-detection prompts, SelfCheckGPT, LMvLM, and Llama-2-13B fine-tuning.The fine-tuned model receives context-response pairs and generates hallucinated spans as its target output.
- Evaluation Data: All algorithms use the same 450-instance test set, randomly sampled with 150 instances from each task.The remaining data is used to fine-tune Llama-2-13B.
- Evaluation Metrics: Detection is assessed at both response and span levels, with span detection evaluated by character-level overlap against human-labeled spans.Response-level and span-level results use precision, recall, and F1 scores.
- Evaluation Metrics: Span localization is treated as more challenging and significant than merely determining whether a response contains hallucinations.
6 Experimental Results
Experiments show that hallucination detection remains difficult in RAG, although fine-tuning Llama-2-13B on RAGTruth substantially improves average F1 and detector-assisted selection reduces hallucination rates.
- 63.4% average F1 is achieved by GPT-4-turbo, while SelfCheckGPT with GPT-3.5-turbo reaches 58.8%.
- 78.7% average F1 is achieved by the fine-tuned Llama-2-13B, the best performance reported in the comparison.
- 18.4% precision is obtained by GPT-4-turbo for span-level detection, indicating substantial misclassification of non-hallucinated content.
- Evident hallucination spans are detected more effectively than subtle hallucination spans.
- The detector-assisted selection strategies reduce hallucination rates relative to random selection, including reductions of 21.6% and 63.2% for smaller models.
7 Conclusion
The paper introduces RAGTruth as a naturally generated, word-level hallucination corpus for RAG and benchmarks hallucination detection with it. Fine-tuning Llama on the corpus is competitive with prompt-based general-model methods, while span-level detection remains challenging.
- RAGTruth is a large-scale corpus of naturally generated hallucinations with detailed word-level annotations tailored for RAG scenarios.
- The corpus supports analysis of hallucinations across task types, models, and contextual settings.
- Fine-tuning Llama with RAGTruth achieves competitive hallucination-detection performance against prompt-based methods using general models such as GPT-4.
- Span-level hallucination detection in RAG remains a formidable challenge, with current methods falling short of reliable detection.
8 Limitations
The authors note that the rapidly evolving hallucination field leaves some practically significant situations outside the study’s coverage.
- Some significant situations for particular practical applications may not be addressed because application scenarios, hallucination sources, and detection techniques continue to evolve.
9 Ethical considerations
The paper states that it complies with the ACL Ethics Policy and reports possible inaccuracies in the LLM-generated responses.
- The authors report full compliance with the ACL Ethics Policy.
- LLM-generated responses in the study may contain inaccuracies.
- The authors report no additional ethical issues to the best of their knowledge.
DATA-TO-TEXT WRITING
The data-to-text task prompts models to generate objective 100–200-word business overviews from structured JSON and customer reviews without inventing information. The stated word-count control limits summarization length but is not used for hallucination annotation.
- The task asks for an objective 100–200-word local-business overview based only on structured JSON data and customer reviews.
- The generation prompt explicitly instructs models not to make up information.
- The word-count requirement controls summarization length and does not determine hallucination annotations.
C Annotation Details
The annotation materials include a privacy-masked interface and analyze implicit truths and null-related hallucinations. Detection prompts clarify that null or None values in JSON represent unknown information rather than negation.
- The annotation-interface screenshot masks annotators’ full names for privacy reasons.
- Implicit truths occur mainly in question-answering and data-to-text writing tasks.
- About 17.7% of hallucination spans in data-to-text writing relate to null values in the JSON data.
- The data-to-text detection prompt treats null or None in JSON as unknown rather than as negation.