PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-grained Hallucination Detection and Editing for Language Models

Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, Hannaneh Hajishirzi

arXiv:2401.06855v4cs.CL

TL;DR

Language models produce diverse factual or unverifiable errors, but existing approaches often detect them only coarsely. The paper introduces a six-type taxonomy, FAVABENCH, and retrieval-augmented FAVA trained on synthetic errors; FAVA outperforms strong baselines on fine-grained detection and improves factuality through editing.

  • Problem

    Existing hallucination systems often use binary or entity-level judgments, although LM hallucinations take diverse forms requiring fine-grained verification.

  • Method

    The paper builds a six-category taxonomy and FAVABENCH, then trains retrieval-augmented FAVA on 35k synthetic instances to detect and edit hallucinations.

  • Results

    FAVA significantly outperforms ChatGPT, Ret-ChatGPT, and GPT-4 on fine-grained detection, while its edits improve factuality scores across Alpaca 7B, Alpaca 13B, and ChatGPT.

  • Takeaways & Limitations

    Fine-grained hallucination detection and editing can address multiple error types beyond binary factuality judgments in information-seeking LM outputs.

  • Takeaways & Limitations

    The study excludes errors in common-sense, numerical, and logical reasoning and defines world knowledge through relevant web documents.

Abstract

from arXiv · show

Large language models (LMs) are prone to generate factual errors, which are often called hallucinations. In this paper, we introduce a comprehensive taxonomy of hallucinations and argue that hallucinations manifest in diverse forms, each requiring varying degrees of careful assessments to verify factuality. We propose a novel task of automatic fine-grained hallucination detection and construct a new evaluation benchmark, FavaBench, that includes about one thousand fine-grained human judgments on three LM outputs across various domains. Our analysis reveals that ChatGPT and Llama2-Chat (70B, 7B) exhibit diverse types of hallucinations in the majority of their outputs in information-seeking scenarios. We train FAVA, a retrieval-augmented LM by carefully creating synthetic data to detect and correct fine-grained hallucinations. On our benchmark, our automatic and human evaluations show that FAVA significantly outperforms ChatGPT and GPT-4 on fine-grained hallucination detection, and edits suggested by FAVA improve the factuality of LM-generated text.

1 Introduction

The paper frames hallucinations as diverse factual or unverifiable errors requiring fine-grained detection, then introduces a taxonomy, benchmark, and retrieval-augmented editor.

  • Motivation: Prior systems often reduce hallucinations to binary factuality judgments or entity-level errors, despite requiring different verification effort.Entity contradictions may need one reference, whereas fabricated entities can require checking multiple sources.
  • Contribution: The proposed task identifies hallucinated spans, assigns taxonomy types, and suggests refinements for information-seeking outputs grounded in world knowledge.The taxonomy hierarchically organizes hallucinations into six categories and was developed through NLP-expert pilot studies.
  • Benchmark: Approximately 1,000 annotations cover responses from ChatGPT and Llama2-Chat models across diverse knowledge-intensive queries.FAVABENCH annotates erroneous spans, error types, and potential refinements.
  • Findings: 59.8% of ChatGPT responses and 70.2% of Llama2-Chat 7B responses contain at least one hallucination.Unverifiable sentences comprise over 60% of LM-generated hallucinations.
  • FAVA: FAVA is a retrieval-augmented LM trained on synthetic data to detect and edit fine-grained hallucinations.It significantly outperforms ChatGPT and improves factuality scores for Alpaca 7B, 13B, and ChatGPT outputs.

2 Related Work

Related work largely studies hallucinations relative to source text, binary factuality, or entity-level errors; this paper targets span-specific factual errors grounded in world knowledge.

  • Hallucination Taxonomies: Prior hallucination taxonomies for summarization, simplification, and dialogue generally assume a source text and assess faithfulness to it.The paper distinguishes these settings from factual hallucinations grounded in world knowledge.
  • Detection and Editing: Recent detection and editing systems commonly use binary factuality labels, focus on entity-level edits, or address detection without fine-grained span tracing.Related surveys may broaden hallucination definitions beyond errors traceable to specific text spans.
  • Fact Verification: Fact-verification research evaluates inaccuracies in human-written claims using datasets built from sources such as Wikipedia, scientific articles, and news.The paper presents this as a related but distinct research area from model-generated hallucination detection.

3 Fine-grained Hallucination Detection

The paper defines hallucinations against retrieved world knowledge, organizes them into six fine-grained types, and formalizes detection and editing tasks with type-specific evaluation.

  • Focus and Definitions: Hallucinations are factual errors or unverified statements in open-ended information-seeking generation, assessed against relevant web documents.Common-sense, numerical, and logical reasoning errors are outside the stated scope.
  • Hallucination Taxonomy: The taxonomy separates contradictory statements from unverifiable statements and includes entity, relation, sentence, invented, subjective, and unverifiable errors.Contradictory errors conflict with world knowledge, while unverifiable errors lack directly supporting or contradicting retrieved evidence.
  • Tasks: The detection task identifies erroneous text spans and their error types, while the editing task proposes changes that improve output factuality.Edits may minimally revise spans, remove hallucinations, or mark statements as unverifiable.
  • Metrics: Fine-grained detection is evaluated with precision, recall, and F1 averaged across the six error types.Binary detection additionally predicts whether a sentence contains any factual error.

4 Benchmark: FAVABENCH

FAVABENCH provides roughly 1,000 fine-grained annotations across three language models and diverse information-seeking prompts, revealing frequent and varied hallucinations.

  • Benchmark Construction: FAVABENCH contains around 1,000 fine-grained annotations on three LM responses to queries spanning multiple domains.The source prompts comprise 200 information-seeking queries from four data sources.
  • Annotation: The benchmark collects 600 zero-shot responses from ChatGPT, Llama2-Chat 7B, and Llama2-Chat 70B, annotated by two annotators per instance.Twenty students performed factual-accuracy annotation using the proposed taxonomy.
  • Analysis: 59.8% of ChatGPT, 70.2% of Llama2-Chat 7B, and 64.9% of Llama2-Chat 70B responses contain at least one hallucination.Entity errors are most frequent, while invented and contradictory errors account for 14.1% and 18.1% of detected errors.
  • Synthetic Data: FAVA’s synthetic-data pipeline generates seed passages, inserts factual errors, and creates diverse training examples aligned with the hallucination taxonomy.The pipeline uses factually accurate passages and produces erroneous outputs for training.

5 Model: FAVA

FAVA combines retrieval with an editing language model to detect and correct fine-grained hallucinations. It is trained using synthetic examples created by inserting taxonomy-controlled errors into diverse passages and post-processing the resulting edits.

  • Model architecture: FAVA retrieves relevant documents before an editing LM identifies hallucinations and, when possible, corrects them in the original output.The edited output interleaves corrections with error tags indicating hallucination types.
  • Training objective: FAVA training instances pair gold context, erroneous output, and a corrected output containing error tags.The editing model learns to generate the tagged, corrected output from the context and erroneous text.
  • Synthetic training data: The synthetic-data pipeline generates seed passages, inserts taxonomy-based factual errors, and post-processes examples for training.Error insertion uses ChatGPT and GPT-4 to add six error types while retaining previously inserted errors.
  • Synthetic training data: Seed passages draw on 35,074 Wikipedia articles and 5,000 Natural Questions pairs, with Wikipedia content paraphrased into varied genres.The genre paraphrasing is intended to expose FAVA to diverse textual formats while retaining control over factuality.
  • Training data statistics: 35,074 training instances were generated, with error types distributed at approximately 15% each and differing average error counts across subsets.Wikipedia subsets averaged 3.1 inserted errors, while question-answer pairs averaged 1.4.
  • Model implementation: FAVA initializes its editing model with Llama2-Chat 7B and retrieves the top five documents during inference.The authors report slightly better performance with this initialization than with pretrained Llama2.

6 Experiments

The experiments evaluate fine-grained and binary hallucination detection on annotated benchmark data, compare FAVA with language-model baselines, and assess editing through factuality scores. Human evaluation is added because automatic metrics may not fully capture detection and refinement quality.

  • Detection evaluation: The detection evaluation uses 902 annotated passages and measures sentence-level per-category classification in both fine-grained and binary settings.Fine-grained detection is presented as the newly proposed task, whereas prior work mainly used binary detection.
  • Evaluation data: The benchmark uses English Wikipedia data from January 2023, with embeddings generated by Contriever.The authors report collecting 1,010 annotated passages including experimental batches, while earlier batches are excluded from evaluation.
  • Detection baselines: The detection baselines include ChatGPT, GPT-4, retrieval-augmented ChatGPT, and FActScore for binary detection only.GPT-4 with retrieval is omitted because of high costs, while FActScore decomposes responses into atomic facts for verification.
  • Editing evaluation: Editing evaluations use biography-generation data and assess whether edited outputs reduce hallucinations through FActScore.The target-task factuality metric is applied to the edited outputs.
  • Editing baselines: Editing baselines include ChatGPT, retrieval-augmented ChatGPT, Llama2-Chat 13B, and retrieval-augmented Llama2-Chat.Baseline prompts use two demonstrations, and retrieval supplies one string-matched document plus four Contriever documents reranked by a cross-encoder.
  • Human evaluation: Automatic evaluations may not fully capture detection and refinement ability because annotations and factuality metrics can be subjective.Human annotators therefore verify randomly sampled outputs from FAVA and the highest-scoring baseline, retrieval-augmented ChatGPT.

7 Results and Analysis

FAVA outperforms strong baselines on fine-grained and binary hallucination detection, while improving factuality through editing. Its performance varies by error type and depends on retrieval quality and training-data scale.

  • Detection results: FAVA significantly outperforms ChatGPT, retrieval-augmented ChatGPT, and GPT-4 on fine-grained and binary error detection.
  • Detection results: FAVA performs well on Sentence, Subjective, and Entity errors, but remains limited on Invented and Unverifiable errors.These latter error types often require searching beyond FAVA’s default top five retrieved documents.
  • Detection results: FAVA outperforms all other models by a large margin on binary hallucination detection.
  • Editing results: FAVA produces the largest FActScore gains in biography editing, outperforming prompted ChatGPT and Llama2-Chat 70B despite being smaller.The authors attribute the gains to editing and error-type detection effectiveness.
  • Human evaluation: FAVA exceeds retrieval-augmented ChatGPT by 46.3% on editing and 31.8% on detection in human evaluation.It also identifies 2.4 errors per passage versus 1.9 for retrieval-augmented ChatGPT.
  • Analysis: Larger synthetic training sets improve fine-grained error detection, while retrieving five documents improves editing over retrieving one by 4.5%.Entity matching raises performance to 50.1%, a 3.1% gain over top-five retrieval and 7.6% over unedited generations.

8 Conclusions

The paper introduces a taxonomy-based framework for fine-grained hallucination detection and editing, supported by a human-annotated benchmark and a retrieval-augmented model. Empirical results show that FAVA outperforms strong baselines on both tasks, although substantial room for improvement remains.

  • The paper introduces a taxonomy-based task, benchmark, and retrieval-augmented model for fine-grained hallucination detection and editing.
  • FAVA significantly outperforms strong baselines on both editing and fine-grained detection, while substantial room for improvement remains.

A Details of Human Annotations

The human-annotation materials document the query sources, annotation interface, training, instructions, and annotated examples. They also summarize the distribution of hallucination labels across the collected datasets.

  • Annotation materials: The annotation materials include seed-query examples and source-prompt tables for constructing the human-annotated data.
  • Annotation materials: The annotation interface and annotator instructions are documented through figures and accompanying descriptions.
  • Annotator training: Ten computer-science undergraduates annotated passages after a 45-minute one-on-one training session covering the six hallucination types and platform use.
  • Annotation results: The appendix provides annotated examples from WebNLG, OpenAssistant, and Instruction Following datasets, together with hallucination-percentage statistics.

B More Details of Synthetic Data Creation and Training

The appendix details synthetic-data quality checks, text diversification prompts, training statistics, hyperparameters, detection metrics, and human evaluation procedures. It also reports limitations in generating nuanced errors and specifies how evaluation data were assessed.

  • Synthetic-data quality: Pilot studies found that ChatGPT often confuses multiple requested error types and struggles to generate plausible perturbations for nuanced errors.
  • Synthetic-data construction: Synthetic-data materials include diversification prompts and statistics describing generated passages and error-type distributions.
  • Training details: The appendix reports the Llama2-Chat 7B training setup, including four A40 GPUs and training hyperparameters.
  • Evaluation details: Additional tables report fine-grained detection F1, precision, recall, and baseline prompts for evaluation.
  • Human evaluation: Human evaluation anonymized outputs and asked annotators to judge detection and editing correctness using extensive web searches.

C.4 Manual Analysis on Generated Data for FAVA training

The analysis evaluates automatically generated training data for validity and quality through human judgments. Iterative insertion produces more valid and realistic edits than one-shot prompting, while the training set contains 35,074 instances with broadly balanced error types.

  • Human evaluation: Human annotators scored 50 generated examples for validity and quality, using 0–2 ratings for each criterion.Validity measures alignment with the annotation scheme; quality measures whether inserted errors are feasible, realistic, and distinct from few-shot examples.
  • Human evaluation: 1.66 validity and 1.36 quality were achieved by iterative insertion, compared with 1.1 and 0.9 for one-shot prompting.The evaluation concludes that iterative insertion produces higher-quality synthetic data and realistic factual errors.
Loading 2401.06855v4…