Source-linked AI summary
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, Bill Dolan
TL;DR
The paper addresses hallucination detection when free-form generation lacks references and sentence- or document-level methods provide insufficiently fine-grained signals. It proposes a reference-free, token-level task with the HADES benchmark, constructed from perturbed Wikipedia text and crowd annotations using iterative model-in-the-loop sampling, and establishes baseline evaluations in offline and online settings.
Problem
Free-form generation often lacks ground-truth references, while sentence- or document-level detection may not pinpoint hallucinated text for real-time use.
Method
The paper perturbs English Wikipedia segments, verifies spans with crowd-sourced annotations, and uses iterative model-in-the-loop sampling to address label imbalance.
Results
The paper introduces HADES and evaluates multiple baseline detectors in both offline and online settings.
Takeaways & Limitations
HADES provides a benchmark for fine-grained, reference-free hallucination detection in free-form text generation.
Takeaways & Limitations
Annotation compares perturbed text with original text because annotators had difficulty reaching consensus on GPT-3-generated text without that original.
Abstract
from arXiv · showhide
Large pretrained generative models like GPT-3 often suffer from hallucinating non-existent or incorrect content, which undermines their potential merits in real applications. Existing work usually attempts to detect these hallucinations based on a corresponding oracle reference at a sentence or document level. However ground-truth references may not be readily available for many free-form text generation applications, and sentence- or document-level detection may fail to provide the fine-grained signals that would prevent fallacious content in real time. As a first step to addressing these issues, we propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HaDes (HAllucination DEtection dataSet). To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations. To mitigate label imbalance during annotation, we utilize an iterative model-in-loop strategy. We conduct comprehensive data analyses and create multiple baseline models.
1 Introduction
The paper targets hallucinations in free-form generation where references are often unavailable and sentence- or document-level judgments lack fine-grained localization. It introduces a reference-free, token-level task and the HADES dataset to support detection research.
- Neural NLG systems can generate fluent but factually inaccurate or untrustworthy content, creating challenges for real-time production deployment.
- Reference-based hallucination detection is difficult to apply when free-form generation tasks lack readily available ground-truth references.
- Sentence- or document-level detection may fail to pinpoint which generated text is hallucinated.
- The paper proposes a reference-free, token-level hallucination detection task for free-form text generation.
- HADES contains approximately 11k English Wikipedia instances, uses iterative data collection to address label imbalance, and includes analyses and baseline models.
2 Task Overview
The task formulates hallucination detection as binary classification over highlighted spans, with separate offline and online settings that differ in available context.
- The task assigns hallucination or not-hallucination labels to highlighted text spans.
- In the offline setting, the detector can use bidirectional context after generation is complete.
- In the online setting, the detector uses only preceding context to simulate on-the-fly generation.
3 Dataset Creation
HADES is constructed by perturbing Wikipedia text into fluent, diverse alternatives and validating target spans through iterative, balanced crowd-sourced annotation. The resulting dataset combines contextual perturbations, human consensus, and analyses of linguistic and statistical hallucination patterns.
- Raw Data Collection: HADES starts from English WIKI-40B Wikipedia paragraphs filtered to at least five sentences, chosen for formal style and broad topic coverage.
- Contextual Perturbation: BERT-based replacement targets fluency and lexical diversity, while top-k sampling with k=10 provides a reported trade-off between diversity and coherence.
- Contextual Perturbation: The perturbation pipeline preserves the first two sentences as preceding context and applies MASK, REPLACE, and RANK operations from the third sentence onward.
- Data Annotation: The authors compare perturbed spans with original text during annotation because a pilot using GPT-3 text produced low consensus without the original reference.
- Data Annotation: The model-in-the-loop strategy addresses an initial imbalance of around 90% hallucination cases by subsampling predicted hallucinations using estimated TPR, TNR, and precision.
- Data Analysis: Hallucinations show higher entropy and higher average probability than factually consistent content, while named-entity perturbations are especially associated with hallucination labels.
4 Baseline Models
The paper benchmarks feature-based and pretrained transformer detectors for binary token-span hallucination classification. Transformer baselines use contextual span representations and are motivated by their ability to leverage context and embedded knowledge.
- Feature-based models: The benchmark includes feature-based classifiers such as logistic regression and SVM using statistical, model-based, POS, and NER features.
- Baseline motivation: The paper uses multiple baselines because the proposed token-level, reference-free setting was not covered by existing literature and lacked a perfectly matching baseline.
- Transformer-based models: The transformer baselines include BERT, GPT-2, XLNet, and RoBERTa, which can leverage context or embedded world knowledge for detection.
- Transformer-based models: Each pretrained model is fine-tuned to predict a binary hallucination label for a target span from its contextualized input representation.
- Transformer-based models: The target span representation is obtained by max pooling the model’s hidden states and passed through a tanh-activated MLP trained with cross entropy.
5 Experimental Setup
Experiments evaluate HADES detectors in offline and online contexts, distinguishing bidirectional from preceding-only context. The setup compares pretrained architectures and feature-based models with classification and calibration metrics.
- Baseline configurations: The transformer configurations use BERT-large, GPT2-medium, XLNet-large, and RoBERTa-large with Adam and model-specific learning rates.
- Baseline configurations: The selected transformer configuration uses max pooling, a two-layer MLP with hidden dimension h/2, and freezes the pretrained model through its last layer.
- Evaluation settings: Offline and online results are reported in Tables 2 and 3, respectively, with percentages and significant tests provided in the appendix.
- Evaluation settings: Offline evaluation provides bidirectional context, whereas online evaluation provides only left context; feature-based baselines are restricted to offline evaluation and GPT-2 to online evaluation.
- Evaluation metrics: The benchmark reports accuracy, precision, recall, F1, AUC, G-Mean, and Brier Skill Score for the imbalanced binary classification task.
6 Results
Baseline detectors perform better offline than online, while contextual information substantially improves detection until performance converges at longer context lengths. Case-study predictions also identify potentially hallucinated spans in GPT-3 text.
- Baseline performance: Offline detection outperforms online detection across baseline models, indicating that succeeding context helps identify hallucinations.Under online detection, GPT-2 achieves the best performance for most metrics.
- Baseline performance: Pretrained models outperform feature-based models by a large margin in the offline setting.The authors attribute this to pretrained models’ contextualized feature extraction.
- Baseline performance: No pretrained model is an obvious offline winner: RoBERTa leads ACC, XLNet leads not-hallucination F1, and BERT leads G-mean.In the online setting, GPT-2’s mean performance differs significantly from BERT, XLNet, and RoBERTa across reported metrics after Bonferroni correction.
- Context matters in HADES: Context-window performance grows rapidly below length 80 and gradually converges thereafter.The offline advantage is largest around context length 75 and vanishes for windows longer than 150 or shorter than 20.
- Model predictions on GPT-3 generated text: BERT-large assigns higher hallucination risk to selected spans in GPT-3 text, including “structures and buildings.”The case study contrasts this span with the related phrase “total greenhouse area including enclosed structures,” which is included in the census instruments.
7 Related Work
Prior hallucination detection commonly relies on references and sentence- or document-level judgments. The proposed task instead targets subtle hallucinations at token level without requiring references.
- Reference-based Hallucination Detection: Reference-based hallucination detection has been applied across summarization, machine translation, data-to-text, and image caption generation.Methods include n-gram matching, edit-distance heuristics, natural language inference, and information extraction.
- Reference-based Hallucination Detection: Reference-based detection is impractical for many free-form generation tasks because suitable references are not readily available.The related-work discussion includes real-time social chatbot responses as an example.
- Reference-based Hallucination Detection: Reference-based hallucination reduction methods include iterative training, post-editing, soft constraints, optimal transport, and plan-guided generation.The cited approaches use explicit or implicit plans to structure generated text.
- Reference-free Detection Approaches: Reference-free detection is related to fake-news detection and fact checking, which commonly involve common-sense and world-knowledge reasoning.These related tasks verify claims or identify disinformation without relying on a supplied reference.
- Reference-free Detection Approaches: The proposed task differs by identifying subtle hallucinations within mostly faithful text at token level rather than judging entire sentences or documents.Fake-news detection and specificity assessment are described as sentence- or document-level tasks.
8 Conclusions
The paper introduces HADES as a benchmark for token-level, reference-free hallucination detection in free-form generation. It constructs the dataset through perturbation and iterative model-in-the-loop annotation, and evaluates baseline systems.
- Conclusions: The paper proposes a token-level, reference-free hallucination detection task and introduces the HADES benchmark dataset.The benchmark targets fine-grained hallucination identification in free-form text generation.
- Conclusions: HADES is created by perturbing text to simulate NLG hallucinations and annotating perturbed examples with an iterative model-in-the-loop approach.The annotation strategy addresses an imbalanced label scenario.
- Conclusions: The paper provides comprehensive HADES analyses and evaluates several baseline models to establish initial benchmarks.The authors hope the task and dataset will support high-resolution hallucination detection and eventually real-time prevention.
Broader Impact and Ethnic Consideration
The study positions HADES as a tool for recognizing potential hallucinations from large pretrained models in free-form generation. Its Wikipedia-based construction is motivated by broad factual coverage and possible transfer to other factual domains.
- Broader Impact: A detector trained with HADES could support online and offline recognition of potential hallucinated content.The authors describe possible uses in guiding beam search, suppressing hallucinated-token probabilities, and expediting human post-examination.
- Broader Impact: Wikipedia is selected because it covers many facts, domains, and topics relevant to factual hallucination detection.The authors also state that learned knowledge may transfer to factual discussions in domains such as social chatbots.
A Detailed Statistical Analysis
The HADES analysis examines annotation and label-rebalancing procedures, including an iterative model-in-the-loop strategy and statistical feature comparisons across linguistic categories.
- Statistical features: Hallucinated content typically has higher word probability and entropy across POS/NER tags.These feature differences persist irrespective of POS/NER tag.
- Statistical features: Strong correlation between hallucination labels and TF-IDF or PPMI features is not observed.
- Annotation procedure: The annotation process uses an even number of annotators so controversial cases can receive additional judgments.With an odd number, every binary-classification outcome already has a majority winner.
- Label rebalancing: The iterative model-in-the-loop procedure trains on prior annotations and selects subsequent batches from the unlabeled candidate pool.The approach addresses label imbalance through subsampling.
- Label rebalancing: The rebalancing estimates TPR, TNR, and precision from annotations collected in previous rounds.Human annotations are treated as the oracle for actual hallucination and non-hallucination labels.
A Detailed Statistical Analysis
The section reports HADES statistical analyses and describes the annotation strategy used to rebalance labels through iterative model-guided sampling.
- Statistical features: Hallucinated content typically associates with higher word probability and entropy irrespective of POS/NER tag.
- Statistical features: Strong correlation between hallucination labels and TF-IDF or PPMI features is not observed.
- Label rebalancing: Initial annotation rounds showed extreme label imbalance, with around 90% of cases assigned to the hallucination class.
- Label rebalancing: Each annotation round retrains a BERT-initialized detection model on prior annotations to select the next batch from remaining unlabeled data.
- Label rebalancing: The method estimates TPR, TNR, and precision from annotations in previous rounds to determine the subsampling ratio.
- Label rebalancing: The procedure subsamples model-predicted hallucination cases to make actual hallucination and non-hallucination distributions roughly equal after resampling.