PaperlayerView exact revision on arXiv

Source-linked AI summary

Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization

Meng Cao, Yue Dong, Jackie Chi Kit Cheung

arXiv:2109.09784v2cs.CL

TL;DR

Abstractive summarization systems often generate content unsupported by their source documents, although some such hallucinations are factual according to world knowledge. The paper detects factual versus non-factual entity hallucinations using prior and posterior probabilities, and reports stronger detection performance plus improved factuality when the detector supplies an offline-RL reward.

  • Problem

    Abstractive summarizers frequently hallucinate content not supported by source documents, while factual hallucinations may provide useful background information.

  • Method

    The paper classifies entity hallucination and factuality using prior and posterior probabilities from masked language models, implemented with an interpretable KNN discriminator and an overlap feature.

  • Results

    The approach outperforms two baselines on factuality classification and hallucination detection tasks, while its reward use vastly improves summary factuality.

  • Takeaways & Limitations

    Factuality-aware reward signals can improve summary factuality while maintaining the level of abstractiveness.

  • Takeaways & Limitations

    The converted Maynez et al. annotation is noisy because hallucinated spans can contain faithful entities, creating false-positive non-factual labels.

Abstract

from arXiv · show

State-of-the-art abstractive summarization systems often generate \emph{hallucinations}; i.e., content that is not directly inferable from the source text. Despite being assumed incorrect, we find that much hallucinated content is factual, namely consistent with world knowledge. These factual hallucinations can be beneficial in a summary by providing useful background information. In this work, we propose a novel detection approach that separates factual from non-factual hallucinations of entities. Our method utilizes an entity's prior and posterior probabilities according to pre-trained and finetuned masked language models, respectively. Empirical results suggest that our approach vastly outperforms two baselines %in both accuracy and F1 scores and strongly correlates with human judgments. % on factuality classification tasks. Furthermore, we show that our detector, when used as a reward signal in an off-line reinforcement learning (RL) algorithm, significantly improves the factuality of summaries while maintaining the level of abstractiveness.

1 Introduction

Abstractive summarizers frequently produce hallucinated content, but hallucinations can be factual additions grounded in world knowledge rather than source text. The paper proposes entity-level detection using prior and posterior probabilities to distinguish factual from non-factual hallucinations and improve summary factuality.

  • 64.1% of summaries generated by a BERT-based abstractive model on XSUM contain hallucinations.
  • Factual hallucinations are verifiable through world knowledge but not directly inferable from the source document.For example, “European Commission President” adds background information about Mr. Juncker while remaining factual.
  • The paper focuses on entities because they express salient summary information and entity hallucinations are common in generated summaries.About 30% of entities generated by BART on the XSUM test set are hallucinated.
  • The detector uses an entity’s prior probability without the source document and posterior probability given the document as features for hallucination and factuality classification.The assumption is that factual errors receive little additional evidence from the source, producing only a small prior-to-posterior change.
  • On two human-annotated datasets, the approach outperforms two baselines for identifying non-factual hallucinations and correlates strongly with human factuality judgments.
  • Using the detector as an offline-RL reward signal improves generated-summary factuality while maintaining abstractiveness.

2 Related Work

Prior work addresses hallucination causes, reduction, factual consistency evaluation, and summary correction using diverse training, verification, and semantic methods. This paper is situated among approaches that seek to control or assess factual errors in abstractive generation.

  • Earlier studies investigate causes of hallucination and propose methods that reduce hallucinations in generated summaries.
  • Some methods filter noisy training samples, verify quantities, use entity chains, or control hallucination through conditional language modeling.
  • The paper differs from prior control work by addressing both hallucination and factuality at the entity level for text summarization.
  • Factual-consistency methods train detectors or correction systems using artificial perturbations, dependency-arc support, or question answering over summaries and documents.
  • Artificial perturbations can differ substantially from hallucinations produced by abstractive summarizers.

3 Method

The method detects entity-level hallucination and factuality using prior and posterior probabilities, then uses factuality predictions as reward signals in offline reinforcement learning.

  • The approach separates factual from non-factual hallucinations of entities and provides a factuality-aware training framework for summarization models.
  • 3.1 Problem Statement: An entity is hallucinated when it is not directly inferable from the source; factual hallucinations are nevertheless correct according to world knowledge.
  • 3.2 The Prior & Posterior Probability of an Entity: The prior is source-independent entity probability, while the posterior conditions on both the entity’s context and the source document.
  • 3.2 The Prior & Posterior Probability of an Entity: A K-nearest-neighbors discriminator uses prior and posterior probabilities as features to classify each entity’s hallucination and factuality status.
  • 3.3 Improving the Factuality of Abstractive Summarization Systems: Offline reinforcement learning frames summary generation as a finite Markov decision process whose policy generates tokens and maximizes discounted cumulative reward.
  • 3.3 Improving the Factuality of Abstractive Summarization Systems: The factuality-aware training method labels non-factual entities as noise and uses classifier predictions, posterior probabilities, and negative rewards to guide learning.

4 Evaluation Tasks and Datasets

The evaluation uses entity-level annotations and compares factuality and hallucination classification with human-judgment correlations and summary-level faithfulness and abstractiveness measures.

  • 4.1 Hallucination and Factuality Assessment: The XENT dataset supplies entity-level hallucination and factuality annotations because fine-grained annotations were unavailable.
  • 4.1 Hallucination and Factuality Assessment: Annotators labeled 800 BART-generated summaries from XSUM, extracting 2,838 entities for annotation.
  • 4.1 Hallucination and Factuality Assessment: The annotations classify entities as non-hallucinated, factual hallucination, or non-factual hallucination after checking source entailment and, when needed, world knowledge.
  • 4.2 MENT Dataset: The converted MENT dataset enables entity-level evaluation by extracting entities and assigning factuality labels from existing summary-level annotations.
  • 4.2 MENT Dataset: The converted MENT dataset is noisy because summary-level hallucinated spans can contain entities faithful to the source, so source-overlapping entities are filtered.
  • 4.3 Evaluating the Factuality of Summarization Systems: Evaluation includes classifier accuracy and macro F1, correlations with human factuality judgments, FEQA and DAE faithfulness scores, ROUGE, and source-novelty rates.

5 Experiments

The experiments evaluate entity-level factuality and hallucination classification, correlation with human judgments, and summarization outcomes on XENT, MENT, and XSUM. The proposed approach consistently improves factuality-related evaluation while preserving abstractiveness.

  • Classification Experiments: The approach uses masked-language-model probabilities and is compared with overlap-based and LM-based baselines for entity classification.The baselines rely on source-summary overlap or prior/posterior probability comparisons.
  • Classification Experiments: The model significantly outperforms two baselines on factuality classification on XENT, with a paired t-test p-value below 3.3e−5.On hallucination detection, word overlap achieves 92.93% accuracy versus 93.09% for the proposed model but cannot distinguish factual from non-factual hallucinations.
  • Classification Experiments: The approach outperforms two baseline models on factuality classification on the converted MENT dataset despite lower performance across models.The authors attribute the lower overall performance to greater noise in the converted dataset.
  • Correlation Experiments: The approach achieves the highest partial Pearson correlation, ρ = 0.183 (p < 1e−8), on Pagnoni et al.’s benchmark and significantly outperforms other automatic metrics on Wang et al.’s dataset.These evaluations compare automatic metrics with human factuality judgments.
  • Factuality Evaluation Results of Summarization Systems: On XSUM, the approach produces fewer non-factual entities, higher faithfulness, the lowest ENFS rate, and more factual hallucinations than the baselines.Its higher novel n-gram rate than loss truncation indicates that the factuality improvement is not achieved simply by making summaries more extractive.
  • Factuality Evaluation Results of Summarization Systems: At the same ROUGE level, the proposed method obtains a higher factuality score than the loss truncation baseline.Figure 2 presents the factuality–ROUGE trade-off under different reward values and loss-truncation drop rates.

6 Analysis

The analysis examines feature importance, summary-model evaluation measures, and the training sources associated with factual hallucinated entities. Posterior probability is central to factuality classification, while training-data differences suggest that many factual hallucinations draw on XSUM knowledge.

  • Ablation Studies: Removing the posterior feature reduces factuality-classification performance from 90.95 to 85.69, indicating that posterior probability is crucial.The ablation evaluates feature combinations using F1 on the XENT test set.
  • Ablation Studies: Word overlap has the largest effect on hallucination-classification performance among the ablated features.This complements the role of posterior probability in factuality classification.
  • Ablation Studies: Table 6 reports factual entities, factual hallucinations, ENFS, and model comparisons on XSUM’s official test set.The loss-truncation baseline is parameterized by the percentage of data dropped.
  • Where Does the Model Learn to Hallucinate?: Figure 3 compares entity posterior-probability distributions from CMLMXSUM and CMLMCNN/DM, with shading indicating classifier boundaries.The distributions are analyzed separately for non-hallucinated entities and factual hallucinations.
  • Where Does the Model Learn to Hallucinate?: Many factual hallucinations have low posterior probability under CMLMCNN/DM but high posterior probability under CMLMXSUM.The pattern suggests that their knowledge comes from the XSUM training set.
  • Where Does the Model Learn to Hallucinate?: For σ(ek) ≥5, factual hallucinated entities appear on average 2.19 times in similar XSUM samples and 0.77 times in similar CNN/DM samples.For σ(ek) ≤0, the corresponding averages are 2.85 and 2.46.

7 Conclusion

The paper finds that hallucinations are common in abstractive summarization and that more than half of hallucinated entities are factual relative to the source and world knowledge. It introduces a probability-based detector that improves factuality evaluation and summarization factuality.

  • Conclusion: About 30% of entities generated by a state-of-the-art summarization model are hallucinated.More than half of these hallucinated entities are factual with respect to the source document and world knowledge.
  • Conclusion: The proposed method uses entities’ prior and posterior probabilities from masked language models to detect hallucination and factuality.The approach distinguishes factual from non-factual hallucinations at the entity level.
  • Conclusion: The approach outperforms two baselines on factuality classification and hallucination detection using human-annotated datasets.The classifier also serves as a reward signal that improves summarization factuality.

A.1 Hallucination Examples

The examples distinguish non-hallucinated entities, factual hallucinations, and non-factual hallucinations using source-document support and world knowledge. Cardiff illustrates a factual hallucination because it is absent from the source but supported by external facts.

  • Hallucination Examples: Cardiff is classified as a factual hallucination because the location is not directly inferable from the source but is factual according to online information.The examples use entity categories to separate source support from factuality based on world knowledge.

A.2 Experimental Setup

The experiments use XSUM and CNN/DailyMail, with Transformer encoder-decoder language models and an offline reinforcement-learning setup initialized from BART large.

  • A.2 Experimental Setup: The study uses XSUM and CNN/DailyMail as summarization datasets.CNN/DailyMail has 287,227 training, 13,368 validation, and 11,490 test samples; XSUM has 226,711 BBC articles split into training, validation, and test sets.
  • A.2 Experimental Setup: All language models use the Transformer encoder-decoder architecture implemented with Fairseq and PyTorch.
  • A.2 Experimental Setup: CMLM training initializes from a large BART checkpoint, uses maximum sequence lengths of 1024, and runs for 15,000 fine-tuning steps.The schedule includes 500 warm-up steps.
  • A.2 Experimental Setup: The offline reinforcement-learning experiment initializes from BART large fine-tuned on XSUM.
  • A.2 Experimental Setup: The reinforcement-learning model uses γ = 1, learning rate 1e−5, 30,000 update steps, 1,000 warm-up steps, and polynomial learning-rate decay.No reward shaping is used.
  • A.2 Experimental Setup: A slowly updated copy of the policy network computes importance weights for more stable training.Polyak updates use an update rate of 0.01.

A.3 Classification Results on XENT Dataset

The classification experiments analyze entity prior and posterior probabilities across hallucination classes and evaluate factuality under noisy training data. Results show distinct probability patterns, classifier separation, and a trade-off between ROUGE and factual consistency.

  • Classification Results on XENT Dataset: Table 8 evaluates ENTFA on XENT using prior probability, posterior probability, and word overlap features.Because the work is presented as the first to distinguish factual from non-factual hallucinations, it has no direct baseline in the three-class comparison.
  • Classification Results on XENT Dataset: Non-factual hallucinated entities cluster near the origin, whereas factual hallucinated and non-hallucinated entities occupy a high-posterior region.The distributions follow the expectation that non-factual hallucinations have lower prior and posterior probabilities.
  • Classification Results on XENT Dataset: KNN separates factual and non-factual entities with clear boundaries, with many factual hallucinations identified through relatively high CMLMXSUM posterior probabilities.The probability distributions and classification boundaries are shown for entities in XENT.
  • Classification Results on XENT Dataset: Non-hallucinated, factual-hallucinated, and non-factual-hallucinated entities have average posterior probabilities of 0.763, 0.599, and 0.133, respectively.Figure 5 compares posterior probabilities from CMLM and prior probabilities from MLM across the three entity classes.
  • Classification Results on XENT Dataset: 50.5% of 7,358 entities absent from the source but predicted factual by ENTFA appear in the reference summary, compared with 12.7% of entities predicted non-factual.Word overlap predicts a lower entity factuality rate because it cannot identify factual hallucinations and produces false negatives.
  • Classification Results on XENT Dataset: Noisier training data increases ROUGE while generating more non-factual entities for both BART and PEGASUS.With 50k clean samples, PEGASUS has ROUGE1 0.450 and a predicted factual entity rate of 84.79%, versus BART’s 0.406 and 91.81%.
  • Classification Results on XENT Dataset: CMLM assigns greater certainty to most factual entities than CLM and achieves a higher true-positive rate as thresholds increase toward the origin.The higher AUC of CMLM is also reported in the comparison.
Loading 2109.09784v2…