PaperlayerView exact revision on arXiv

Source-linked AI summary

Mutual Information Alleviates Hallucinations in Abstractive Summarization

Liam van der Poel, Ryan Cotterell, Clara Meister

arXiv:2210.13210v2cs.CL

TL;DR

Abstractive summarization models can generate unsupported facts, and robust in-generation detection and prevention techniques remain lacking. This paper links hallucination onset to uncertainty and uses conditional PMI decoding to reduce hallucinated-token probability while maintaining summary metrics. The evidence is limited to English XSum, and the authors did not conduct human evaluations.

  • Problem

    Robust techniques for identifying and preventing unsupported content during summarization generation remain lacking despite prior work on hallucination causes.

    Source: §1 · p. 1

  • Method

    The proposed CPMI decoder switches from target-token log probability to pointwise mutual information when model uncertainty is high.

    Source: §1 · p. 1

  • Results

    On XSum, CPMI lowers the revised log probability of hallucinated tokens while maintaining ROUGE and BERTS scores.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors propose CPMI as a decoding procedure to reduce unfaithful-token likelihood and suggest investigating its combination with post-hoc correction and other decoding procedures.

    Source: §6 · p. 5

    The results are shown only for English XSum, the only open dataset with the required annotations, and the authors conducted no human evaluations.

    Source: Limitations · p. 6

Abstract

from arXiv · show

Despite significant progress in the quality of language generated from abstractive summarization models, these models still exhibit the tendency to hallucinate, i.e., output content not supported by the source document. A number of works have tried to fix--or at least uncover the source of--the problem with limited success. In this paper, we identify a simple criterion under which models are significantly more likely to assign more probability to hallucinated content during generation: high model uncertainty. This finding offers a potential explanation for hallucinations: models default to favoring text with high marginal probability, i.e., high-frequency occurrences in the training set, when uncertain about a continuation. It also motivates possible routes for real-time intervention during decoding to prevent such hallucinations. We propose a decoding strategy that switches to optimizing for pointwise mutual information of the source and target token--rather than purely the probability of the target token--when the model exhibits uncertainty. Experiments on the XSum dataset show that our method decreases the probability of hallucinated tokens while maintaining the Rouge and BertS scores of top-performing decoding strategies.

1 Introduction

Abstractive summarization models can produce fluent summaries but may hallucinate unsupported facts, and robust ways to detect or prevent this during generation remain lacking. This work links hallucination onset to model uncertainty and proposes uncertainty-triggered PMI decoding, which lowers hallucinated-token probability while retaining strong summary metrics.

1 Introduction

  • The work links hallucination onset during generation to high uncertainty about the next token, quantified by conditional entropy.It hypothesizes that uncertain models favor frequent training-corpus tokens, even when they may be unrelated to the source.Source: §1 · p. 1
  • The proposed decoder switches to pointwise mutual information between the source and target token when uncertainty is high.This encourages tokens relevant to the source rather than optimizing only target-token probability.Source: §1 · p. 1
  • The conditional, temporary PMI change drops ROUGE-L by 0.977%, versus 3.13% for a full PMI change, while increasing factuality according to FACTScore.The paper also reports that the proposed strategy maintains ROUGE and BERTS scores.Source: §1 · p. 2
  • On an annotated XSUM subset, conditional entropy strongly correlates with hallucination onset, and CPMI lowers the revised log probability of hallucinated tokens.The proposed decoding strategy maintains ROUGE and BERTS scores.Source: §1 · p. 2

2 Preliminaries

The section describes probabilistic, autoregressive summarization models and the score-based search used to generate summaries. It also introduces automatic metrics used to assess summarization quality and factuality.

2 Preliminaries

  • A summary is generated token by token from a model distribution conditioned on the source document and prior tokens.The probability of a complete string is the product of its token probabilities.Source: §2 · p. 2
  • Because searching all valid sequences is impractical, beam search approximately optimizes a score that can be decomposed into token-level scores.The described beam-search inputs include a source document, maximum beam size and hypothesis length, and a scoring function.Source: §2 · p. 2
  • ROUGE is quick to compute but correlates poorly with human judgments and only weakly with factuality; entailment and contextual-embedding metrics are also used as factuality indicators.The section names FactCC and BERTScore as examples.Source: §2 · p. 2

3 Finding and Combating Hallucinations

The section proposes that hallucinations begin when summarization models are uncertain and may favor frequent but source-unrelated tokens. It measures uncertainty with conditional entropy and uses it to trigger pointwise mutual information decoding.

3 Finding and Combating Hallucinations

  • Hallucination onset is hypothesized to coincide with high uncertainty, when models may favor frequent training-corpus tokens that are unrelated to the source.The section frames this as an extrapolation from findings about model behavior on less familiar data.Source: §3 · pp. 2–3
  • Conditional entropy quantifies next-token uncertainty, while a language model estimates marginal token probability irrespective of the source.The section notes that entropy is not a holistic measure of uncertainty.Source: §3 · p. 3, §3.1 · p. 3

3.1 Pointwise Mutual Information Decoding

  • CPMI switches a token’s score to pointwise mutual information when conditional entropy exceeds a threshold, subtracting its marginal log probability.The PMI score uses the source-conditioned token probability relative to the marginal probability.Source: §3.1 · p. 3

4 Related Work

Prior work has linked hallucinations to training and inference factors, and has proposed post-hoc detection or decoding interventions. This paper applies mutual-information decoding to summary faithfulness, with an uncertainty-triggered intervention.

4 Related Work

  • Prior work attributes hallucinations partly to exposure bias and explores minimum-risk training, while other work finds that pre-training and more data can improve faithfulness.The cited exposure-bias results are tentative and concern neural machine translation rather than summarization.Source: §4 · pp. 3–4
  • Token- and sentence-level detection, entailment systems, and span replacement identify or correct unfaithful text after generation, unlike this paper’s decoding-time intervention.The paper characterizes its approach as real-time detection and prevention during decoding.Source: §4 · p. 4
  • Keyword constraints and focus attention also modify decoding, but this method biases away from likely yet potentially irrelevant candidates only when conditional entropy is high.The approach remains abstractive rather than resorting to extractive methods.Source: §4 · p. 4
  • The paper presents its use of mutual information to increase faithfulness in abstractive summaries as a first.Prior mutual-information decoding work addressed settings such as zero-shot generation and dialogue diversity or relevance.Source: §4 · p. 4

5 Experiments

Experiments on XSUM examine whether conditional entropy marks the start of hallucinations and whether CPMI decoding can reduce unfaithful-token likelihood while preserving summary quality. High entropy is associated with initial hallucinated tokens, and CPMI's reported results show little change in performance metrics alongside higher FACTScore for both models.

5 Experiments

  • The experiments use XSUM's 226,711 BBC articles and single-sentence summaries, with 500 test summaries annotated for faithfulness and converted to token-level hallucination labels.The evaluated systems are TRANS2S trained on XSUM and BARTS2S fine-tuned on XSUM; the language model is Transformer-based.Source: §5 · p. 4
  • CPMI and beam search generate summaries, while automatic factuality detection supplies labels beyond the 500 human-annotated summaries for token-level scoring.Evaluation includes ROUGE, BERTScore, FactCC, FACTScore, and estimated hallucination incidence; the authors note that hallucination lacks a good direct operationalization.Source: §5 · pp. 4–5

5.1 Preliminary Analysis

  • Initial hallucinated tokens have higher average conditional entropy than non-hallucinated tokens: 4.197±0.065 vs 3.689±0.021 for TRANS2S and 3.115±0.051 vs 2.390±0.013 for BARTS2S.The authors suggest a threshold on conditional entropy could identify where hallucinations begin.Source: §5.1 · p. 5

5.2 Results

  • CPMI leaves BARTS2S performance metrics with very little change and TRANS2S changes within the margin of error, while FACTScore increases for both models.FactCC decreases, but the authors treat FACTScore as the better indicator because its model is trained specifically for XSUM, unlike FactCC's CNN/DM-trained model.Source: §5.2 · p. 5, Table 1 · p. 5
  • For both models, CPMI decreases scores and increases ranks of hallucinated tokens versus standard log-probability scoring, with stronger effects at the starts of unfaithful spans.For BARTS2S, initial versus non-hallucinated token score changes were −0.13 ± 0.03 versus −0.07 ± 0.01, and ranking changes were 275 ± 114 versus 552 ± 134.Source: §5.2 · p. 5

6 Conclusion

The conclusion links hallucination starts to high conditional entropy in next-token predictions and proposes CPMI, which switches to PMI decoding when uncertainty is high. It reports reduced likelihood of unfaithful tokens while retaining high-quality summaries.

6 Conclusion

  • The authors link hallucination starts to high conditional entropy and propose CPMI, switching to pointwise mutual information decoding at high uncertainty to reduce unfaithful-token likelihood while retaining high-quality summaries.They suggest future work combining CPMI with post-hoc correction and other decoding procedures.Source: §6 · p. 5

Limitations

The evaluation is limited to English XSUM because it is the only open dataset with the required annotations; the study also reports no human evaluation.

Limitations

  • Results cover only English on XSUM, the only open dataset with the annotations required for these experiments.The authors suggest further work on other architectures and datasets, including CNN/DM.Source: Limitations · p. 6
  • The study lacks human evaluation, which the authors say could add a qualitative assessment of CPMI’s efficacy.They note that human judgments of summary faithfulness are themselves imperfect.Source: Limitations · p. 6
  • XSUM articles and summaries may include Gaelic, and reference summaries can contain spans not directly inferable from their source articles.The models also truncate sources at 4096 tokens, losing information due to this training constraint.Source: Limitations · p. 6

Ethical Concerns

The authors foresee no ethical concerns beyond those already documented for abstractive summarization systems and other text generators.

Ethical Concerns

  • The authors foresee no ethical concerns beyond those documented for abstractive summarization systems and other text generators.They cite Smiley et al. (2017), Zellers et al. (2019), and Kreps et al. (2022).Source: Ethical Concerns · p. 6

A Additional Results

The appendix points to additional evaluation and analysis tables and identifies the standard beam-search algorithm.

A Additional Results

  • Table 3 reports average scores and rankings by hallucination label for ground-truth tokens under beam search and CPMI.The appendix says these labels were processed to generate Table 2.Source: A Additional Results · p. 9
  • Table 4 reports the preliminary analysis correlating average conditional entropy with token labels.The appendix identifies this as the full set of preliminary correlation results.Source: A Additional Results · p. 9
  • Algorithm 1 provides the standard beam-search algorithm.Source: A Additional Results · p. 9

B Implementation Details

Implementation covers preprocessing and model training, followed by a CPMI hyperparameter search on annotated XSUM examples. The experiments use automatic factuality detection and compare beam search with CPMI.

B Implementation Details

  • Preprocessing tokenizes with Moses, applies model-specific BPE, and binarizes the resulting data with Fairseq’s preprocessing tool.TRANS2S uses FastBPE; BARTS2S follows its provided BPE steps.Source: B Implementation Details · p. 9
  • Sources are truncated at 4096 tokens, and both beam search and CPMI use a beam size of 5.Training uses early stopping after validation performance fails to improve for five consecutive runs.Source: B Implementation Details · p. 9
  • TRANS2S uses Fairseq’s transformer model, while BARTS2S uses a pretrained BART summarization model fine-tuned on XSUM.The TRANS2S grid search selected dropout 0.3 and learning rate 7 × 10^-5.Source: B Implementation Details · p. 9
  • The language models are Fairseq transformers trained on source and target text, with selected learning rates of 1 × 10^-4 for TRANS2S and 5 × 10^-4 for BARTS2S.The models use a maximum length of 2048 tokens.Source: B Implementation Details · p. 9
  • The CPMI search tunes λ and τ on 500 XSUM examples with factuality annotations, balancing the average PMI log probability of initial hallucinated tokens against ROUGE-L.The search selects parameters using a heat plot; λ and τ control language-model influence and the conditional-entropy threshold for triggering PMI decoding.Source: B Implementation Details · p. 9
  • In their evaluation, the authors use automatic factuality detection to obtain measures such as FACTScore, using code provided by Zhou et al. (2021) for this purpose.Source: B Implementation Details · p. 10

Related papers

Loading 2210.13210v2…