Source-linked AI summary
On Faithfulness and Factuality in Abstractive Summarization
Joshua Maynez, Shashi Narayan, Bernd Bohnet, Ryan McDonald
TL;DR
Abstractive summarizers often hallucinate content that is not faithful to their source documents, creating a gap between fluent generation and reliable summarization. The paper conducts a large-scale human evaluation of these errors across systems, finding that pretrained models are more faithful and factual and that entailment-based metrics correlate better with these qualities. The study therefore identifies semantic inference as a stronger evaluation signal than ROUGE or BERTScore, while noting that entailment measures require reference-based safeguards.
Problem
Neural abstractive summarizers can generate content inconsistent with their input documents despite reasonable likelihood, making faithfulness and factuality central evaluation concerns.
Method
The paper performs a large-scale human evaluation of hallucinations in RNN, CNN, and Transformer summarizers on the XSum extreme-summarization task.
Results
More than 70% of single-sentence summaries hallucinate, over 90% of extrinsic hallucinations are erroneous, and pretrained models perform best on automatic and human evaluations.
Takeaways & Limitations
Entailment-based measures correlate better with faithfulness and factuality than ROUGE and BERTScore, supporting their use in evaluation, training, and decoding research.
Takeaways & Limitations
Entailment is reference-less and can be gamed, so it should be combined with reference-based measures such as ROUGE.
Abstract
from arXiv · showhide
It is well known that the standard likelihood training and approximate decoding objectives in neural text generation models lead to less human-like responses for open-ended tasks such as language modeling and story generation. In this paper we have analyzed limitations of these models for abstractive document summarization and found that these models are highly prone to hallucinate content that is unfaithful to the input document. We conducted a large scale human evaluation of several neural abstractive summarization systems to better understand the types of hallucinations they produce. Our human annotators found substantial amounts of hallucinated content in all model generated summaries. However, our analysis does show that pretrained models are better summarizers not only in terms of raw metrics, i.e., ROUGE, but also in generating faithful and factual summaries as evaluated by humans. Furthermore, we show that textual entailment measures better correlate with faithfulness than standard metrics, potentially leading the way to automatic evaluation metrics as well as training and decoding criteria.
1 Introduction
Abstractive summarizers generate fluent text but frequently produce hallucinated content that is unfaithful or factually incorrect. A large-scale human evaluation examines these hallucinations and finds that pretrained models and entailment-based measures offer advantages over standard evaluation metrics.
- Abstractive summarization models can generate fluent, coherent text that is nevertheless unfaithful to the source document.
- The study conducts a large-scale human evaluation of RNN, CNN, Transformer, and human-written summaries for extreme summarization.
- Over 90% of extrinsic hallucinations are erroneous, so most hallucinations are neither faithful nor factual.
- Pretrained models perform best on automatic metrics and human judgments of faithfulness and factuality.
- Entailment-based metrics correlate better with faithfulness and factuality than ROUGE and BERTScore.
2 Hallucinations in Summarization
Summarization requires source-faithful output, but word-level likelihood training can reward summaries that are inconsistent with the document. The paper defines hallucinations by their relationship to the source and distinguishes intrinsic, extrinsic, and factual cases.
- Unlike open-ended generation, document summarization requires outputs to remain factual and faithful to the source text.
- Word-level likelihood training and reference divergence can reward summaries that conflict with the input while retaining reasonable model likelihood.
- Hallucination definitions: A summary is hallucinated when it contains a span unsupported by the document; paraphrases and document-inferable information are excluded.
- Hallucination definitions: Intrinsic hallucinations misrepresent information present in the document, whereas extrinsic hallucinations add information not introduced by the source.
- Hallucination definitions: A factual hallucination contains information absent from the document that is nevertheless factually correct, and it may be intrinsic or extrinsic.
- Background knowledge: The study permits background-knowledge integration when it produces summaries factual with respect to the document and associated knowledge.
3 Extreme Document Summarization
The study focuses on XSum extreme summarization, where models produce a single introductory sentence for each news article. This setting facilitates annotation and requires genuinely abstractive generation.
- XSum contains 226,711 BBC articles paired with journalist-written single-sentence summaries.
- The dataset uses a standard split of 90% training, 5% validation, and 5% test articles.
- Single-sentence summaries are easier to annotate and analyze than longer summaries.
- Because each gold summary introduces the article and the task is not amenable to extraction, XSum is a benchmark for faithful and factual abstraction.
4 Abstractive Summaries
The paper evaluates RNN, CNN, and Transformer abstractive systems alongside journalist-written references on XSum. Transformer systems compare randomly initialized models with BERT-initialized models.
- The evaluation covers state-of-the-art RNN, CNN, and Transformer abstractive methods plus human-written reference summaries.
- Model families: PTGEN is an attention-based RNN sequence-to-sequence model that generates target-vocabulary words or copies source words.
- Model families: TCONVS2S conditions convolutional sequence-to-sequence generation on article topics and captures long-range document dependencies.
- Evaluation: Table 1 reports ROUGE and BERTScore F1 for nonpretrained and pretrained models on 500 sampled human-evaluation items.
- Model families: TRANS2S uses randomly initialized Transformer encoder and decoder weights, whereas BERTS2S initializes both from BERT checkpoints.
5 Experiments and Results
The experiments compare automatic and human assessments of abstractive summarization, focusing on hallucination, faithfulness, factuality, and metric validity. Pretrained BERTS2S performs strongly on these dimensions, while textual entailment better tracks human judgments than ROUGE or BERTScore.
- Experimental Setup: 500 articles and five summarization systems were evaluated with ROUGE, BERTScore, textual entailment, question answering, and human judgments.Human annotators assessed hallucination spans, faithfulness, and factuality across generated summaries.
- Automatic Evaluation: BERTS2S outperformed randomly initialized systems and GPT-TUNED on the reported automatic measures.Differences among PTGEN, TCONVS2S, and TRANS2S were not significant, while other differences were significant.
- Hallucination Assessment: Intrinsic hallucinations occurred in 16.9% of BERTS2S summaries, compared with 19.9% for PTGEN, whose copy mechanism produced the lowest extrinsic hallucination rate at 63.3%.BERTS2S had the fewest intrinsic hallucinations among the four abstractive systems, while PTGEN remained prone to unsupported synthesis.
- Factuality Assessment: 34.7% of BERTS2S abstracts were faithful and/or factual, 7.4% absolute better than PTGEN.BERTS2S also had 7.8% unfaithful yet factual summaries, versus 2.2% for PTGEN among extrinsic hallucinations.
- Factuality Assessment: Over 90% of BERTS2S hallucinations were erroneous despite its superior factual-summary performance.The authors attribute the model’s advantage most likely to exposure to extensive text during pretraining.
- Automatic Measures: Textual entailment correlated best with human faithfulness and factuality scores, whereas ROUGE, BERTScore, and question-answering scores correlated weakly.Faithfulness correlations were moderate at 0.40 ≤|rs| ≤0.59, while factuality correlations were weak at 0.20 ≤|rs| ≤0.39.
- Automatic Measures: Selecting among four candidates using entailment improved faithfulness by almost 5% absolute, but introduced a trade-off with ROUGE.The authors caution that reference-less entailment can be gamed and should be coupled with reference-based measures such as ROUGE.
6 Related Work
Prior summarization research largely evaluated content overlap and linguistic quality, while faithfulness and factuality remained insufficiently assessed. Recent work explored reference-less, natural-language-inference, and question-answering approaches, but their effectiveness and applicability varied.
- ROUGE measures unigram and bigram overlap plus longest common subsequence, but can mislead when used alone to assess informativeness.
- Human assessment, pyramid annotation, question-answering evaluation, and reference-less source comparison were proposed to complement automatic overlap metrics.
- Natural-language-inference and question-answering models were explored to detect factual consistency in generated text.
- Reinforcement-learning approaches used textual-entailment or question-answering rewards, but did not evaluate whether those rewards improved faithfulness.
7 Conclusion
The study identifies hallucination as a central challenge in abstractive summarization and finds that pretrained generators improve summary quality without eliminating unfaithful or unfactual content. It further concludes that semantic-inference measures better represent summarization quality than ROUGE or BERTScore alone.
- The study presents hallucination as a critical, potentially most critical, challenge for abstractive summarization.
- NLU-driven pretraining is key to generating informative, coherent, faithful, and factual abstracts, but remains far from solving hallucination.
- ROUGE and BERTScore are insufficient for studying hallucination, whereas semantic-inference measures better represent true summarization quality.
A Model Hyperparameters and Predictions
This appendix documents the vocabulary configurations used for the evaluated prediction systems and identifies the annotation-agreement table. It provides implementation and measurement context rather than new modeling conclusions.
- PTGEN and TCONVS2S use a 50k Stanford-tokenized vocabulary, while TRANS2S and BERTS2S use approximately 30k BERT-matched WordPieces.
- GPT-TUNED uses approximately 50k SentencePieces to match the GPT-2 pretrained vocabulary.
- Table 6 reports Fleiss’s Kappa scores for word-level agreement across hallucination, factuality, repetition, and incoherence assessments.
B Inter annotator agreement
Annotators showed substantial agreement when categorizing hallucinations and almost perfect agreement when judging factuality across all models.
- Fleiss’s Kappa was used to assess agreement among raters for hallucination categorization and factuality judgments.
- 0.61 ≤ k ≤ 0.80 indicates substantial agreement for word-level faithfulness, intrinsic-hallucination, and extrinsic-hallucination labels across all models.
- 0.81 ≤ k ≤ 1.0 indicates almost perfect agreement for factuality annotations across all models.
C Highlighted Span Characteristics
The study characterizes hallucinated spans and evaluates linguistic irregularities in neural summaries. It also finds that repetition and incoherence metrics do not correlate with faithfulness or factuality assessments.
- Highlighted Span Characteristics: GOLD abstracts had the fewest intrinsically hallucinated spans, at 0.55 per document.Table 7 reports intrinsic and extrinsic hallucination spans across 500 document-summary pairs.
- Highlighted Span Characteristics: PTGEN summaries had the fewest extrinsically hallucinated spans, at 2.85 per document, versus 3.04 for BERTS2S.PTGEN’s extrinsic hallucination rate was also lower by document percentage: 63.3% versus 64.1%.
- Highlighted Span Characteristics: PTGEN hallucinated spans averaged 8.48 words, compared with 6.12 words for BERTS2S summaries.The authors characterize BERTS2S hallucinations as more local than PTGEN hallucinations.
- Assessment of Linguistic Irregularities: Annotators identified repeated or incoherent spans in 2,500 document-summary pairs, using three judgments per pair.Each assessment presented annotators with only a single-sentence summary.
- Assessment of Linguistic Irregularities: TRANS2S and BERTS2S produced more repetition-free and coherent summaries than PTGEN and TCONVS2S, but these metrics failed to correlate with faithfulness and factuality.Repetition and incoherence assessments showed almost perfect Fleiss’s Kappa agreement.
E Full Hallucination Results
Table 10 presents the study’s complete human-evaluation results for hallucinations.
- Full Hallucination Results: Table 10 reports the full results from the human study of hallucinations.The table summarizes the paper’s hallucination assessment results.