Source-linked AI summary
Entity-level Factual Consistency of Abstractive Text Summarization
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, Bing Xiang
TL;DR
Abstractive summarization models often produce summaries that are not factually consistent with their source documents, including hallucinated entities. The paper introduces entity-level metrics, filters hallucinated entities from training data, and adds summary-worthy entity classification and joint generation; these approaches reduce entity hallucination with minimal ROUGE degradation.
Problem
Abstractive summarization models can generate factually inconsistent summaries, including entities absent from the source, while ROUGE inadequately measures factual consistency.
Method
The paper proposes entity-level precision and recall metrics, entity-based data filtering, summary-worthy entity classification, and joint entity-summary generation.
Results
Below 94% to above 98%: filtering increases XSUM precision-source, while the combined approaches improve entity-level metrics with minimal ROUGE degradation.
Takeaways & Limitations
Filtering training data and adding entity-focused modeling can significantly reduce entity hallucination while preserving ROUGE scores closely.
Takeaways & Limitations
The metrics depend on automatic NER tools and heuristic matching rules, although manual checks on 10 XSUM validation examples found correct entity identification and matching.
Abstract
from arXiv · showhide
A key challenge for abstractive summarization is ensuring factual consistency of the generated summary with respect to the original document. For example, state-of-the-art models trained on existing datasets exhibit entity hallucination, generating names of entities that are not present in the source document. We propose a set of new metrics to quantify the entity-level factual consistency of generated summaries and we show that the entity hallucination problem can be alleviated by simply filtering the training data. In addition, we propose a summary-worthy entity classification task to the training process as well as a joint entity and summary generation approach, which yield further improvements in entity level metrics.
1 Introduction
Abstractive summarization models can generate factually inconsistent summaries, especially by hallucinating entities absent from the source. The paper proposes entity-level consistency metrics and techniques to reduce this problem.
- About 30% of summaries generated by neural sequence-to-sequence models suffer from fact fabrication.
- The widely used ROUGE score is inadequate for quantifying factual consistency.
- Entity hallucination occurs when a generated summary names an entity absent from the source document, such as “UK” in an example about Italy and the Netherlands.
- Entity relations can also be inconsistent even when the entities appear in the source, and existing OpenIE and dependency-parsing methods are not accurate enough for practical use.
- The paper proposes simple metrics for entity-level factual consistency and evaluates BART summaries on three news datasets.
- Data filtering, multi-task learning, and joint sequence generation are proposed to improve entity-level metrics, while relation-level consistency is left for future work.
2 Related work
Related work establishes factual inconsistency as a recognized problem in abstractive summarization and connects it to consistency classification and response-control methods. BART is highlighted as a strong pretrained encoder-decoder model for text generation.
- BART is a pretrained encoder-decoder transformer model jointly trained to reconstruct corrupted input text.
- BART has shown superior performance in several text-generation tasks, including abstractive summarization.
- Prior work trained models to classify summary-source factual consistency similarly to natural language inference.
- In dialogue generation, unlikelihood training was proposed to suppress logically inconsistent responses.
- The paper positions its approach as complementary to these existing methods.
3 Entity-level factual consistency metrics
The paper introduces entity-level metrics for measuring hallucination and entity accuracy in abstractive summaries, then describes filtering and generation strategies to improve these measures.
- Entity-level metrics: The metrics use named-entity counts from the target, hypothesis, and source, with shortened multiword entities counted as matches when any n-gram matches.Matching is case-insensitive and excludes unigram stop words such as “the”.
- Entity-level metrics: The proposed precision-source (precs) metric measures the percentage of generated-summary named entities found in the source document.Low precs indicates more severe entity hallucination.
- Entity-level metrics: Ground-truth summaries in XSUM have the lowest precision-source score among Newsroom, CNN/DailyMail, and XSUM, motivating entity-based data filtering.Filtering removes summary sentences containing entities unmatched in the source and ensures precs = 1 in the retained ground-truth summaries.
- Entity-level metrics: Precision-target (prect) and recall-target (recallt) measure generated-summary entity accuracy against the ground-truth summary.Their harmonic combination is F1t = 2 · prect · recallt/(prect + recallt).
- Entity-level metrics: The paper also adds summary-worthy entity classification to BART and jointly generates salient entities before the summary through JAENS.The classification task uses BIO labels and an additional encoder loss; JAENS lets the summary attend to generated salient entities through decoder self-attention.
6 Experiment results
Experiments compare entity-based filtering, multi-task learning, and JAENS across three summarization datasets using entity-level metrics and ROUGE. Filtering improves entity consistency, while the added methods further improve target-oriented metrics, with possible ROUGE costs for JAENS.
- Model comparisons: Table 3 compares original-data training with entity-based filtering, an additional classification task, and JAENS across three datasets.Scores are percentages averaged over five runs with standard deviations, including micro and macro entity-level metrics.
- Entity-based filtering: 93.6% to 98.2%: XSUM precs increases after entity-based data filtering, indicating reduced entity hallucination.The filtered model is evaluated on the clean test set, and filtering removes about 34% of XSUM training data.
- Entity-based filtering: Entity-based filtering improves precs across Newsroom, CNN/DailyMail, and XSUM, while generally improving other entity-level metrics.The comparison uses models trained before and after filtering and evaluates them on test data filtered for entity consistency.
- Additional training objectives: Multi-task training further increases prect, recallt, and F1t after filtering.The additional task classifies summary-worthy named entities in the source document.
- Additional training objectives: JAENS produces similar entity-level gains and outperforms the multi-task approach on CNN/DailyMail and Newsroom, but added decoding complexity may hurt ROUGE.JAENS generates salient named entities before generating the summary.
7 Conclusion
The paper introduces entity-level metrics and training approaches for reducing hallucinated entities in abstractive summaries. Combined, these approaches improve entity-level metrics while minimally degrading ROUGE.
- Contributions: The paper proposes precs for entity hallucination and prect and recallt for entity-level accuracy against ground-truth summaries.The conclusion frames these metrics as measures of entity-level factual consistency and summary accuracy.
- Findings: Below 94% to above 98%: entity-based data filtering substantially improves XSUM precs.The filtering technique removes entity hallucinations from training data.
- Findings: Combining data filtering, multi-task learning, and joint sequence generation significantly reduces entity hallucination and raises entity-level metrics with minimal ROUGE degradation.The joint generation approach generates salient entities together with the summary.
A.1 Details of NER filtering
The NER filtering details define which named-entity categories are considered and which entity types are excluded from matching.
- Entity categories: The filtering considers PERSON, FAC, GPE, ORG, NORP, LOC, and EVENT named entities.These categories include people, facilities, geopolitical entities, organizations, groups, locations, and events.
- Excluded types: Dates, times, and numerals are ignored because their representations vary substantially and are difficult to match in the source.The scope is therefore limited to the listed named-entity types.
A.2 Details of experimental setup
The experiments fine-tune pretrained BART-large models with validation-based early stopping and dataset-specific training and decoding settings.
- Training: BART-large models from Fairseq are fine-tuned on Newsroom, CNN/DailyMail, and XSUM.The setup uses a standard 3e-5 learning rate, linear decay, and 500 warmup steps.
- Training: Models early-stop at the epoch with the highest validation ROUGE score.Validation ROUGE is used to select the training checkpoint.
- Hyperparameters: The multi-task loss weight α is selected on validation ROUGE, with best values 0.3, 0.3, and 0.15 for Newsroom, CNN/DailyMail, and XSUM.The values are searched from 0.1 to 0.5 in increments of 0.05.
- Decoding: Beam sizes are 1 for Newsroom, 4 for CNN/DailyMail, and 6 for XSUM.These settings follow the referenced BART configuration for CNN/DailyMail and XSUM.
A.3 Evaluation of PEGASUS (Zhang et al., 2020)
This section evaluates Huggingface PEGASUS checkpoints on NER-filtered test sets and reports comparable performance on CNNDM and XSUM, but worse performance on Newsroom than BART-large.
- PEGASUS checkpoints from Huggingface are evaluated on NER-filtered test sets.The evaluated checkpoints include PEGASUS models for Newsroom, CNN/DailyMail, and XSUM.
- PEGASUS performs similarly to BART-large on CNNDM and XSUM but worse on Newsroom.
- Table 5 summarizes PEGASUS evaluation on the NER-filtered test sets.