Source-linked AI summary
Factual Error Correction for Abstractive Summarization Models
Meng Cao, Yue Dong, Jiapeng Wu, Jackie Chi Kit Cheung
TL;DR
Factual inconsistency remains a challenge for abstractive summarization, motivating a post-editing corrector that reconstructs summaries from heuristic artificial corruptions. The model performs well on artificial data and outperforms previous models on manually annotated factual-consistency evaluation, but transfer to real system errors remains difficult.
Problem
Ensuring that abstractive summaries remain factually consistent with their source documents is challenging, despite progress from self-supervised models.
Method
The paper trains a post-editing corrector on reference summaries corrupted by heuristic transformations, using a denoising-style sequence-to-sequence model conditioned on the source.
Results
The model shows promising artificial-test performance, outperforms previous models on manually annotated evaluation, and corrects some inconsistent summaries generated by abstractive systems.
Takeaways & Limitations
Post-editing can improve factual consistency and provide factuality evaluation, although artificial-error training does not fully match real summarization errors.
Takeaways & Limitations
Human evaluation finds low recall on inconsistent summaries and false positives that can corrupt some consistent summaries.
Abstract
from arXiv · showhide
Neural abstractive summarization systems have achieved promising progress, thanks to the availability of large-scale datasets and models pre-trained with self-supervised methods. However, ensuring the factual consistency of the generated summaries for abstractive summarization systems is a challenge. We propose a post-editing corrector module to address this issue by identifying and correcting factual errors in generated summaries. The neural corrector model is pre-trained on artificial examples that are created by applying a series of heuristic transformations on reference summaries. These transformations are inspired by an error analysis of state-of-the-art summarization model outputs. Experimental results show that our model is able to correct factual errors in summaries generated by other neural summarization models and outperforms previous models on factual consistency evaluation on the CNN/DailyMail dataset. We also find that transferring from artificial error correction to downstream settings is still very challenging.
1 Introduction
Abstractive summarization has improved with self-supervised models, but generated summaries can remain factually inconsistent with their source documents. The paper proposes post-editing correction trained on artificially corrupted summaries.
- Factual consistency remains challenging despite improvements in automatic metrics such as ROUGE.
- About 30% of summaries generated by abstractive models contain factual errors, limiting their practicality.
- Existing approaches detect or evaluate factual consistency, while directly editing inconsistent summary content remains underexplored.
- The proposed model post-edits a draft summary generated by an abstractive summarizer to produce a corrected summary.
- Training uses artificial factual errors introduced into reference summaries through heuristic transformations.
- Automatic and human evaluations indicate that the corrector can correct factual errors and serve as a factuality evaluation model.
2 Background and Related Work
Prior work addresses factual consistency through specialized summarizers and separate evaluation models. These methods use extracted facts, knowledge graphs, entailment, question answering, or heuristic transformations.
- Prior factual-consistency work includes summarizers tailored to factuality and models designed to evaluate factual consistency.
- Specialized summarizers attend to source documents and extracted relation triples or use knowledge graphs to support factual generation.
- Evaluation approaches check fact-triple overlap, adapt entailment models, or compare question-answering outputs from summaries and documents.
3 Proposed Approach
The proposed corrector learns to reconstruct reference summaries from artificially corrupted versions conditioned on the source document. Corruptions target common entity, number, date, and pronoun errors.
- 3.1 Dataset of Artificial Corruptions: The training dataset is weakly supervised, using heuristic text transformations inspired by error analysis of summarization systems.
- 3.1 Dataset of Artificial Corruptions: With probability α = 0.3, a reference summary is corrupted into s′; otherwise, s′ = s, producing training triplets (s′, s, d).
- 3.1 Dataset of Artificial Corruptions: The corruption process introduces entity, number, date, and pronoun errors, identified as common through manual inspection.
- 3.1 Dataset of Artificial Corruptions: Entity, number, and date errors swap same-type source entities, while pronoun errors swap pronouns with matching syntactic case.
- 3.2 Training Objective and Models: The encoder receives the inconsistent summary and source document, and the decoder generates the correct summary by maximizing P(s|s′, d).
- 3.2 Training Objective and Models: BART is used as the corrector because its denoising pre-training resembles reconstructing a clean summary from noisy inconsistent input.
4 Experiments
The experiments evaluate factuality checking and error correction on artificial corruptions and manually labeled summaries from the K2019 dataset. Evaluation uses classification metrics, exact matching, and human judgments.
- 4.1 Evaluation Tasks and Measures: The study evaluates two tasks: factual consistency checking and correction of summary inconsistencies.
- 4.1 Evaluation Tasks and Measures: For consistency checking, the model performs binary classification and is evaluated with accuracy, precision, recall, and F1.
- 4.1 Evaluation Tasks and Measures: Any model edit is treated as an inconsistent prediction; no edit is treated as a consistent prediction.
- 4.1 Evaluation Tasks and Measures: Correction accuracy measures the proportion of summaries correctly changed, with exact reference matching required on the artificial test set.
- 4.2 Datasets: The artificial dataset contains 5,780 corrupted and 5,710 clean test samples, while K2019 contains 503 manually labeled summaries from neural abstractive summarizers.
- 4.2 Datasets: The K2019 evaluations are more meaningful because the artificial test set is intended only to demonstrate performance in the artificial setting.
5 Results
The corrector performs well on artificially corrupted summaries and factuality checking, but its correction performance drops substantially on real system summaries.
- Artificial corruptions: High classification accuracy and F1 scores show that the corrector identifies artificially injected factual errors.
- Artificial corruptions: 62.13% of 5780 corrupted test summaries were corrected to exactly match the reference summary.Among 5710 clean summaries, 73.73% were correctly left unchanged under the stated evaluation.
- K2019: On the K2019 test set, the corrector outperformed BERT but was slightly worse than FactCC for consistency checking.
- K2019: 17.74% of inconsistent real summaries were successfully corrected, while 1.13% of consistent summaries were corrupted.The human evaluation covered 62 inconsistent and 441 consistent summaries.
6 Conclusions
The paper presents an end-to-end corrector trained on artificially corrupted reference summaries to repair inconsistent abstractive summaries. It performs promisingly on artificial data and factuality evaluation, but real-summary correction remains limited.
- The paper proposes post-editing correction for inconsistent content in summaries generated by abstractive summarization models.
- The end-to-end correction model is trained on artificial examples created by corrupting reference summaries.
- The model achieves promising performance on the artificial test set and outperforms previous models on the manually annotated test set.
- Human evaluation finds that the model corrects some inconsistent summaries, but low recall and false positives remain.
A.1 Summary Correction Examples
The appendix examples illustrate corrections to factual errors, additions of supported entities, and edits that fail to resolve or may alter summary content.
- The corrector also modifies summaries that were originally consistent, including changes to names or wording.Examples include adding Britt McHenry and changing the Iran summary’s subject from Iran to America.
- Some corrections successfully repair entity or role confusion, such as changing Steve Woolfenden to Leo Woolfenden.
- The model sometimes leaves an inconsistent summary unchanged or produces another incorrect statement after correction.Examples include the Sebastian Vettel summary, the Haiti victim summary, and the nitrogen-gas summary.