PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Fact Correction in Abstractive Text Summarization

Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, Jingjing Liu

arXiv:2010.02443v1cs.CL

TL;DR

Abstractive summaries can be factually inconsistent with their sources despite strong automatic scores. SpanFact uses QA-informed span selection to replace entities in generated summaries; across datasets, it improves factuality with only small or slight ROUGE drops.

  • Problem

    Abstractive summaries can contain facts inconsistent with their sources, and high ROUGE scores do not establish factual consistency.

    Source: §1 · pp. 1–2

  • Method

    SpanFact uses QA-style span selection to replace masked entities in generated summaries, either iteratively one at a time or auto-regressively after masking all entities.

    Source: §3 · p. 3, §3.2 · p. 4, §3.4 · p. 5

  • Results

    Across multiple datasets, SpanFact improves factuality measures while keeping ROUGE drops small or slight, including average drops of -0.5 to -1.5 on XSum and Gigaword.

    Source: §4.4 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    The models can correct factual errors in summaries generated by state-of-the-art abstractive systems without a huge drop in ROUGE.

    Source: §5 · p. 9

    SpanFact currently corrects entities; applying the method to noun phrases, verbs, or clauses is left for future work.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Pre-trained neural abstractive summarization systems have dominated extractive strategies on news summarization performance, at least in terms of ROUGE. However, system-generated abstractive summaries often face the pitfall of factual inconsistency: generating incorrect facts with respect to the source text. To address this challenge, we propose Span-Fact, a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text, while retaining the syntactic structure of summaries generated by abstractive summarization models. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.

1 Introduction

Abstractive summarization offers flexibility and vocabulary diversity, but fluent, high-ROUGE summaries can still contain facts inconsistent with their sources. SpanFact addresses this problem by correcting entities in generated summaries, using QA-informed span selection to improve factuality while retaining summary informativeness.

1 Introduction

  • Pre-trained abstractive systems have surpassed extractive systems on automatic metrics such as ROUGE, despite extractive summaries being simpler and often more grammatically and semantically correct.Abstractive methods are increasingly used for their flexibility, coherence, and vocabulary diversity.Source: §1 · p. 1
  • High ROUGE does not ensure factual consistency: abstractive summaries can be fluent yet unfaithful to their source documents.Generation must balance source faithfulness against the language-modeling prior for fluency.Source: §1 · p. 2
  • QA-based factuality metrics correlate strongly with human judgments, while factual-control models using triples or entailment can lower ROUGE by sacrificing informativeness.QA evaluation results motivate using QA knowledge for correction.Source: §1 · p. 2
  • SpanFact uses single- or multi-span selection to correct erroneous entities in summaries from any summarization system, without retraining that system.The two methods perform correction iteratively or auto-regressively while retaining generated summary syntax.Source: §1 · p. 2

2 Related Work

Summarization research progressed from neural encoder-decoder models and pointer-generators to pretrained Transformers. Related factuality work has sought to control generation with knowledge or entailment signals, while SpanFact instead post-corrects summaries produced by any model.

2 Related Work

  • Neural encoder-decoder summarizers evolved through LSTM and GRU encoders and reinforcement learning, while pointer-generators added source-word copying.Rush et al. (2015) introduced the general encoder-decoder structure; See et al. (2017) proposed the pointer-generator framework.Source: §2 · pp. 2–3
  • Self-supervised Transformers including BERT, GPT, and BART achieved state-of-the-art abstractive summarization performance after supervised fine-tuning on source-summary pairs.The cited work includes Liu and Lapata (2019), Lewis et al. (2020), Zhang et al. (2020a), Shi et al. (2019), and Fabbri et al. (2019).Source: §2 · p. 3
  • Abstractive summaries can be fluent but unfaithful, and 8% to 30% have factual errors that ROUGE does not detect.Prior responses include decoders that use extracted knowledge triples and training or reranking methods based on entailment.Source: §2 · p. 3
  • SpanFact differs from generation-time controls by post-correcting summaries from any model through masked-entity prediction and span selection.Meng et al. (2020) concurrently proposed a post-editing model fine-tuned on distant-supervision examples.Source: §2 · p. 3

3 Multi-Fact Correction Models

SpanFact comprises two span-selection factual correctors for abstractive summaries. They edit an informative but incorrect draft toward greater source-summary factual consistency.

3 Multi-Fact Correction Models

  • SpanFact comprises QA-span and auto-regressive fact-correction models, both built around span selection.The two models use different masking and prediction strategies.Source: §3 · p. 3

3.1 Problem Formulation

  • The correction objective is to turn an informative but incorrect system summary into one with higher factual consistency against its source.Factual consistency is represented by a metric comparing the source with the system summary.Source: §3.1 · p. 4

3.2 Span Selection Dataset

  • Training treats a masked reference summary as a QA query and its source document as the passage, learning to predict the masked entity span.When an entity appears multiple times in the source, candidates are ranked using fuzzy string matching with the query sentence.Source: §3.2 · p. 4

3.3 QA-Span Fact Correction Model

  • QA-span correction masks and replaces one entity at a time; the paper reports it works better when the draft contains only a few errors.The predicted span fills the masked slot, and the updated summary becomes the query for the next correction step.Source: §3.2 · p. 4, §3.3 · p. 4

3.4 Auto-regressive Fact Correction Model

  • When many entities are wrong, the auto-regressive model masks them all and fills them sequentially with corrected context, using beam search to reduce greedy error accumulation.Its decoder predicts a source span for each mask, and the earlier predicted entities inform later predictions.Source: §3.4 · pp. 5–6

4 Experiment

The experiments evaluate SpanFact correction on three datasets using factuality and ROUGE metrics, compare against summarization and correction baselines, and include human assessment.

4.1 Experimental Setup

  • Training data come from CNN/DailyMail, XSum, and Gigaword; entities without source spans map to [CLS] during training and are restored unchanged if predicted at inference.Models use BERT-large encoders initialized from an English and SQuAD-pretrained checkpoint.Source: §4.1 · p. 6

4.2 Evaluation Metrics

  • Evaluation combines ROUGE for summarization quality with FactCC and QGQA as factuality metrics; QGQA compares source- and summary-derived answers across questions.FactCC predicts whether a system-generated summary is consistent with its source.Source: §4.2 · pp. 6–7

4.4 Experimental Results

  • Across CNN/DailyMail, XSum, and Gigaword, correction models boost factuality measures with small ROUGE drops; XSum and Gigaword average drops are -0.5 to -1.5.On CNN/DailyMail, QA-span performs better than auto-regressive correction.Source: Table 3 · p. 7, Table 4 · p. 7, Table 5 · p. 8

4.3 Baselines

  • Comparisons include established abstractive systems and Split Encoder, whose pointer-generator regenerates the masked query rather than selecting replacement spans.The baselines vary by dataset, including BertSum and TransformerAbs on CNN/DailyMail and XSum, and pointer-generator and GenParse on Gigaword.Source: §4.3 · p. 7

4.4 Experimental Results

  • On XSum, factuality scores are especially low; source-absent reference entities can impair QA-span context, and auto-regressive performs better.The authors attribute the low scores in part to XSum's construction, which uses the source's first sentence as the summary and the remainder as source.Source: §4.4 · p. 8

4.5 Human Evaluation

  • On 508 FactCC sentences, QA-span corrected 18 of 62 incorrect sentences and auto-regressive corrected 16; they made 3 and 4 wrong entity changes among 446 correct sentences.The outputs were manually checked against the human-annotated FactCC test set.Source: Table 7 · p. 9, §4.5 · p. 9
  • In 450 sampled CNN/DailyMail article comparisons, human raters chose both corrected variants more often than originals, with comparable preferences between correction models.Three annotators judged each anonymized pair for factual correctness against the source document.Source: §4.5 · pp. 8–9, Table 6 · p. 8

5 Conclusion

SpanFact improves the factuality of summaries from state-of-the-art abstractive systems without a huge drop in ROUGE, using span selection to replace entity masks.

5 Conclusion

  • SpanFact improves factuality in summaries from state-of-the-art abstractive systems without a huge drop in ROUGE.The conclusion reports this empirical outcome without specifying metric values.Source: §5 · p. 9
  • SpanFact's two factual correction models use span selection to replace one or multiple entity masks at a time.The models can be used to correct facts in any abstractive summaries.Source: §5 · p. 9
  • Future work will apply the method to other spans, including noun phrases, verbs, and clauses.Source: §5 · p. 9

Related papers

Loading 2010.02443v1…