PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization

Shuyang Cao, Lu Wang

arXiv:2109.09209v1cs.CL

TL;DR

Abstractive summarizers are fluent but often factually inconsistent, and maximum-likelihood training does not ensure discrimination between references and incorrect generations. CLIFF trains with contrastive learning over references and automatically generated erroneous summaries, achieving consistent factuality improvements across datasets and models, with human judges finding more corrected errors.

  • Problem

    Abstractive summaries often contain factual inconsistencies, while maximum-likelihood training does not guarantee distinguishing references from incorrect generations.

  • Method

    CLIFF uses contrastive learning with references as positive samples and automatically constructed erroneous summaries as negative samples.

  • Results

    CLIFF consistently outperforms post-error correction, entailment reranking, and unlikelihood training on QuestEval across XSum and CNN/DM, with human evaluation confirming fewer errors.

  • Takeaways & Limitations

    Contrastive training helps summarizers discriminate factual summaries from erroneous peers and improves faithfulness across datasets and different large-model backbones.

  • Takeaways & Limitations

    Constructed mask-and-fill negatives can contain drifted topics that summarization models detect easily, producing less efficient training signals.

Abstract

from arXiv · show

We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive training data, and automatically generated erroneous summaries, as negative training data, to train summarization systems that are better at distinguishing between them. We further design four types of strategies for creating negative samples, to resemble errors made commonly by two state-of-the-art models, BART and PEGASUS, found in our new human annotations of summary errors. Experiments on XSum and CNN/Daily Mail show that our contrastive learning framework is robust across datasets and models. It consistently produces more factual summaries than strong comparisons with post error correction, entailment-based reranking, and unlikelihood training, according to QA-based factuality evaluation. Human judges echo the observation and find that our model summaries correct more errors.

1 Introduction

CLIFF addresses factual inconsistency in abstractive summaries by training models to distinguish reference summaries from factually incorrect generations. It uses contrastive learning with error-like negative samples and consistently improves factuality across XSum and CNN/DM.

  • Motivation: Large pre-trained Transformers produce fluent abstractive summaries but often include factually inconsistent content.Existing remedies include separate error correction, noisy-sample removal, and architecture modification.
  • Results: CLIFF’s contrastive-learning model correctly generates the full name in the illustrated XSum example, unlike the cross-entropy and entailment-reranking comparisons.The example shows those comparison summaries producing unfaithful content.
  • Motivation: Maximum-likelihood training optimizes references without guaranteeing that models distinguish references from incorrect generations.CLIFF targets this gap by explicitly informing preferences for factual summaries over incorrect ones.
  • CLIFF framework: CLIFF uses contrastive learning to expand the margin between factually consistent summaries and incorrect peers.References serve as positive samples, while erroneous summaries serve as negative samples.
  • Negative samples: Four negative-sample strategies edit references or use system outputs, reflecting errors annotated in BART and PEGASUS summaries on XSum and CNN/DM.The strategies include entity- and relation-anchored rewriting and system-generated summaries with unfaithful errors.
  • Results: Across XSum and CNN/DM, CLIFF models uniformly outperform post-error correction, entailment reranking, and unlikelihood training on QuestEval.Human evaluation also finds consistent reductions in extrinsic and intrinsic errors.

2 Related Work

Prior work improves summary factuality through correction models, architectural changes, and other training strategies, while evaluation increasingly uses entailment and QA-based methods. CLIFF applies contrastive learning to distinguish factual from incorrect summaries.

  • Factuality improvement: Prior factuality remedies learn separate correction models, modify sequence-to-sequence architectures, or remove noisy training samples.These approaches target errors through post-generation correction, architectural information, or data filtering.
  • Factuality evaluation: ROUGE and other word-matching metrics correlate poorly with human-rated factual consistency, motivating automated entailment- and QA-based evaluation.Entailment scorers operate at summary or dependency-relation levels, while QA models assess consistency by reading articles.
  • Negative sampling: Negative sampling has been studied in grammatical error detection, dialogue systems, word representations, and masked language-model training.Unlikelihood training penalizes negative tokens or sentences in text generation.
  • Contrastive learning: CLIFF uses contrastive learning to improve representations that distinguish factual summaries from incorrect ones and encourage more faithful generation.This extends contrastive-learning applications from vision and language representation learning to summarization factuality.

3 CLIFF: Contrastive Learning Framework for Summarization

CLIFF fine-tunes BART and PEGASUS with contrastive learning that separates faithful reference summaries from erroneous ones. It combines this objective with cross-entropy and explores methods for constructing positive samples and representing summaries.

  • Contrastive Learning Objective: CLIFF trains summarizers to prefer faithful summaries over summaries containing factual errors.The contrastive objective compares positive reference summaries with negative erroneous summaries in the same training batch.
  • Contrastive Learning Objective: Reference summaries form positive samples, while erroneous summaries form negative samples for contrastive training.The objective uses summary representations and cosine similarity, with temperature τ set to 1.0.
  • Contrastive Learning Objective: Positive and negative summaries are placed in the same batch to maximize positive-sample probabilities and minimize corresponding negative-sample likelihoods.This batch construction helps the model differentiate correct summaries from summaries with errors.
  • Sample Construction: Back-translation is selected to create multiple positive samples because it preserves meaning while providing language variation.Synonym substitution and masked-language-model replacement were also evaluated in a pilot study.
  • Summary Representation: Summary representations use decoder last-layer outputs averaged over all tokens, named entities, or the final token, optionally followed by a one-hidden-layer MLP.Entity and parsing information is obtained with spaCy.
  • Training Objective: The final objective is L = LCE + λLCL, with λ set to 1.0 in all experiments.This combines the typical cross-entropy loss with the contrastive learning loss.

4 Summary Error Annotation and Model Behavior Analysis

The authors annotate unfaithfulness in BART- and PEGASUS-based summaries on XSum and CNN/DM, then analyze generation behavior to guide negative-sample design. Extrinsic errors dominate, and low confidence is associated with extrinsic error spans.

  • Annotation Study: 600 summaries are annotated for hallucinated content, intrinsic errors, extrinsic errors, and world knowledge across XSum and CNN/DM.Intrinsic errors misconstruct source phrases or clauses, while extrinsic errors introduce unverifiable or unsupported words.
  • Annotation Study: Two experienced English speakers label error and world-knowledge spans while reviewing each article and two system summaries.They discuss disagreements every 50 samples; Fleiss’s Kappa is 0.35 on XSum and 0.45 on CNN/DM.
  • Error Statistics: Extrinsic errors dominate both datasets: 58.7% of BART summaries and 44.0% of PEGASUS summaries contain at least one.The difference is especially pronounced on XSum.
  • Model Behavior Analysis: Figure 3 groups first-token generation probabilities for proper nouns and numbers by extrinsic errors, intrinsic errors, world knowledge, and correct tokens.The analysis probes model behavior that can guide negative-sample construction.
  • Model Behavior Analysis: Extrinsic-error spans tend to begin with lower-confidence proper-noun or number tokens than world-knowledge spans.In one BART example, “Manchester City” receives probability 0.10, whereas “Wayne” receives 0.92.

5 Negative Sample Construction

CLIFF constructs negative summaries through reference editing, mask-and-fill generation, source-conditioned regeneration, and low-confidence system outputs. These strategies target different error types but vary in source grounding and realism.

  • Overview: Four negative-sample strategies modify references or use system-generated summaries to approximate unfaithful model outputs.The strategies are designed around errors observed in BART and PEGASUS annotations.
  • Entity Swap: SWAPENT replaces each reference named entity with a randomly selected source entity of the same type to imitate intrinsic errors.It creates one sample per reference entity and does not depend on a trained model.
  • Entity Swap: SWAPENT covers intrinsic errors but lacks extrinsic-error coverage, motivating the subsequent generation-based methods.Its model independence is an advantage, but its error scope is limited.
  • Mask-and-fill with BART: MASKENT masks reference entities and uses an unfine-tuned BART model to generate replacement entities, while MASKREL masks governor and dependent spans to introduce new relations.MASKREL retains samples containing dependency relations absent from both the source and reference.
  • Mask-and-fill with BART: MASKENT and MASKREL can create more extrinsic errors because their generations are not grounded in source articles.They may also drift in topic, producing errors that are easy for the summarizer to detect and therefore weaker training signals.
  • Source-conditioned Regeneration: REGENENT and REGENREL conditionally regenerate content after a reference prefix using a source-reading summarizer and retain samples with new entities or relations.These regenerated negatives are more article-relevant than mask-and-fill samples but may miss errors and differ from real outputs because they modify references.
  • System-generated Negatives: SYSLOWCON selects BART- or PEGASUS-generated summaries when proper-noun or number spans begin below a threshold tuned for maximum F1 on annotated errors.All beams at the last decoding step are considered, using beam sizes 6 for XSum and 4 for CNN/DM.

6 Experiment Setup

The experiments evaluate factual consistency with QA-based and entailment metrics, compare CLIFF-related alternatives, and analyze metric correlations with annotated summary errors.

  • Evaluation Metrics: QuestEval evaluates factual consistency through bidirectional question answering between article and summary, using the harmonic mean of two QA scores.The learned-weight version is reported to correlate highly with human judgments of consistency and relevance.
  • Evaluation Metrics: FactCC measures whether summaries are entailed by the source, while ROUGE-L provides a word-overlap baseline that also correlates with human factuality judgments.
  • Evaluation Metrics: QuestEval correlates better than other metrics with both annotated error-token rates and raw error counts.
  • Comparisons: Comparisons include cross-entropy training, FactCC beam reranking, post-hoc correction, dependency-clean subset fine-tuning, knowledge-graph modeling, and unlikelihood training.Unlikelihood training penalizes probabilities of tokens in negative samples and is combined with cross-entropy using equal weights.
  • Comparisons: CLIFF also compares alternative negative samples, including FactCC scorer samples and other summaries in the same batch.

7 Results

Across XSum and CNN/DM with BART and PEGASUS, CLIFF generally improves factuality while preserving informativeness, outperforming or exceeding several factuality-improvement comparisons.

  • Automatic Evaluation: CLIFF models use averaged all-token summary representations with MLP projection, while experiments compare BART and PEGASUS objectives and negative samples across XSum and CNN/DM.
  • Automatic Evaluation: CLIFF models generally achieve higher QuestEval scores across datasets and both base models, with larger improvements on XSum.
  • Automatic Evaluation: CLIFF maintains comparable or better ROUGE than cross-entropy baselines, whereas SUBSETFT significantly sacrifices ROUGE and correction or FASUM provide only incremental gains.
  • Automatic Evaluation: CLIFF beats unlikelihood training on QuestEval in 12 of 14 BART comparisons and 11 PEGASUS setups using the same negative samples.Similar trends occur for FactCC and ROUGE-L, while unlikelihood training sometimes substantially harms factuality or ROUGE.
  • Negative Samples: Low-confidence summaries produce the best QuestEval scores among CLIFF variants on the more abstractive dataset and improve FactCC on both datasets.They also enhance ROUGE-L on CNN/DM; system-generated summaries provide diverse errors suited to the contrastive framework.
  • Negative Samples: Combining SYSLOWCON with other strategies generally improves QuestEval over individual strategies, indicating that diverse error coverage matters.
  • Human Evaluation: Human judges more often rate low-confidence CLIFF summaries as more informative and factual than cross-entropy summaries on XSum, while negative-sample models improve both aspects on CNN/DM.Entailment-reranked summaries are less distinguishable from cross-entropy outputs because of more ties.
  • Error Analysis: Contrastive learning reduces extrinsic errors more effectively than unlikelihood training, though summaries show slight decreases in world knowledge.

8 Conclusion

CLIFF trains abstractive summarizers to distinguish factual, salient content from errors using contrastive learning. Its automatically constructed erroneous summaries support consistent improvements across datasets and pretrained models.

  • CLIFF uses references and factually inconsistent summaries to train models to discriminate errors from factual and salient content.
  • Four strategies automatically create erroneous summaries by editing references or leveraging system outputs, based on summary-error analysis.
  • Automatic evaluation and human ratings show consistent improvements over competitive comparison methods across datasets and models fine-tuned from different large models.

A Additional Analysis for Summary Error Annotation

The authors annotate summaries from BART and PEGASUS on XSum and CNN/DailyMail, then analyze token-generation probabilities for different error types. Error tokens can remain highly probable when they are generated after the first token.

  • Two fluent English speakers annotate 100 summaries from BART and PEGASUS, using 50 articles per dataset batch across XSum and CNN/DailyMail.
  • The analysis reports first-token probability distributions for nouns and verbs and non-first-token distributions for proper nouns, numbers, nouns, and verbs.
  • Tokens within extrinsic and intrinsic errors have high generation probabilities when they are non-first tokens.

B Statistics for Datasets and Training Samples

The appendix documents datasets, sample construction, training settings, decoding configurations, computational costs, and model sizes for XSum and CNN/DailyMail experiments.

  • The experiments follow official XSum and CNN/DailyMail data splits, with split sizes reported in Table 8.
  • 182,114 and 91,468 positive samples are obtained by back-translation on XSum and CNN/DailyMail after filtering unfaithful paraphrases.
  • A 0.21 selection threshold yields F1 scores of 73.99 on XSum and 40.49 on CNN/DailyMail for BART-generated annotated samples.
  • SYSLOWCON constructs the fewest negative samples but achieves the best reported results, indicating greater training effectiveness per constructed sample.
  • BART and PEGASUS use beam search with dataset-specific settings, while their models contain 400M and 568M parameters, respectively.
  • BART-based training takes 6 and 13 hours on XSum and CNN/DailyMail, whereas PEGASUS-based training takes 8 and 25 hours.

D Human Evaluation

Human evaluation examines world-knowledge content in summaries from XSum and CNN/DailyMail. CLIFF-related models reduce such content on XSum while matching or exceeding cross-entropy training on CNN/DailyMail.

  • On XSum, all models produce less world knowledge than the model trained with cross-entropy loss.
  • Figure 8 reports the percentages of samples containing world knowledge, as labeled by human evaluators, for both datasets.
  • On CNN/DailyMail, models generate similar or greater amounts of world knowledge compared with cross-entropy training.

E Sample Outputs

This section illustrates the human-evaluation setup, summary-error categories, and operations used to compare generated summaries against a baseline. Examples show how informativeness and factuality judgments distinguish corrections, omissions, and unsupported content.

  • Error Categories: Intrinsic errors misconstruct phrases or clauses from the article, while extrinsic errors introduce unsupported content.The evaluation instructions identify spans contradicted by or not implied by the article.
  • Figures: The figures provide guidelines for summary-error annotation and human evaluation, while sample outputs visualize intrinsic and extrinsic errors in generated summaries.MASKENT and SYSLOWCON are used to construct negative samples for unlikelihood training and contrastive learning.
  • Human Evaluation: Human evaluation presents an article, a baseline summary with highlighted errors, and four system summaries judged for informativeness and factual consistency.Summaries are shown in random order without system names to support fair comparison.
  • Sample Outputs: In the Lexi example, the baseline incorrectly changes the original £2,000 appeal target to £10,000 and adds unsupported details about the death.The article states that Lexi died at home and that the appeal was originally hoped to raise £2,000.
  • Evaluation Criteria: Informativeness rewards retaining salient correct content: mentioning the appeal target wins, omitting incorrect content ties, and omitting the girl's death loses.Incorrect content is treated as invalid when judging whether salient article content was captured.
  • Correction Operations: Baseline errors can be handled through deletion or substitution, illustrated by removing the incorrect £10,000 figure or replacing it with £2,000.Both operations preserve the surrounding account of the appeal and the girl's death.
Loading 2109.09209v1…