Improving Factuality of Abstractive Summarization via Contrastive Reward Learning

1 Introduction

• Prior approaches augment inputs, post-process outputs, or modify learning [Cao and Wang, 2021; Liu et al., 2021].The passage notes that learning-based methods need not change model architecture or add modules.Source: §1 · p. 1
• Reward learning from automatic metrics or human preferences has gained attention, using reinforcement learning, minimum risk training, and contrastive reward learning.Source: §1 · p. 1
• The paper asks whether contrastive reward learning can use existing factuality metrics effectively and whether factuality improvements appear in human evaluations.Source: §1 · p. 1
• The framework learns directly from factuality-metric feedback sample-efficiently, avoids complex negative-sample construction, and uses quality metrics to rank candidates more finely.The investigated factuality metrics are BARTScore and DAE.Source: §1 · pp. 1–2
• Automatic and human evaluations show that the framework enables summarization models to generate significantly more factual summaries.Source: §1 · p. 2

2.1 Contrastive Learning for Abstractive Summarization

• The contrastive loss ranks pretrained-model candidate summaries by quality metric and penalizes disagreement between metric ordering and length-normalized model likelihood.The rank-gap term is scaled by λ, usually set to 1, and the likelihood is length-normalized using α.Source: §2.1 · p. 2
• Fine-tuning combines maximum-likelihood and contrastive losses, weighted by γ; models trained with this objective are called CRL-COM.The maximum-likelihood term trains against reference summaries, while the contrastive term incorporates metric-based quality.Source: §2.1 · p. 2

2.2 Reward from Factuality Metrics

• BARTScore measures the log-likelihood of a summary given its source, while DAE uses the softmax output for the summary’s least-factual dependency arc.The authors select these factuality metrics for relative computational efficiency because they are evaluated many times during training.Source: §2.2 · p. 2
• QA-based factuality metrics are less feasible for reward learning because they are computationally inefficient.Source: §2.2 · p. 2

3.1 Experimental Setup

• The setup fine-tunes BART on CNNDM and PEGASUS on XSUM, using ROUGE, BARTScore, or DAE as the contrastive-learning quality metric.Few-shot fine-tuning uses 1,000 training samples per dataset; human evaluation samples 100 CNNDM and 200 XSUM summaries, rating factuality, coherence, and relevance.Source: §3.1 · p. 3

Contrastive reward learning can enforce models to learn from feedback of factuality metrics

• Automatic evaluation indicates that contrastive reward learning moves models toward outputs aligned with the factuality metrics used for feedback.Source: Contrastive reward learning can… · p. 3
• Human evaluations on both datasets find CRL-COM (B) and CRL-COM (D) superior in factuality to CRL-COM (R).The comparison is reported in Table 2; the authors note factuality gains may come with lower ROUGE scores.Source: Contrastive reward learning can… · p. 3, Table 2 · p. 4
• Human evaluations find CRL-COM (B) and CRL-COM (D) comparable to CRL-COM (R) in coherence and relevance.The authors also interpret this as comparable metric ability for BARTScore and DAE versus ROUGE on those dimensions.Source: Contrastive reward learning can… · pp. 3–4

4.1 Factuality Metrics for Abstractive Summarization

• QA-based metrics compare answers to summary-generated questions across the source and summary, while entailment-based metrics test whether summary information is entailed by the source.Pretrained language model evaluation has also achieved competitive factuality-evaluation performance.Source: §4.1 · p. 4

4.2 Improving Factuality of Abstractive Summarization via Contrastive Learning

• CLIFF and CO2Sum learn factuality from positive and negative samples, with CO2Sum using more sophisticated negative-sample construction.Source: §4.2 · p. 4

• The framework trains abstractive summarization models from feedback provided by existing factuality metrics through contrastive reward learning.Source: §5 · p. 4
• Empirical studies find that models trained from factuality-metric feedback generate more factual summaries without sacrificing coherence or relevance.Source: §5 · p. 4
• The conclusion suggests advances in reward learning and factuality metrics can facilitate more factually consistent abstractive summaries.Source: §5 · p. 4

• The experiments cover two datasets, CNNDM and XSUM; more non-news datasets could be included in future studies.Source: §6 · p. 5
• Comparing RL-based and contrastive reward learning for factuality improvement is another proposed direction for future work.Source: §6 · p. 5

7 Ethics Statement

• High factuality on CNNDM does not guarantee off-the-shelf factual consistency.The passage cautions against treating dataset performance as sufficient for deployment.Source: §7 · p. 5
• Thorough evaluation should precede using these models in high-stakes settings.The stated aim is to ensure their reliability.Source: §7 · p. 5
• The deployment caveat applies even to systems that achieve high factuality on CNNDM.The passage frames that level of factuality as insufficient assurance for off-the-shelf use.Source: §7 · p. 5