Source-linked AI summary
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Yixin Liu, Pengfei Liu
TL;DR
Abstractive summarization faces a mismatch between Seq2Seq training objectives and holistic evaluation metrics. SimCLS separates candidate generation from contrastive, reference-free evaluation, and reports improved performance over baseline systems, including a 2.2 ROUGE-1 improvement against the baseline model on CNN/DailyMail.
Problem
Seq2Seq training with MLE optimizes local token predictions, while summarization metrics evaluate holistic similarity, leaving training and evaluation objectives mismatched.
Method
SimCLS uses a two-stage framework that generates candidate summaries with a Seq2Seq model and ranks them with a contrastively trained reference-free evaluation model.
Results
2.2 ROUGE-1 improvement against the baseline model is reported on CNN/DailyMail, alongside improvements over the baseline on all evaluated metrics.
Takeaways & Limitations
The results indicate that pretrained abstractive systems can generate candidates better than their original outputs and support further two-stage optimization beyond MLE.
Takeaways & Limitations
RL-based alternatives are described as unstable and sensitive to hyper-parameters, while estimated loss accuracy can be restricted by the number of sampled outputs.
Abstract
from arXiv · showhide
In this paper, we present a conceptually simple while empirically powerful framework for abstractive summarization, SimCLS, which can bridge the gap between the learning objective and evaluation metrics resulting from the currently dominated sequence-to-sequence learning framework by formulating text generation as a reference-free evaluation problem (i.e., quality estimation) assisted by contrastive learning. Experimental results show that, with minor modification over existing top-scoring systems, SimCLS can improve the performance of existing top-performing models by a large margin. Particularly, 2.51 absolute improvement against BART and 2.50 over PEGASUS w.r.t ROUGE-1 on the CNN/DailyMail dataset, driving the state-of-the-art performance to a new level. We have open-sourced our codes and results: https://github.com/yixinL7/SimCLS. Results of our proposed models have been deployed into ExplainaBoard platform, which allows researchers to understand our systems in a more fine-grained way.
1 Introduction
SimCLS addresses the mismatch between Seq2Seq training objectives and holistic evaluation metrics by separating generation from metric-oriented candidate evaluation. Its two-stage contrastive framework improves abstractive summarization performance over baseline systems.
- Motivation: Seq2Seq summarization models trained with MLE optimize local token predictions, whereas ROUGE evaluates holistic similarity between references and outputs.Autoregressive generation also accumulates errors during testing, creating exposure bias.
- Motivation: Existing alternatives face optimization challenges: RL-based training has noisy gradient estimates, while minimum-risk estimates depend on the number of sampled outputs.Sentence-level extensions of MLE can also leave the relation between objectives and evaluation metrics indirect and implicit.
- Contribution: SimCLS introduces a generate-then-evaluate framework that directly optimizes toward corresponding evaluation metrics using contrastive learning.The approach formulates candidate selection as reference-free evaluation and aims to mitigate training–test gaps.
- Contribution: The framework separates MLE and contrastive loss into different stages rather than using contrastive learning only as an augmentation of MLE.This design trains generation and evaluation modules separately with supervised learning, bypassing RL-based optimization.
- Results: 2.2 ROUGE-1 improvement against the baseline model is reported on CNN/DailyMail, raising state-of-the-art performance.The paper presents this result as evidence for the potential of two-stage optimization beyond maximum likelihood estimation.
2 Contrastive Learning Framework for Abstractive Summarization
SimCLS decomposes summarization into candidate generation and reference-free evaluation. A contrastively trained evaluator ranks candidates using their similarity to the source document.
- Framework: The framework uses a generation model g to produce candidate summaries and an evaluation model h to score and select the best candidate.The final output is the candidate receiving the highest evaluation score.
- Stage I: Candidate Generation: Stage I trains a Seq2Seq generator with likelihood of the reference summary and uses sampling strategies such as Beam Search to produce multiple candidates.The number of candidates is denoted by n.
- Stage II: Reference-free Evaluation: Stage II assigns candidate scores solely from source–candidate similarity, without using the reference summary at evaluation time.The evaluator computes scores r_i = h(S_i, D) and selects the highest-scoring candidate.
- Stage II: Reference-free Evaluation: RoBERTa instantiates the evaluator by separately encoding each candidate and source document and comparing their first-token representations with cosine similarity.This similarity becomes the candidate score r_i.
- Contrastive Training: Contrastive training ranks candidates sorted by their reference-based metric scores using margins λ_ij = (j−i)∗λ.The paper uses ROUGE as M, while noting that automated metrics or human judgments can define the ranking signal.
3 Experiments
The experiments use CNN/DailyMail and XSum to evaluate summarization with ROUGE and semantic similarity metrics. CNN/DailyMail is large-scale news summarization, while XSum is highly abstractive.
- Datasets: The experiments use CNN/DailyMail, a large-scale news-article dataset, and XSum, a highly abstractive dataset of BBC online articles.Dataset statistics are reported in Appendix A.
- Evaluation Metrics: ROUGE-1/2/L are the main evaluation metrics used in the experiments.The paper abbreviates these metrics as R-1/2/L.
- Evaluation Metrics: BERTScore and MoverScore provide additional semantic similarity evaluations.These metrics complement the main ROUGE-based evaluation.
3.3 Base Systems
The experiments use pretrained abstractive systems as generators and evaluate candidates with multiple metrics and candidate-selection settings on CNNDM.
- Base Systems: BART and Pegasus serve as pretrained generation models, with GSum and ProphetNet included for comparison.The baseline checkpoints come from Transformers.
- Base Systems: Sixteen diverse-beam-search groups generate 16 candidate summaries for evaluation-model training.The evaluation-model checkpoint is selected using validation-set performance.
- Base Systems: Table 1 reports ROUGE, BERTScore, and MoverScore, with Origin denoting the baseline’s original performance.Min, Max, and Random are ROUGE-based candidate-selection oracles.
- Base Systems: Figure 2 evaluates test performance as the number of candidate summaries changes, relative to the baseline’s original performance.Origin is the original baseline result.
3.5 Results on CNNDM dataset
On CNNDM, SimCLS improves over the original abstractive baseline across evaluation metrics and remains effective with fewer candidate summaries.
- Results on CNNDM dataset: SimCLS outperforms the baseline on all reported ROUGE and semantic-similarity metrics, with statistically significant improvement beyond ROUGE artifacts.The semantic metrics are BERTScore and MoverScore.
- Results on CNNDM dataset: The Max oracle substantially exceeds original outputs, indicating that diverse sampling exposes additional potential in the pretrained abstractive system.The comparison uses candidates selected by ROUGE scores.
- Results on CNNDM dataset: Table 2 compares sentence alignments among source articles, reference summaries, SimCLS summaries, and baseline summaries.The caption specifies bold alignment for the reference and proposed summaries and italic alignment for baseline summaries.
- Results on CNNDM dataset: SimCLS remains able to outperform the baseline with fewer candidates during testing, showing robustness to the specific candidate count.Training uses as many candidates as computation permits.
3.6 Fine-grained Analysis
Fine-grained analyses examine semantic entities, sentence alignments, and positional bias to characterize how SimCLS differs from the baseline on CNNDM.
- Entity-level: SimCLS better captures salient entities from source documents than the baseline.Salient entities are selected using entities appearing in reference summaries and compared with candidate-summary entities.
- Sentence-level: SimCLS-generated summaries are more similar to reference summaries at the sentence level.The analysis matches summary sentences to source sentences using ROUGE-based similarity, then compares matched-sentence overlap.
- Sentence-level: In the case study, SimCLS focuses on the same source sentences as the reference, whereas the baseline focuses on some different sentences.The aligned reference and SimCLS sentences are identical in the example.
- Sentence-level: Figure 3 plots the relative source position of matched sentences against their ratio after truncating articles to the generator’s maximum input length.Origin denotes the original baseline performance.
- Sentence-level: For articles longer than 30 sentences, baseline summaries favor head sentences relative to references, while SimCLS mitigates this positional bias.Diverse beam search generates candidates differing from original outputs, and the evaluation model assesses their holistic quality.
3.7 Results on XSum dataset
On XSum, SimCLS still outperforms the Pegasus baseline, but the improvement margin is smaller because short, highly abstractive summaries limit candidate diversity.
- Results on XSum dataset: Pegasus is used as the XSum base system because it performs better than BART on that dataset.The same sampling strategy is used for training data, while validation and test data use four candidates from four diverse groups.
- Results on XSum dataset: SimCLS outperforms the XSum baseline with a smaller margin than on CNNDM.XSum summaries are shorter and more abstractive, restricting semantic candidate diversity and making meaningful improvement harder.
4 Conclusion
The framework improves abstractive summarization by optimizing summary quality at the summary level and evaluating candidate summaries with semantic metrics. Experiments on CNN/DailyMail and XSum examine performance gains and the potential of existing systems to generate better candidates.
- The framework targets the discrepancy between training and test stages in maximum-likelihood summarization.
- The authors comprehensively evaluate improvements at different semantic levels to explain where the method's gains arise.
- Existing abstractive systems can generate candidate summaries substantially better than their original outputs.
- The results motivate extending the two-stage strategy to other datasets and developing more holistic optimization algorithms for abstractive models.
A Dataset Statistics
The dataset-statistics table reports token lengths, while preprocessing lower-cases source documents and reference summaries and truncates some source documents because of input-length limits.
- Table 5 reports dataset statistics, with Len denoting token length.
- Source documents and reference summaries are lower-cased before processing.
- Some source documents are truncated during training because of the input-length limitation.
B Experiment Details
Experiments generate diverse candidate summaries and use a pretrained RoBERTa evaluation model, with Adam optimization and validation-based checkpoint selection. Training converges in five epochs, requiring approximately 40 hours on CNN/DailyMail and 20 hours on XSum.
- Diverse beam search uses 16 diversity groups and 16 beams, producing 16 candidate summaries.
- The evaluation model is a pretrained roberta-base model with 125M parameters.
- Training uses Adam with learning-rate scheduling and selects checkpoints using validation performance.
- Training converges in 5 epochs, taking around 40 hours on CNN/DailyMail and 20 hours on XSum using 4 GTX-1080-Ti GPUs.