Source-linked AI summary
BRIO: Bringing Order to Abstractive Summarization
Yixin Liu, Pengfei Liu, Dragomir Radev, Graham Neubig
TL;DR
MLE concentrates probability on the reference, although inference must compare imperfect generated candidates whose prefixes may deviate from the reference. BRIO assigns probability to candidates according to quality through contrastive learning while retaining generation with cross-entropy, achieving state-of-the-art summarization results and improved candidate-quality estimation.
Problem
MLE does not train models to rank non-reference candidate summaries by quality, even though inference compares such candidates and can suffer from exposure bias.
Method
BRIO combines standard MLE with a contrastive loss that coordinates candidate probabilities with ROUGE-based quality.
Results
BRIO achieves new state-of-the-art performance on CNN/DailyMail and XSum and estimates candidate-summary quality more accurately.
Takeaways & Limitations
The method gives one abstractive model both generation and reference-free candidate-evaluation roles.
Takeaways & Limitations
Generating candidate summaries with large neural models across the entire training set can introduce non-trivial overhead.
Abstract
from arXiv · showhide
Abstractive summarization models are commonly trained using maximum likelihood estimation, which assumes a deterministic (one-point) target distribution in which an ideal model will assign all the probability mass to the reference summary. This assumption may lead to performance degradation during inference, where the model needs to compare several system-generated (candidate) summaries that have deviated from the reference summary. To address this problem, we propose a novel training paradigm which assumes a non-deterministic distribution so that different candidate summaries are assigned probability mass according to their quality. Our method achieves a new state-of-the-art result on the CNN/DailyMail (47.78 ROUGE-1) and XSum (49.07 ROUGE-1) datasets. Further analysis also shows that our model can estimate probabilities of candidate summaries that are more correlated with their level of quality.
1 Introduction
Abstractive models trained with MLE may rank imperfect candidate summaries poorly because MLE concentrates on the reference. BRIO introduces a quality-aware, non-deterministic target distribution and dual generation-evaluation role.
- Motivation: 54.80% accuracy: original BART often fails to assign higher probability to the higher-ROUGE candidate.The study compares the best and worst candidates generated by BART on CNN/DailyMail.
- Motivation: MLE training assigns high probability to the reference but does not train relative comparisons among non-reference summaries.This leaves model scores potentially misaligned with the quality metric used during evaluation.
- Contribution: The abstractive model serves both as an autoregressive generation model and as an evaluation model scoring candidate outputs.This dual role supports selecting among generated candidates without a separate evaluation model.
- Contribution: BRIO changes the target from a one-point deterministic distribution to a non-deterministic distribution assigning candidate summaries probability according to quality.The paradigm is designed for candidate ranking during inference.
- Results: BRIO reports new state-of-the-art performance on CNN/DailyMail and XSum.The supplied introduction states the benchmark-level result without listing the scores.
2 Neural Abstractive Summarization
Neural abstractive summarization generates summaries autoregressively, while standard MLE trains against reference prefixes. At inference, errors in the model’s own prefixes create exposure bias and can degrade performance.
- Background: The goal is to learn a function g that maps a source document D to an appropriate summary S.
- Training Objective: MLE maximizes the likelihood of the reference summary S∗ given gold preceding subsequences.The training objective is implemented through token-level negative log-likelihood or cross-entropy.
- Training Objective: Under standard MLE, ptrue is a one-hot distribution centered on the reference output.Label smoothing can instead assign probability mass β to other tokens.
- Inference: Beam search reduces the intractable space of possible autoregressive candidate outputs during inference.The model estimates each next word from its previously predicted sequence.
- Inference: Exposure bias arises because inference conditions on the model’s own predictions rather than the reference, so deviations can cause performance degradation.
3 Coordinating Abstractive Models
BRIO coordinates candidate probabilities with quality by generating varied candidates and fine-tuning with a contrastive loss. It combines this sequence-level objective with cross-entropy to retain generation ability.
- Motivation: MLE can assign zero probability to non-reference candidates, whereas BRIO seeks probabilities correlated with candidate quality.The desired ordering is evaluated using an automatic metric M.
- Contrastive Learning for Coordination: ROUGE defines candidate quality, and generated candidates are ranked so better candidates should receive higher estimated probabilities.Candidates are produced from a pretrained abstractive model before contrastive fine-tuning.
- Contrastive Learning for Coordination: The contrastive loss gives the model a reference-free evaluation role for scoring candidates in a two-stage summarization pipeline.Contrastive-only fine-tuning does not preserve the model’s generation role.
- Multi-task Fine-tuning: The combined objective Lmul = Lxent + γLctr preserves generation ability while training sequence-level coordination.Cross-entropy provides token-level normalization, while γ weights the contrastive loss.
- Implementation: Diverse beam search generates the candidate summaries used for contrastive learning.The paper notes that candidate generation introduces overhead when applied across the training set.
4 Related Work
Prior Seq2Seq work uses structured or margin-based losses, reinforcement learning, generated-text contrastive examples, and separate rerankers. BRIO differs through supervised contrastive training and shared Seq2Seq parameterization.
- Structured Losses: Margin-based Seq2Seq losses require higher probability for better outputs but commonly assume a deterministic target distribution.
- Reinforcement Learning: Reinforcement learning addresses non-differentiable quality optimization but is sensitive to reward-shaping and related design choices.BRIO is presented as supervised learning in contrast to this family.
- Contrastive Learning: BRIO constructs contrastive examples from model-generated texts rather than rule-based perturbations of reference outputs.This makes the contrastive task closer to generation.
- Discriminative Reranking: Existing discriminative reranking methods use large pretrained language models as evaluators, whereas BRIO factorizes the Seq2Seq model trained on the same dataset.This maximizes parameter sharing across generation and reranking stages.
5 Experiments
Experiments show that BRIO improves abstractive summarization across datasets, backbones, evaluation metrics, inference widths, and candidate-quality estimation. Analyses also examine optimization behavior, calibration, abstraction, noise filtering, and iterative fine-tuning.
- 5.2 Results: BRIO is not restricted to one backbone: experiments use BART on CNNDM and NYT and PEGASUS on XSum.The XSum result specifically supports applicability beyond the BART base model.
- 5.2 Results: BRIO-Ctr outperforms SimCLS as an evaluation model, while BRIO-Mul establishes new state-of-the-art performance on CNNDM.BRIO-Ctr uses BART for both candidate generation and scoring; BRIO-Mul outperforms methods improving upon BART.
- 5.3 Analysis: Increasing the contrastive-loss weight improves performance, but cross-entropy remains necessary to preserve generation ability; iterative candidate regeneration further improves BRIO.BRIO-Loop is trained on candidates generated by BRIO-Mul and reaches its best performance quickly.
- 5.3 Analysis: BRIO benefits from larger beam widths, whereas BART performance decreases as beam width increases, consistent with better probability-quality coordination.The comparison evaluates BART and BRIO-Mul on CNNDM using different beam-search widths.
- 5.3 Analysis: BRIO improves performance when candidate summaries are ordered by either ROUGE or BERTScore, and gains transfer across the two metrics.The authors interpret this as evidence that the method can target specific evaluation metrics without merely exploiting weaknesses of one metric.
- 5.3 Analysis: BRIO better estimates candidate quality and calibration, produces more novel n-grams, and filters the noisy “click here” pattern generated by BART.Its probability estimates have better rank correlation, its calibration improves, and it never generated that pattern across the test set.
6 Conclusion and Future Work
The paper concludes that BRIO assigns probability mass to candidate summaries according to quality through contrastive learning. It also identifies extensions to other generation settings and candidate-generation methods as future work.
- BRIO assigns candidate outputs probability mass according to their quality using contrastive learning.
- BRIO may extend to other conditional text generation tasks, including machine translation, because its method makes no task-specific assumptions.
- The method could be applied in reinforcement learning with dynamically generated candidate summaries.
- Experiments used diverse beam search for candidate generation, leaving other generation methods as a possible source of further improvements.
A Datasets Statistics
Table 12 presents dataset statistics for the datasets used in the experiments.
- Table 12 is titled “Datasets Statistics.”
- The table concerns statistics associated with the paper’s experimental datasets.
- No individual dataset values are stated in the supplied table passage.
B Implementation Details
The implementation uses diverse beam search and pretrained abstractive models to generate candidates and perform fine-tuning, with specified optimization, scoring, and evaluation settings.
- Diverse beam search generates 16 candidate summaries for each data sample.
- BART12 and PEGASUS13 serve as base abstractive models for CNNDM and XSum candidate generation and fine-tuning, while NYT uses an MLE-fine-tuned BART model14.
- The scoring function retains the original beam-search length penalty, while the contrastive-loss margin λ is selected from [1 × 10−5, 1] using validation performance.
- Evaluation uses the standard ROUGE Perl package with lower-cased, tokenized reference summaries and system outputs.
- Table 13 reports the hyper-parameter setting used in the experiments.
C Details of Few-shot Fine-tuning
Few-shot fine-tuning uses dataset-specific sample sizes, repeated runs, validation-based hyper-parameter selection, Adam optimization, and fixed training durations.
- CNNDM fine-tuning randomly selects 100 training examples, whereas XSum requires at least 1000 examples to outperform the baseline.
- All experiments are repeated three times, with 1000 validation examples randomly selected for hyper-parameter selection.
- Adam optimization uses a learning rate of 1 × 10−6, with training for 15 epochs on CNNDM and 10 epochs on XSum.
- The PTB tokenizer is used for tokenization.