PaperlayerView exact revision on arXiv

Source-linked AI summary

SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization

Mathieu Ravaut, Shafiq Joty, Nancy F. Chen

arXiv:2203.06569v2cs.CL

TL;DR

Beam search selects one summary even though discarded candidates may score better on evaluation measures. SummaReranker reranks candidate summaries, yielding relative ROUGE improvements of 1.31%–9.34% across three datasets, depending on the dataset.

  • Problem

    Beam search outputs a single top-log-probability candidate, although discarded candidates can score better on different evaluation measures.

    Source: §1 · p. 1

  • Method

    SummaReranker trains a second-stage re-ranker that encodes the source with each summary candidate and predicts whether it maximizes each optimized evaluation metric.

    Source: §6 · p. 9

  • Results

    1.31%–9.34% relative improvements on ROUGE metrics across CNN-DailyMail, XSum, and Reddit TIFU, depending on the dataset.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Selected summaries improve ROUGE state-of-the-art on CNN/DM and XSum and are more abstractive and more likely to be preferred by human evaluators than base outputs.

    Source: §6 · p. 9

    For long-document summarization, SummaReranker would need better long-range modeling because source-candidate encoding truncates sources to the encoder's fixed context window (512 tokens for RoBERTa-large).

    Source: §5 · p. 9

Abstract

from arXiv · show

Sequence-to-sequence neural networks have recently achieved great success in abstractive summarization, especially through fine-tuning large pre-trained language models on the downstream dataset. These models are typically decoded with beam search to generate a unique summary. However, the search space is very large, and with the exposure bias, such decoding is not optimal. In this paper, we show that it is possible to directly train a second-stage model performing re-ranking on a set of summary candidates. Our mixture-of-experts SummaReranker learns to select a better candidate and consistently improves the performance of the base model. With a base PEGASUS, we push ROUGE scores by 5.44% on CNN-DailyMail (47.16 ROUGE-1), 1.31% on XSum (48.12 ROUGE-1) and 9.34% on Reddit TIFU (29.83 ROUGE-1), reaching a new state-of-the-art. Our code and checkpoints will be available at https://github.com/ntunlp/SummaReranker.

1 Introduction

Autoregressive decoding can miss stronger summaries already present among generated candidates, motivating a second-stage model that learns to select candidates across evaluation measures. SummaReranker uses a multi-task mixture-of-experts architecture and reports improvements across three datasets and two base models.

  • Beam search discards all but its top-ranked candidate, although the other hypotheses can score better on evaluation measures; autoregressive errors can also propagate under exposure bias.The passage attributes exposure bias to the discrepancy between teacher-forced training and inference, and notes that global properties such as coherence and factual consistency are difficult to encode.Source: §1 · p. 1
  • Oracle ROUGE-1 scores on CNN-DailyMail are up to 10 points (+22.8%) above the top-beam baseline, and mixing decoding methods yields gains exceeding 13 points (+30.5%).These oracle scores are the maximum over generated candidates, indicating potential gains from identifying better candidates in the pool.Source: §1 · p. 2
  • SummaReranker trains a second-stage model to select the best candidate and uses multi-task mixture-of-experts learning to optimize jointly over several evaluation measures.The framework is designed to support any set of evaluation metrics and is less computationally expensive to train than the single-stage models it builds on.Source: §1 · p. 2
  • 1.31% to 9.34% relative ROUGE improvements across three datasets, plus new state-of-the-art results on CNN-DailyMail and XSum, are reported for SummaReranker.The system is evaluated with PEGASUS and BART and outperforms the second-stage approaches RefSum and SimCLS.Source: §1 · p. 2

2 Related Work

Re-ranking has a long history across NLP, while second-stage re-ranking for abstractive summarization remains underexplored. SummaReranker is presented as a multi-task approach that differs from prior summarization frameworks in its architecture, loss, and candidate-generation process.

2 Related Work

  • Prior NLP work applies re-ranking to parsing, passage and answer selection in question answering, and machine translation (Collins and Koo (2005); Charniak and Johnson (2005); Kratzwald and Feuerriegel (2018); Nogueira and Cho (2019); Kratzwald et al. (2019); Iyer et al. (2021); Pandramish and Sharma (2020); Bhattacharyya et al. (2021)).Checkpoint re-ranking uses multiple translation-model checkpoints because their oracle is higher quality than the last checkpoint alone; an energy-based model on BERT selects translation candidates with higher BLEU (Bhattacharyya et al., 2021).Source: §2 · p. 2
  • 46.18 ROUGE-1 for RefSum and 46.67 ROUGE-1 for SimCLS on CNN-DailyMail mark recent second-stage summarization results (Liu et al., 2021; Liu and Liu, 2021).RefSum addresses train-test distribution mismatch in second-stage models; SimCLS uses contrastive learning and a ranking loss to select from 16 diverse beam-search candidates.Source: §2 · p. 2
  • Unlike RefSum, SummaReranker uses one base model while mixing decoding methods; unlike SimCLS, it predicts the probability that a candidate is best rather than using a ranking loss (Liu et al., 2021; Liu and Liu, 2021).The authors describe their approach as a multi-task re-ranking system and say it enables practitioners to leverage work in automatic abstractive summarization evaluation (Lin, 2004; Zhang et al., 2019a; Zhao et al., 2019a; Yuan et al., 2021).Source: §2 · pp. 2–3

3 Model

SummaReranker trains a second-stage model to select the best summary from a candidate pool, using metric-specific predictions and shared experts. It addresses the training–inference gap by generating reranker training candidates out of sample and evaluates transfer and lower-resource inference setups.

3.1 Re-ranking Framework

  • Given a source and candidate summaries, the reranker learns to identify the candidate that scores best under a chosen evaluation metric.The candidates are generated by a base model using one or more decoding methods.Source: §3.1 · p. 3
  • Binary classification treats the best candidate as positive and the rest as negative, since tied candidate scores often leave too little signal for multi-way classification.The model is trained with binary cross-entropy.Source: §3.1 · p. 3

3.2 Model Architecture

  • For multiple evaluation metrics, separate prediction towers optimize their respective losses, while shared experts and task-specific gates allocate expert weights.The architecture uses a RoBERTa-large encoder over concatenated source and candidate text, followed by a shared representation and mixture-of-experts layers; it has 370.09 million trainable parameters, a 4.14% increase over RoBERTa-large.Source: §3.2 · pp. 3–4, Figure 1 · p. 3

3.3 Tackling Training and Inference Gap

  • To address distribution bias, the training set is split in half and each fine-tuned base model generates candidates for the half it did not train on.At test time, the reranker can be paired with a half-data base model or transferred to a full-data base model; the latter still considerably improves base-model performance despite a possible quality mismatch.Source: §3.3 · p. 4
  • For lower-resource inference, the paper separates training and test decoding-method sets and requires the test methods to be a subset of the training methods.Multiple decoding methods can produce a large candidate set and slow inference.Source: §3.3 · p. 4

4 Experiments

SummaReranker improves summarization results across CNN/DM, XSum, and Reddit TIFU, while also improving candidate recall and producing summaries that human raters more often judge as faithful. The experiments test different base models, decoding methods, scoring metrics, training setups, and evaluation criteria.

4.1 Scope & Datasets

  • Experiments use PEGASUS and BART, four decoding methods, and five metrics across CNN/DM, XSum, and Reddit TIFU.The metrics are R-1, R-2, R-L, BERTScore, and BARTScore; R-1 and R-L correlate strongly (Pearson 0.977), while BARTScore is least correlated with the others. XSum is more abstractive than CNN/DM, and Reddit TIFU uses a random 80:10:10 split.Source: §4.1 · pp. 4–5

4.2 Training & Inference Details

  • Training samples the top and bottom candidates by summed normalized evaluation scores, while inference selects the candidate with the highest sum of predicted metric probabilities.With mtop = 1 and mbottom = 1, training sees two candidates sampled from 60 candidates generated across four decoding methods; inference scores each candidate.Source: §4.2 · p. 5

4.3 Base Setup Results

  • 4.57%-7.21% with 15 candidates and 8.70%-9.36% with 30 candidates improve PEGASUS; BART improves by 3.94%-11.65% and 7.98%, respectively.With SummaReranker, models trained on 50% of CNN/DM surpass counterparts trained on the whole set, achieving 46.19 R-1 with PEGASUS and 45.96 R-1 with BART, better than GSum.Source: §4.3 · p. 6

4.4 Transfer Setup Results

  • 47.16 R-1, 22.55 R-2, and 43.87 R-L establish a new CNN/DM state of the art for PEGASUS + SummaReranker in the transfer setup.These correspond to gains of 2.60/1.65/2.29 R-1/2/L, or +5.44% from the diverse beam search baseline.Source: §4.4 · p. 6, Table 5 · p. 6
  • 1.31% improves PEGASUS on XSum to 48.12/24.95/40.00 R-1/2/L; 9.34% improves PEGASUS on Reddit TIFU to 29.83/9.50/23.47 R-1/2/L.On Reddit TIFU, BART improves by 4.22%, reaching 28.99/9.82/22.96 R-1/2/L; beam and diverse beam candidates are consistently effective across datasets.Source: §4.4 · p. 7, Table 6 · p. 7

4.5 Ranking Evaluation

  • +14.90 absolute recall@5 on CNN/DM, +9.54 on XSum, and +5.23 on Reddit TIFU show that SummaReranker significantly improves recall at all thresholds.The base PEGASUS ranking is not significantly above random on CNN/DM and Reddit TIFU, but does improve over random on XSum.Source: Figure 3 · p. 7, §4.5 · p. 8

4.6 Qualitative Evaluation

  • SummaReranker’s example summary includes a whole sentence missed by the base summary.The paper also refers to full re-ranking demonstrations on all three datasets in Appendix K.Source: §4.6 · p. 8, Figure 4 · p. 8
  • Humans were more likely on average to choose SummaReranker’s candidate as more faithful than the PEGASUS beam-search summary.Three raters evaluated 50 randomly sampled test summaries per dataset and could choose a tie when summaries were similar or identical.Source: §4.6 · p. 8, Figure 5 · p. 8

5 Discussion

SummaReranker selects more abstractive candidates, while its inference cost and gains depend on how many candidates are scored. Its current context-window limit constrains long-document use, and more complex loss balancing remains future work.

Abstractiveness

  • SummaReranker selects more abstractive candidates across datasets, with the clearest gains over PEGASUS decoding on CNN/DM.The gains appear across all novel n-gram metrics and are stronger with diverse beam search; on XSum, near-ground-truth abstractiveness makes gains harder to obtain, except for 1-grams.Source: §5 · pp. 8–9, Figure 6 · p. 9

Speed/Performance trade-off

  • 38ms per candidate is the average scoring time, and six to eight candidates generally offer a good speed–performance trade-off.Re-ranking two candidates suffices on CNN/DM; XSum needs three to eight and Reddit TIFU three to four. Combining all four decoding methods does not outperform using only beam search and diverse beam search, and inference cost grows linearly with candidate count.Source: §5 · p. 9

Further Work

  • The 512-token RoBERTa-large encoder context window limits SummaReranker on long documents, and more complex loss balancing is left for future work.Applying it to scientific-article summarization would require better long-range modeling; future work could explore complex weight balancing or multi-task objectives (Cohan et al., 2018; Lin et al., 2019).Source: §5 · p. 9

6 Conclusion · A Hyper Parameters & Packages · B Oracle Scores

SummaReranker is a multi-task re-ranking framework for abstractive summarization that selects candidates across evaluation metrics. The conclusion reports improved ROUGE state-of-the-art on CNN/DM and XSum, while the appendices document evaluation packages, hyperparameter tables, and oracle-score trends.

6 Conclusion

  • SummaReranker is presented as the first multi-task re-ranking framework for abstractive summarization, predicting whether a candidate maximizes each optimized metric.It encodes the source with the candidate.Source: §6 · p. 9
  • The framework works across datasets, models, decoding methods, and evaluation metrics, and its selected summaries improve ROUGE state-of-the-art on CNN/DM and XSum.Source: §6 · p. 9
  • Selected summaries are more abstractive and more likely to be preferred by human evaluators than base-model outputs.Source: §6 · p. 9

A Hyper Parameters & Packages

B Oracle Scores

  • Oracle scores are widely above the top-beam baseline and increase when several decoding methods are mixed, on XSum and Reddit TIFU.The tables report oracle scores for four decoding methods and five evaluation measures using base PEGASUS.Source: Table 9 · p. 13, Table 10 · p. 13, B Oracle Scores · p. 13

C Unique Candidates Scores · D Identical Candidates Scores

Across candidate pools, embedding-based BERTScore and BARTScore usually assign distinct scores, while identical-score sets are a small minority and are more likely with ROUGE-2. The analyses cover pools of 15 candidates across datasets, base models, and decoding methods.

C Unique Candidates Scores

  • Table 11 counts unique metric scores among pools of 15 candidates across datasets, base models, and decoding methods.A score count of 1 means all candidates share a score; 15 means every candidate has a different score.Source: Table 11 · p. 14
  • BERTScore and BARTScore produce unique-score counts closer to 15, making exact score ties between candidates unlikely.Both metrics use pre-trained language model embeddings, which vary when input summaries differ.Source: C Unique Candidates Scores · p. 14, Table 11 · p. 14
  • ROUGE can assign the same score to different summaries when their differences involve n-grams absent from the target.Unlike embedding-based metrics, ROUGE measures n-gram overlap.Source: C Unique Candidates Scores · p. 14

D Identical Candidates Scores

  • Table 12 reports the fraction of 15-candidate sets with identical scores across datasets, base models, and decoding methods.The caption identifies the scope but does not provide the fraction values in the supplied passage.Source: Table 12 · p. 14
  • Sets in which all candidate scores are identical constitute a small minority.The supplied passage does not give a numerical fraction.Source: D Identical Candidates Scores · p. 14, Table 12 · p. 14
  • ROUGE-2 is more likely than other metrics to produce sets in which all scores are identical.This describes relative likelihood, not the proportion of such sets.Source: D Identical Candidates Scores · p. 14

E Metrics Correlation

The section reports Pearson correlations among five evaluation metrics for beam-decoded base PEGASUS on XSum and Reddit TIFU, noting that both datasets follow the Table 2 pattern.

E Metrics Correlation

F Base Setup Results · G Recall Curves

The sections report SummaReranker base-setup results for PEGASUS and BART on XSum and Reddit TIFU, and provide the values underlying the recall curves in Fig. 3.

F Base Setup Results

  • Table 15 reports SummaReranker base-setup results for PEGASUS and BART on XSum.Its caption identifies beam search and diverse beam search as the two decoding methods shown.Source: Table 15 · p. 15
  • For Table 15, Gain is the mean relative gain over R-1, R-2, and R-L compared with the best decoding method.Source: Table 15 · p. 15
  • Table 15 defines decoding method {1} as beam search and {2} as diverse beam search.Source: Table 15 · p. 15
  • Table 16 reports SummaReranker base-setup results for PEGASUS and BART on Reddit TIFU.Source: Table 16 · p. 15
  • Tables 15 and 16 complement the base-setup results in Table 4.Source: F Base Setup Results · p. 15

G Recall Curves

  • Table 17 gives the values of the recall curves plotted in Fig. 3.Source: Table 17 · p. 16

H Human Evaluation · I Candidate Selection

Human-evaluation numbers are referenced in Figure 5, while candidate selection compares reranked summaries with base and oracle-best candidates and reports greater adherence to the base candidate under diverse beam search.

H Human Evaluation

  • Table 18 presents the numbers for the human evaluation shown in Figure 5.Source: Table 18 · p. 16

I Candidate Selection

J Speed/Performance Trade-off · K Re-ranking Examples

The speed/performance analysis identifies a failure case for beam-search-trained reranking on XSum, which is avoided by mixing beam-search and diverse-beam-search candidates; the examples section presents reranking scores for candidate summaries across three datasets.

J Speed/Performance Trade-off

  • Figure 7 plots CNN/DM ROUGE-1 for k sampled candidates at inference time, with k ranging from 1 to 15.The caption identifies SR as SummaReranker, BS as beam search, and DBS as diverse beam search.Source: Figure 7 · p. 17
  • On XSum, PEGASUS reranking trained with beam-search candidates decreases performance, but training on a mixture of beam-search and diverse-beam-search candidates removes the problem.Figure 8 reports ROUGE-1 for k sampled candidates at inference time, with k from 1 to 15.Source: J Speed/Performance Trade-off · p. 17, Figure 8 · p. 18
  • In a rare XSum case, reranking a single candidate beats the top-beam baseline, indicating that the baseline is below random sampling on average.The passage says the top-beam baseline is usually better than the random baseline.Source: J Speed/Performance Trade-off · p. 17, Figure 9 · p. 18
  • When the top-beam baseline is lower than random sampling, retaining all candidates and using a second-stage method to identify a better one is especially important.The passage says such cases are rare.Source: J Speed/Performance Trade-off · p. 17
  • Figure 9 plots Reddit TIFU ROUGE-1 for k sampled candidates at inference time, with k ranging from 1 to 15.The caption identifies SR as SummaReranker, BS as beam search, and DBS as diverse beam search.Source: Figure 9 · p. 18

K Re-ranking Examples

  • The CNN/DM re-ranking example table presents diverse-beam-search PEGASUS candidates alongside ground truth and SummaReranker re-ranking scores.Source: Table 20 · p. 19
  • The XSum and Reddit TIFU example tables present beam-search PEGASUS candidates alongside ground truth and SummaReranker re-ranking scores.Source: Table 21 · p. 20, Table 22 · p. 21

Related papers

Loading 2203.06569v2…