PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

1 Introduction

• Although summarization datasets increasingly cover abstractive summaries, longer and multiple documents, and diverse domains, systematic evaluation across these settings remains limited.The work addresses this gap by evaluating models across 12 downstream datasets.Source: §1 · pp. 1–2
• Gap Sentences Generation masks important whole sentences and generates them from the remaining document, with important-sentence selection outperforming lead or random selection.The objective is designed to resemble abstractive summarization by encouraging whole-document understanding and summary-like generation.Source: §1 · p. 2
• With only 1000 supervised examples, PEGASUS surpasses previous state-of-the-art results on 6 datasets.This demonstrates rapid adaptation across broad-domain summarization tasks with little supervision.Source: §1 · p. 2
• Human evaluations find PEGASUS summaries at least as good as reference summaries on XSum, CNN/DailyMail, and Reddit TIFU, including under low supervision.These studies validate the model’s qualitative output quality and experimental findings.Source: §1 · p. 2
• PEGASUS surpasses or matches state of the art on all 12 downstream summarization datasets considered.The evaluation spans news, science, short stories, instructions, emails, patents, and legislative bills.Source: §1 · p. 2

Transformer pre-training

• Large external corpora paired with Transformer sequence models substantially improved fine-tuned natural language understanding and text generation over earlier in-domain pre-training for RNNs.Earlier work used language-model and autoencoder pre-training on in-domain data, whereas later approaches used corpora such as Wikipedia, books, and Web pages.Source: §2 · p. 2

Prior objectives

• MASS reconstructed a randomly selected sentence fragment, while UniLM jointly trained unidirectional, bidirectional, and sequence-to-sequence language-modeling tasks.These objectives provide closely related masked or multi-task pre-training approaches for sequence models.Source: §2 · pp. 2–3

Span-corruption objectives

• T5 scaled text-to-text pre-training to 11 billion parameters using C4 and randomly corrupted spans, while BART reconstructed text corrupted by arbitrary noising or text infilling.PEGASUS also uses C4 in some models but replaces random span corruption with importance-based sentence masking.Source: §2 · p. 3

Objective distinction

• PEGASUS masks multiple whole sentences, deterministically selects them by importance, and generates only the masked sentences as one sequence for downstream summarization.Unlike MASS, UniLM, BART, and T5, the objective targets summarization and is not evaluated on NLU classification tasks.Source: §2 · p. 3

Low-resource summarization

• ROUGE-2 reached 8.27 with prompted Web-text pre-training and 13.1 with Wikipedia pre-training fine-tuned on 3000 examples in prior low-resource CNN/DailyMail work.These studies demonstrated that large pretrained Transformer language models can generate summaries with limited supervision.Source: §2 · p. 3

3 Pre-training Objectives

• The study evaluates GSG alongside BERT’s masked-language-model objective, both in isolation and in conjunction with GSG.Source: §3 · p. 3

3.1 Gap Sentences Generation (GSG)

• GSG masks important whole sentences and trains the model to generate them together as a pseudo-summary, better matching abstractive summarization than copying sentences.Selected sentences are replaced by [MASK1] tokens, while the concatenated gap-sentences form the target sequence. Importance is approximated with ROUGE1-F1, using random, lead, or principal selection strategies with independent/sequential and original/unique variants.Source: §3.1 · p. 3

3.2 Masked Language Model (MLM)

• MLM did not improve downstream tasks at large numbers of pre-training steps, so it was excluded from the final PEGASUSLARGE model.MLM was evaluated as the sole encoder objective or alongside GSG, following BERT’s 15% token corruption scheme.Source: §3.2 · p. 4

4 Pre-training Corpus

• C4 contains text from 350M web pages totaling 750GB.C4 is the Colossal and Cleaned version of Common Crawl, introduced in Raffel et al. (2019).Source: §4 · p. 4
• HugeNews contains 1.5B articles totaling 3.8TB, collected from news and news-like websites between 2013 and 2019.Its construction used a curated domain whitelist to seed a crawler, heuristics to identify news-like articles, and extraction of only main article text as plain text.Source: §4 · p. 4

• The datasets were accessed through TensorFlow Summarization Datasets with public processing code; unspecified splits used 80/10/10 train/validation/test, or 10% training data for validation when needed.The study used only public abstractive summarization datasets.Source: §5 · p. 4
• News datasets covered XSum’s 227k BBC articles with single-sentence summaries, CNN/DailyMail’s 313k articles with bullet summaries, NEWSROOM’s 1.3M article-summary pairs, and Multi-News’s 56k multi-document pairs.XSum spans 2010–2017; CNN/DailyMail uses the non-anonymized variant from See et al. (2017); NEWSROOM covers 38 publications; Multi-News uses Newser.com articles.Source: §5 · p. 4
• Additional news tasks included Gigaword’s 4M examples for headline generation, while arXiv and PubMed use 113k and 215k scientific papers to generate abstracts from paper bodies.Gigaword examples come from seven publishers; arXiv and PubMed are long-document datasets.Source: §5 · p. 4
• BIGPATENT contains 1.3 million U.S. patents with human summaries across nine classification categories, and WikiHow contains 200k instructional examples requiring concatenated summary sentences.WikiHow examples comprise multiple instruction-step paragraphs paired with a summarizing sentence.Source: §5 · p. 4
• Reddit TIFU provides 120K informal-story posts; the study uses its TIFU-long subset with TL;DR summaries, judged higher quality than the broader V¨olske et al. (2017) collection by manual inspection.The posts come from the TIFU subreddit between 2013-Jan and 2018-Mar and follow a descriptive TL;DR rule.Source: §5 · pp. 4–5
• AESLC contains 18k Enron email bodies and subjects, while BillSum contains 23k U.S. Congressional bills with summaries from 1993–2018; California’s out-of-distribution test set was excluded.BillSum covers the 103rd–115th Congressional sessions.Source: §5 · p. 5

6.1.1 PRE-TRAINING CORPUS

• HugeNews transferred better to the two news datasets, whereas C4 transferred better to WikiHow and Reddit TIFU, indicating that domain alignment improves downstream transfer.The corpus comparison used C4’s 350M web pages and HugeNews’s 1.5B news-like documents.Source: §6.1.1 · p. 5, Figure 3 · p. 5

6.1.2 EFFECT OF PRE-TRAINING OBJECTIVES

• Ind-Orig was the best gap-sentence objective, while MLM alone performed significantly worse and adding MLM inhibited later gains; PEGASUSLARGE therefore used Ind-Orig without MLM.Ind-Orig and Seq-Uniq were consistently better or similar to Random and Lead across four datasets, and MLM helped at 100k–200k steps but inhibited gains by 500k steps.Source: §6.1.2 · pp. 5–6, Figure 4 · p. 6
• The optimal gap-sentence ratio stayed below 50% but varied by dataset: 15% for CNN/DailyMail, 30% for XSum and Reddit TIFU, and 45% for WikiHow; 30% was used when scaling up.Higher ratios lose contextual information, whereas lower ratios make pre-training less challenging and more computationally efficient.Source: §6.1.2 · pp. 5–6

6.1.3 EFFECT OF VOCABULARY

• Unigram and BPE were comparable on news datasets, while Unigram outperformed BPE on the non-news datasets in the vocabulary study.Unigram vocabulary sizes ranged from 32k to 256k; the scaled model adopted a 96k Unigram vocabulary.Source: §6.1.3 · p. 6, Figure 5 · p. 6, §6.2 · p. 6

6.2 Larger Model Results

• PEGASUSLARGE achieved better-than-state-of-the-art results on all downstream datasets, while C4 outperformed HugeNews on WikiHow.Compared with the non-pretrained TransformerBASE, ROUGE2-F1 nearly tripled on AESLC and quintupled on Reddit TIFU, with larger gains on smaller datasets.Source: §6.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7

6.3 Zero and Low-Resource Summarization

• With only 1,000 examples, PEGASUSLARGE surpassed previous state-of-the-art results on 6 of 12 datasets and matched fully supervised TransformerBASE quality using far fewer examples.On CNN/DailyMail, zero-shot ROUGE2-F was 13.28 versus GPT-2’s 8.27, and 1,000 examples produced 19.35 versus 13.1 from Khandelwal et al. (2019) with 3,000 examples.Source: §6.3 · pp. 7–8, Figure 6 · p. 8

6.4 Qualitative Observations and Human Evaluation

• Human evaluations found PEGASUSLARGE outputs at least as good as reference summaries across tested cases, although Reddit TIFU required full supervision to reach human performance.Low-supervision outputs were not measurably worse than human summaries on XSum and CNN/DailyMail, and qualitative assessment found fluent, coherent summaries with rare repetition.Source: §6.4 · p. 8, Table 3 · p. 8

6.6 Additional PEGASUSLARGE Improvements

• The mixed, stochastic PEGASUSLARGE variant achieved the best results on almost all downstream tasks after combining corpora, stochastic gap selection, longer training, and tokenizer updates.It used C4 and HugeNews, sampled gap ratios from 15%–45%, added noise to sentence importance, trained for 1.5M steps, and encoded newline characters.Source: §6.6 · p. 9, Table 4 · p. 9

7 Conclusion

• PEGASUS achieved state-of-the-art results on all 12 diverse downstream datasets considered.The best configuration was selected after studying pretraining corpora, gap-sentence ratios, and vocabulary sizes, then scaling it up.Source: §7 · p. 9
• Principle sentence selection was identified as the optimal gap-sentence selection strategy.The study compared several methods for selecting gap sentences within the proposed sequence-to-sequence pretraining objective.Source: §7 · p. 9
• PEGASUS adapted quickly to unseen summarization datasets, achieving strong results with as few as 1000 examples.Source: §7 · p. 9
• Human evaluation showed that PEGASUS summaries achieved human performance on multiple datasets.Source: §7 · p. 9

A Datasets Statistics

• Extractive fragment coverage and density are calculated for all downstream datasets following Grusky et al.The analysis defines these measures using articles, summaries, and extractive fragments.Source: A Datasets Statistics · p. 13
• Higher density indicates a more extractive dataset, whereas lower coverage suggests that summaries contain more novel words.Source: A Datasets Statistics · p. 13
• Figure A.1 compares downstream datasets by extractive fragment coverage and density, with darker blocks denoting higher percentages and n denoting dataset size.Source: Figure A.1 · p. 13

B Pre-training Steps

• Performance increases on downstream datasets as PEGASUSBASE trains for more steps on C4.The result is shown in Figure B.1.Source: Figure B.1 · p. 13
• Figure B.1 evaluates how additional pre-training steps affect PEGASUSBASE performance on downstream datasets.The figure specifically concerns training on C4.Source: Figure B.1 · p. 13

C PEGASUS Hyper Parameters

• The pre-training and fine-tuning stages use reported hyperparameters summarized in Table C.1.The table reports the settings referenced in section 6.Source: Table C.1 · p. 14
• Fine-tuning PEGASUSLARGE uses hyperparameters selected through grid search.The table distinguishes this selection procedure from the approach used for the other settings.Source: Table C.1 · p. 14
• Other hyperparameters were selected using empirically default commonly used values.This applies to settings other than fine-tuning PEGASUSLARGE.Source: Table C.1 · p. 14
• Table C.1 defines maximum input and target token counts as Linput and Ltarget in Section 6.These limits are reported for the pre-training and fine-tuning stages.Source: Table C.1 · p. 14

D Experiment Figures’ Numbers

• Table D.1 reports the raw ROUGE1-F1, ROUGE2-F1, and ROUGEL-F1 scores from the corresponding experiment figures.Source: Table D.1 · p. 15

E Low Resource Numbers

• With 100 examples, PEGASUSLARGE beats previous state of the art on ROUGE2-F1 for BIGPATENT, Reddit TIFU, and BillSum.Source: Table E.1 · p. 16
• With 1000 examples, PEGASUSLARGE beats previous state-of-the-art metrics on Multi-News, WikiHow, Reddit TIFU, BigPatent, AESLC, and BillSum.Source: Table E.1 · p. 16

F Human Evaluation Details

• Human evaluations rated four summaries per document on a 1–5 scale, using three independent workers, randomized order, median scores, and $1 payment per task.Workers had to meet a 95% minimum approval rate.Source: F Human Evaluation Details · p. 17
• The evaluations compared three fully supervised models on 100 examples each from XSum, CNN/DailyMail, and Reddit TIFU, and tested PEGASUS with 10, 100, 1000, or all examples.Experiment 2 randomly selected four of five summaries per example to preserve the common template without increasing task cost.Source: F Human Evaluation Details · p. 17
• Paired t-tests assessed significance between summary-rating sets, while repeated runs with qualifying workers produced highly similar conclusions.The HITT evaluation template was provided publicly.Source: F Human Evaluation Details · p. 17

G Example of summary with relatively low ROUGE2-F but qualitatively good.

• ROUGE2-F of 16 accompanied a fluent, coherent, highly abstractive, and factually accurate CNN/DailyMail summary that paraphrased Chelsea references while preserving four matches.The example linked Chelsea with “Jose Mourinho’s side” and “The Blues” and highlighted the same four matches.Source: Figure G.1 · p. 18, G Example of summary with relatively… · p. 18

H Abstractiveness of Summaries

• Model summaries were generally less abstractive than human summaries, although models fine-tuned on XSum and Reddit TIFU generated more abstractive outputs than humans on other datasets.Abstractiveness was measured using average extractive coverage and extractive density.Source: H Abstractiveness of Summaries · p. 19, Figure H.1 · p. 19

I Example Model Outputs

• Example outputs were automatically selected from the first 300 test examples, ranked by ROUGE1-F1, ROUGE2-F1, or ROUGEL-F1, bucketed by document length, and sampled once per bucket.Documents were truncated at 300 words for visualization, and examples containing bad words were filtered out.Source: I Example Model Outputs · p. 19
• The appendix presents PEGASUSLARGE (HugeNews) examples across datasets including XSum, CNN/DailyMail, NEWSROOM, Multi-News, Gigaword, WikiHow, Reddit TIFU, BIGPATENT, arXiv, PubMed, AESLC, and BillSum.Examples are organized by ROUGE1-F1, ROUGE2-F1, and ROUGEL-F1 samples.Source: Table I.1 · p. 20, Table I.4 · p. 23, Table I.7 · p. 26