PaperlayerView exact revision on arXiv

Source-linked AI summary

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

Jingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. Liu

arXiv:1912.08777v3cs.CL

TL;DR

Abstractive summarization lacked task-specific pre-training objectives and systematic evaluation across broad settings. PEGASUS masks important whole sentences and generates them as a pseudo-summary, achieving state-of-the-art results on all 12 evaluated datasets.

  • Problem

    Abstractive summarization lacked task-specific pre-training objectives and systematic evaluation across broad settings.

  • Method

    PEGASUS masks selected whole sentences and generates the resulting gap-sentences as a pseudo-summary from the remaining document.

  • Results

    PEGASUS achieved state-of-the-art or on-par performance on all 12 evaluated summarization datasets and strong results with as little as 1000 examples.

  • Takeaways & Limitations

    The proposed objective supports strong abstractive summarization across diverse domains and rapid adaptation to unseen datasets with limited supervision.

  • Takeaways & Limitations

    The pre-training corpora may overlap with downstream test sets, raising a possible memorization concern for evaluation results.

Abstract

from arXiv · show

Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been explored. Furthermore there is a lack of systematic evaluation across diverse domains. In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective. In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We evaluated our best PEGASUS model on 12 downstream summarization tasks spanning news, science, stories, instructions, emails, patents, and legislative bills. Experiments demonstrate it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores. Our model also shows surprising performance on low-resource summarization, surpassing previous state-of-the-art results on 6 datasets with only 1000 examples. Finally we validated our results using human evaluation and show that our model summaries achieve human performance on multiple datasets.

1 Introduction

PEGASUS introduces a self-supervised pre-training objective tailored to abstractive summarization and evaluates it systematically across 12 diverse downstream datasets. Its 568M-parameter model reaches state-of-the-art or comparable performance, including in low-resource settings, with human-level quality on several datasets.

  • Evaluation: The study evaluates the objective across 12 summarization datasets spanning news, science, short stories, instructions, emails, patents, and legislative bills.The introduction identifies systematic evaluation across broad settings as an unmet need.
  • Low-resource summarization: 6 datasets achieve state-of-the-art results with only 1000 supervised examples, demonstrating strong low-resource adaptation across broad domains.The setting addresses the scarcity and expense of large-scale document-summary datasets.
  • Human evaluation: PEGASUS summaries are at least as good as reference summaries on XSum, CNN/DailyMail, and Reddit TIFU in human evaluations, even with low supervision.The human studies also validate the experimental design and demonstrate human-level summarization performance.
  • Contributions: PEGASUS proposes gap-sentences generation, masking whole important sentences and generating them as one output sequence, while studying sentence-selection strategies.The objective is designed specifically for abstractive summarization.
  • Results: A 568M-parameter PEGASUS model surpasses or matches state-of-the-art results on all 12 considered downstream datasets.The best model is selected using careful ablations and is trained on the C4 corpus; HugeNews further advances results on competitive XSum and CNN/DailyMail tasks.

2 Related Work

Prior work established that large-scale pre-training with external corpora and Transformer-based models improves downstream language understanding and generation. PEGASUS differs from existing objectives by masking important whole sentences and generating them jointly, focusing exclusively on abstractive summarization.

  • General Pre-training: Large external-corpus pre-training with Transformer sequence models substantially improved fine-tuned natural language understanding and text generation.Earlier approaches used in-domain language-model and autoencoder pre-training for RNN sequence models.
  • Related Objectives: MASS reconstructs a randomly selected sentence fragment from the remaining portion of that sentence.Its masked sequence-to-sequence generation objective selects a single fragment at random.
  • Related Objectives: UniLM jointly trains unidirectional, bidirectional masked-language-modeling, and sequence-to-sequence prediction tasks.The bidirectional objectives include word-level masking with next-sentence prediction.
  • Related Objectives: T5 scales text-to-text pre-training to 11 billion parameters and uses randomly corrupted spans with varying mask ratios and lengths.T5 introduced C4, a massive corpus derived from Common Crawl, which PEGASUS also uses in some models.
  • PEGASUS Objective: PEGASUS masks multiple important whole sentences deterministically, generates them as one sequence without reconstructing the full input, and evaluates only summarization tasks.This contrasts with MASS, UniLM, BART, and T5, which mask smaller or randomly sampled continuous spans.

3 Pre-training Objectives

PEGASUS introduces Gap Sentence Generation (GSG), a sequence-to-sequence objective that masks important document sentences and generates them as a summary-like output. It compares GSG with BERT’s masked language modeling (MLM), which does not improve downstream tasks at large pre-training-step counts and is excluded from PEGASUSLARGE.

  • Masked Language Modeling: MLM is evaluated alone and alongside GSG, but it does not improve downstream tasks at large numbers of pre-training steps and is omitted from PEGASUSLARGE.The MLM objective follows BERT’s 15% token-selection corruption scheme.
  • Gap Sentence Generation: Selecting important sentences makes the pre-training objective more closely resemble abstractive summarization than simply training a model to copy sentences.The objective combines the benefits of masking with anticipation of the downstream task’s form.
  • Gap Sentence Generation: GSG generates summary-like text by masking whole document sentences and concatenating the masked sentences into a pseudo-summary.Each selected sentence position is replaced with [MASK1] to inform the model.
  • Gap Sentence Generation: GSG uses Gap Sentence Ratio (GSR), defined as the number of selected gap sentences divided by the document’s total number of sentences.GSR is analogous to the mask rate used in other pre-training methods.
  • Gap Sentence Selection: The paper evaluates Random, Lead, and Principal strategies for selecting m gap sentences, with Principal ranking sentences by ROUGE1-F1 against the rest of the document.Principal selection can score sentences independently or sequentially, and uses Orig or Uniq n-gram counting variants.

4 Pre-training Corpus

The pre-training corpus study considered two large-scale text collections: C4, containing cleaned Common Crawl text, and HugeNews, a web-crawled corpus of news and news-like articles.

  • Candidate corpora: C4 contains text from 350M web pages, totaling 750GB.C4 is the Colossal and Cleaned version of Common Crawl.
  • Candidate corpora: HugeNews contains 1.5B articles totaling 3.8TB, collected from news and news-like websites between 2013 and 2019.Its sources ranged from high-quality news publishers to high-school newspapers and blogs.
  • Candidate corpora: HugeNews was built by seeding a web crawler with a curated domain whitelist, identifying news-like articles heuristically, and extracting only main article text as plain text.The whitelist covered publishers from high-quality news outlets to lower-quality sites, including high-school newspapers and blogs.

5 Downstream Tasks/Datasets

The study evaluates downstream summarization using publicly reproducible abstractive datasets with standardized split handling. The benchmark spans news, scientific papers, patents, instructions, informal stories, emails, and congressional bills.

  • Dataset preparation: Public abstractive summarization datasets use reproducible TensorFlow processing, with 80/10/10 train/validation/test splits when none are provided.When no validation split exists, 10% of the training data is used for validation.
  • News: News benchmarks include XSum, CNN/DailyMail, NEWSROOM, Multi-News, and Gigaword, ranging from 56k to 4M examples and covering single- and multi-document tasks.XSum uses single-sentence summaries; CNN/DailyMail uses bullet-point summaries; Gigaword generates headlines from first sentences.
  • Scientific and technical documents: Scientific and technical benchmarks include arXiv and PubMed for abstract generation and BIGPATENT, which contains 1.3 million patents with human summaries.The scientific datasets contain 113k arXiv and 215k PubMed publications, while BIGPATENT spans nine patent classification categories.
  • Instructions, stories, and emails: WikiHow, Reddit TIFU, and AESLC cover instructional paragraphs, informal stories with TL;DR summaries, and Enron email bodies paired with subjects.WikiHow contains 200k examples, Reddit TIFU uses the TIFU-long subset, and AESLC contains 18k email-body/subject pairs.
  • Legislative bills: BillSum contains 23k U.S. Congressional bills with human-written summaries from the 103rd–115th sessions, excluding the out-of-distribution California test set.The covered sessions span 1993–2018.

6 Experiments

Experiments identify domain-aligned pre-training, Ind-Orig gap-sentence selection, and a 30% gap-sentence ratio as strong settings, while PEGASUSLARGE achieves state-of-the-art results across downstream and low-resource summarization tasks. Human evaluation finds its outputs comparable to reference summaries, with high fluency and coherence.

  • Experimental setup: A 223M-parameter PEGASUSBASE was used for ablations on four diverse datasets before scaling the best settings to the 568M-parameter PEGASUSLARGE.The four datasets were XSum, CNN/DailyMail, WikiHow, and Reddit TIFU.
  • Pre-training ablations: Domain alignment mattered: HugeNews outperformed C4 on the two news datasets, whereas WikiHow and Reddit TIFU preferred C4 pre-training.This suggests transfer was more effective when pre-training and downstream domains aligned better.
  • Pre-training ablations: Ind-Orig achieved the best GSG performance, followed by Seq-Uniq, and both were consistently better or similar to Random and Lead across four datasets.Lead performed reasonably on news but significantly worse on the two non-news datasets.
  • Pre-training ablations: The best GSR was always below 50%; 15% was best on CNN/DailyMail, 30% on XSum and Reddit TIFU, and 45% on WikiHow, so 30% was selected for PEGASUSLARGE.Different downstream datasets had slightly different optima.
  • Downstream results: PEGASUSLARGE beat previous state-of-the-art results on 6 of 12 datasets with only 1000 fine-tuning examples and achieved ROUGE2-F of 19.35 on CNN/DailyMail.The CNN/DailyMail result exceeded the 13.1 reported by Khandelwal et al. (2019) using 3000 examples.
  • Human evaluation: PEGASUSLARGE outputs were at least as good as reference summaries in all human-evaluation cases at p < 0.01, with high fluency and coherence and rare repetition.The comparisons included HugeNews, C4, TransformerBASE, and fine-tuning with 10, 100, 1000, or all supervised examples.

7 Conclusion

PEGASUS introduces gap-sentence generation as a pretraining objective tailored to abstractive summarization. The study identifies principle sentence selection as optimal and reports state-of-the-art results across 12 diverse downstream datasets.

  • 7 Conclusion: PEGASUS uses gap-sentence generation as a pretraining objective for abstractive text summarization.It is implemented in a sequence-to-sequence model.
  • 7 Conclusion: The study finds principle sentence selection to be the optimal gap-sentence selection strategy.Several gap-sentence selection methods were evaluated.
  • 7 Conclusion: The best scaled-up configuration achieves state-of-the-art results on all 12 diverse downstream datasets.The study also examines pretraining corpora, gap-sentence ratios, and vocabulary sizes.

A Datasets Statistics

The authors characterize all downstream datasets using extractive fragment coverage and density. These measures indicate how extractive summaries are and how much novel wording they contain, with Figure A.1 visualizing the comparison.

  • Dataset characterization: Extractive fragment coverage and density are calculated for all downstream datasets following Grusky et al.The measures use extractive fragments between each article and its summary.
  • Dataset characterization: High density indicates a more extractive dataset, whereas low coverage suggests summaries contain more novel words.Coverage and density therefore describe different aspects of extractiveness and summary novelty.
  • Dataset characterization: Figure A.1 compares downstream datasets by extractive fragment coverage and density using darker blocks for higher percentages.The figure also reports n as the number of examples in each dataset.

B Pre-training Steps · C PEGASUS Hyper Parameters

PEGASUS performance on downstream datasets increases as PEGASUSBASE trains for more steps. Pre-training and fine-tuning use reported hyperparameters, with PEGASUSLARGE fine-tuning selected by grid search and other settings chosen empirically from common defaults.

  • B Pre-training Steps: B Pre-training Steps: PEGASUSBASE performance on downstream datasets increases with additional training steps on C4.The relationship is shown in Figure B.1.
  • B Pre-training Steps: B Pre-training Steps: Figure B.1 evaluates performance changes as PEGASUSBASE trains for more steps.The evaluation concerns downstream datasets and C4 pre-training.
  • C PEGASUS Hyper Parameters: C PEGASUS Hyper Parameters: The paper reports hyperparameters for both pre-training and fine-tuning stages.These hyperparameters are reported in section 6.
  • C PEGASUS Hyper Parameters: C PEGASUS Hyper Parameters: Fine-tuning PEGASUSLARGE hyperparameters were selected through grid search.The grid-search decision applies specifically to PEGASUSLARGE fine-tuning.
  • C PEGASUS Hyper Parameters: C PEGASUS Hyper Parameters: Other hyperparameters were chosen using empirically determined commonly used default values.The passage contrasts these defaults with the PEGASUSLARGE grid-search procedure.
  • C PEGASUS Hyper Parameters: C PEGASUS Hyper Parameters: Maximum input and target token settings correspond to Linput and Ltarget in Section 6.The table explicitly maps these token limits to the notation used in Section 6.

D Experiment Figures’ Numbers · E Low Resource Numbers

Section D provides the raw ROUGE scores underlying the experiment figures, while Section E reports low-resource PEGASUSLARGE comparisons against previous state of the art across several datasets and sample sizes.

  • D Experiment Figures’ Numbers: Table D.1 reports the raw ROUGE1-F1, ROUGE2-F1, and ROUGEL-F1 scores from the corresponding experiment figures.
  • D Experiment Figures’ Numbers: The experiment-figure results include ROUGE1-F1 scores for the evaluated summarization tasks.
  • D Experiment Figures’ Numbers: The experiment-figure results include ROUGE2-F1 scores for the evaluated summarization tasks.
  • D Experiment Figures’ Numbers: The experiment-figure results include ROUGEL-F1 scores for the evaluated summarization tasks.
  • E Low Resource Numbers: With 100 examples, PEGASUSLARGE beats previous SOTA on ROUGE2-F1 for BIGPATENT, Reddit TIFU, and BillSum.
  • E Low Resource Numbers: With 1000 examples, PEGASUSLARGE beats previous SOTA metrics on Multi-News, WikiHow, Reddit TIFU, BigPatent, AESLC, and BillSum.

F Human Evaluation Details … I Example Model Outputs

The paper details a controlled human-evaluation protocol, illustrates a fluent but low-overlap CNN/DailyMail summary, and documents analyses of abstractiveness and automatically selected model outputs.

  • F Human Evaluation Details: Human evaluations rated four summaries per document on a 1–5 scale, using randomized presentation, three independent workers, median aggregation, and 1 USD payment per task.Workers required a minimum approval rate of 95%.
  • F Human Evaluation Details: Multiple runs with different qualifying workers produced very similar human-evaluation conclusions, indicating high reproducibility.Statistical significance between two summary sets was assessed with a paired t-test.
  • F Human Evaluation Details: Experiment 1 compared three fine-tuned models and human references on 100 examples each from XSum, CNN/DailyMail, and Reddit TIFU.The models were PEGASUSLARGE trained on HugeNews or C4 and TransformerBASE.
  • F Human Evaluation Details: Experiment 2 compared PEGASUSLARGE models trained with 10, 100, 1000, or all supervised examples alongside human summaries.Each task randomly selected four of five summaries, reducing ratings per model without increasing task work or cost.
  • G Example of summary with relatively low ROUGE2-F but qualitatively good.: The CNN/DailyMail example achieved ROUGE2-F of 16 despite high fluency and coherence, because it paraphrased Chelsea references while preserving the same four matches.The summary was described as highly abstractive and factually accurate.
  • H Abstractiveness of Summaries: Across downstream datasets, model summaries were less abstractive than human-written summaries according to extractive coverage and extractive density.More abstractive summaries have smaller values for both measures, reflecting more novel words and shorter copied spans.
  • I Example Model Outputs: Example model outputs were automatically sampled from the first 300 test-set triplets, ranked by ROUGE1-F1, ROUGE2-F1, and ROUGEL-F1, bucketed by document length, and randomly selected.Inputs were truncated at 300 words for visualization, with bad-word examples filtered out.
Loading 1912.08777v3…