Source-linked AI summary
Text Summarization with Pretrained Encoders
Yang Liu, Mirella Lapata
TL;DR
The paper addresses how pretrained BERT can be adapted to summarization, which requires document-level understanding beyond individual sentences and may also require generation. It proposes a shared BERT-based framework with specialized extractive and abstractive training, achieving state-of-the-art results across extractive and abstractive settings.
Problem
Text summarization requires wide-coverage document understanding, while abstractive summarization additionally requires generating a shorter version that preserves most of the document’s meaning.
Method
The paper introduces a BERT-based document encoder for sentence representations, inter-sentence Transformer layers for extraction, separate encoder-decoder optimizers, and two-stage extractive-then-abstractive fine-tuning.
Results
The proposed models achieve state-of-the-art results across three datasets under both extractive and abstractive settings, with strong preferences in human evaluations.
Takeaways & Limitations
BERT can be used within a general framework spanning both extractive and abstractive summarization.
Takeaways & Limitations
The encoder learning-rate schedule is based on the assumption that the pretrained encoder should use a smaller learning rate and smoother decay than the decoder.
Abstract
from arXiv · showhide
Bidirectional Encoder Representations from Transformers (BERT) represents the latest incarnation of pretrained language models which have recently advanced a wide range of natural language processing tasks. In this paper, we showcase how BERT can be usefully applied in text summarization and propose a general framework for both extractive and abstractive models. We introduce a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers. For abstractive summarization, we propose a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two (the former is pretrained while the latter is not). We also demonstrate that a two-staged fine-tuning approach can further boost the quality of the generated summaries. Experiments on three datasets show that our model achieves state-of-the-art results across the board in both extractive and abstractive settings. Our code is available at https://github.com/nlpyang/PreSumm
1 Introduction
The paper investigates how pretrained BERT can support text summarization, which requires document-wide understanding and, for abstractive systems, language generation. It introduces a unified extractive and abstractive framework and reports state-of-the-art results across three datasets.
- Summarization must condense documents while preserving meaning, requiring broader understanding than sentence-level language-understanding tasks.
- The proposed framework applies BERT to both extractive and abstractive summarization through a document-level encoder that represents documents and their sentences.
- The extractive model stacks inter-sentence Transformer layers on the BERT-based encoder to capture document-level features for sentence selection.
- The abstractive model uses separate encoder and decoder optimizers to accommodate a pretrained encoder and a decoder trained from scratch.
- Across three news summarization datasets, the models achieve state-of-the-art results in both extractive and abstractive settings.
2 Background
Pretrained language models learn contextual representations from large corpora and are fine-tuned with task-specific parameters. Text summarization is framed as either sentence selection or sequence-to-sequence generation, while BERT provides contextual token representations that can be adapted to document and sentence encoding.
- Pretrained language models: Pretrained language models learn contextual representations from large-scale corpora through language-modeling objectives, with BERT additionally trained on masked language modeling and next-sentence prediction.BERT was trained on a corpus containing 3,300M words.
- Pretrained language models: BERT prepends [CLS] and inserts [SEP] tokens so its Transformer layers can aggregate sequence information and mark sentence boundaries.The top layer produces a contextual output vector for each token.
- Extractive summarization: Extractive summarization selects important sentences, commonly treating the task as sentence classification over neural sentence representations.Prior systems use recurrent encoders, reinforcement learning, latent-variable inference, structured attention, or joint sentence scoring.
- BERTSUM architecture: BERTSUM extends BERT for summarization by inserting multiple [CLS] symbols to learn sentence representations and interval segmentation embeddings.Figure 1 contrasts the original BERT architecture with this document-level adaptation.
- Abstractive summarization: Abstractive summarization uses an autoregressive encoder-decoder model in which source representations support token-by-token generation of the target summary.Pointer-generator and coverage mechanisms extend this framework by copying source words and tracking summarized content.
3 Fine-tuning BERT for Summarization
This section adapts BERT to summarization by producing document-level sentence representations for extractive modeling and pairing a pretrained encoder with a randomly initialized decoder for abstractive modeling. It further addresses encoder–decoder mismatch through separate optimizers and uses two-stage fine-tuning from extractive to abstractive summarization.
- 3.1 Summarization Encoder: BERTSUM addresses BERT’s token-grounded outputs and limited sentence-pair segmentation embeddings by encoding multi-sentential documents with sentence-level representations.External [CLS] tokens collect features for each sentence, while interval segment embeddings distinguish sentences and added position embeddings extend the 512-position limit.
- 3.2 Extractive Summarization: Extractive summarization assigns each document sentence a binary inclusion label indicating whether it should appear in the summary.The formulation assumes selected sentences represent the document’s most important content and uses a sigmoid classifier with binary classification entropy.
- 3.2 Extractive Summarization: BERTSUMEXT represents each sentence with its top-layer [CLS] vector and stacks inter-sentence Transformer layers to capture document-level extraction features.The model uses sinusoidal sentence-position embeddings before the inter-sentence layers; two layers performed best in experiments.
- 3.3 Abstractive Summarization: The abstractive model combines a pretrained BERTSUM encoder with a randomly initialized six-layer Transformer decoder, whose mismatch can make fine-tuning unstable.Separate Adam optimizers use different warmup schedules and learning rates for the encoder and decoder; the encoder uses a smaller learning rate and smoother decay.
- 3.3 Abstractive Summarization: The two-stage BERTSUMEXTABS model fine-tunes the encoder first on extractive summarization and then on abstractive summarization.The approach is intended to exploit information shared between the two tasks without fundamentally changing the architecture.
4 Experimental Setup
The experiments evaluate BERTSUM on three news summarization datasets spanning different document conventions, summary styles, and degrees of abstractiveness. Models use BERT-based implementations with specified extractive and abstractive training and decoding procedures, and CNN/DailyMail results are reported with ROUGE F1 comparisons.
- Datasets: The evaluation uses CNN/DailyMail, NYT, and XSum, whose summaries range from highlights to brief one-sentence summaries and from cut-and-paste to genuinely abstractive rewriting.CNN/DailyMail and NYT are somewhat extractive, whereas XSum is highly abstractive.
- Datasets: CNN/DailyMail uses standard training, validation, and test splits, while NYT is split by publication date with articles from January 1, 2007 onward in the test set.CNN/DailyMail entities were not anonymized, and input documents were truncated to 512 tokens.
- Extractive Summarization: Extractive models are trained for 50,000 steps, select top-3 scored sentences, and apply trigram blocking to reduce redundancy.Checkpoints are evaluated every 1,000 steps, and the top three validation-loss checkpoints are averaged for test reporting.
- Abstractive Summarization: Abstractive models are trained for 200,000 steps and decoded with beam search of size 5, length-penalty tuning, end-of-sequence termination, and repeated-trigram blocking.The decoder uses neither copy nor coverage mechanisms, while subword tokenization helps reduce out-of-vocabulary issues.
- Evaluation: CNN/DailyMail comparisons report ROUGE F1 for unigram overlap, bigram overlap, and longest common subsequence.The table includes comparison systems and the dataset’s extractive ORACLE upper bound.
5 Results
Across CNN/DailyMail, NYT, and XSum, BERT-based summarizers achieve strong results in both extractive and abstractive settings, with analyses examining learning rates, sentence positions, and novel n-grams. Human evaluations further compare informativeness, fluency, and succinctness across systems.
- Automatic Evaluation: ROUGE evaluates informativeness with unigram and bigram overlap and fluency with longest common subsequence.
- Automatic Evaluation: BERT-based models outperform previously proposed extractive and abstractive systems on CNN/DailyMail, with BERTSUMEXT performing best among BERT variants.They remain behind the extractive ORACLE upper bound.
- Automatic Evaluation: BERT-based models outperform previous approaches on NYT, where abstractive BERT models generally perform better than BERTSUMEXT and nearly approach ORACLE performance.
- Automatic Evaluation: On XSum, extractive models perform poorly because summaries are highly abstractive, while the proposed BERT abstractive summarizers outperform previously reported models by a wide margin.XSum summaries consist of a single sentence conveying the document gist.
- Analysis: Model perplexity is best with separate encoder and decoder learning rates of lr_E = 2e-3 and lr_D = 0.1.
- Analysis: BERTSUMEXT selects sentences in positions more like Oracle summaries than TransformerEXT, suggesting less reliance on shallow position features.
- Analysis: Generated summaries contain fewer novel n-grams than references on CNN/DailyMail, whereas this gap is much smaller on XSum; BERTEXTABS is less novel than BERTABS.
- Human Evaluation: Participants overwhelmingly prefer BERTSUM outputs over comparison systems across datasets and evaluation paradigms, with nearly all differences statistically significant.The exception is TCONVS2S on XSum in the QA evaluation setting.
6 Conclusions
The paper applies pretrained BERT to summarization through a document-level encoder and a general extractive–abstractive framework, achieving state-of-the-art results across three datasets under automatic and human-based evaluation.
- BERT is applied to text summarization through a novel document-level encoder within a general framework for extractive and abstractive models.
- The proposed model achieves state-of-the-art results across three datasets under both automatic and human-based evaluation protocols.