Source-linked AI summary
Fine-tune BERT for Extractive Summarization
Yang Liu
TL;DR
Extractive summarization must identify important sentences while recent neural systems faced a performance bottleneck. The paper introduces BERTSUM, adapts BERT with sentence representations and summarization layers, and evaluates variants on large datasets. BERTSUM with inter-sentence Transformer layers achieves the best reported performance, including state-of-the-art results across the reported metrics.
Problem
Neural extractive summarization models had reached a bottleneck on automatic metrics such as ROUGE, motivating improved approaches.
Method
BERTSUM adapts BERT for extractive summarization using [CLS]-based sentence representations and jointly fine-tuned summarization-specific layers.
Results
BERTSUM with inter-sentence Transformer layers achieves the best performance, outperforming previous state-of-the-art models across all three reported metrics.
Takeaways & Limitations
The paper’s experiments support using inter-sentence Transformer layers with BERT for extractive summarization.
Abstract
from arXiv · showhide
BERT, a pre-trained Transformer model, has achieved ground-breaking performance on multiple NLP tasks. In this paper, we describe BERTSUM, a simple variant of BERT, for extractive summarization. Our system is the state of the art on the CNN/Dailymail dataset, outperforming the previous best-performed system by 1.65 on ROUGE-L. The codes to reproduce our results are available at https://github.com/nlpyang/BertSum
1 Introduction
Extractive summarization selects important spans, usually sentences, from a document, but recent neural models had reached a ROUGE improvement bottleneck. The paper investigates BERT-based variants to further improve this task.
- Extractive summarization forms summaries by copying and concatenating important spans, usually sentences, from a document.
- ROUGE improvement had reached a bottleneck despite many recently proposed neural models for extractive summarization.
- BERT’s pre-training and architecture are proposed as a way to further boost extractive summarization performance.
- The paper evaluates different BERT-based extractive summarization variants on the CNN/Dailymail and NYT datasets.
- A flat architecture with inter-sentence Transformer layers achieves the state-of-the-art results reported for the task.
2 Methodology
BERTSUM adapts BERT for extractive summarization by representing each sentence with a dedicated [CLS] vector and adding summarization-specific layers. These layers predict sentence inclusion scores, with inter-sentence Transformer layers providing document-level modeling.
- Extractive Summarization with BERT: Extractive summarization assigns each document sentence a binary label indicating whether it belongs in the summary.The labels target sentences representing the document’s most important content.
- Encoding Multiple Sentences: BERTSUM inserts [CLS] before and [SEP] after each sentence, using multiple [CLS] symbols to obtain sentence representations.This modifies BERT’s input sequence so features can be extracted separately for multiple sentences.
- Encoding Multiple Sentences: The i-th [CLS] vector from BERT’s top layer, T_i, serves as the representation of sentence i.
- Fine-tuning with Summarization Layers: BERTSUM stacks summarization-specific layers on BERT outputs, predicts each sentence score Ŷ_i, and jointly fine-tunes these layers with BERT using binary classification entropy.
- Summarization Layers: The simple classifier applies a linear layer and sigmoid to BERT outputs, whereas the inter-sentence Transformer adds Transformer layers over sentence representations to capture document-level features.The Transformer uses positional embeddings and multi-head attention on sentence vectors.
- Inter-sentence Transformer: A two-layer inter-sentence Transformer performs best among the tested configurations with 1, 2, and 3 layers.
3 Experiments
The experiments evaluate BERTSUM on CNN/DailyMail and NYT datasets using oracle labels, ROUGE-based protocols, and sentence-selection procedures designed for extractive summaries.
- Training: Models are jointly fine-tuned with BERT and summarization layers, trained for 50,000 steps, and evaluated through averaged top-3 checkpoints.Training uses three GPUs, gradient accumulation, validation every 1,000 steps, and checkpoint selection by validation loss.
- Inference: At prediction time, sentences are ranked by scores, the top three are selected, and trigram blocking reduces redundancy.A candidate sentence is skipped when it shares a trigram with the selected summary.
- Datasets: BERTSUM is evaluated on CNN/DailyMail and NYT, using standard or date-based dataset splits and validation sets.CNN/DailyMail uses standard training, validation, and test splits; NYT uses publication-date splits with 4,000 training examples for validation.
- Data preparation: Oracle extractive labels are generated greedily by selecting sentences that maximize ROUGE scores.Selected oracle sentences receive label 1, while all other sentences receive label 0.
- Evaluation: CNN/DailyMail quality is evaluated with ROUGE Recall after truncating predicted summaries to gold-summary lengths.The table caption identifies the CNN/DailyMail results as ROUGE F1, while the cited evaluation description specifies ROUGE Recall and truncation.
4 Experimental Results
Results compare BERTSUM variants with extractive and abstractive baselines on CNN/DailyMail, while ablations examine interval segments and trigram blocking; NYT results use a limited-length recall protocol.
- CNN/DailyMail results: All BERT-based models outperform previous state-of-the-art models by a large margin on CNN/DailyMail.BERTSUM with Transformer achieves the best performance on all three reported metrics.
- CNN/DailyMail results: BERTSUM with Transformer achieves the best performance on all three CNN/DailyMail metrics.The comparison includes LEAD, REFRESH, NEUSUM, PGN, DCA, and a randomly initialized non-pretrained Transformer baseline.
- Model variants: The LSTM variant has no obvious summarization-performance influence compared with the Classifier model.The paper contrasts this result with the stronger performance of the inter-sentence Transformer variant.
- Ablation studies: Interval segments improve the base model, while trigram blocking greatly improves summarization results.The paper presents trigram blocking as a simple, robust alternative to a sequential extractive decoder.
- NYT results: NYT experiments use limited-length recall evaluation, with predicted summaries truncated to gold-standard summary lengths.Table 3 reports test-set results on the NYT50 dataset using ROUGE Recall.
5 Conclusion
The paper introduces BERTSUM for extractive summarization, evaluates several summarization layers on two large-scale datasets, and finds inter-sentence Transformer layers perform best.
- Conclusion: The paper explores how to use BERT for extractive summarization and proposes the BERTSUM model.BERTSUM is evaluated with several summarization layers applied on top of BERT.
- Conclusion: Experiments are conducted on two large-scale datasets.The conclusion summarizes the empirical evaluation across both datasets without specifying their names here.
- Conclusion: BERTSUM with inter-sentence Transformer layers achieves the best performance.This conclusion matches the reported finding that the flat architecture with inter-sentence Transformer layers performs best.