Fine-tune BERT for Extractive Summarization

• Extractive summarization forms summaries by copying and concatenating important spans, usually sentences, from a document.Source: §1 · p. 1
• ROUGE improvement had reached a bottleneck despite many recently proposed neural models for extractive summarization.Source: §1 · p. 1
• BERT’s pre-training and architecture are proposed as a way to further boost extractive summarization performance.Source: §1 · p. 1
• The paper evaluates different BERT-based extractive summarization variants on the CNN/Dailymail and NYT datasets.Source: §1 · p. 1
• A flat architecture with inter-sentence Transformer layers achieves the state-of-the-art results reported for the task.Source: §1 · p. 1

Extractive Summarization with BERT

• Extractive summarization assigns each document sentence a binary label indicating whether it belongs in the summary.The labels target sentences representing the document’s most important content.Source: §2 · p. 1

Encoding Multiple Sentences

• BERTSUM inserts [CLS] before and [SEP] after each sentence, using multiple [CLS] symbols to obtain sentence representations.This modifies BERT’s input sequence so features can be extracted separately for multiple sentences.Source: §2.1 · pp. 1–2
• The i-th [CLS] vector from BERT’s top layer, T_i, serves as the representation of sentence i.Source: §2.1 · p. 2

Fine-tuning with Summarization Layers

• BERTSUM stacks summarization-specific layers on BERT outputs, predicts each sentence score Ŷ_i, and jointly fine-tunes these layers with BERT using binary classification entropy.Source: §2.2 · p. 2

Summarization Layers

• The simple classifier applies a linear layer and sigmoid to BERT outputs, whereas the inter-sentence Transformer adds Transformer layers over sentence representations to capture document-level features.The Transformer uses positional embeddings and multi-head attention on sentence vectors.Source: §2.2 · p. 2

Inter-sentence Transformer

• A two-layer inter-sentence Transformer performs best among the tested configurations with 1, 2, and 3 layers.Source: §2.2 · p. 2

Training

• Models are jointly fine-tuned with BERT and summarization layers, trained for 50,000 steps, and evaluated through averaged top-3 checkpoints.Training uses three GPUs, gradient accumulation, validation every 1,000 steps, and checkpoint selection by validation loss.Source: §3.1 · p. 3

Inference

• At prediction time, sentences are ranked by scores, the top three are selected, and trigram blocking reduces redundancy.A candidate sentence is skipped when it shares a trigram with the selected summary.Source: §3.1 · p. 3

Datasets

• BERTSUM is evaluated on CNN/DailyMail and NYT, using standard or date-based dataset splits and validation sets.CNN/DailyMail uses standard training, validation, and test splits; NYT uses publication-date splits with 4,000 training examples for validation.Source: §3.2 · p. 3

Data preparation

• Oracle extractive labels are generated greedily by selecting sentences that maximize ROUGE scores.Selected oracle sentences receive label 1, while all other sentences receive label 0.Source: §3.2 · pp. 3–4

Evaluation

• CNN/DailyMail quality is evaluated with ROUGE Recall after truncating predicted summaries to gold-summary lengths.The table caption identifies the CNN/DailyMail results as ROUGE F1, while the cited evaluation description specifies ROUGE Recall and truncation.Source: Table 1 · p. 4

CNN/DailyMail results

• All BERT-based models outperform previous state-of-the-art models by a large margin on CNN/DailyMail.BERTSUM with Transformer achieves the best performance on all three reported metrics.Source: §4 · p. 4
• BERTSUM with Transformer achieves the best performance on all three CNN/DailyMail metrics.The comparison includes LEAD, REFRESH, NEUSUM, PGN, DCA, and a randomly initialized non-pretrained Transformer baseline.Source: §4 · p. 4

Model variants

• The LSTM variant has no obvious summarization-performance influence compared with the Classifier model.The paper contrasts this result with the stronger performance of the inter-sentence Transformer variant.Source: §4 · p. 4

Ablation studies

• Interval segments improve the base model, while trigram blocking greatly improves summarization results.The paper presents trigram blocking as a simple, robust alternative to a sequential extractive decoder.Source: §4 · p. 4, Table 2 · p. 4

NYT results

• NYT experiments use limited-length recall evaluation, with predicted summaries truncated to gold-standard summary lengths.Table 3 reports test-set results on the NYT50 dataset using ROUGE Recall.Source: §4 · p. 4, Table 3 · p. 5

Conclusion

• The paper explores how to use BERT for extractive summarization and proposes the BERTSUM model.BERTSUM is evaluated with several summarization layers applied on top of BERT.Source: §5 · p. 5
• Experiments are conducted on two large-scale datasets.The conclusion summarizes the empirical evaluation across both datasets without specifying their names here.Source: §5 · p. 5
• BERTSUM with inter-sentence Transformer layers achieves the best performance.This conclusion matches the reported finding that the flat architecture with inter-sentence Transformer layers performs best.Source: §5 · p. 5