Source-linked AI summary
SciBERTSUM: Extractive Summarization for Scientific Documents
Athar Sefid, C Lee Giles
TL;DR
Scientific-paper summarization must handle documents far longer and more structured than typical news inputs. SciBERTSUM extends BERTSUM with section embeddings and sparse local-global attention, using author-generated slides as reference summaries. The model improves ROUGE scores on paper-slide pairs and is computationally efficient, while evaluation on other scholarly-document settings remains future work.
Problem
Summarizing scientific papers is difficult because they can exceed 500 sentences and contain multiple sections, making full-document attention costly.
Method
SciBERTSUM extends BERTSUM with section embeddings, sparse local-global inter-sentence attention, and author-generated presentation slides as reference summaries.
Results
SciBERTSUM improves ROUGE scores on a dataset of paper-slide pairs and is computationally efficient.
Takeaways & Limitations
Section-aware sparse attention provides an efficient extractive summarization framework for long scientific documents with multiple sections.
Takeaways & Limitations
Future work includes applying SciBERTSUM to existing summarization datasets and other long scholarly documents.
Abstract
from arXiv · showhide
The summarization literature focuses on the summarization of news articles. The news articles in the CNN-DailyMail are relatively short documents with about 30 sentences per document on average. We introduce SciBERTSUM, our summarization framework designed for the summarization of long documents like scientific papers with more than 500 sentences. SciBERTSUM extends BERTSUM to long documents by 1) adding a section embedding layer to include section information in the sentence vector and 2) applying a sparse attention mechanism where each sentences will attend locally to nearby sentences and only a small number of sentences attend globally to all other sentences. We used slides generated by the authors of scientific papers as reference summaries since they contain the technical details from the paper. The results show the superiority of our model in terms of ROUGE scores.
1 Introduction
SciBERTSUM addresses the difficulty of summarizing long, multi-section scientific documents by extending BERTSUM with section information, sparse attention, and slide-based reference summaries.
- 1 Introduction: SciBERTSUM targets scientific papers exceeding 500 sentences, where full-attention summarization requires substantial memory and computation.Existing summarization methods commonly use news documents with fewer than 30 sentences, while scientific papers are longer and structurally different.
- 1 Introduction: SciBERTSUM builds on BERTSUM, whose document-level full attention is not efficient for scientific papers with more than 500 sentences.BERTSUM extends BERT to cover all document sentences, but its full-attention formulation does not scale efficiently to these inputs.
- 1 Introduction: The framework adds a section embedding layer so sentence representations incorporate the document’s hierarchical section structure.Tokens in the same section share a section embedding, extending BERTSUM’s sentence representation with section information.
- 1 Introduction: Its sparse inter-sentence attention lets sentences attend locally to nearby sentences while selected random sentences attend globally across the document.This design addresses the inefficiency of full attention for long scientific documents.
- 1 Introduction: The model uses authors’ presentation slides as reference summaries because they contain technical details and generally follow the paper’s structure.The slides serve as a less compressed reference than abstracts and preserve technical content from the paper.
2 Related Work
Scientific-article summarization is less studied than news summarization because suitable full-paper training and reference-summary data are limited. Prior references and models have coverage, noise, or scalability limitations, motivating slide-based summaries for long scientific documents.
- 2 Related Work: Scientific-article summarization has received less investigation than news summarization, mainly because training data for complete scientific papers are scarce.
- 2 Related Work: Abstracts are highly compressed and may omit contributions, citation-based summaries reflect citing-paper context, and TalkSum transcripts can be noisy.These reference-summary types provide different coverage but do not consistently preserve the paper’s full technical content.
- 2 Related Work: Presentation slides aim to cover important content across an entire paper, including highlights and valuable images or tables, while avoiding transcript noise.The authors use the PS5k dataset to build their summarizer.
- 2 Related Work: Transformer summarizers such as BART often target short news datasets, while their quadratic space and computational complexity limits use on scientific articles.BART and related models are not designed for long scientific inputs.
- 2 Related Work: BERTSUM and related transformer extractive models improve sentence representations but generally do not scale to long documents with thousands of tokens or full scientific papers.BERTSUM expands positional and segmentation embeddings for multiple sentences, while HIBERT [25] learns context-aware sentence representations.
3 Method - SciBERTSUM
SciBERTSUM extends BERTSUM for long scientific documents by incorporating section information into sentence embeddings and replacing expensive full attention with sparse sentence-level attention.
- 3 Method - SciBERTSUM: SciBERTSUM extends BERTSUM to generate sentence embeddings for full documents with multiple sections and models inter-sentence relations using linear sparse attention.The method is designed for scientific documents that can contain more than 500 sentences, where full attention is expensive.
- 4.1 Embedding Layer: Section embeddings augment semantic, positional, and segmentation embeddings so sentences carry information about their document sections.All tokens in a section share the same section embedding, while sentence embeddings are taken from the [CLS] representations.
- 4.2 Attention Mechanism: Sparse attention combines local and global schemes: each sentence attends to nearby sentences, while selected sentences attend globally across the document.This design captures salient sentences within local windows and identifies globally important sentences across sections.
- Building the Attention Matrix: Window-based attention requires padding documents to fixed and window-compatible lengths, plus an attention matrix distinguishing padding, local, and combined local-global attention.The corpus fixes scientific-document length to 500 sentences, and the attention matrix uses 0 for padding, 1 for local attention, and 2 for combined attention.
- Calculating Attention Value: Local attention computes query, key, and value vectors from sentence vectors, applies normalized sliding-window scores, and masks padding positions.The query, key, and value projections use learned weight matrices and a bias term; global attention follows the same approach with adjusted sentence vectors.
5. Softmax is applied to attention scores to generate the attention probabilities
The sparse attention mechanism computes attention within each transformation layer while using sentence embeddings that include section information.
- Attention values are multiplied by value vectors chunk by chunk within a sliding window.
- Sparse attention replaces full attention in each transformation layer, using sentence embeddings that include section information.
5 Sentence Extractor
The sentence extractor combines language-module sentence embeddings with document-dependent features to predict final sentence scores.
- Final sentence scores combine language-module sentence embeddings with features calculated from the document embedding.
- The score-prediction module concatenates the features and applies a linear layer to generate the final scores.
3. Section: section of the sentence i in the document
The model represents sentence relationships and importance using correlation, saliency, and document-level embeddings, then combines these features for score prediction.
- Sentence correlations encode relationships between sentences and help identify highly correlated sentences for exclusion.
- Saliency embeddings represent sentence importance relative to the document embedding through a learned saliency matrix.
- The document embedding is a learned weighted average of the sentence vectors.
- The score predictor concatenates section, correlation, and saliency features before applying a linear layer.
6 Reinforcement Learning
The reinforcement-learning formulation addresses the mismatch between extractive-label training and ROUGE-based evaluation by incorporating ROUGE rewards into training.
- Training minimizes cross-entropy on extractive labels, whereas testing evaluates similarity to abstractive summaries with ROUGE scores.
- ROUGE scores are incorporated into a reinforced setting to reduce the discrepancy between training and testing objectives.
- The reward r(y) averages ROUGE-F1 and ROUGE-F2 scores, encouraging predictions whose extracted sentences overlap strongly with the abstractive summary.
7 Experimental Results
Experiments show that larger local and global attention settings improve ROUGE recall, while reinforcement learning and trigram blocking do not improve results. SciBERTSUM outperforms the tested extractive and abstractive baselines on paper-slide summaries.
- Larger local attention windows and higher global-attention ratios improve ROUGE recall and accelerate convergence, subject to available hardware.Table 1 evaluates these attention settings with a summary limit of 20% of the input document.
- Reinforcement learning does not improve ROUGE scores on this dataset, mainly because it reduces bias toward sentence position and length.
- SciBERTSUM outperforms many tested extractive and abstractive models on the paper-slide dataset, with comparisons reported using ROUGE recall.BERTSUM was processed chunk by chunk, while BART and T5 were processed section by section because of long-document limitations.
- Trigram blocking worsens results, indicating that the model’s predicted scores are sufficient to assess whether shared-token sentences improve ROUGE.
8 Conclusions and Future Work
SciBERTSUM extends BERTSUM for long, multi-section documents by incorporating section information and sparse attention. The model is computationally efficient, improves ROUGE on paper-slide pairs, and leaves broader scholarly-document evaluation for future work.
- 8 Conclusions and Future Work: SciBERTSUM generates section-aware sentence vectors and uses sparse attention to process long documents efficiently.The model uses local attention together with a small number of random sentences attending globally.
- 8 Conclusions and Future Work: SciBERTSUM improves ROUGE scores on the dataset of paper-slide pairs while remaining computationally efficient.
- 8 Conclusions and Future Work: Applying SciBERTSUM to existing summarization datasets and other long scholarly documents remains future work.