Source-linked AI summary
A Supervised Approach to Extractive Summarisation of Scientific Papers
Ed Collins, Isabelle Augenstein, Sebastian Riedel
TL;DR
Scientific summarisation lacked large datasets suitable for supervised neural models. The paper builds a large computer-science dataset from author highlights, extends it with automatic labeling, and finds that context-aware extractive models outperform established baselines.
Problem
Existing scientific-publication summarisation datasets contain only tens of documents, insufficient for supervised neural models typically trained on many thousands.
Method
The paper creates a dataset from author-provided highlights, extends training data with HighlightROUGE, and uses sentence, contextual, positional, and abstract-based features.
Results
Models encoding sentences, global context, and position significantly outperform well-established summarisation methods, while AbstractROUGE and automatic dataset extension increase performance.
Takeaways & Limitations
The dataset supports extractive summarisation research on large technical documents and can be extended automatically to additional domains.
Takeaways & Limitations
The approach remains challenged by overfitting on automatically generated labels and by incomplete modeling of global context and cross-sentence dependencies.
Abstract
from arXiv · showhide
Automatic summarisation is a popular approach to reduce a document to its main arguments. Recent research in the area has focused on neural approaches to summarisation, which can be very data-hungry. However, few large datasets exist and none for the traditionally popular domain of scientific publications, which opens up challenging research avenues centered on encoding large, complex documents. In this paper, we introduce a new dataset for summarisation of computer science publications by exploiting a large resource of author provided summaries and show straightforward ways of extending it further. We develop models on the dataset making use of both neural sentence encoding and traditionally used summarisation features and show that models which encode sentences as well as their local and global context perform best, significantly outperforming well-established baseline methods.
1 Introduction
The paper addresses the lack of large scientific-publication summarisation datasets by introducing a scalable computer-science dataset and methods for extending and exploiting it.
- Dataset: Over 10k computer-science publications form a new dataset that supports both extractive and abstractive summarisation.The dataset is built from author-provided highlight statements and can be extended automatically to 26 additional domains.
- Dataset extension: HighlightROUGE automatically extends training data by identifying body sentences similar to author highlights.The authors report that this extension improves summarisation performance.
- Summarisation features: AbstractROUGE uses similarity between a sentence and the paper abstract as a feature for extracting summary sentences.It exploits the abstract as a preexisting summary of the paper.
- Model evaluation: Models encoding sentences, local context, global context, and position significantly outperform well-established extractive-summarisation baselines.The paper benchmarks neural and traditional methods and reports the strongest comparison for its best-performing model.
2 Dataset and Problem Formulation
The authors construct large training and evaluation resources from 10,148 computer-science papers, formulate extraction as sentence classification, and extend labels with HighlightROUGE.
- 2 Dataset and Problem Formulation: The dataset contains 10,148 computer-science publications with titles, abstracts, author highlights, and keywords.The papers come from ScienceDirect and can be extended across its 27 domains.
- 2 Dataset and Problem Formulation: Author highlight statements serve as gold summaries because they are intended to convey each paper’s main takeaway.The dataset is designed for extractive summarisation using these author-written statements.
- 2.1 Problem Formulation: Extractive summarisation is formulated as binary sentence classification, assigning each document sentence a label y ∈ 0, 1.Training examples contain sentences, context-encoding features, and labels in a randomly ordered list.
- 2.2 Creation of the Training and Testing Data: The two resulting datasets are CSPubSum and its HighlightROUGE-extended version, CSPubSumExt.The authors use both resources to train and test summarisation models.
- 2.2 Creation of the Training and Testing Data: CSPubSum uses highlight statements as positives and randomly samples an equal number of low-ROUGE sentences as negatives, yielding 85,490 training instances.Its test set contains 150 full papers for evaluating summary quality rather than sentence-classification accuracy.
- 2.2 Creation of the Training and Testing Data: CSPubSumExt addresses missing body-sentence labels by adding HighlightROUGE-selected examples, producing 263k training and 132k test instances.Models are evaluated for classification on CSPubSumExt Test, while summariser quality is measured on CSPubSum Test with ROUGE-L.
3 ROUGE Metrics
The paper uses ROUGE-L to evaluate extractive summaries and introduces HighlightROUGE and AbstractROUGE to identify or score sentences using author highlights and abstracts.
- ROUGE metrics: ROUGE metrics measure summarisation quality and correspond well with human judgements, with ROUGE-L selected for scientific-article evaluation.The choice follows prior scientific-summarisation research.
- 3.1 HighlightROUGE: HighlightROUGE selects body sentences with the highest ROUGE-L similarity to author highlights as ideal extractive-summary candidates.The method treats these sentences as oracle-like positive examples and combines them with equal numbers of low-scoring negatives.
- 3.1 HighlightROUGE: Abstract sentences are excluded from HighlightROUGE training examples because the goal is to extract salient sentences from the main paper.The abstract is treated as an already existing summary rather than a source of additional body-sentence labels.
- 3.2 AbstractROUGE: AbstractROUGE scores a sentence by its ROUGE-L overlap with the abstract and uses that score as a summarisation feature.The feature assumes sentences summarising the abstract are likely to summarise the author highlights.
4 Method
The method represents sentences with averaged word vectors or recurrent encodings, combines these with abstract information and handcrafted contextual features, and compares neural, feature-based, and ensemble summarisers. SAFNet extends the feature-and-LSTM architecture by also encoding the abstract, while ensemble models combine summariser outputs with a weighted average.
- Summariser features: Eight handcrafted features provide local and global context because randomly ordered sentences lack surrounding-sentence context.The feature set includes document TF-IDF, sentence length, AbstractROUGE, section location, numeric count, title overlap, keyphrase overlap, and TF-IDF.
- Sentence representations: The models represent sentences using averaged word vectors, ordered-word bidirectional LSTMs, or both with handcrafted features.Word2Vec uses a 100-dimensional averaged sentence vector, while SNet uses a bidirectional LSTM with 128 hidden units and dropout.
- SAFNet: SAFNet extends SFNet by encoding the abstract alongside the sentence and handcrafted features.SFNet combines an LSTM sentence representation with handcrafted features; SAFNet further encodes the abstract.
- Ensemble methods: SAF+F and S+F ensembles compute weighted averages of two summarisers’ outputs, with C controlling the weighting.SAF+F combines SAFNet with FNet, while S+F combines SNet with FNet.
5 Results and Analysis
Results show that no single paper section definitively supplies summary sentences, while models combining neural sentence encoding with contextual features outperform established baselines. Performance also depends on evaluation data quality: automatically generated labels improve training scale but can misalign accuracy with gold-summary ROUGE-L.
- 5.1 Most Relevant Sections to a Summary: No definitive paper section should supply summary sentences after excluding the abstract, although some sections are slightly more likely to contain good candidates.The title has the highest average ROUGE, while the introduction has low ROUGE but high Copy/Paste frequency.
- 5.2 Comparison of Model Performance and Error Analysis: The best model significantly outperforms LexRank, TextRank, KL-Sum, LSA, and SumBasic on CSPubSum Test.It combines neural sentence encoding with simple global-context and positional features.
- 5.2 Comparison of Model Performance and Error Analysis: LSTM sentence encoding beats averaged word embeddings by 6.7% accuracy and 2.1 ROUGE points.The result indicates that word order matters when identifying summary sentences.
- 5.2 Comparison of Model Performance and Error Analysis: Accuracy on CSPubSumExt Test and ROUGE-L on CSPubSum Test are strongly correlated (Pearson R=0.8738), but the highest accuracy does not yield the best ROUGE-L.SAFNet has the highest automatically generated-test accuracy but performs worse than AbstractROUGE Summariser on the gold test set, likely because of imperfect labels.
- 5.3 Effect of Using ROUGE-L to Generate More Data: HighlightROUGE expansion improves summarisation performance, with FNet and SFNet suffering significant drops when trained without the expanded data.The reported drops are statistically significant for FNet (p = 0.0147) and SFNet (p < 0.0001), while SAFNet is more stable.
- 5.4 Effect of the AbstractROUGE Metric on Summariser Performance: AbstractROUGE improves feature-engineered summarisation, while models learning from raw text and features remain more stable when it is unavailable.This reduces reliance on the presence of an abstract during training.
6 Related Work
Scientific summarisation datasets have historically been small, limiting supervised neural approaches, while newer methods combine statistical features with neural document encoding. This work addresses the data and encoding challenge with a large dataset and a hybrid context-modelling strategy.
- Extractive Summarisation Methods: AbstractROUGE-based training comparisons evaluate Features Only, SAFNet, and SFNet models with and without the abstract-derived metric on CSPubSum Test.The FNet classifier shows a statistically significant performance decrease without AbstractROUGE (p=0.0279).
- Datasets: Existing scientific-document summarisation datasets range from 9 to 80 papers, far smaller than datasets used for data-intensive neural summarisation.Examples include 21 publications from Kupiec et al., 30 in CL-SciSumm 2017, and 80 in Teufel and Moens.
- Datasets: The introduced dataset is, to the authors’ knowledge, the only large dataset for extractive summarisation of scientific publications.Its size enables training data-intensive neural methods and creates challenges around encoding very large documents.
- Extractive Summarisation Methods: Recent extractive summarisation methods increasingly use neural representations, including word embeddings and CNN/RNN encoders for whole documents.Earlier supervised systems instead classified sentences using statistical cues such as word frequency, position, and TF-IDF.
- Extractive Summarisation Methods: Because scientific publications are very large, this work encodes target sequences with an RNN but represents global context with simpler features rather than a fully neural document encoder.Fully neural approaches to encoding publications are left for future work.
7 Conclusion
The paper introduces a substantially larger scientific-publication summarisation dataset, develops extractive models using sentence, global-context, and positional information, and reports improved performance over established methods. AbstractROUGE and automatic dataset extension further improve summarisation, while modelling global context and cross-sentence dependencies remains challenging.
- 7 Conclusion: Models encoding sentences, global context, and position significantly outperform well-established summarisation methods.The conclusion identifies these combined representations as the strongest reported modelling result.
- 7 Conclusion: The paper introduces a substantially larger computer-science publication summarisation dataset by exploiting an existing resource.The dataset is intended to support both the data scale and document-encoding challenges of scientific summarisation.
- 7 Conclusion: AbstractROUGE increases summarisation performance, and automatically extending the dataset further increases performance.These improvements are reported as separate contributions alongside the main model comparison.
- 7 Conclusion: Remaining challenges are better modelling global summary context and capturing dependencies across sentences.The conclusion identifies both as unresolved directions for future work.