Source-linked AI summary
Extractive Summarization of Long Documents by Combining Global and Local Context
Wen Xiao, Giuseppe Carenini
TL;DR
Long documents make extractive summarization difficult because models must identify important sentences across extended, multi-topic sources. This paper combines global document context with local topic context in a neural extractive model. On Pubmed and arXiv, it outperforms prior approaches, performs relatively better on longer documents, and its ablation suggests the gains come from local context.
Problem
Neural summarization models struggle with long documents because their decoders must construct relevant context over longer source sequences.
Method
The paper proposes a neural extractive model that combines whole-document global context with local context from the topic or section containing each sentence.
Results
The model achieves state-of-the-art results on Pubmed and arXiv, becomes more competitive as documents lengthen, and outperforms prior extractive and abstractive approaches.
Takeaways & Limitations
The ablation study indicates that the model's benefits appear to come exclusively from modeling local context, even for the longest documents.
Takeaways & Limitations
The current proposal does not explicitly address redundancy, which remains present in extracted summaries and is left for future work.
Abstract
from arXiv · showhide
In this paper, we propose a novel neural single document extractive summarization model for long documents, incorporating both the global context of the whole document and the local context within the current topic. We evaluate the model on two datasets of scientific papers, Pubmed and arXiv, where it outperforms previous work, both extractive and abstractive models, on ROUGE-1, ROUGE-2 and METEOR scores. We also show that, consistently with our goal, the benefits of our method become stronger as we apply it to longer documents. Rather surprisingly, an ablation study indicates that the benefits of our model seem to come exclusively from modeling the local context, even for the longest documents.
1 Introduction
Long-document summarization is difficult because neural models struggle with long sequences, while documents often span multiple topics. The paper therefore combines global document context with local topic context to select informative sentences, especially in scientific articles.
- The approach focuses on extracting informative sentences from long documents without explicitly addressing redundancy.
- Long-document summarization remains challenging because neural decoders struggle to construct relevant context vectors over longer source sequences.
- Scientific papers provide a structured long-document setting, typically organizing problem, methodology, results, and conclusions across sections.
- The model captures both global document context and local topic context when deciding whether to include each sentence.
- The authors apply LSTM-minus to capture local context and evaluate the method on Pubmed and arXiv, where gains become stronger for longer documents.
2 Related work
Prior summarization research largely targeted shorter documents, while scientific-paper datasets introduced longer, structured documents requiring models that use document organization. The paper positions its approach as a simpler alternative that combines local topic representations with whole-document context.
- Scientific-paper summarization: Earlier scientific-paper summarizers used rhetorical categories, while later work also exploited citation contexts and finer-grained discourse labels.
- Scientific-paper datasets: Most large-scale summarization datasets contain relatively short documents, whereas arXiv and PubMed provide substantially longer scientific-paper corpora.
- Structured neural models: Hierarchical abstractive models represent words and sections, but the discussed approach uses them without whole-document global context for extractive decisions.
- Limitations of prior work: A prior hierarchical implementation processed only the first 2000 words, imposing a hard threshold that discarded later information.
- LSTM-minus: LSTM-minus represents a text span by subtracting hidden states, allowing segment embeddings to use information from inside and outside the segment.
- LSTM-minus: The paper applies LSTM-minus at document level to represent the topic or section containing each target sentence.
3 Our Model
The model extracts sentences from long documents using sentence-level, global document, and local topic-segment representations. It combines these signals through concatenation or attention before predicting each sentence’s inclusion confidence.
- Model architecture: The model sequentially encodes each sentence, the whole document, and its current topic before classifying sentence inclusion.Its components are a sentence encoder, document encoder, and sentence classifier.
- Sentence encoder: Average word embeddings serve as the sentence encoder because prior experiments found them competitive across domains and summarizer architectures.The paper also tried pre-trained BERT sentence embeddings, but initial results were poor.
- Document encoder: A bidirectional recurrent encoder represents the document by concatenating the final forward and backward hidden states.This document representation supplies global information to sentence classification.
- Topic segment encoder: LSTM-Minus represents each topic segment by subtracting hidden states at its boundaries, capturing local context for sentences in that segment.Although the method is named LSTM-Minus, this implementation uses GRUs as recurrent units.
- Sentence classifier: The model combines sentence, topic, and document representations by concatenation or attention, then uses an MLP with sigmoid activation to produce sentence-selection confidence.Attention assigns weights to contextual vectors before the final prediction.
4 Experiments
Experiments evaluate the model on Pubmed and arXiv using ROUGE and METEOR, comparing it with extractive and abstractive systems. Results show stronger performance on long documents, while ablations attribute the gains primarily to local context.
- Experimental setup: Experiments use Pubmed and arXiv scientific-paper datasets, comparing traditional extractive, neural extractive, and neural abstractive models.ROUGE and METEOR are used for evaluation.
- Experimental setup: The model selects sentences by ranking model probabilities until a 200-word length limit is met or exceeded.Evaluation compares outputs with ground-truth abstractive summaries using ROUGE and METEOR.
- Results: Neural extractive models outperform traditional extractive models on ROUGE-1 and ROUGE-2, while results are mixed on ROUGE-L.Neural extractive models also outperform neural abstractive models on ROUGE-1 and ROUGE-2, whereas abstractive models tend to achieve the highest ROUGE-L.
- Results: Our models outperform other neural extractive models on ROUGE-1, ROUGE-2, ROUGE-L, and METEOR.The baseline also slightly exceeds previous work, possibly because autoregressive decoding is more detrimental for long documents.
- Results: The performance gain over the current state-of-the-art extractive summarizer is more pronounced for documents with >= 3000 words in both datasets.Figure 2 compares the model, SummaRuNNer, and Oracle across increasing document lengths using ROUGE-1 and ROUGE-2 on Pubmed and arXiv.
- Ablation study: Ablations show that local topic information significantly improves performance on both datasets, especially for long documents, whereas global context never significantly improves it.The authors conclude that the benefits appear to come exclusively from modeling local context, including for the longest documents.
5 Conclusions and Future Work
The paper proposes an extractive model for long documents that combines local topic context with global document context. It achieves state-of-the-art results on two scientific-paper datasets, becomes more competitive as documents lengthen, and attributes its benefits primarily to local-context modeling.
- The model combines local context within each topic with global context from the whole document for long-document extractive summarization.Its architecture is described as parameter lean and modular.
- The model achieves state-of-the-art performance on two scientific-paper datasets and is compared with extractive and abstractive systems.The datasets contain documents longer than those in previously used corpora.
- The model becomes more competitive as document length increases.This additional experiment tests documents with increasing length.
- The ablation study indicates that the model's benefits come only from modeling local context.This finding is reported as applying even when documents are longest.
- Future work includes neural methods for redundancy, richer document structure, human evaluation, and integrating extractive with abstractive techniques.The authors also propose combining traditional and neural models and adding explicit features such as sentence position and salience.
A Extractive Label Generation
The extractive-label generation algorithm iteratively selects sentences that maximize ROUGE against a reference abstract while respecting a word-length limit. It returns the selected sentence indices as extractive labels.
- The algorithm generates extractive labels from human-made abstractive summaries, specifically scientific-paper abstracts.
- At each iteration, it evaluates every candidate sentence by the ROUGE score of the current hypothesis plus that sentence against the reference.
- Selection continues while the hypothesis remains within the length limit and stops when no candidate improves the score.
- The algorithm selects the highest-scoring sentence, appends it to the hypothesis, records its index, and updates the word count.
- The procedure returns the list of selected sentence indices.