Source-linked AI summary
SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
Ramesh Nallapati, Feifei Zhai, Bowen Zhou
TL;DR
Extractive summarization needs effective document-level sentence selection while remaining simpler and more reliable than abstractive generation. SummaRuNNer uses a recurrent sequence model with interpretable predictions and trains extractively or from abstractive reference summaries. It matches or outperforms state-of-the-art deep-learning models, although abstractive training remains weaker than extractive training on most datasets.
Problem
Extractive summarization requires selecting document sentences while abstractive methods are more complex and expensive despite their growing popularity.
Method
SummaRuNNer models sentence inclusion as sequential binary classification with a two-layer bidirectional GRU-RNN and can train from abstractive reference summaries through a training-time RNN decoder.
Results
SummaRuNNer matches or outperforms state-of-the-art deep-learning models for extractive summarization.
Takeaways & Limitations
The model provides interpretable visualizations of information content, salience, and novelty while supporting extractive training without sentence-level extractive labels.
Takeaways & Limitations
Abstractive training remains a couple of ROUGE points below extractive training on most datasets.
Abstract
from arXiv · showhide
We present SummaRuNNer, a Recurrent Neural Network (RNN) based sequence model for extractive summarization of documents and show that it achieves performance better than or comparable to state-of-the-art. Our model has the additional advantage of being very interpretable, since it allows visualization of its predictions broken up by abstract features such as information content, salience and novelty. Another novel contribution of our work is abstractive training of our extractive model that can train on human generated reference summaries alone, eliminating the need for sentence-level extractive labels.
Introduction
Document summarization supports information retrieval and natural language understanding, with extractive methods remaining attractive despite growing interest in abstractive neural approaches. SummaRuNNer addresses this setting with an interpretable recurrent sequence classifier and abstractive-summary-based training.
- Extractive summarization selects salient sentences or passages, whereas abstractive summarization paraphrases document information concisely.
- Traditional extractive approaches include greedy, graph-based, and constraint-optimization methods, while neural approaches increasingly use learned sentence representations.
- Extractive techniques remain attractive because they are less complex, less expensive, and usually produce grammatically and semantically correct summaries.
- SummaRuNNer is a recurrent sequence classifier for single-document sentential extraction that matches or outperforms state-of-the-art extractive models.
- Its contributions include interpretable decision visualizations and end-to-end training from abstractive summaries without sentence-level extractive labels.
SummaRuNNer
SummaRuNNer treats extractive summarization as sequential sentence classification using a two-layer bidirectional GRU-RNN. It combines sentence content, document salience, summary novelty, and position, and can be trained from abstractive reference summaries through a temporary decoder.
- Sequence classification: Each sentence is visited in document order, and a binary summary-membership decision incorporates previous decisions.At test time, the model emits a sentence-level probability used as its soft extractive prediction.
- Hierarchical encoder: The model uses bidirectional GRU-RNN layers at word and sentence levels to encode sentence and document representations.Word-level hidden states are average-pooled into sentence-level inputs, while sentence-level states encode the document.
- Interpretable decision features: Sentence decisions combine information content, document salience, redundancy against the current summary, and absolute and relative position.The summary representation is a running weighted sum of prior sentence states, with weights given by summary-membership probabilities.
- Extractive training: Training minimizes the negative log-likelihood of observed binary sentence labels for extractive supervision.The formulation requires ground-truth sentence-level labels for each document.
- Abstractive training: An abstractive training variant couples SummaRuNNer to an RNN decoder during training, eliminating the need for approximate extractive labels.The decoder uses the final summary representation as context and predicts words in the reference summary; it is removed at test time.
Related Work
Earlier work framed extractive summarization as sequential sentence classification, used large news corpora for neural training, and contrasted encoder-decoder designs with SummaRuNNer’s label-free abstractive training. Fully abstractive systems instead generate summaries at test time, whereas SummaRuNNer predicts sentence-extraction probabilities.
- Conditional Random Fields previously classified sentences sequentially, whereas SummaRuNNer uses RNNs without handcrafted sentence or document features.
- The CNN/Daily Mail corpus contains around 280,000 documents with human-generated summaries, addressing the limited size of earlier single-document datasets.
- SummaRuNNer’s extractively trained model uses a single sequence model without a decoder, potentially reducing parameters relative to Cheng and Lapata’s encoder-decoder.
- Its abstractive trainer models the likelihood of abstractive gold summaries during training, eliminating the need for extractive labels unlike Cheng and Lapata’s approach.
- Nallapati et al.’s model is fully abstractive at test time, while SummaRuNNer’s abstractive trainer produces sentence-extraction probabilities.
Experiments and Results
Experiments evaluate SummaRuNNer on CNN/Daily Mail and out-of-domain DUC 2002 datasets using Rouge metrics, with comparisons across extractive, abstractive, and graph-based baselines. Results show strong in-domain performance, while cross-domain performance is lower than state-of-the-art graph-based systems.
- Experimental setup: The experiments use CNN/Daily Mail and DUC 2002, evaluating limited-length Rouge recall and full-length Rouge F1 against multiple baselines.Daily Mail comparisons use 75- and 275-byte limits; CNN/Daily Mail uses full-length F1; DUC 2002 uses 75-word limited-length recall.
- Experimental setup: SummaRuNNer selects sentences by predicted probability rather than a fixed 0.5 threshold, using length limits for limited-length Rouge and validation-selected sentence counts for full-length F1.This strategy addresses sentence-label imbalance in the training data.
- Daily Mail results: At 75 bytes on DailyMail, the abstractively trained model performs on par with Cheng and Lapata, while the extractively trained model significantly improves over it.Table 1 evaluates Rouge recall against abstractive ground truth.
- Daily Mail results: At 275 bytes on DailyMail, the abstractively trained model underperforms Cheng and Lapata, while the extractively trained model is statistically indistinguishable from it.The authors suggest SummaRuNNer is better at selecting the best sentence than subsequent sentences at this length.
- CNN/Daily Mail results: On CNN/Daily Mail, SummaRuNNer significantly outperforms the abstractive state-of-the-art, while its top-three-sentence strategy favors recall over precision under full-length F1.The authors note that dynamically adjusting summary length might improve the precision–recall balance, but they did not test it.
- DUC 2002 results: On out-of-domain DUC 2002, SummaRuNNer is statistically on par with Cheng and Lapata but performs worse than graph-based TGRAPH and URANK.The authors associate this pattern with domain adaptation issues for supervised models and greater robustness to domain variation in graph-based unsupervised approaches.
Qualitative Analysis
SummaRuNNer offers interpretable extractive summarization by decomposing sentence-selection decisions into abstract feature scores and visualizing them for example documents. Qualitative examples from Daily Mail and DUC compare selected sentences with gold summaries and indicate reasonably good identification of key points.
- The model visualizes normalized scores for abstract features contributing to each sentence’s final classification.The visualization uses a representative validation document to expose factors behind individual sentence decisions.
- The feature-level visualization helps explain the system’s sentence-selection decisions to end-users.
- Examples from Daily Mail and DUC compare SummaRuNNer’s selected sentences with the corresponding gold summaries.
- The qualitative examples demonstrate reasonably good identification of a document’s key points.
Conclusion
The paper presents an interpretable neural sequence model for extractive document summarization that performs at or above comparable state-of-the-art deep-learning models. It also introduces abstractive training without extractive labels, although that approach remains below extractive training on most datasets.
- SummaRuNNer is a very interpretable neural sequence model for extractive document summarization.Its design supports intuitive visualization of the model’s decisions.
- The model performs better than or comparably to state-of-the-art deep-learning models.
- The abstractive training mechanism eliminates the need for extractive labels during training.
- The abstractive training approach remains a couple of Rouge points below extractive training on most datasets.