Source-linked AI summary
Neural Summarization by Extracting Sentences and Words
Jianpeng Cheng, Mirella Lapata
TL;DR
Extractive summarization traditionally depends on human-engineered features, creating a need for data-driven alternatives. The paper develops hierarchical neural encoder-extractor models that select sentences or words, trains them on large-scale DailyMail-derived data, and reports performance comparable to state-of-the-art systems on two datasets.
Problem
Traditional extractive summarization methods rely heavily on human-engineered features, while neural extraction requires large labeled corpora that have been scarce.
Method
The paper uses a hierarchical neural document encoder and attention-based extractor to select sentences or words from single documents.
Results
The models achieve performance comparable to state-of-the-art systems on the DUC 2002 and DailyMail summarization datasets.
Takeaways & Limitations
The framework learns informativeness from continuous representations without linguistic annotations and supports both sentence and word extraction.
Takeaways & Limitations
Human evaluation did not include TGRAPH or URANK because the authors lacked their system outputs.
Abstract
from arXiv · showhide
Traditional approaches to extractive summarization rely heavily on human-engineered features. In this work we propose a data-driven approach based on neural networks and continuous sentence features. We develop a general framework for single-document summarization composed of a hierarchical document encoder and an attention-based extractor. This architecture allows us to develop different classes of summarization models which can extract sentences or words. We train our models on large scale corpora containing hundreds of thousands of document-summary pairs. Experimental results on two summarization datasets demonstrate that our models obtain results comparable to the state of the art without any access to linguistic annotation.
1 Introduction
The paper replaces hand-engineered extractive summarization features with neural networks and continuous sentence representations. Its hierarchical encoder and attention-based extractor support sentence- and word-level extraction, achieving performance comparable to state-of-the-art systems on two datasets.
- Automatic summarization seeks shorter versions of documents that preserve their information content, commonly by concatenating salient extracted sentences.
- Most extractive methods identify sentences using human-engineered surface, content, and event features with classifiers, hidden Markov models, graphs, or integer programming.
- The proposed approach uses neural networks and continuous sentence features instead of manually designed summarization features.
- A hierarchical document encoder represents sentences and their words, while an attention-based extractor directly selects input sentences or words for the summary.
- Training data are built from hundreds of thousands of DailyMail articles and highlights, producing datasets for sentence and word extraction.
- On DUC 2002 and DailyMail, the neural summarizers perform comparably to state-of-the-art systems using hand-engineered features and linguistic constraints.
2 Problem Formulation
The paper formulates summarization at sentence and word granularity. Sentence extraction labels selected sentences, while word extraction orders selected document words to form shorter summaries.
- 2 Problem Formulation: The task considers summaries at two granularities: selecting sentences or selecting and ordering words from a document.
- 2 Problem Formulation: Sentence extraction selects fewer than m document sentences by scoring each sentence and predicting an inclusion label yL ∈ {0,1}.
- 2 Problem Formulation: Supervised sentence extraction maximizes the likelihood of the document’s sentence labels under model parameters θ.
- 2 Problem Formulation: Extracted sentences are grammatical but can produce long summaries containing redundant information.
- 2 Problem Formulation: Word extraction selects a subset of document words and orders them to form a summary.
- 2 Problem Formulation: The word-extraction vocabulary may be extended with a small set of commonly used high-frequency words.
- AFL star blames vomiting cat for speeding: The example article concerns an AFL defender who attributed speeding to distraction from his vomiting cat.
- AFL star blames vomiting cat for speeding: Word extraction is formulated as language generation with an output vocabulary restricted to the document, and training maximizes generated-sentence likelihood.
3 Training Data for Summarization
Because labeled training data for extractive neural summarization were scarce, the authors construct large-scale sentence- and word-extraction datasets from DailyMail articles and highlights.
- 3 Training Data for Summarization: Existing labeled training corpora were limited to hundreds of examples, motivating construction of two large-scale extraction datasets.
- 3 Training Data for Summarization: The authors retrieve hundreds of thousands of DailyMail news articles and corresponding editor-created highlights.
- 3 Training Data for Summarization: Because highlights are genuinely abstractive, they are not directly suited to supervised extraction training.
- 3 Training Data for Summarization: Sentence-extraction labels are approximated with a rule-based system that matches document sentences to highlights and assigns labels 1 or 0.
- 3 Training Data for Summarization: Word-extraction examples are retained when highlight words overlap the article, with semantically equivalent in-document substitutions attempted for out-of-vocabulary words.
- 3 Training Data for Summarization: The resulting word-extraction dataset contains 170K DailyMail articles after unmatched examples are discarded.
4 Neural Summarization Model
The model uses hierarchical neural representations to encode documents and attention-based recurrent extractors to select sentences or generate summary words. Sentence extraction accounts for relevance and redundancy, while word extraction uses hierarchical attention over sentences and words.
- Document Reader: A hierarchical document reader builds sentence representations with CNNs and composes them into document representations with an RNN.The CNN uses temporal convolution, max-over-time pooling, multiple feature maps, and multiple kernel widths; the document-level RNN models sentence transitions.
- Sentence Extractor: The sentence extractor applies attention directly to encoded sentences and labels them sequentially using document context and previously labeled sentences.Its recurrent decisions jointly account for sentence relevance and mutual redundancy.
- Sentence Extractor: Curriculum learning reduces training–testing mismatch by shifting the previous-sentence input from the true label toward the model’s predicted label.This addresses error accumulation caused by relying on true previous labels during training but predicted labels during testing.
- Word Extractor: Word extraction extends sequential labeling into generation by directly predicting the next summary word with hierarchical attention over document sentences and their words.The decoder first attends to sentences, then to words, and computes the probability of the next included word.
- Word Extractor: The word extractor constrains output to document vocabulary but is not powerful enough to enforce grammaticality because highlights are lexically diverse and sparse.The paper suggests pairing it with a neural language model or another decoding strategy to improve grammaticality.
5 Experimental Setup
The experiments train and evaluate neural sentence- and word-extraction models on DailyMail-derived data, using DUC-2002 and DailyMail test articles. Evaluation combines ROUGE with human rankings and comparisons against extractive, abstractive, and published systems.
- Datasets: The models were trained on two DailyMail-derived datasets, each split approximately 90% training, 5% validation, and 5% testing.The datasets supported sentence- and word-based summarization models.
- Datasets: Evaluation used 567 DUC-2002 documents and 500 DailyMail test articles, with human summaries or highlights as gold standards.The DUC-2002 documents belong to 59 news-topic clusters and have two versions of 100-word manual summaries.
- Model operation: At test time, sentence extraction selected the three sentences with the highest softmax scores, subject to evaluation word or byte limits.The method uses relative sentence ranking rather than binary labels.
- System comparisons: The comparison systems included a lead baseline, logistic-regression extraction with engineered features, neural abstractive summarization, and published ILP and graph-based systems.The published systems included Woodsend and Lapata’s phrase-based extraction model, TGRAPH, and URANK.
- Automatic evaluation: ROUGE-1 and ROUGE-2 measured informativeness through unigram and bigram overlap, while ROUGE-L measured fluency through longest common subsequence.The evaluation reported ROUGE scores for the DUC-2002 and DailyMail samples.
- Human evaluation: Human evaluation ranked summaries for 20 DUC-2002 documents by informativeness and included five responses per document.Participants compared neural extraction, neural abstractive, lead, phrase-based ILP, and human-authored summaries.
6 Results
The neural sentence extractor performs strongly against established baselines, while the word extractor is less favorable on ROUGE but outperforms the abstractive model in direct comparison. Human evaluation also places NN-SE near human-written summaries.
- Sentence extraction: NN-SE outperforms LEAD and LREG by a significant margin and performs slightly better than ILP.The model uses embedding features from raw text rather than manually selected features or syntactic constraints.
- Sentence extraction: NN-SE achieves performance comparable to state-of-the-art systems without sophisticated constraint optimization or sentence-ranking mechanisms.The model’s sentence weights identify text portions contributing most to the document’s overall meaning.
- Word extraction: NN-WE’s ROUGE scores are less promising because ROUGE is not well suited to paraphrastic summaries.The metric may penalize summaries that express similar meaning while deviating from the reference wording.
- Word extraction: NN-WE consistently outperforms the similar-in-spirit NN-ABS model.Because NN-WE selects words from the source document, its decoding is easier than open-vocabulary generation.
- Dataset results: NN-SE performs best across all ROUGE metrics on the 500 DailyMail test articles.Scores are generally lower than on DUC 2002 because the DailyMail highlights are more laconic and involve substantial paraphrasing.
- Human evaluation: 27% of human rankings placed human-written descriptions first, compared with 22% for NN-SE.Post-hoc Tukey tests found NN-SE and ILP significantly better than LEAD, NN-WE, and NN-ABS, while not differing significantly from each other or the human gold standard.
7 Conclusions
The paper concludes that encoder-extractor models support both sentence and word extraction while learning informativeness from continuous representations without linguistic annotations. Extraction also avoids the large-vocabulary difficulties associated with rare words and named entities.
- Conclusions: The framework uses an encoder-extractor architecture with separate sentence- and word-extraction model classes.The models are trained on large-scale datasets and learn informativeness features from continuous representations.
- Conclusions: The hierarchical neural structures reflect the sentence-and-word structure of the summarization task.This design is presented as one of two important ideas underlying the framework.
- Conclusions: Extraction avoids searching over a vocabulary containing many low-frequency words and named entities.The conclusion identifies generation by extraction as the mechanism that sidesteps this difficulty.
- Future work: Future work includes adding structural information to word generation, applying the models to phrase-based settings, and developing an unsupervised selection approach.The proposed directions include tree-based algorithms and an information-theoretic perspective.
8 Appendix
The appendix reports additional evaluation on the entire 500-article DailyMail test set using three ROUGE length limits.
- Appendix: The entire 500-article DailyMail test set is evaluated with ROUGE limits of 75 bytes, 275 bytes, and full length.The task has no established evaluation standard, motivating experiments with multiple length limits.