Source-linked AI summary
Content Selection in Deep Learning Models of Summarization
Chris Kedzie, Kathleen McKeown, Hal Daume
TL;DR
Deep learning summarizers must select content effectively, but it is unclear how well their embedding-based features support that task. The paper compares neural sentence encoders and extractors across news, personal stories, meetings, and medical articles. Across domains, simpler representations and non-auto-regressive extraction often perform as well as or better than sophisticated alternatives, motivating new sentence representations and external knowledge sources.
Problem
It is not well understood how deep learning summarizers perform content selection using word- and sentence-embedding features across domains.
Method
The paper empirically compares neural sentence encoders and extractors across news, personal stories, meetings, and medical articles.
Results
Across domains, word embedding averaging and non-auto-regressive extraction are as good as or better than their more sophisticated alternatives.
Takeaways & Limitations
The findings suggest that new sentence representations or external knowledge sources better suited to summarization are needed.
Abstract
from arXiv · showhide
We carry out experiments with deep learning models of summarization across the domains of news, personal stories, meetings, and medical articles in order to understand how content selection is performed. We find that many sophisticated features of state of the art extractive summarizers do not improve performance over simpler models. These results suggest that it is easier to create a summarizer for a new domain than previous work suggests and bring into question the benefit of deep learning models for summarization for those domains that do have massive datasets (i.e., news). At the same time, they suggest important questions for new research in summarization; namely, new forms of sentence representations or external knowledge sources are needed that are better suited to the summarization task.
1 Introduction
The paper investigates how deep learning summarizers select content across news, personal stories, meetings, and medical articles. Its main results show that simpler representations and non-auto-regressive extraction often match or outperform more sophisticated alternatives.
- The study examines content selection across news, personal stories, meetings, and medical articles using several neural extractive architectures.
- Sentence position bias dominates learning in news, while other domains show less reliance on position.
- Word embedding averaging is as good or better than RNNs or CNNs for sentence embeddings across all domains.
- Pre-trained word embeddings are as good as or better than learned embeddings in five of six datasets.
- Non-auto-regressive sentence extraction performs as well as or better than auto-regressive extraction in all domains.
- The findings question whether deep learning models learn robust, meaningful content features and motivate better sentence representations or external knowledge sources.
2 Related Work
Prior work includes neural and non-neural summarization methods, mostly focused on news. This paper extends the analysis to personal narratives, workplace meetings, and medical articles while avoiding domain-specific features.
- Earlier neural summarization work developed sentence extractors using CNN or bidirectional RNN sentence representations and sequence models.
- Other neural work studied graph-convolutional structures for multi-document summarization rather than sentence-encoder choices.
- Non-neural approaches commonly learn n-gram feature weights alongside lexical and structural features.
- The paper broadens evaluation beyond news to Reddit personal stories, workplace meetings, and medical journal articles.
- The experiments deliberately avoid domain-specific features to test content selection using only surface lexical features.
3 Methods
The method models sentence extraction as sequence tagging and compares sentence encoders and extractors. It evaluates averaging, recurrent, and convolutional representations alongside independent and autoregressive prediction designs.
- Extractive summarization selects sentences under a word budget, with each sentence assigned a binary inclusion label.
- Sentence Encoders: The study compares averaging, RNN, and CNN architectures for mapping word-embedding sequences to fixed-length sentence vectors.
- Sentence Encoders: The RNN encoder concatenates final forward and backward GRU states, while the CNN encoder concatenates max-pooled convolutional outputs.
- Sentence Extractors: Earlier Cheng and Lapata and SummaRunner extractors use autoregressive dependencies through previous extraction decisions or an iteratively constructed summary representation.
- Sentence Extractors: The proposed RNN and Seq2Seq extractors make independent sentence predictions conditioned on sentence embeddings.
4 Datasets
Experiments span six corpora covering news, personal narratives, workplace meetings, and medical articles. The datasets use varied source summaries and, when needed, greedily optimized ROUGE-1 labels.
- The six corpora comprise CNN-DailyMail, New York Times, DUC, Reddit, AMI, and PubMed.
- The collection covers three news datasets plus personal narratives, workplace meetings, and medical journal articles.
- The New York Times corpus contains archival abstracts and shorter online teasers, with date-based training, validation, and test splits.
- PubMed contributes 25,000 randomly sampled medical articles, retaining articles at least 1,000 words long with abstracts of at least 50 words.
- When extractive gold labels are unavailable, the method greedily optimizes ROUGE-1 to construct label sequences.
5 Experiments
Across six datasets, the experiments find little advantage for sophisticated encoders or extractors over simpler alternatives, while sentence position strongly shapes performance in news and medical articles.
- 5.2 Baselines: The lead summary is a competitive baseline for single-document summarization, especially on newswire.It takes the first x words of each document, where x is the target summary length.
- 5.3 Results: CNN and RNN sentence encoders provide no major advantage over word embedding averaging across the evaluated datasets.The best encoder/extractor pair uses averaging on five of six datasets, or differences are not statistically significant.
- 5.3 Results: The Seq2Seq extractor is among the best systems on three of six datasets or is statistically indistinguishable from the best extractor.Across news and medical journal domains, the best-to-worst spread is small; CNN/DM spans only .56 ROUGE points, while Reddit differences are not significant and AMI configurations overlap with the best.
- 5.3 Results: In all but one case, fixed pretrained embeddings are as good as or better than embeddings learned during training.The embeddings are initialized with pretrained GloVe vectors; learning produces significantly worse results on PubMed.
- 5.3 Results: Removing word classes has small effects in news but larger effects in non-news datasets, with nouns most important on AMI and PubMed and adjectives/adverbs on Reddit.Maximum differences are .18 on CNN/DM, .41 on NYT, and .3 on DUC, versus 1.89 on Reddit, 2.56 on AMI, and 1.3 on PubMed.
- 5.3 Results: Sentence position is learned in news and medical-article domains: shuffling document order significantly reduces performance, unlike Reddit and AMI.Shuffling improves performance on AMI, while the news domains and PubMed suffer significant drops.
6 Discussion
The discussion finds that news models are strongly shaped by lead bias, while non-news domains show more evidence of content-sensitive selection. Across domains, sophisticated sentence encoders and extraction histories offer little consistent advantage, motivating new sentence representations.
- Content selection and domain differences: Shuffling reduces lead overlap to 35.2%, but overall performance drops significantly because models cannot identify important information without position.
- Content selection and domain differences: News models largely learn lead-specific stylistic features rather than robust content features.Part-of-speech ablations have relatively small effects in news, while learned news-domain embeddings underperform recognition of the lead.
- Content selection and domain differences: Non-news domains show larger part-of-speech ablation effects, and shuffling does not significantly inhibit selection or can modestly improve it.The authors note that stronger word-level representations may require much larger corpora, especially for personal stories and meetings.
- Model design implications: Word embedding averaging is difficult to outperform, while similarity computations and previous-selection histories generally do not improve extraction performance.These findings motivate rethinking sentence representations for summarization.
- Observed failure modes: News systems often ignore quoted material in lead sentences but select quote attributions that contain little information.Manual examination found no broad failure pattern beyond lead bias.
7 Conclusion
The paper empirically studies deep-learning content selection for summarization and finds stark limitations in the models’ ability to learn robust task features. It concludes that better sentence representations are needed.
- The study presents an empirical analysis of deep-learning content-selection algorithms for summarization.
- The models face stark limitations in learning robust features for summarization.
- The findings motivate further work on sentence representations for summarization.
A Details on Sentence Encoders
The encoder experiments use 200-dimensional word embeddings and apply dropout to embeddings during training. Wherever dropout is used, its probability is 0.25.
- The models use 200-dimensional word embeddings.
- Dropout is applied to the embeddings during training.
- The dropout probability is 0.25 wherever dropout is applied.
A.1 Details on RNN Encoder
The RNN sentence encoder represents each sentence by concatenating forward and backward GRU states. The two directional GRUs use separate parameters and 300-dimensional hidden layers, with dropout during training.
- The sentence embedding concatenates the final forward GRU state with the initial backward GRU state.
- The forward and backward GRUs process the sentence in opposite directions with separate parameters.
- Each GRU uses a 300-dimensional hidden layer, and dropout is applied during training.
A.2 Details on CNN Encoder
The section describes convolutional, recurrent, and sequence-to-sequence extractive summarization components, along with oracle label generation and shared training settings. The CNN encoder uses multiple filter widths and feature maps to produce a 250-dimensional sentence embedding.
- CNN encoder: The CNN encoder uses filter window sizes K = {1, 2, 3, 4, 5, 6} and feature-map sizes 25, 25, 50, 50, 50, 50, producing a 250-dimensional sentence embedding.The filters use learned bias and weight parameters with ReLU activation, and dropout is applied to the CNN output during training.
- Recurrent encoders: Bidirectional GRU encoders process sentence embeddings in both directions, with separate learned parameters and 300 hidden units per direction.The resulting directional states are concatenated into sentence representations or passed between encoder and decoder components.
- Sequence-to-sequence extractor: The unidirectional sequence-to-sequence extractor feeds each decoder the previous sentence embedding weighted by its extraction probability, making each output depend on earlier outputs.A multilayer perceptron transforms recurrent states before a logistic sigmoid produces extraction probabilities.
- SummaRunner extractor: SummaRunner combines sentence representations with document-level, position, salience, novelty, and quartile sources to determine extraction probabilities.Its iterative summary representation sums previously selected sentence representations weighted by their extraction probabilities, while position and quartile embeddings encode document location.
- Oracle extraction labels: The oracle extractor greedily adds the next sentence when it improves ROUGE while respecting a summary word budget, assigning extracted sentences label 1.It initializes an empty summary and zero labels, then stops when no candidate improves the ROUGE score or the word budget is reached.
- Training settings: All models use a learning rate of .0001, dropout rate .25, gradient clipping between −5 and 5, Xavier initialization, and batch size 32 except for AMI and PubMed.AMI and PubMed use batch sizes two and four respectively because their documents are longer and consume more memory.