Source-linked AI summary
Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
Shashi Narayan, Shay B. Cohen, Mirella Lapata
TL;DR
Existing summarization datasets and systems often favor extractive behavior, while extreme summarization requires highly compressed, abstractive summaries that identify pertinent document content. The paper introduces a BBC-based dataset and a topic-conditioned convolutional model using document-level information and long-range dependencies. The model outperforms extractive and recurrent abstractive baselines in automatic and human evaluations, though the dataset is limited to one outlet and a uniform style.
Problem
Existing large-scale summarization datasets often favor extractive models, while extreme summarization requires highly compressed summaries that distill pertinent content.
Method
The paper constructs a BBC article dataset and proposes a topic-conditioned convolutional encoder-decoder that uses word and document topic distributions.
Results
The topic-aware convolutional model outperforms an oracle extractive system and state-of-the-art RNN-based abstractive systems on ROUGE, and human evaluations find its summaries more informative and complete.
Takeaways & Limitations
The experiments support using abstractive capabilities, document topics, and long-range dependencies to recognize pertinent content and generate informative summaries.
Takeaways & Limitations
XSum focuses on one news outlet and a uniform single-sentence summarization style, so it is not diverse.
Abstract
from arXiv · showhide
We introduce extreme summarization, a new single-document summarization task which does not favor extractive strategies and calls for an abstractive modeling approach. The idea is to create a short, one-sentence news summary answering the question "What is the article about?". We collect a real-world, large-scale dataset for this task by harvesting online articles from the British Broadcasting Corporation (BBC). We propose a novel abstractive model which is conditioned on the article's topics and based entirely on convolutional neural networks. We demonstrate experimentally that this architecture captures long-range dependencies in a document and recognizes pertinent content, outperforming an oracle extractive system and state-of-the-art abstractive approaches when evaluated automatically and by humans.
1 Introduction
The paper introduces extreme summarization, a single-document task requiring short, one-sentence abstractive summaries that answer what an article is about. It builds a BBC dataset and proposes a topic-aware convolutional model that outperforms extractive and recurrent abstractive systems in automatic and human evaluations.
- Extreme summarization creates a short, one-sentence news summary answering what the article is about.The task is designed to require abstraction rather than sentence extraction.
- The task draws on information distributed across documents and requires paraphrasing, fusion, synthesis, and inference.Its summaries differ from headlines, which primarily encourage readers to read the story.
- The dataset is built by harvesting online BBC articles that often include a first-sentence summary.
- The topic-conditioned model uses convolutional encoders and decoders rather than recurrent neural networks.The encoder assigns words topic vectors, while the decoder conditions predictions on a document topic vector.
- Convolutional layers are used to capture long-range document dependencies for document-level inference, abstraction, and paraphrasing.
- The model outperforms an oracle extractive system and state-of-the-art RNN-based abstractive systems on ROUGE, while human evaluations find its summaries more informative and complete.
2 The XSum Dataset
XSum is a large BBC-based dataset of single-sentence summaries designed to reduce extractive bias. Its summaries contain substantially more novel n-grams than comparison datasets, and extractive baselines perform poorly, although the dataset is limited to one outlet and a uniform style.
- XSum pairs BBC articles with professionally written introductory sentences that serve as single-sentence summaries.These summaries are typically written by the article author and identified through the article structure.
- 226,711 BBC articles from 2010 to 2017 cover diverse domains and are split into training (90%, 204,045), validation (5%, 11,332), and test (5%, 11,334) sets.
- 36% of XSum reference unigrams are novel, compared with 17% in CNN, 17% in DailyMail, and 23% in NY Times.The authors use novel n-grams as evidence that XSum summaries are more abstractive.
- XSum summaries contain approximately 83% novel bigrams, 96% novel trigrams, and 98% novel 4-grams.Comparison datasets show around 47–55% novel bigrams, 58–72% novel trigrams, and 63–80% novel 4-grams.
- LEAD and EXT-ORACLE perform poorly on XSum, indicating that the dataset is less biased toward extractive methods.EXT-ORACLE selects the single best sentence for XSum, while LEAD selects initial document content.
- XSum is not diverse because it focuses on BBC and a uniform single-sentence summarization style.The authors nevertheless describe it as sufficiently large for neural-network training.
3 Convolutional Sequence-to-Sequence Learning for Summarization
This section presents a topic-aware convolutional encoder-decoder for extreme summarization. It combines hierarchical convolutional representations, topic information, and multi-hop attention to capture long-range dependencies and foreground pertinent content.
- Topic-Sensitive Embeddings: The model adapts convolutional sequence-to-sequence learning to summarization by incorporating topic salience into encoding and document topics into decoding.Each word receives a topic-salience vector, while every word prediction is conditioned on the document topic.
- Model Overview: Stacked convolutional layers build hierarchical document representations in which distant words interact at higher layers, capturing long-range dependencies.Lower layers model nearby interactions, while higher layers combine information over larger contexts.
- Multi-hop Attention: The decoder uses hierarchical convolutions and multi-hop attention to select source context while tracking previously attended words.Each decoder layer attends to encoder representations before passing its output upward.
- Topic-Sensitive Embeddings: The encoder combines word embeddings, absolute position embeddings, local word-topic distributions, and the global document-topic distribution.Local topic distributions capture a word’s topicality, whereas the document distribution represents the overall theme.
- Convolutional Sequence Modeling: The architecture retains convolutional computational advantages because each convolution block operates over a fixed-size input window and supports parallel computation.Its topic enhancement is intended to combine globally salient content with efficient long-range modeling.
4 Experimental Setup
The experiments evaluate T-CONVS2S on XSum against extractive and abstractive systems. The setup uses standardized one-line summaries, specified preprocessing and optimization choices, and LDA-derived topic inputs.
- Experimental Setup: The experiments assess the Topic-aware Convolutional Sequence-to-Sequence model, abbreviated T-CONVS2S, against comparison systems on XSum.All systems generate a one-line summary from an input news article.
- Comparison Systems: The comparison includes RANDOM, LEAD, and EXT-ORACLE extractive baselines, alongside neural abstractive systems.EXT-ORACLE selects a single best sentence and is commonly used as an upper bound for extractive methods.
- Preprocessing: The XSum inputs were lowercased, truncated to 400 tokens, and paired with summaries limited to 90 tokens.Entities were not anonymized during preprocessing.
- Topic Modeling: LDA topic distributions were supplied as model inputs, with 512 topics producing the best held-out-data results.The LDA model was trained on the XSum training documents, and document topics could be inferred at training and test time.
- Optimization and Decoding: All neural models used 50,000-word vocabularies, single-GPU training, batch size 32, and beam search with beam size 10 at test time.The convolutional models used 512-dimensional hidden states and word and position embeddings.
5 Results
On XSum, convolutional abstractive systems outperform extractive and RNN-based alternatives across automatic and human-centered evaluations. Topic conditioning further improves the convolutional model, while the outputs remain strongly abstractive and preserve more answerable information.
- Automatic evaluation: ROUGE evaluates informativeness through unigram and bigram overlap and fluency through longest common subsequence.The reported metrics are ROUGE-1, ROUGE-2, and ROUGE-L F1 scores.
- Automatic evaluation: CONVS2S significantly outperforms all RNN-based abstractive systems.The authors associate this result with convolutional representations of long-range document dependencies.
- Automatic evaluation: All four T-CONVS2S variants outperform CONVS2S, with the best variant constraining both encoder and decoder by the document topic.This variant is called T-CONVS2S in the remainder of the paper.
- Abstractive rewriting: Convolutional models produce the highest proportion of novel n-grams among the evaluated systems.PTGEN trained on XSum copies 4% of source 4-grams, 10% of trigrams, 27% of bigrams, and 73% of unigrams.
- Human evaluation: Human judgments rank T-CONVS2S second behind gold summaries, while PTGEN ranks last with a score of −0.218.The ranking continues with EXT-ORACLE and CONVS2S.
- Human evaluation: 46.05% of questions are answered correctly from T-CONVS2S summaries, compared with 30.90% for CONVS2S, 21.40% for PTGEN, and 15.70% for EXT-ORACLE.All pairwise differences are statistically significant at p < 0.01 except PTGEN versus EXT-ORACLE.
- Human evaluation: ROUGE-selected extractive sentences can contain the correct answer yet omit the context needed to answer document questions.T-CONVS2S extracts important entities with the right theme, whereas CONVS2S and PTGEN can fail despite containing “Dick Advocaat.”
6 Conclusions
The study concludes that extreme summarization favors abstractive models and depends on document-level topic knowledge and long-range dependencies. Future work will incorporate co-reference and entity linking into linguistically aware encoders and decoders.
- Abstractive models perform better on the extreme summarization task.
- High-level document knowledge, including topics and long-range dependencies, is critical for recognizing pertinent content and generating informative summaries.
- Future work will develop linguistically aware encoders and decoders incorporating co-reference and entity linking.