Source-linked AI summary
Improving Abstraction in Text Summarization
Wojciech Kryściński, Romain Paulus, Caiming Xiong, Richard Socher
TL;DR
Abstractive summarization seeks concise, readable summaries, but existing systems often produce limited abstraction by copying source text. The paper separates extraction from generation with a contextual network and pretrained language model, and directly rewards novelty through policy learning. Its model maintains ROUGE performance comparable to or above state of the art while producing substantially more abstractive summaries than prior approaches.
Problem
Existing abstractive summarization approaches often copy source passages, while word-overlap metrics do not fully capture the paraphrasing expected of abstractive summaries.
Method
The model separates source extraction from paraphrase generation and combines maximum-likelihood training with policy learning that rewards ROUGE overlap and novel words.
Results
The model achieves state-of-the-art ROUGE-L, comparable ROUGE-1 and ROUGE-2, and significantly higher abstraction than previous abstractive approaches.
Takeaways & Limitations
The approach improves summary abstraction while maintaining high ROUGE performance close to or above the state of the art.
Abstract
from arXiv · showhide
Abstractive text summarization aims to shorten long text documents into a human readable form that contains the most important facts from the original document. However, the level of actual abstraction as measured by novel phrases that do not appear in the source document remains low in existing approaches. We propose two techniques to improve the level of abstraction of generated summaries. First, we decompose the decoder into a contextual network that retrieves relevant parts of the source document, and a pretrained language model that incorporates prior knowledge about language generation. Second, we propose a novelty metric that is optimized directly through policy learning to encourage the generation of novel phrases. Our model achieves results comparable to state-of-the-art models, as determined by ROUGE scores and human evaluations, while achieving a significantly higher level of abstraction as measured by n-gram overlap with the source document.
1 Introduction
The paper targets abstractive summaries that avoid copying long source passages while preserving important information and word-overlap performance. It introduces a decoder decomposition and a novelty-oriented training objective, achieving strong ROUGE results and greater abstraction than prior approaches.
- 1 Introduction: Existing abstractive summarization models achieve high word-overlap scores but often copy long source passages instead of generating genuinely abstractive summaries.This motivates measuring and improving abstraction beyond standard ROUGE evaluation.
- 1 Introduction: The decoder is factored into a contextual extraction network and a language model that generates concise paraphrases.This separates source-document extraction from generation and supports preserving word-overlap performance while increasing abstraction.
- 1 Introduction: A mixed objective combines maximum-likelihood learning with policy gradients that reward both ROUGE overlap and novel words.The novelty reward encourages words absent from the source document.
- 1 Introduction: State-of-the-art ROUGE-L and comparable ROUGE-1 and ROUGE-2 performance accompany significantly higher abstraction than previous abstractive approaches on CNN/DailyMail.Table 1 also shows less copying and more abstraction in the generated summaries.
2 Model
The model separates source extraction from language generation and combines maximum-likelihood, ROUGE-L policy learning, and novelty rewards. Its contextual network extracts and compacts the source, while an external language model generates vocabulary-based paraphrases.
- 2.1 Base Model and Training Objective: The base architecture uses a bidirectional LSTM encoder with temporal and intra-attention over source and previously decoded states.Temporal attention penalizes repeatedly attended input tokens, while intra-attention attends to earlier decoder states.
- 2.1 Base Model and Training Objective: The decoder combines pointer-based copying from the source document with selection from a fixed output vocabulary.The model assigns probabilities to generating vocabulary words and copying source words through temporal attention.
- 2.1 Base Model and Training Objective: The training objective mixes maximum-likelihood estimation with policy learning, using ROUGE-L and greedy decoding as a self-critical baseline.The final loss is weighted by hyperparameter γ, while policy learning uses sampled summaries and greedy predictions for variance reduction.
- 2.2 Language Model Fusion: The decoder is factorized into contextual and language models, separating source extraction from fixed-vocabulary generation.The contextual network focuses on attention and extraction, while the external language model generates concise paraphrases and can incorporate pretrained fluency or domain knowledge.
- 2.3 Abstractive Reward: The novelty metric rewards summary n-grams absent from the source document and is normalized using the generated-to-ground-truth length ratio.The metric is incorporated alongside ROUGE-L so policy learning encourages both overlap with human summaries and novel wording.
3 Experiments
Experiments use both anonymized and full-text CNN/Daily Mail versions, with preprocessing aligned to prior work where possible. The study also evaluates a random-word novelty baseline and trains language models on reference summaries.
- Datasets: Both anonymized and full-text CNN/Daily Mail versions are used to compare performance across differing dataset formats.The authors note that format differences make direct comparison with prior results difficult.
- Datasets: Named entities, out-of-vocabulary words, and source-document numbers supervise when the model points to the source rather than generates.This supervision uses named-entity lists from Hermann et al. (2015).
- Implementation: The language model uses a 400-dimensional embedding layer and a 3-layer LSTM, while the main model limits articles to 400 tokens and summaries to 100 tokens.The output vocabulary is limited to 50,000 tokens and scheduled sampling uses probability 0.25.
- Language-model training: The language model is trained on CNN/Daily Mail ground-truth summaries using the same training, validation, and test splits as the main experiments.
- Baselines: The novelty baseline inserts random out-of-article words after summary tokens with probability r = 0.0005, increasing novelty while keeping outputs similar.It is constructed from base-model outputs without reinforcement learning or the language model.
4 Results
The final model preserves strong ROUGE performance while producing more novel n-grams than prior abstractive systems. Ablations and trade-off analyses attribute this balance to combining the language model with mixed ROUGE-and-novelty reinforcement learning, while human ratings remain competitive.
- 4.1 Quantitative analysis: State-of-the-art ROUGE-L and substantially higher novelty than prior abstractive approaches are achieved across CNN/Daily Mail dataset versions.ROUGE-1 and ROUGE-2 are close to state-of-the-art, while human summaries remain more novel.
- 4.2 Ablation study: The language model plus mixed ROUGE-and-novelty rewards yields higher ROUGE scores and more novel unigrams than the corresponding model without the language model.The result supports combining decoder decomposition with a novelty reward.
- 4.3 ROUGE vs novelty trade-off: The final model offers the best ROUGE-1-to-novel-unigram Pareto trade-off and one of the best ROUGE-2-to-novel-bigram trade-offs.The same model without the language model produces more novel bigrams but has lower ROUGE-2.
- 4.3 ROUGE vs novelty trade-off: ROUGE and novelty scores are inversely correlated across model types, making simultaneous optimization of both metrics difficult.
- 4.4 Qualitative evaluation: Human evaluators found relevance matching prior systems, while readability was slightly lower than their scores.Five evaluators rated 100 full-text test summaries on both criteria.
5 Related work
Related work distinguishes extractive and abstractive summarization, surveys datasets and training strategies, and situates methods that optimize generation quality or evaluation metrics. The field includes both sequence-generation and adversarial approaches.
- Text summarization: Extractive systems select and combine source passages, whereas abstractive systems paraphrase them with phrases absent from the source.Extractive methods are generally robust and coherent but cannot create new paraphrasing expressions.
- Text summarization: Attention-based sequence-to-sequence models underpin state-of-the-art abstractive summarization, with self-attention and coverage extensions addressing generation quality.
- Datasets: CNN/Daily Mail supports longer, multi-sentence summaries up to 100 words, complementing shorter headline-generation datasets such as Gigaword and DUC.The New York Times dataset is another benchmark for long summaries.
- Training strategies for sequential models: Maximum-likelihood teacher forcing is common for sequence generation, while reinforcement learning directly optimizes evaluation metrics to mitigate objective mismatch.The cited literature connects policy-learning methods with improvements in image captioning and abstractive summarization.
- Training strategies for sequential models: Generative adversarial approaches add a discriminator that distinguishes natural from generated outputs to guide generation toward human-like text.
6 Conclusions
The model combines an external language model with a reinforcement-learning reward to encourage more abstractive summaries. On CNN/Daily Mail, it produces substantially more abstraction than prior approaches while retaining high ROUGE scores, though it remains below human abstraction levels.
- The model generates much more abstractive summaries than previous approaches while maintaining high ROUGE scores close to or above state of the art.Experiments were conducted on the CNN/Daily Mail dataset.
- The approach uses an external language model in the decoder and a new reinforcement-learning reward that encourages summary abstraction.
- Human summaries remain far ahead in novel n-grams, leaving the gap to human-level abstraction as an open direction for future work.The paper also suggests mechanisms that promote paraphrase generation as a possible direction.