Source-linked AI summary
A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss
Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, Min Sun
TL;DR
Extractive summarization can select important content but may be less readable, while abstractive summarization can be readable yet lose or mistake facts. The paper unifies both through cross-level attention modulation and inconsistency loss, achieving state-of-the-art ROUGE scores and the strongest reported human-evaluation informativity and readability on CNN/Daily Mail.
Problem
Extractive summaries can be less readable, whereas abstractive summaries may lose or mistake factual details.
Method
The model combines sentence-level and word-level attention, modulates word attention using sentence attention, and trains with inconsistency loss.
Results
The model achieves state-of-the-art ROUGE scores and is reported as the most informative and readable summarization on CNN/Daily Mail in human evaluation.
Takeaways & Limitations
The inconsistency loss enables extractive and abstractive summarization to be mutually beneficial.
Abstract
from arXiv · showhide
We propose a unified model combining the strength of extractive and abstractive summarization. On the one hand, a simple extractive model can obtain sentence-level attention with high ROUGE scores but less readable. On the other hand, a more complicated abstractive model can obtain word-level dynamic attention to generate a more readable paragraph. In our model, sentence-level attention is used to modulate the word-level attention such that words in less attended sentences are less likely to be generated. Moreover, a novel inconsistency loss function is introduced to penalize the inconsistency between two levels of attentions. By end-to-end training our model with the inconsistency loss and original losses of extractive and abstractive models, we achieve state-of-the-art ROUGE scores while being the most informative and readable summarization on the CNN/Daily Mail dataset in a solid human evaluation.
1 Introduction
Text summarization condenses text while preserving important points, with extractive methods favoring simplicity and abstractive methods favoring readable, concise generation. The paper unifies both approaches by combining sentence- and word-level attention and adding inconsistency loss.
- Text summarization condenses text while maintaining important points for applications including news digests, search results, and reports.
- Extractive methods select source sentences and are typically simpler, whereas abstractive methods can generate novel words and phrases.
- Abstractive summaries can be more coherent and concise than extractive summaries, but abstractive systems may inaccurately reproduce factual details.
- The unified model uses sentence-level attention to modulate word-level attention, making words in less attended sentences less likely to be generated.
- The proposed inconsistency loss encourages consistency between attention levels without requiring additional human annotation.
- The model achieves the best ROUGE scores on CNN/Daily Mail and outperforms recent state-of-the-art methods in informativity and readability in human evaluation.
2 Related Work
Related work covers neural extractive and abstractive summarization, including mechanisms for handling unknown words and repetition. The paper combines sentence-level attention from an extractive model with word-level attention from an abstractive model and adds inconsistency loss.
- Extractive summarization: Prior work includes neural models that represent sentences and select them for extractive summarization.
- Abstractive summarization: Pointer-generator and related models address out-of-vocabulary words and repeated phrases in generated summaries.
- Unified model: The proposed model combines sentence-level attention from Nallapati et al. (2017) with word-level attention from See et al. (2017).
- Unified model: An inconsistency loss is introduced to enhance cooperation between the extractive and abstractive models.
3 Our Unified Model
The unified model combines sentence-level extraction with word-level abstractive generation, using sentence attention to modulate word attention and an inconsistency loss to align them during training.
- 3 Our Unified Model: The model combines sentence-level attention from an extractor with dynamic word-level attention from an abstracter.The extractor produces sentence probabilities, while the abstracter generates summary text word by word.
- 3.1 Combining Attentions: Sentence-level attention modulates word-level attention so words in less attended sentences are less likely to be generated.The attentions are combined by scalar multiplication and renormalization; high word attention requires both levels to be high.
- 3.2 Inconsistency Loss: The inconsistency loss encourages sentence attention to be high when word attention is high and makes the two attention levels mutually beneficial.It uses the top K attended words and summary length T, while the original extractor, abstracter, and coverage losses prevent degenerate attention distributions.
- 3.3 Extractor: The extractor uses a hierarchical bidirectional GRU and predicts sentence-level attention, with labels selected to favor highly informative sentences.Ground-truth labels are constructed using ROUGE-L recall against the reference abstractive summary, prioritizing sentences that increase collective informativity.
- 3.4 Abstracter: The updated word attention affects decoding by producing a context vector that updates the final word distribution.The abstracter uses a pointer-generator network that can copy article words or generate words from a fixed vocabulary, including OOV words.
- 3.5 Training Procedure: End-to-end training minimizes extractor, abstractive, coverage, and inconsistency losses together.The authors also pre-train both components; during end-to-end training, sentence attention is soft and the extractor loss may receive a larger weight.
4 Experiments
The experiments use the CNN/Daily Mail news dataset and describe the model’s training and implementation settings, including pre-training and two-stage training.
- The CNN/Daily Mail dataset contains news stories paired with human-written multi-sentence summaries, with anonymized and non-anonymized versions.
- The extractor and abstracter use 128-dimensional word embeddings, vocabularies of 50k, hidden dimensions of 200 and 256, Adagrad, early stopping, and summaries limited to 120 tokens.
- Pre-training: During pre-training, the extractor runs for 27k iterations, while the abstracter uses ground-truth extracted sentences and trains with and without coverage mechanisms.
- Two-stages training: During two-stage training, the abstracter receives sentences whose pre-trained extractor probabilities βn exceed 0.5 before fine-tuning.
5 Results
The experiments evaluate extracted sentences, generated summaries, attention consistency, and human judgments. End-to-end training with inconsistency loss achieves strong ROUGE results, reduces inconsistency, and improves reported human-evaluation quality.
- Extracted sentences: The extractor is evaluated using full-length ROUGE recall between selected sentences and reference abstractive summaries, selecting sentences with β greater than 0.5.
- Abstractive summarization: The two-stage model outperforms the pointer-generator model on ROUGE-1 and ROUGE-2, while end-to-end training with inconsistency loss exceeds the lead-3 baseline.
- Extracted sentences: The ground-truth extracted sentences are an upper bound but cannot reach 100% recall because abstractive references may contain words absent from the article.
- Extracted sentences: The end-to-end extractor performs best when trained with inconsistency loss.
- Abstractive summarization: The end-to-end model without inconsistency loss already improves ROUGE through cooperation between the extractor and abstracter, while inconsistency loss yields state-of-the-art ROUGE scores.
- Attention consistency: The inconsistency rate counts decoder steps whose maximally attended word belongs to a sentence with below-mean sentence attention.
- Attention consistency: 20% to 4%: inconsistency loss significantly decreases the average inconsistency rate on the test set.
- Human evaluation: Human evaluation scores informativity, conciseness, and readability on a 1-to-5 scale, using a random-summary trap to reject unreliable evaluations.
6 Conclusion
The paper concludes that its unified model combines extractive and abstractive summarization, with inconsistency loss aligning their attentions. End-to-end training achieves strong ROUGE performance and favorable human-evaluation results on CNN/Daily Mail.
- The unified model combines the strengths of extractive and abstractive summarization through sentence-level and word-level attention.
- The inconsistency loss penalizes disagreement between the two attention levels and enables extractive and abstractive summarization to be mutually beneficial.
- End-to-end training achieves the best reported ROUGE-recall and ROUGE results while producing the most informative and readable summaries in the CNN/Daily Mail human evaluation.