Source-linked AI summary
Controllable Abstractive Summarization
Angela Fan, David Grangier, Michael Auli
TL;DR
Existing summarization models do not let readers specify preferences such as length, entities, style, or reading progress. The paper introduces a controllable neural summarizer using input controls for these attributes, and reports improved ROUGE and human-evaluation results on CNN/DailyMail.
Problem
Existing document summarization models disregard reader preferences for summary length, style, entities of interest, and already-read content.
Method
The model conditions abstractive summarization on input markers representing desired length, entities, source style, and document portions to summarize.
Results
+0.85 F1-ROUGE1 on full-text CNN-Dailymail and +0.76 F1-ROUGE1 on entity-anonymized data outperform previous pointer-based models, with human readers preferring the outputs.
Takeaways & Limitations
Control variables follow specified preferences and improve generation even when automatically assigned during inference.
Takeaways & Limitations
The paper reports that source-style control is least impactful in ROUGE and may benefit from datasets with richer style variation.
Abstract
from arXiv · showhide
Current models for document summarization disregard user preferences such as the desired length, style, the entities that the user might be interested in, or how much of the document the user has already read. We present a neural summarization model with a simple but effective mechanism to enable users to specify these high level attributes in order to control the shape of the final summaries to better suit their needs. With user input, our system can produce high quality summaries that follow user preferences. Without user input, we set the control variables automatically. On the full text CNN-Dailymail dataset, we outperform state of the art abstractive systems (both in terms of F1-ROUGE1 40.38 vs. 39.53 and human evaluation).
1 Introduction
The paper introduces controllable abstractive summarization, allowing readers to shape summaries according to their preferences and reading progress. Its model follows specified controls and improves benchmark and human-evaluation results.
- Readers can control summary length, focus on entities, select source style, and summarize only unread portions of an article.
- The model builds on sequence-to-sequence encoder-decoder systems that generate summaries by attending to document representations.
- +0.76 F1-ROUGE1 on entity-anonymized data and +0.85 F1-ROUGE1 on full-text CNN/DailyMail outperform previous pointer-based maximum-likelihood models.
- Human readers preferred the model’s summaries in a blind evaluation study.
2 User Controllable Summarization
The model controls summaries through input markers encoding desired length, entities, source style, and document remainder. Experiments show these controls affect generation and that using both read and unread text benefits remainder summarization.
- Model architecture: The convolutional encoder-decoder uses source attention, multi-hop attention across decoder layers, and intra-attention over previously generated words.
- Model architecture: Shared encoder-decoder representations and BPE tokenization provide an alternative to pointer networks for handling rare entities.
- Length-Constrained Summarization: Length control quantizes summaries into bins and prepends a marker identifying the desired length during training and inference.
- Length-Constrained Summarization: Changing the length marker easily controls output length, while ground-truth markers substantially improve summary quality.
- Entity-Centric Summarization: Entity markers request entities in the output, and supplying more entities improves summarization quality.
- Source-Specific and Remainder Summarization: Source-style markers condition generation on the desired source, while remainder summarization uses aligned document-summary data and read/remainder position distinctions.
- Remainder Summarization: The read-and-remainder model outperforms the compared remainder-summarization methods.
3 Related Work
The work extends controllable text generation into abstractive summarization while drawing on convolutional, attention-based, and subword techniques. It favors conditional language modeling over more complex adversarial, latent-variable, or pointer-based approaches.
- Neural Summarization: Prior summarization research uses sequence-to-sequence models but must address repetition caused by long-range dependence across generated sentences.
- Positioning of This Work: The paper uses convolutional networks for faster training, multi-hop intra-attention, shared word representations, and BPE tokenization.
- Controllable Generation: Controllable generation extends unconditioned generation by steering outputs such as tense, sentiment, politeness, length, style, and domain.
- Positioning of This Work: The approach relies on conditional language modeling without adversarial training, variational auto-encoders, or pointer networks.
4 Experimental Setup
Experiments evaluate controllable summarization primarily on CNN/DailyMail, with an additional length-control comparison on DUC-2004. Automatic metrics, baseline comparisons, and human preferences provide the evaluation framework.
- Datasets: CNN/DailyMail contains 287k training, 13k validation, and 11k test articles, averaging 758 tokens per article and 55 tokens per summary.
- Datasets: Most CNN/DailyMail experiments truncate articles at 400 tokens and evaluate entity-anonymized and full-text versions.
- Datasets: The DUC-2004 length-control experiment trains on 3.6 million English Gigaword article-headline pairs and evaluates 500 task-1 documents.
- Architecture and Training: CNN/DailyMail models use eight encoder and decoder layers with 512 hidden units, 340-dimensional embeddings, and dropout 0.2.
- Architecture and Training: Summaries use beam search with beam size 5 and prevent repeated trigrams during decoding.
- Evaluation: Evaluation reports F1-ROUGE-1, ROUGE-2, and ROUGE-L against abstractive baselines and Lead-3, while human raters choose preferred outputs.
- Evaluation: DUC-2004 reports recall ROUGE-1, ROUGE-2, and ROUGE-L at 30, 50, and 75 byte lengths.
5 Results
The experiments show that controllable summarization improves quality while allowing users to specify summary length, entities, and source style. Automatic controls also improve generation, including remainder summarization and benchmark performance.
- Controllable Summarization: Control variables improve summary quality, with length control contributing most, followed by entity and source-style control.Combining control variables yields +2.2 F1-ROUGE1.
- Controllable Summarization: 1.68 F1-ROUGE1: length control improves accuracy because length mismatch is penalized and baseline length prediction is uncertain.The model respects the 10 possible length markers when decoding test examples.
- Controllable Summarization: +0.69 vs. +1.68 F1-ROUGE1: entity control has less ROUGE impact than length control, partly because summaries already contain many reference entities.The model has difficulty generating summaries containing entities unlikely to appear in human references.
- Controllable Summarization: +0.2 F1-ROUGE1: source-style control is least impactful in ROUGE, while changing the style variable changes generated summaries.Dailymail-style summaries are generally more repetitive and slightly longer than CNN-style summaries.
- Automatic Control: 40.38 vs. 39.53 F1-ROUGE1: on original CNN-DailyMail text, the model outperforms the cited abstractive baseline.On entity-anonymized text, it reports 39.06 versus 38.30 for the cited maximum-likelihood setting.
- Remainder Summarization: 30.70 F1-ROUGE1: remainder summarization improves when models receive only the remainder and use partition-specific length control.The baseline scores 28.12, while remainder-only models score 30.30 before the additional length-control improvement.
- Human Evaluation: 59%: human raters prefer the model with fixed control variables in about 2.5k judgments.ROUGE and human ratings agree in two-thirds of cases where at least four of five humans agree.
6 Conclusion
The paper proposes a controllable summarization model that lets users specify high-level attributes including length, source style, entities of interest, and which document portion to summarize. These controls work with user input and can also be assigned automatically, outperforming comparable state-of-the-art systems in ROUGE and human evaluation.
- The model lets users control summary length, source style, entities of interest, and whether to summarize only the document’s remaining portion.
- The authors simulate user preferences with oracle control values and report large ROUGE gains.
- Fixed control values tuned on held-out data remain effective without user input.
- The controllable model outperforms comparable state-of-the-art summarization models in both ROUGE and human evaluation.