Source-linked AI summary
Bottom-Up Abstractive Summarization
Sebastian Gehrmann, Yuntian Deng, Alexander M. Rush
TL;DR
Neural abstractive summarizers are fluent but can select content poorly. The paper uses a data-efficient content selector as bottom-up attention to constrain copying, yielding stronger compression and ROUGE improvements across CNN-DM and NYT while retaining fluent generation. The selector can also adapt a trained summarizer using only 1,000 in-domain sentences.
Problem
Neural abstractive summarizers produce fluent outputs but have mixed success selecting what content to summarize.
Method
A separate sequence-tagging content selector identifies likely summary phrases, and bottom-up attention masks copying to those selected source tokens.
Results
The combined system improves ROUGE scores by over two points on both CNN-DM and NYT, with a CNN-DM-to-NYT transfer improvement of over 5 ROUGE-L points using 1,000 in-domain sentences.
Takeaways & Limitations
The two-step method provides a data-efficient way to adjust a trained summarizer for a new domain while retaining the fluency advantages of abstractive summarization.
Takeaways & Limitations
The abstractive system requires the full source document, and distilling content-selector output into abstractive-model inputs caused a drastic performance decrease.
Abstract
from arXiv · showhide
Neural network-based methods for abstractive summarization produce outputs that are more fluent than other techniques, but which can be poor at content selection. This work proposes a simple technique for addressing this issue: use a data-efficient content selector to over-determine phrases in a source document that should be part of the summary. We use this selector as a bottom-up attention step to constrain the model to likely phrases. We show that this approach improves the ability to compress text, while still generating fluent summaries. This two-step process is both simpler and higher performing than other end-to-end content selection models, leading to significant improvements on ROUGE for both the CNN-DM and NYT corpus. Furthermore, the content selector can be trained with as little as 1,000 sentences, making it easy to transfer a trained summarizer to a new domain.
1 Introduction
Neural abstractive summarizers generate fluent text but struggle with content selection. The paper adds a data-efficient bottom-up selector that constrains copying to likely summary phrases, improving compression and ROUGE while preserving fluency.
- Neural abstractive models produce fluent summaries but have mixed success deciding what content to summarize.
- With bottom-up attention, summaries show more explicit sentence compression, whereas models without it copy whole sentences verbatim.
- Bottom-up attention first selects a source-document mask, then constrains a standard neural summarizer with that mask.
- The content selector treats selection as sequence tagging and identifies summary tokens with over 60% recall and over 50% precision.
- 38.3 ROUGE-L versus 36.4 for the See et al. (2017) baseline on CNN-DM, while bottom-up attention is simpler to train.
- A content selector trained on only 1,000 in-domain sentences improves a CNN-DM-trained model by over 5 ROUGE-L points on NYT.
2 Related Work
Prior summarization work spans select-and-compress systems, extractive sentence or phrase selection, neural abstractive models, reinforcement learning, and multi-pass architectures. These approaches trade off source fidelity, compression, grammaticality, training complexity, and content selection.
- Select-and-compress systems extract phrases or sentences and then shorten them, combining source fidelity with compressive modification.
- Neural extractive systems commonly select and order full sentences, but word-level extraction can over-extract or produce ungrammatical output.
- Neural abstractive sequence-to-sequence models generate summaries while copy mechanisms combine source-word copying with abstraction.
- Reinforcement-learning approaches optimize alternative summarization objectives, but RL-based training can be difficult to tune and slow to train.
- Multi-pass extractive-abstractive systems first create or identify important source content and then use an abstractive model to generate or order the summary.
- Other content-selection methods use hierarchical attention, keywords, loss functions, or sentence-level selection to guide summarization.
3 Background: Neural Summarization
Neural summarization models generate a shorter target sequence from source tokens, using attentional sequence-to-sequence decoding one word at a time. Copy models combine source copying and word generation through a latent binary switch.
- The framework represents each example as source tokens x_1,...,x_n and summary tokens y_1,...,y_m, with m much smaller than n.
- An attentional sequence-to-sequence model generates the summary one word at a time while conditioning on previously generated words.
- At decoding step j, the attention distribution p(a_j|x,y_1:j-1) acts as a soft focus over source tokens.
- The copy mechanism predicts a binary switch z_j that determines whether the decoder copies from the source or generates a word.
- The joint output distribution combines copy and generation distributions as a convex combination weighted by the switch probabilities.
- The pointer-generator model reuses attention as its copy distribution and trains with marginal likelihood over the latent switch variable.
4 Bottom-Up Attention
Bottom-up attention uses a separately trained word-level content selector to identify likely summary tokens, then masks the pointer-generator’s copy attention to constrain copying. This two-step design preserves fluency while improving content selection and remaining simple and data-efficient.
- 4.1 Content Selection: The content selection task treats each source token as a binary sequence-tagging decision indicating whether it is copied into the target summary.Training labels are generated by aligning summaries with source documents and identifying copied tokens.
- 4.1 Content Selection: A bidirectional LSTM content selector combines static and contextual token embeddings to estimate each word’s selection probability.The selector computes q_i from the contextual representation h_i using trainable parameters.
- 4.1 Content Selection: The selector remains data-efficient because fine-tuning contextual embeddings adds only four trainable parameters to the tagger.The contextual representation is a learned linear combination of LSTM states and the token embedding.
- 4.2 Bottom-Up Copy Attention: Masking targets a common copy-model error: copying excessively long sequences, which occur among over 50% of baseline copied tokens but only 10% of reference-summary tokens.The authors retain a standard full-text encoder and limit bottom-up processing to attention masking.
- 4.2 Bottom-Up Copy Attention: At inference, selection probabilities produce a hard threshold mask that restricts the pointer-generator’s copy attention to selected source tokens.The masked distribution is normalized before replacing the model’s copy probabilities.
- 4.3 End-to-End Alternatives: The two-step system is simpler to train than end-to-end alternatives, which include fixed training-time masks, shared multi-task training, and differentiable predicted masks.End-to-end variants differ in whether masking is applied only during training, jointly through a shared encoder, or softly during training and testing.
5 Inference
The inference procedure modifies beam-search scoring to address long-form generation’s length and repetition problems. It combines length normalization with a coverage penalty and a no-repeated-trigram constraint.
- 5 Inference: Inference scoring adds a length penalty and a coverage penalty to the model’s log probability.The score is defined as s(x, y) = log p(y|x)/lp(x) + cp(x; y).
- Length: Increasing the length-penalty parameter α encourages longer generated summaries, alongside a training-data-based minimum length.Length normalization is applied during beam search.
- Repeats: The coverage penalty increases when cumulative attention to one encoded token exceeds 1.0, discouraging repeated phrases.A sufficiently high β blocks summaries that would produce repetitions.
- Repeats: Beam search additionally forbids repeated trigrams to reduce recurring text in generated summaries.This constraint follows the approach used by Paulus et al. (2017).
6 Data and Experiments
The experiments evaluate bottom-up attention on the CNN-DM and NYT news-summarization corpora using established abstractive baselines and attention-masking variants. The setup includes controlled inference tuning, implementation comparisons, and reported reproduction resources.
- Datasets: The evaluation uses CNN-DM and NYT, two standard news-summarization corpora with different summary formats and lengths.CNN-DM contains website bullet-point summaries averaging 66 tokens and 4.9 bullet points; NYT summaries were written by library scientists.
- Implementation: The content selector uses 100-dimensional GloVe and 1024-dimensional ELMo embeddings with a two-layer bidirectional LSTM of hidden size 256.Training uses dropout 0.5 and limits examples to 100,000 per corpus with little reported performance impact.
- Implementation: The base system reimplements the Pointer-Generator model with a one-layer bidirectional encoder and a one-layer decoder.The encoder uses 256 hidden states per direction, while the decoder uses 512 hidden states.
- Results: Table 1 organizes CNN-DM results into cross-entropy abstractive baselines, reinforcement-learning methods, and the authors’ baselines and attention-masking methods.The table caption defines these three result sections.
- Implementation: The CopyTransformer uses one randomly selected attention head as its copy distribution and otherwise follows the big Transformer configuration.Larger models and Transformers slightly improve performance but increase training time and parameter count.
- Inference Setup: Inference parameters are tuned on a 200-example validation subset, with bottom-up attention using beam size 10 versus beam size 5 for Pointer-Generator.The mask threshold ranges from 0.1 to 0.2, while the length penalty parameter ranges from 0.6 to 1.4.
- Reproducibility: Code and reproduction instructions are provided in the paper’s cited public repository.The repository URL is included as a footnote.
- Dataset Version: The reported comparison uses the non-anonymized CNN-DM version used by See et al. (2017), distinct from results on the anonymized version.The footnote separately reports the best anonymized-version scores as R1:41.69, R2:19.47, and RL:37.92.
7 Results
Bottom-up attention improves summarization results across CNN-DM and NYT, while remaining data-efficient for domain transfer.
- Bottom-up attention produces a major improvement across all three CNN-DM scores, unlike end-to-end masking models.The end-to-end models did not improve, while bottom-up attention did.
- Bottom-up attention outperforms the reinforcement-learning model in ROUGE-1 and ROUGE-2 on NYT, but not ROUGE-L.
- 52.8% copied-word precision versus 50.0% for the best Pointer-Generator models, with average summary length decreasing from 13 to 12 words.The copied-word precision improvement is statistically significant with t=14.7 (p < 10^-5).
- 74 AUC is achieved with only 1,000 CNN-DM sentences, after which additional training data yields only slight increases.
- On NYT domain transfer, the smallest content-selector subset improves performance by almost 5 points, increasing to up to 7 points with larger subsets.The transferred model remains below models trained directly on NYT but produces readable summaries.
8 Analysis and Discussion
The analysis examines how bottom-up attention changes copying, extractive behavior, novel-word composition, and inference-time performance. It improves copied-phrase distributions and summary scores, but generated phrases remain unlike reference summaries and the abstractive model remains necessary for fluency.
- Extractive analysis: Top-3 extraction places only 7.1% of selected sentence sets outside the first three sentences, reinforcing the strength of the LEAD-3 baseline.The naive sentence extractor performs slightly worse than a specialized extractive system that scores sentence combinations.
- Extractive analysis: The content selector identifies important words effectively but is less effective at chaining them together, reflected in lower ROUGE-2 and reduced fluency.An ungrammatical example receives ROUGE-1 of 29.3; the combined system uses the abstractive model to chain selected content fluently.
- Inference penalty analysis: All three inference penalties improve all three reported scores, including when added on top of the other two penalties.This suggests the unmodified Pointer-Generator represents the task appropriately but is limited by ineffective inference behavior.
9 Conclusion
The paper presents a data-efficient content selector that identifies likely summary phrases and uses bottom-up attention to restrict copying. Across CNN-DM and NYT, the combined system improves ROUGE while supporting adaptation to a new domain with few additional data points.
- Conclusion: The content selector identifies document phrases likely to appear in summaries and supplies bottom-up attention that restricts source copying.The combined system improves ROUGE scores by over two points on both CNN-DM and NYT.
- Conclusion: The technique’s data-efficiency enables adjustment of a trained summarizer with few data points for transfer to a new domain.The paper also reports preliminary promise for related content-selection tasks such as grammar correction and data-to-text generation.
A Domain Transfer Examples
In CNN-DM-to-NYT transfer, content selection substantially improves a Pointer-Generator with Coverage Penalty without fine-tuning the abstractive model.
- Domain transfer examples: 27.7 ROUGE-L with content selection exceeds 20.6 ROUGE-L for the CNN-DM-trained S2S model on NYT, without fine-tuning S2S.The comparison uses a Pointer-Generator with Coverage Penalty trained on CNN-DM and evaluated on NYT.