Source-linked AI summary
A Neural Attention Model for Abstractive Sentence Summarization
Alexander M. Rush, Sumit Chopra, Jason Weston
TL;DR
Extractive summarization is limited, while abstractive sentence summarization is difficult to build. The paper proposes a fully data-driven local attention model trained end-to-end on input-summary pairs, and reports gains over strong baselines, including on DUC-2004. The model remains limited in transferring unseen proper-noun phrases from the input.
Problem
Extractive summarization is limited, while generation-based abstractive sentence summarization is challenging to build.
Method
The paper combines a neural language model with an attention-based contextual encoder that learns a soft alignment and trains both components jointly.
Results
The approach outperforms a machine translation system trained on the same large-scale dataset and improves substantially over the highest-scoring DUC-2004 system.
Takeaways & Limitations
Attention-Based Summarization provides a simple data-driven approach that scales to around 4 million article-summary pairs for headline generation.
Takeaways & Limitations
The abstractive model cannot reliably find extractive matches when needed, such as transferring unseen proper-noun phrases from the input.
Abstract
from arXiv · showhide
Summarization based on text extraction is inherently limited, but generation-style abstractive methods have proven challenging to build. In this work, we propose a fully data-driven approach to abstractive sentence summarization. Our method utilizes a local attention-based model that generates each word of the summary conditioned on the input sentence. While the model is structurally simple, it can easily be trained end-to-end and scales to a large amount of training data. The model shows significant performance gains on the DUC-2004 shared task compared with several strong baselines.
1 Introduction
The paper presents a fully data-driven, attention-based approach to abstractive sentence summarization that learns to generate summaries from input sentences. ABS scales to millions of article-summary pairs and outperforms strong abstractive and extractive baselines.
- Motivation: Abstractive summarization generates condensed representations that may include content absent from the original, unlike extractive systems that reuse input text.Human summarizers also paraphrase, generalize, and reorder content while condensing sentences.
- Approach: The model jointly trains a contextual input encoder and neural language model, using a learned latent soft alignment to inform each generated word.The approach is fully data-driven and also incorporates beam search and features for extractive elements.
- Approach: Attention-Based Summarization uses less linguistic structure than comparable abstractive methods while scaling to large training datasets.The system can train directly on document-summary pairs without assumptions about the generated-summary vocabulary.
- Data: Around 4 million Gigaword article pairs support training a headline-generation summarization model.The corpus consists of approximately 4 million articles.
- Results: The approach outperforms a same-data machine translation system and yields a large improvement over the highest-scoring DUC-2004 competition system.The evaluation compares against multiple abstractive and extractive baselines, including syntax-based, integer-linear-program, information-retrieval, and phrase-based systems.
2 Background
The paper frames sentence summarization as selecting a shortened output under a scoring function, contrasting abstractive generation with extraction and deletion-based compression. Its factored models score locally using a fixed context of previous summary words.
- Task definition: Given an input sentence of M words, the task is to produce a condensed summary of length N < M.Input and summary words are represented using indicators from a fixed vocabulary V.
- Assumptions: The model assumes the summary length N is fixed and known before generation.This differs from related tasks such as machine translation.
- Task definition: Abstractive summarization searches over possible output sentences, whereas extractive summarization transfers input words and compression deletes input words.The abstractive formulation allows more freedom in generation but creates a more difficult generation challenge.
- Implications: Abstractive summarization can fit a wider range of training data because it lacks hard constraints on copying or deleting input words.The literature's distinction between abstractive and extractive systems is formally inconsistent.
- Scoring: The scoring function models the local conditional distribution p(y_i+1|x, y_c; θ) using a fixed window of previous summary words.The context length is bounded by C, with a special start symbol used before the first output word.
3 Model
The model directly parameterizes a conditional language model that combines a neural language model with an encoder of the input sentence. Its encoders range from bag-of-words to convolutional and attention-based representations, trained jointly with generation.
- Conditional language model: The model directly parameterizes p(y_i+1|x, y_c; θ) with a neural probabilistic language model and an input encoder.This replaces separately estimated noisy-channel components with one neural network.
- Neural language model: The NNLM estimates the contextual probability of each next summary word from a fixed window of preceding words.Its parameters include word embeddings and weight matrices, while the encoder supplies an input-dependent representation.
- Bag-of-words encoder: The bag-of-words encoder embeds input words while ignoring their original order and neighboring relationships.It can distinguish content words from stop words but is inherently limited in representing contiguous phrases.
- Convolutional encoder: The convolutional encoder adds local word interactions through alternating temporal convolution and max-pooling layers.Unlike the bag-of-words encoder, it does not require the generation context while encoding the input.
- Attention-based encoder: The attention-based encoder uses generation context to learn a soft alignment over the input and weight a smoothed local input representation.A context aligned with position i gives greater weight to words from x_i−Q through x_i+Q.
- Training: The encoder and generation model are trained jointly by minimizing token-level negative log-likelihood over input-summary pairs, using mini-batch stochastic gradient descent.The training objective relies on gold-standard preceding summary contexts.
4 Generating Summaries
Summary generation searches for a high-scoring output sequence under the factored model. Because the vocabulary is large, beam search approximates exact decoding while retaining multiple candidate summaries.
- Exact and greedy decoding: Exact Viterbi decoding is theoretically tractable in O(NV^C) time, but the large vocabulary makes it difficult in practice.Greedy or deterministic decoding provides a simpler approximation.
- Beam search: Beam search retains K potential hypotheses at each summary position while considering the full vocabulary V.It is presented as a compromise between exact and greedy decoding.
- Beam-search procedure: The beam-search algorithm expands hypotheses left to right and uses hypothesis recombination during decoding.Its pseudocode initializes an empty hypothesis, selects an abstractive vocabulary, and iterates for N positions.
- Computational cost: Beam search requires O(KNV) time, with each round dominated by computing p(y_i|x, y_c) for the K hypotheses.The per-hypothesis conditional probabilities can be computed as a mini-batch.
5 Extension: Extractive Tuning
The extractive-tuning extension addresses the model's difficulty copying unseen input phrases by adding features that adjust the abstractive–extractive balance. These feature weights are tuned after neural training.
- Motivation: The attention-based model can miss extractive word matches, including unseen proper-noun phrases that appear in human-generated references.The paper relates this limitation to rare-word problems observed in neural machine translation.
- Feature extension: The extension adds a small set of features to trade off the system's abstractive and extractive tendencies.This directly targets the missing extractive behavior without replacing the main neural model.
- Scoring function: The modified log-linear scoring function uses α ∈ R^5 and a feature function f to estimate summary probabilities.The resulting best-summary search maximizes a factored scoring function.
- Extractive features: The added features indicate unigram, bigram, and trigram matches with the input, together with input-word reordering.Setting α = ⟨1, 0, . . . , 0⟩ recovers standard ABS.
- Parameter tuning: After neural training, θ is fixed and α is tuned with minimum-error-rate training on tuning data for the summarization metric.The procedure follows the statistical machine translation setup and matches the tuning used for the phrase-based baseline.
6 Related Work
The paper situates its neural abstractive summarizer among headline-generation, compression, statistical machine translation, and neural language-model approaches. It differs by using a larger corpus and fully abstractive decoding without requiring monotonic alignment.
- Earlier headline-generation work used statistical machine translation-inspired methods trained on article-headline pairs.
- The model extends earlier headline-generation work with a neural summarization model, training on 4 million articles, and fully abstractive decoding.
- Syntax-based approaches learn transformations from parsed or aligned texts, whereas this system does not directly use input syntax.
- Unlike strictly extractive compression, the system uses the text directly and does not require monotonic alignment between input and output.
- The model is closely related to neural language models and attention-based neural machine translation, especially soft alignment between input positions and generated output.
7 Experimental Setup
The experiments evaluate headline generation on DUC shared-task data and Gigaword, using ROUGE-based comparisons against extractive, compression, retrieval, and translation baselines. Training uses article-first-sentence pairs from large-scale Gigaword data, with ABS and tuned ABS+ variants.
- Systems are evaluated with ROUGE-1, ROUGE-2, and ROUGE-L, using a 75-character cutoff for the DUC task.
- Training pairs each article’s headline with its first sentence, using filtered Gigaword data containing about 9.5 million news articles.
- Baselines: Baselines include PREFIX, TOPIARY, COMPRESS, IR, and MOSES+, covering extractive, syntax-based, retrieval, compression, and phrase-based translation approaches.
- The DUC task includes human reference summaries, with the best human evaluator scoring 31.7 ROUGE-1.
- The main model is ABS, while ABS+ adds MERT tuning using DUC-2003 development data.
8 Results
ABS and ABS+ outperform major baselines on the reported summarization tasks, while ablations show benefits from attention-based encoding and beam search. Example outputs nevertheless reveal substantial grammaticality and human-performance gaps.
- ABS and MOSES+ outperform TOPIARY, especially on ROUGE-2 and ROUGE-L for DUC, while ABS+ performs best on these tasks.
- ABS+ is significantly better than TOPIARY on all reported metrics and surpasses MOSES+ on selected DUC and Gigaword ROUGE metrics.
- Encoder ablations: The attention-based encoder reduces validation perplexity to below 30, outperforming the bag-of-words and convolutional encoder variants.
- Model and decoding ablations: The largest ablation gains come from a more powerful attention encoder and beam-search decoding, although purely extractive generation remains effective for ROUGE.
- Qualitative analysis: Generated summaries often identify key words but can reorder them incorrectly, produce wrong subjects, and make attachment mistakes despite outperforming baselines.
9 Conclusion
The paper presents a neural attention-based model and generation algorithm for accurate abstractive summarization. It identifies improved grammaticality and paragraph-level generation as future challenges involving efficient alignment and consistency.
- The paper combines a neural attention-based probabilistic model with a generation algorithm that produces accurate abstractive summaries.
- Future work targets more grammatical summaries and paragraph-level generation.
- Paragraph-level generation raises additional challenges in efficient alignment and consistency during generation.