Source-linked AI summary
Deep Recurrent Generative Decoder for Abstractive Text Summarization
Piji Li, Wai Lam, Lidong Bing, Zihao Wang
TL;DR
Abstractive summarization needs models that can generate new sentences while representing latent summary structure. This paper introduces a seq2seq encoder-decoder with a deep recurrent generative decoder and variational inference, combining latent and deterministic states for generation. Experiments on multilingual benchmark datasets report improvements over state-of-the-art methods.
Problem
Existing seq2seq summarization models use deterministic decoding states, while few works explicitly model latent structure information in target summaries.
Method
The framework adds a deep recurrent generative decoder with recurrent latent variables to a seq2seq encoder-decoder and jointly decodes from latent and deterministic states.
Results
The framework achieves improvements over state-of-the-art methods on benchmark datasets in different languages.
Takeaways & Limitations
Modeling latent structural information together with deterministic states supports abstractive summary generation.
Takeaways & Limitations
Baseline methods may differ across datasets because their reported results were extracted from the original papers.
Abstract
from arXiv · showhide
We propose a new framework for abstractive text summarization based on a sequence-to-sequence oriented encoder-decoder model equipped with a deep recurrent generative decoder (DRGN). Latent structure information implied in the target summaries is learned based on a recurrent latent random model for improving the summarization quality. Neural variational inference is employed to address the intractable posterior inference for the recurrent latent variables. Abstractive summaries are generated based on both the generative latent variables and the discriminative deterministic states. Extensive experiments on some benchmark datasets in different languages show that DRGN achieves improvements over the state-of-the-art methods.
1 Introduction
Abstractive summarization constructs new sentences but requires deeper text understanding and generation ability. The paper models latent summary structures alongside deterministic decoding states to improve generated summaries.
- 1 Introduction: Abstraction-based summarization generates new sentences, offering better focus, reduced redundancy, and good compression.Unlike extraction- and compression-based methods, it requires deeper text understanding and sentence-generation capability.
- 1 Introduction: Human-written abstractive summaries often follow structures such as “What,” “What-Happened,” and “Who Action What.”CNN Technology headlines illustrate these recurring summary patterns.
- 1 Introduction: Existing seq2seq decoders are deterministic, limiting their ability to represent latent structure information in summaries.The paper identifies limited consideration of latent summary structure as a gap in prior work.
- 1 Introduction: The proposed framework adds a deep recurrent generative decoder to a seq2seq encoder-decoder, using variational inference for recurrent latent variables.Historical dependencies are added to VAE latent variables, and the model is trained end to end by back-propagation.
- 1 Introduction: Summary generation jointly uses generative latent structural information and discriminative deterministic variables.Experiments on benchmark datasets in different languages report better performance than state-of-the-art models.
2 Related Works
Related work spans extraction, compression, abstraction, neural seq2seq models, and topic models. Prior neural approaches remain limited by deterministic representations or insufficient modeling of recurrent latent dependencies.
- 2 Related Works: Traditional summarization methods include extraction-based, compression-based, and abstraction-based approaches.Abstraction-based methods can generate new sentences by fusing facts or salient phrases from different source sentences.
- 2 Related Works: Neural abstractive systems introduced attention, copying, distraction, and vocabulary-control mechanisms for sequence-to-sequence summarization.These methods target summary quality, source understanding, or training efficiency.
- 2 Related Works: Prior neural methods calculate decoding states deterministically, limiting representation ability.A generative model captured latent summary information, but omitted recurrent dependencies in its generative component.
- 2 Related Works: Topic models capture latent information from source documents or sentences for sentence selection and extraction.Examples use term-document and term-sentence associations, hierarchical topic models, and regression.
3 Framework Description
The framework combines a sequence-to-sequence encoder-decoder with deterministic decoding and recurrent latent structure modeling. Neural variational inference learns recurrent latent variables, and generation uses both latent structural information and deterministic hidden states.
- Overview: The model uses a neural encoder-decoder with GRUs for sequence-to-sequence abstractive summarization.The encoder processes variable-length source sequences, while the decoder generates summary sequences.
- Motivation: Deterministic recurrent states alone may not effectively represent high-level syntactic features and latent topics in target summaries.The framework addresses this limitation by introducing latent random variables with historical dependencies.
- Recurrent generative decoder: The recurrent inference model estimates qφ(zt|y<t, z<t) from previous summary words and latent variables, capturing historical dynamic latent structure information.A recognition model approximates the intractable posterior and is jointly learned with the generative model.
- Summary generation: The generative process draws each target word from pθ(yt|zt), while the deterministic decoder incorporates the previous word, previous hidden state, and attention context.The final deterministic state is produced by a second GRU layer.
- Training: The variational objective maximizes a lower bound on marginal likelihood, and the fully differentiable framework is optimized end-to-end by back-propagation.The objective combines generated-summary likelihood with a variational term.
- Decoder architecture: The decoder contains separate discriminative deterministic decoding and generative latent structure modeling components.The deterministic decoder uses attention, while the generative component models latent structure in target summaries.
4 Experimental Setup
The experiments evaluate the framework on English and Chinese summarization datasets using ROUGE F-measures and comparisons with published baselines and state-of-the-art methods. Dataset-specific baseline sets may differ because comparative results are extracted from prior papers.
- Datasets: The study uses Gigawords, DUC-2004, and LCSTS, covering English sentence summarization, English testing, and Chinese short-text summarization.Gigawords contains roughly 3.8M training pairs, while LCSTS uses filtered pairs with human relevance scores of at least 3.
- Evaluation: ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-SU4 F-measures are reported with standard options.ROUGE counts overlapping units between generated and reference summaries.
- Comparative methods: The model is compared with baselines and state-of-the-art methods drawn from prior summarization studies.Compared systems include length control, latent compression, vocabulary control, attention, copying, and alternative recurrent architectures.
- Comparative methods: Baseline methods may differ across datasets because the reported comparison results are extracted from the corresponding prior papers.The authors describe the datasets and baselines as sufficiently standard for this procedure.
- Implementation: The implementation varies by dataset in embedding dimensions, maximum sequence lengths, and other training settings.For example, English word embeddings use dimension 300, whereas LCSTS uses dimension 350.
5 Results and Discussions
DRGD improves abstractive summarization over standard decoders and achieves the best reported performance across English and Chinese benchmark evaluations. Its generated summaries also better capture latent structures consistent with reference summaries.
- ROUGE Evaluation: DRGD obtains obvious improvements over standard decoders on Gigawords and LCSTS validation sets using ROUGE-F1.The standard decoders perform similarly to popular baseline methods.
- ROUGE Evaluation: DRGD achieves the best summarization performance on all ROUGE metrics for the English Gigawords and DUC-2004 datasets.The compared lvt2k-1sent and lvt5k-1sent systems use additional linguistic and statistical features.
- ROUGE Evaluation: The lvt2k and lvt5k baselines are not end-to-end models and are more complicated in practical applications than DRGD.The paper notes that extracting their linguistic features is time-consuming, especially on large-scale datasets such as Gigawords.
- ROUGE Evaluation: DRGD achieves the best performance on the Chinese LCSTS dataset, outperforming CopyNet and RNN-distract.The comparison includes models using copying mechanisms and attention-information diversity.
- Summary Case Analysis: DRGD-generated summaries capture latent structures consistent with golden summaries, whereas standard decoders can produce looser sentences that miss the main points.For example, DRGD matches a “Who Action What” structure in one Wuhan sports summary, while StanD does not.
6 Conclusions
The paper proposes DRGD, a sequence-to-sequence encoder-decoder with latent structure modeling, and reports improved abstractive summarization performance over state-of-the-art methods.
- DRGD combines a sequence-to-sequence encoder-decoder framework with a latent structure modeling component.
- Abstractive summaries are generated using both latent variables and deterministic states.
- DRGD achieves improvements over state-of-the-art methods on benchmark datasets.