Source-linked AI summary
Language as a Latent Variable: Discrete Generative Models for Sentence Compression
Yishu Miao, Phil Blunsom
TL;DR
The paper asks whether sentence compression can be modeled with discrete language-valued latent representations rather than only conditional string prediction. It develops a variational auto-encoder with generative and discriminative compression models, achieving improved Gigaword performance and competitive benchmark results with much less labelled data. The approach is constrained by a language-model prior that favors short compressions.
Problem
The paper addresses the limitation of conditional sequence-to-sequence models by modeling the joint distribution of string pairs with discrete language-valued latent representations.
Method
The approach uses a generative auto-encoding sentence compression model with a latent language model summary, variational inference, pointer networks, and a supervised forced-attention model.
Results
The combined models improve state-of-the-art performance on Gigaword compression and outperform previously reported benchmarks when supervised training uses little labelled data alongside large unlabelled data.
Takeaways & Limitations
Discrete latent language variables support effective sentence compression while enabling combinations of generative and discriminative training and semi-supervised learning.
Takeaways & Limitations
The pre-trained language-model prior favors short compressions, and KL regularisation can restrict encoder learning and cause the decoder to rely less on compression information.
Abstract
from arXiv · showhide
In this work we explore deep generative models of text in which the latent representation of a document is itself drawn from a discrete language model distribution. We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences. In this application the generative model first draws a latent summary sentence from a background language model, and then subsequently draws the observed sentence conditioned on this latent summary. In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data. Further, we explore semi-supervised compression scenarios where we show that it is possible to achieve performance competitive with previously proposed supervised models while training on a fraction of the supervised data.
1 Introduction
The paper introduces a generative, variational auto-encoding approach to sentence compression in which summaries are discrete language-model sequences. It combines abstractive and extractive mechanisms and supports semi-supervised learning through unlabelled data.
- The model jointly represents pairs of strings with a deep generative model and uses a discrete variational auto-encoder for inference.
- The variational framework enables semi-supervised learning by marginalising latent compressed text for unlabelled data.
- The generative auto-encoding sentence compression model treats a variable-length natural-language summary as a latent representation of the observed utterance.Its inference network approximates the posterior over summaries conditioned on the observed sentence and samples compressions to optimise the variational lower bound.
- Pointer networks constrain latent compression samples toward words appearing in the source sentence, reducing the sampling space from the full vocabulary.
- The forced-attention compression model combines source-word copying with generation from a background vocabulary distribution while training on labelled pairs.It shares the pointer network with the generative model.
- With all supervised data, the mixed discriminative and generative objective significantly outperforms previously published sentence-compression results.With small amounts of supervised data, the model matches previous benchmarks.
2 Auto-Encoding Sentence Compression
The ASC model treats a sentence compression as a discrete latent language sequence: an inference network extracts a compression, and a reconstruction network generates the source sentence from it. Variational training combines reconstruction with language-model prior regularization, using sampling-based inference for the discrete latent variables.
- Model structure: The model comprises recurrent encoder, compressor, decoder, and language-model components for extracting and reconstructing compressed sentences.The language model prior regularizes latent compressions toward reasonable natural-language sequences.
- Model structure: ASC represents compression as a latent sentence c inferred from source sentence s, then reconstructs s conditioned on c.The inference network is qφ(c|s), while the generative network is pθ(s|c).
- Compression model: The pointer-network inference model generates extractive compressions by attending to source words, so sampled compression words come from the source sentence.Its predictive distribution conditions each compression word on prior compression words and the source sentence.
- Reconstruction model: The reconstruction model uses a soft-attention sequence-to-sequence network to generate the source sentence from sampled compressions.The decoder receives compression samples from qφ(c|s) and predicts reconstructed source words with a softmax.
- Inference and optimization: Discrete latent variables prevent the standard VAE reparameterization trick, so ASC uses REINFORCE with sampled compressions and variance-reduction baselines.The learning signal depends on samples from qφ(c|s), producing high estimator variance that centered and input-dependent baselines address.
- Inference and optimization: The variational objective balances reconstruction quality against KL regularization, while λ = 0.1 scales the KL term and controls compression rate.The authors state that the reconstruction error is the main objective for learning an effective compression model.
- Limitation: Because the language-model prior prefers short compressions, KL regularization can restrict encoder learning and make the decoder rely on previously decoded context instead of compression information.This behavior can reduce sentence-compression performance, motivating adjustment of the KL contribution.
3 Forced-attention Sentence Compression
The FSC model addresses unstable early variational training by teaching the compressor to generate coherent sentences while combining source copying with full-vocabulary generation.
- Training motivation: High-variance gradients make reasonable compression samples difficult early in variational training.The difficulty arises because sequence sampling exposes many possible vocabulary words at each hidden state.
- Training motivation: The FSC model uses supervised training to teach the compression model to generate coherent compressed sentences.
- Model design: FSC combines a pointer network with a full-vocabulary softmax, unlike directly replicating ASC’s pointer network or using a typical sequence-to-sequence model.
- Model design: At each recurrent state, a force-attention strategy chooses between selecting a word from the source sentence and predicting one from the full vocabulary V.
- Model design: The FSC probability distribution combines source-selection and vocabulary-prediction probabilities using a state-specific selection factor.
4 Semi-supervised Training
The paper trains FSC on labelled source-compression pairs and ASC on unlabelled source sentences, joining them through a semi-supervised objective.
- Data and objectives: The semi-supervised framework uses labelled parallel sentences L and an unlabelled dataset U containing only source sentences.
- Data and objectives: FSC is trained on L by maximising the compression model’s log-probability.
- Data and objectives: ASC is trained on U by maximising a modified variational lower bound.
- Joint training: The joint semi-supervised objective trains the models together across labelled and unlabelled data.
- Joint training: The pointer network receives a mixed training criterion combining REINFORCE with cross-entropy on U and L.
5 Related Work
The related work contrasts earlier discriminative sentence summarisation models with variational auto-encoder approaches used for generative modelling in vision and language.
- Sentence summarisation: Earlier sentence-compression work includes extractive deletion with LSTMs and abstractive generation with convolutional encoders and attentional feed-forward decoders.
- Sentence summarisation: Subsequent studies improved Gigaword performance with RNN encoder-decoder variants and combined pointer networks with softmax outputs.
- Sentence summarisation: The paper distinguishes its approach from these discriminative models, though the supplied passage does not complete the stated distinction.
- Variational auto-encoders: Variational auto-encoders have been applied to image generation and variable-sized object identification, as well as document and sentence modelling in NLP.
6 Experiments
Experiments on Gigaword evaluate extractive and abstractive compression under supervised and semi-supervised training. Adding generative objectives and unlabelled data improves performance, with full-data models surpassing prior published results.
- Experimental setup: The models are evaluated on Gigaword, using roughly 3.8M training pairs, 400K validation examples, and a 400K test set, with results reported on a 2K subset.Models are decoded with k = 5 beam search; hyperparameters are tuned for summary perplexity rather than reconstructed-source perplexity.
- Experimental setup: ASC+FSC1 uses equivalent labelled and unlabelled datasets, whereas ASC+FSC2 adds the full unlabelled dataset in the true semi-supervised setting.ASC+FSC1 therefore adds no additional unlabelled article sentences beyond those paired with labelled data.
- Extractive Summarisation: In extractive compression, ASC+FSC1 significantly improves Rouge F-1 over supervised FSC, while ASC+FSC2 further benefits from additional unlabelled data.The added unlabelled data mainly increases precision, while recall does not benefit, producing significant F-1 improvements.
- Abstractive Summarisation: For abstractive compression, adding the generative objective significantly boosts all Rouge scores, and extra unlabelled data increases performance further.These results support transferring knowledge learned from unlabelled data to supervised abstractive summarisation.
- Language modelling: ASC+FSC1 achieves better validation perplexity than FSC, and the full-data joint ASC+FSC1 model performs best among benchmark models on compression-language modelling.The difference between ASC+FSC1 and ASC+FSC2 is small as the labelled dataset grows.
- Abstractive Summarisation: ASC+FSC2 outperforms NABS with 500K supervised pairs, approximately one-eighth of the supervised data, while full-data ASC+FSC1 significantly exceeds previous state-of-the-art performance.The semi-supervised model can use unlimited unlabelled data, leaving scope for further improvement.
7 Discussion
The paper’s generative contribution is a variance-reduction process for discrete sampling-based variational inference, centered on pointer-network estimation and supervised guidance. In sentence compression, this enables the latent language model to learn from unlabelled data.
- Generative-model contribution: The proposed process reduces variance in discrete sampling-based variational inference, where the reparameterisation trick is unavailable.It introduces control-variate baselines before applying a pointer-network biased estimator.
- Empirical discussion: Validation perplexity compares ASC+FSC1, ASC+FSC2, and FSC as measures of how well the models learn compression languages.ASC+FSC1 achieves better perplexities than FSC, while ASC+FSC1 and ASC+FSC2 differ little as labelled data grows.
- Generative-model contribution: Pointer networks bound the latent estimator’s state space by the source-sentence length rather than the full vocabulary.The source sentence is mostly 20–50 tokens long, making sampling more tractable.
- Generative-model contribution: Jointly applying supervised FSC training transfers knowledge to the pointer network and biases the unsupervised latent space toward the supervised distribution.This further reduces sampling variance and helps ASC summarise sentences from large unlabelled data.
- Sequence-level interpretation: The compression model acts as an agent generating words, while the reconstruction model evaluates the compressed sentence as a reward signal.The paper uses a variational lower bound combining reconstruction error and KL-divergence regularisation instead of an explicit Rouge score.
8 Conclusion
The paper introduces a generative model that jointly models sequence pairs and applies it to sentence compression. Combining generative ASC with discriminative FSC improves reported Gigaword performance and can exploit large unlabelled datasets alongside limited labelled data.
- Conclusion: The variational auto-encoding framework supports joint generative ASC and discriminative FSC compression models.The paper evaluates their combination on the Gigaword compression dataset.
- Conclusion: Supervised training of the combined models improves upon state-of-the-art performance on Gigaword sentence compression.This is the paper’s principal reported supervised-training conclusion.
- Conclusion: With limited labelled data and a large unlabelled set, the combined model outperforms benchmarks trained on substantially more supervised data.The result demonstrates effective use of both supervised and unsupervised data in sequence-to-sequence tasks.
- Examples: Table 5 lists source, reference, abstractive, and extractive compression examples from the joint model alongside abstractive outputs from FSC.The joint model outputs are labelled asca and asce, while FSC outputs are labelled fsca.