Source-linked AI summary
Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, Taylor Berg-Kirkpatrick
TL;DR
Textual VAEs have produced weaker results than LSTM language models, with decoder architectures implicated in ignoring latent information. This paper uses dilated CNN decoders to vary contextual capacity and studies their effects across language modeling and labeling tasks. Carefully selected decoders improve language modeling and labeling performance over strong baselines.
Problem
Textual VAEs have often underperformed LSTM language models because LSTM decoders may ignore latent representations and collapse into simpler language models.
Method
The paper uses dilated CNNs as VAE decoders, varying dilation architectures to control contextual capacity and the effective history available to the decoder.
Results
Carefully selected CNN decoders improve VAE language modeling over LSTM language models and outperform strong baselines on semi-supervised classification and unsupervised clustering.
Takeaways & Limitations
Controlling decoder contextual capacity can improve VAE performance on language modeling and semi-supervised classification by preventing degenerate training collapse.
Abstract
from arXiv · showhide
Recent work on generative modeling of text has found that variational auto-encoders (VAE) incorporating LSTM decoders perform worse than simpler LSTM language models (Bowman et al., 2015). This negative result is so far poorly understood, but has been attributed to the propensity of LSTM decoders to ignore conditioning information from the encoder. In this paper, we experiment with a new type of decoder for VAE: a dilated CNN. By changing the decoder's dilation architecture, we control the effective context from previously generated words. In experiments, we find that there is a trade off between the contextual capacity of the decoder and the amount of encoding information used. We show that with the right decoder, VAE can outperform LSTM language models. We demonstrate perplexity gains on two datasets, representing the first positive experimental result on the use VAE for generative modeling of text. Further, we conduct an in-depth investigation of the use of VAE (with our new decoding architecture) for semi-supervised and unsupervised labeling tasks, demonstrating gains over several strong baselines.
1. Introduction
Textual VAEs have struggled because LSTM decoders can ignore latent information, while simpler decoders sacrifice long-range dependencies. The paper introduces dilated CNN decoders to vary contextual capacity and finds settings that improve language modeling and labeling tasks.
- 1. Introduction: Textual VAEs often underperform LSTM language models because LSTM decoders may ignore latent representations during training.Bag-of-words decoders use latent representations more effectively but cannot model long-range textual dependencies well.
- 1. Introduction: The paper proposes dilated CNN decoders whose dilation architecture controls the effective context from previously generated words.Different dilation choices span contexts from bag-of-words-like MLP behavior to conditioning on the entire history.
- 1. Introduction: Varying decoder contextual capacity reveals a trade-off between modeling text accurately and preserving reliance on the latent representation.The authors seek a contextual-capacity sweet spot where the decoder does not overpower the latent representation.
- 1. Introduction: With carefully selected CNN decoders, textual VAEs outperform simple LSTM language models and several strong baselines on text categorization and sentiment analysis.The paper also investigates semi-supervised classification and unsupervised clustering using dilated CNN VAEs.
- 1. Introduction: The paper contributes a dilated CNN VAE framework and empirically evaluates decoder architectures with different contextual capacities.Its evaluations cover language modeling, semi-supervised classification, and unsupervised clustering.
2. Model
This section presents VAEs for language modeling, explains training collapse with text, and introduces dilated CNN decoders whose receptive field can be controlled for generative and labeling tasks.
- 2.1. Background on Variational Autoencoders: VAEs generate text from a latent vector z sampled from a Gaussian prior and decoded through pθ(x|z).The encoder qφ(z|x) approximates the posterior and is optimized jointly with the generative model.
- 2.1. Background on Variational Autoencoders: The VAE objective uses a variational lower bound because directly computing the marginal likelihood is intractable for many decoder choices.Training jointly optimizes model and approximation parameters using stochastic gradient ascent.
- 2.1. Background on Variational Autoencoders: The KL-divergence term discourages memorizing each input as a single latent point, distinguishing the VAE from an autoencoder.Without the KL term, the posterior variance can become small and training reduces to an autoencoder.
- 2.2. Training Collapse with Textual VAEs: Textual VAEs can collapse when the KL term becomes vanishingly small, causing the decoder to ignore z and reducing the model to a simpler language model.The authors hypothesize that this condition is related to decoder contextual capacity.
- 2.3. Dilated Convolutional Decoders: Dilated CNN decoders control contextual capacity through filter size, dilation, and network depth, spanning bag-of-words-like to LSTM-like behavior.Dilation expands the effective receptive field exponentially with network depth.
- 2.3. Dilated Convolutional Decoders: The causal CNN decoder shifts inputs so each generated token conditions only on past tokens, with effective filter size (k −1) × n + 1 for n layers.This preserves autoregressive generation while allowing the receptive field to grow with depth.
- 2.3. Dilated Convolutional Decoders: The VAE uses an LSTM encoder and concatenates sampled z with every decoder input word embedding.Residual connections are used to speed convergence and enable deeper decoder models.
- 2.4. Semi-supervised VAE: The semi-supervised extension jointly models labels and continuous latent variables, combining labeled and unlabeled objectives with α controlling their trade-off.Unsupervised clustering minimizes reconstruction loss plus KL regularization on q(y|x), while a modified KL term addresses degenerate optima.
3. Experiments
Experiments evaluate dilated CNN decoders, encoder initialization, latent representations, and semi-supervised and unsupervised uses of VAEs on Yahoo and Yelp15 text datasets. Results show decoder context and encoder quality affect language modeling and classification performance, with improvements over LSTM language models and strong baselines.
- 3.1. Data sets: Yahoo Answer and Yelp15 review provide topic- and sentiment-classification datasets, with 100k training and 10k validation and test samples drawn from each partition.
- 3.2. Models: Dilated CNN decoders vary contextual capacity through SCNN, MCNN, LCNN, and VLCNN configurations, while the semi-supervised model predicts labels and continuous latent variables jointly.
- 3.3. Language modeling results: LCNN-VAE reaches Yahoo NLL 333.9 versus 334.9 for LSTM-LM, while encoder initialization further improves LCNN-VAE from 334.9 to 332.1 NLL and from 66.2 to 63.9 PPL.On sentiment data, encoder-initialized LCNN-VAE improves over LSTM-LM from 362.7 to 359.1 NLL and from 42.6 to 41.1 PPL.
- 3.3. Language modeling results: Latent representations cluster documents by topic, whereas documents with different ratings form a continuum ordered by review rating.
- 3.4. Semi-supervised VAE results: With 500 labeled Yahoo samples, SCNN-VAE-Semi obtains the best classification accuracy at 65.5, while LCNN-VAE-Semi obtains the best NLL.Increasing decoder contextual capacity lowers classification accuracy, supporting a trade-off between classification accuracy and NLL.
- 3.4. Semi-supervised VAE results: Semi-supervised VAE improves NLL over simple VAE from 337.8 to 335.7 for SCNN, from 336.2 to 332.8 for MCNN, and from 333.9 to 332.8 for LCNN.The reported improvement mainly comes from the KL-divergence component.
- 3.4. Semi-supervised VAE results: SCNN-VAE-Semi outperforms LM-LSTM and LA-LSTM, with a larger advantage at fewer labeled samples; at 25k labels, its accuracy is 70.4 versus 70.5 for LM-LSTM.
4. Related work
The paper situates its approach among variational text models, decoder architectures, and unsupervised sentence-encoding methods, addressing gaps in prior work.
- Bowman et al. combine VAEs with LSTM language-model decoders but report negative text-modeling results.
- Miao et al. improve text modeling with a bag-of-words VAE, but their model cannot generate text.
- The paper positions itself between LSTM-based generative modeling and bag-of-words approaches by combining generation with variational text representations.
- Variational inference has also been applied to dialogue modeling, machine translation, and recurrent models with variational units at every step.
- CNN decoders draw inspiration from PixelCNN, WaveNet, Video Pixel Network, and ByteNet, but here CNN capacity is controlled rather than simply increased.
- The work uses Dai and Le’s pretrained-model approach as a baseline for semi-supervised VAE classification.
5. Conclusion
The conclusion reports that controlling decoder contextual capacity improves VAE performance on language modeling and semi-supervised classification by preventing degenerate training collapse.
- Controlling decoder contextual capacity improves VAE performance on language modeling and semi-supervised classification tasks.
- The authors identify characterizing decoder capacity and its relationship to variational training procedures as an avenue for future unsupervised problems.