Source-linked AI summary
A Hybrid Convolutional Variational Autoencoder for Text Generation
Stanislau Semeniuta, Aliaksei Severyn, Erhardt Barth
TL;DR
Text VAEs struggle because recurrent decoders can ignore latent variables, causing KL-term collapse. The paper introduces a hybrid convolutional/deconvolutional architecture with a recurrent language-model component and reports easier training, better handling of long sequences, and realistic generated text.
Problem
Text VAE decoders can ignore latent vectors, causing KL-term collapse and reducing the model to a standard language model.
Method
The model combines a convolutional encoder, deconvolutional decoder, and recurrent language-model component, with an auxiliary reconstruction objective to encourage latent-vector use.
Results
The hybrid model is easier to train than the fully recurrent alternative, preserves latent information better on long texts, and produces realistic sampled texts.
Takeaways & Limitations
Feed-forward components help avoid KL-term collapse, while the recurrent component captures dependencies that are difficult for a fully feed-forward architecture.
Abstract
from arXiv · showhide
In this paper we explore the effect of architectural choices on learning a Variational Autoencoder (VAE) for text generation. In contrast to the previously introduced VAE model for text where both the encoder and decoder are RNNs, we propose a novel hybrid architecture that blends fully feed-forward convolutional and deconvolutional components with a recurrent language model. Our architecture exhibits several attractive properties such as faster run time and convergence, ability to better handle long sequences and, more importantly, it helps to avoid some of the major difficulties posed by training VAE models on textual data.
1 Introduction
Text generation models support major NLP applications, while VAEs add latent-variable control but are difficult to train for text because powerful RNN decoders can ignore latent representations. The paper proposes a hybrid convolutional-recurrent VAE to address this difficulty.
- Generative text models underpin machine translation, dialogue modelling, and abstractive summarization.
- RNN-based generative models generate outputs sequentially, conditioning each prediction on previously generated elements.
- VAEs generate examples from noise through a decoder and provide latent representations for controlling attributes such as sentiment or writing style.
- VAE training for natural text is difficult because the KL divergence can collapse to zero, causing the generator to ignore latent representations.
- The proposed hybrid VAE combines convolutional and deconvolutional components with an RNN language model to encourage latent-vector use and improve training on long texts.The authors report faster convergence than the fully recurrent alternative, which fails to converge on longer texts, and realistic sampled texts.
2 Related Work
Related work spans VAEs, autoregressive models, and GANs, with autoregressive RNNs dominating NLP generation. Prior text VAEs established latent-variable generation but required training techniques to prevent latent-vector neglect.
- Generative modeling includes VAEs, autoregressive models, and GANs.
- Autoregressive models predict each data element from sufficient generated history, and RNN language models dominate NLP generation.
- Recurrent and convolutional decoders provide effective solutions for autoregressive generation.
- GANs have been highly effective in computer vision but have gained little traction in NLP.
- Bowman et al. applied VAEs to text, enabling realistic sampling and latent-space interpolation, while other work used VAEs for document representations and sequence-to-sequence tasks.
- Prior VAE training methods controlled the reconstruction–KL trade-off through KL-term annealing, input dropout, or structured sparsity.The paper presents another technique for controlling this trade-off.
3 Model
The model combines convolutional and deconvolutional VAE components with a recurrent language model, then adds optimization techniques to preserve useful latent representations.
- 3.3 Hybrid Convolutional-Recurrent VAE: The proposed hybrid architecture uses a fully convolutional encoder, deconvolutional decoder, and recurrent language-model component.The recurrent component is implemented with either an LSTM or a ByteNet decoder.
- 3.1 Variational Autoencoder: The VAE replaces deterministic encodings with posterior distributions, samples latent vectors for reconstruction, and regularizes them toward a prior using KL divergence.Its objective combines reconstruction cost with a KL regularizer, supporting generation from latent-space samples.
- 3.2 Deconvolutional Networks: Deconvolutional layers provide parallel up-sampling, making the feed-forward portion computationally efficient and potentially easier to optimize than recurrent architectures.These benefits become more significant as generated text sequences grow longer.
- 3.3 Hybrid Convolutional-Recurrent VAE: The recurrent decoder captures dependencies between sequence elements that a fully feed-forward decoder cannot richly represent under conditional independence assumptions.This lets the latent vector focus more on high-level semantic and stylistic features rather than every text detail.
- 3.4 Optimization Difficulties: Adding the recurrent component can cause KL collapse, so the model uses KL-term annealing and input dropout to encourage meaningful latent representations.The paper notes that KL annealing gradually transitions from a deterministic autoencoder toward a full VAE.
- 3.4 Optimization Difficulties: The auxiliary reconstruction term is computed from deconvolutional activations without previous outputs, forcing reliance on the latent vector while preserving decoder expressiveness.The final objective combines the VAE bound with this intermediate reconstruction term weighted by α.
4 Experiments
Experiments compare the Hybrid model with LSTM-based VAEs on training, latent-information retention, and text generation. The Hybrid architecture trains faster, preserves latent information better for long texts, and uses auxiliary losses and KL annealing to avoid KL collapse.
- Comparison with LSTM VAE: At 30-character inputs, the historyless LSTM autoencoder fails to fit well while the convolutional model converges almost instantaneously.For 50-character sequences, LSTM results are worse; the authors report searching learning rates, clipping thresholds, and layer sizes without matching the convolutional model.
- Comparison with LSTM VAE: The Hybrid model is computationally faster by roughly a factor of two than the LSTM-based model.The authors attribute this to running only one recurrent network per sample and negligible convolutional-part time complexity.
- Comparison with LSTM VAE: For texts longer than 48 characters, the LSTM VAE’s KL component drops to almost zero while the Hybrid model’s KL-to-reconstruction ratio stays roughly constant.The authors interpret this as better latent encoding of long texts by the Hybrid model; LSTM VAE fails to produce meaningful latent vectors above 50 characters in both experiments.
- Controlling the KL term: With sufficiently large α, the Hybrid model finds non-trivial latent vectors under the full VAE loss, while KL annealing enables non-zero KL solutions at smaller α.Increasing α slightly harms the likelihood bound but makes the model rely more on the latent vector.
- Controlling the KL term: When the ByteNet decoder’s receptive field exceeds 3 and α = 0, the KL term collapses to zero; the auxiliary reconstruction term prevents this across receptive-field sizes.The receptive field of the recurrent layer is N + 1 characters for N convolutional layers, and increasing α strengthens the KL component.
- Qualitative tweet generation: In tweet generation, LSTM VAE produces limited and repetitive samples, whereas the Hybrid model produces more diverse samples.Greedy decoding makes sample variation depend only on the latent vector; the reported data use 1M training tweets and a held-out set of 10k samples.
5 Conclusions
The paper introduces a hybrid VAE for text whose convolutional and deconvolutional components support training stability, latent-vector use, and long-sequence generation.
- The model combines a convolutional encoder, deconvolutional decoder, and recurrent layer within a VAE framework.
- The feed-forward architecture helps avoid KL-term collapse, preventing the decoder from reverting to a standard language model.
- An additional cost term encourages the model to rely on the latent vector during training.
- The hybrid model works well on long sequences, which the paper describes as difficult for purely RNN-based VAEs.
- The authors evaluate the trade-off between KL-term and reconstruction loss, including how receptive-field size affects latent-vector adherence.
A Supplementary Material
The supplementary material adds generated samples and latent-space linear operations for the hybrid VAE and examines transitions between generated texts.
- The supplement includes more random samples from the hybrid VAE model and the LSTM VAE baseline.
- It also performs linear operations in latent space to examine transitions between generated texts.
- The reported transitions are smooth and syntactically correct.
A.1 Random Samples
Random-sample comparisons show more varied hybrid-model outputs, while the LSTM VAE produces redundant and uninteresting samples associated with near-zero KL terms.
- The hybrid model produces samples with substantially more variation in both length and content.
- The LSTM VAE more often has KL terms close to zero, resulting in redundant and uninteresting generated samples.
- Increasing dropout rates did not lead to better LSTM VAE samples.
- Table 4 presents hybrid-model samples, whereas Table 5 presents LSTM VAE samples.
A.2 Linear operations in the latent space
The hybrid model’s latent space supports interpolation, with generated texts changing gradually in length and topic while retaining sensible structure.
- Linear interpolation examples are generated between two randomly selected latent-space points.
- Interpolated texts show gradual changes in generated-tweet length along the latent-space trajectories.
- The trajectories maintain consistent topic usage and generally sensible structure.
- The supplementary examples include generated texts with repeated @userid sequences and varied conversational content.
- Table 4 contains hybrid-model random samples, while Table 5 contains LSTM VAE samples, including truncated repetitions of hashtags or @userid sequences.