Source-linked AI summary
Z-Forcing: Training Stochastic Recurrent Networks
Anirudh Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, Yoshua Bengio
TL;DR
The paper addresses how to train autoregressive latent-variable models whose powerful decoders can make latent representations difficult to learn. It combines timestep-specific latent variables, a backward recurrent posterior, and an auxiliary reconstruction cost, achieving state-of-the-art speech results and interpretable language-modeling representations.
Problem
Powerful autoregressive decoders can capture most data entropy, making meaningful latent representations difficult to extract.
Method
The model uses timestep-specific latent variables, a backward RNN posterior, latent-conditioned recurrent dynamics, and an auxiliary cost reconstructing backward-network states.
Results
The model achieves state-of-the-art results on TIMIT and Blizzard, while the auxiliary cost outperforms KL annealing in the explored tasks.
Takeaways & Limitations
The auxiliary cost promotes more effective use of latent variables and helps learn interpretable representations in language modeling.
Takeaways & Limitations
The posterior approximation drops dependence on future latent variables, introducing intrinsic bias and potentially excluding the true posterior from its function class.
Abstract
from arXiv · showhide
Many efforts have been devoted to training generative latent variable models with autoregressive decoders, such as recurrent neural networks (RNN). Stochastic recurrent models have been successful in capturing the variability observed in natural sequential data such as speech. We unify successful ideas from recently proposed architectures into a stochastic recurrent model: each step in the sequence is associated with a latent variable that is used to condition the recurrent dynamics for future steps. Training is performed with amortized variational inference where the approximate posterior is augmented with a RNN that runs backward through the sequence. In addition to maximizing the variational lower bound, we ease training of the latent variables by adding an auxiliary cost which forces them to reconstruct the state of the backward recurrent network. This provides the latent variables with a task-independent objective that enhances the performance of the overall model. We found this strategy to perform better than alternative approaches such as KL annealing. Although being conceptually simple, our model achieves state-of-the-art results on standard speech benchmarks such as TIMIT and Blizzard and competitive performance on sequential MNIST. Finally, we apply our model to language modeling on the IMDB dataset where the auxiliary cost helps in learning interpretable latent variables. Source Code: \url{https://github.com/anirudh9119/zforcing_nips17}
1 Introduction
The paper combines stochastic recurrent modeling with a backward posterior and an auxiliary objective that makes latent variables encode future sequence information. This approach targets stronger performance and more interpretable latent representations in sequential data.
- Autoregressive RNNs model variable-length sequences by sharing weights across timesteps and updating hidden states to summarize prior observations.
- Latent variables can encode higher-level factors such as objects or speakers, potentially supporting control, semi-supervised learning, transfer learning, and interpretability.
- The proposed model assigns one latent variable to each timestep, uses a backward RNN for inference, and conditions future recurrent dynamics on the latent variables.
- An auxiliary task-agnostic loss forces latent variables to predict backward-encoder states, thereby encoding future information in the sequence.
- The paper unifies backward posteriors, conditional priors, and latent-conditioned hidden dynamics, while reporting stronger results than KL annealing in explored tasks.
2 Background
The background frames the model as a variational latent-variable approach with an autoregressive decoder. It motivates posterior approximation and decoder design as central choices for sequential generative modeling.
- The VAE framework introduces latent variables z and trains generative and inference networks to model observed data.
- Because marginal likelihood is generally intractable, variational training maximizes an evidence lower bound using an approximate posterior qφ(z|x).
- The ELBO becomes tight when the approximate posterior matches the true posterior, linking variational optimization to marginal log-likelihood.
- The KL term measures dependence between observations and latent variables; zero KL indicates independence in the stated formulation.
- Neural-network parameterizations and reparameterizable distributions enable gradient-based optimization through latent-variable sampling.
- Autoregressive decoders model each observation conditioned on prior observations and, in sequential latent-variable models, prior latent variables.
3 Proposed Approach
The proposed stochastic recurrent model combines backward posterior inference, conditional priors, and latent variables that condition forward recurrent dynamics. An auxiliary reconstruction objective trains latents to encode future information while retaining an expressive autoregressive decoder.
- 3 Proposed Approach: The model unifies backward posterior inference, conditional priors, and timestep-specific latent variables that condition the forward recurrent dynamics.These choices combine techniques associated with earlier stochastic recurrent models into one framework.
- 3.1 Generative Model: A forward autoregressive LSTM predicts the next observation from its hidden state, while latent variables influence recurrent dynamics rather than directly producing the output.The authors report better performance when the latent variables are excluded from the direct computation of next-output probabilities.
- 3.1 Generative Model: The conditional prior for each latent variable is computed from the previous hidden state of the forward network.The prior uses a diagonal multivariate Gaussian parameterization as a common VAE choice.
- 3.2 Inference Model: A backward LSTM provides states containing future-sequence information, which shapes the approximate posterior and lets each latent act as a future plan for forward predictions.The posterior network combines the previous forward state with the corresponding backward state.
- 3.2 Inference Model: The posterior approximation drops dependence on future latent variables z_t+1:T, introducing intrinsic bias because the true posterior may fall outside the approximator’s function space.This design contrasts with SRNN, which omits latent variables from forward recurrent dynamics.
- 3.3 Auxiliary Cost: The auxiliary reconstruction model trains each z_t to encode future information represented by the backward state b_t.It adds a task-agnostic training signal intended to help escape apparent local minima caused by short-term reconstructions.
- 3.4 Learning: Training regularizes the variational lower bound with the auxiliary cost and uses backpropagation through time with reparameterized posterior samples.The auxiliary prediction gradients are disconnected from the backward network during optimization.
4 Experiments
Experiments evaluate the stochastic recurrent model on speech, sequential MNIST, and IMDB language modeling. The auxiliary cost improves speech-modeling performance relative to KL annealing, while latent-variable behavior is examined through likelihood bounds and interpolation.
- Speech Modeling and Sequential MNIST: The model achieves state-of-the-art results on Blizzard and TIMIT speech modeling and competitive results on sequential MNIST.Sequential MNIST evaluation uses test negative log-likelihood, while Blizzard and TIMIT use average log-likelihood per sequence.
- Speech Modeling and Sequential MNIST: Auxiliary-cost models outperform KL-annealed models on both Blizzard and TIMIT.On Blizzard, KL annealing eventually converges to slightly lower log-likelihood than training without annealing; TIMIT shows a slight synergy between annealing and auxiliary cost.
- Speech Modeling and Sequential MNIST: KL annealing hurts overall performance on sequential MNIST, despite the model remaining competitive with strong baselines such as DRAW.The sequential MNIST task generates binarized digits pixel by pixel.
- Speech Modeling and Sequential MNIST: Figure 2 shows that auxiliary cost produces higher and more stable KL divergence during Blizzard and TIMIT training.The curves compare models selected using validation-set hyperparameters for KL annealing and auxiliary-cost weights.
- Language Modeling: On IMDB, IWAE substantially improves perplexity over the ELBO, but training with IWAE yields no noticeable validation-perplexity improvement.The ELBO may severely underestimate likelihood when the approximate posterior loosely matches the true posterior.
- Language Modeling: Latent-space interpolation uses inferred timestep latents to generate intermediate sentences, with latent variables generally capturing sentence length.Table 3 contrasts greedy argmax decoding with random samples across interpolation values.
5 Conclusion
The paper proposes a recurrent stochastic generative model combining backward inference, latent-conditioned recurrent dynamics, and an auxiliary reconstruction cost. It reports state-of-the-art speech results and suggests the auxiliary cost promotes latent-variable use more effectively than KL annealing.
- Conclusion: The model combines a backward inference RNN with latent variables that condition recurrent dynamics and an auxiliary cost reconstructing backward-network states.The auxiliary objective explicitly encodes a summary of future observations in the latent variables.
- Conclusion: The model achieves state-of-the-art results on the TIMIT and Blizzard speech benchmarks.The conclusion also reports that the auxiliary cost appears more effective than similar strategies such as KL annealing.