Source-linked AI summary
Learning Stochastic Recurrent Networks
Justin Bayer, Christian Osendorfer
TL;DR
The paper addresses limitations of factorised recurrent output distributions for high-dimensional, tightly coupled sequences. It combines RNNs with latent variables and SGVB, producing STORNs that improve over factorised-output RNNs and generalize deterministic RNNs. Experiments cover polyphonic music and motion-capture data, while the authors identify stochastic optimization and evaluation scope as limitations.
Problem
Factorised recurrent output distributions can miss dependencies among components of high-dimensional, tightly coupled sequences.
Method
STORNs combine recurrent neural networks with latent variables, using SGVB with recurrent recognition and generating models.
Results
STORNs greatly improve over sRNNs that factorise output variables and generalize deterministic sRNNs.
Takeaways & Limitations
More sophisticated recurrent modeling methods can improve the representative capabilities and performance of RNNs.
Takeaways & Limitations
The method uses a stochastic objective, and its motion-capture evaluation includes MSE despite concerns that MSE can overfit and captures only one residual-distribution moment.
Abstract
from arXiv · showhide
Leveraging advances in variational inference, we propose to enhance recurrent neural networks with latent variables, resulting in Stochastic Recurrent Networks (STORNs). The model i) can be trained with stochastic gradient methods, ii) allows structured and multi-modal conditionals at each time step, iii) features a reliable estimator of the marginal likelihood and iv) is a generalisation of deterministic recurrent neural networks. We evaluate the method on four polyphonic musical data sets and motion capture data.
1 INTRODUCTION
RNNs model sequence distributions flexibly, but factorising output components can miss dependencies in high-dimensional, tightly coupled data. STORNs address this by adding latent variables and training with SGVB.
- 1 INTRODUCTION: RNNs represent sequence distributions by decomposing them into conditional distributions modeled at successive time steps.Each conditional distribution is produced by an RNN output whose components provide distribution statistics.
- 1 INTRODUCTION: Naive Bayes output factorisation simplifies modeling but omits dependencies among components of each time step.The limitation is especially relevant for high-dimensional, tightly coupled sequence spaces.
- 1 INTRODUCTION: Mixture distributions can model dependencies among components, but their parameter count scales at least linearly with the number of mixture components.This approach is described as potentially powerful enough to represent arbitrary component dependencies.
- 1 INTRODUCTION: RBM-based approaches address component dependencies but impose tighter assumptions, use an intractable objective, and require costly MCMC for learning and sampling.The cited restriction includes modeling data with exponential-family posteriors.
- 1 INTRODUCTION: The paper adds latent variables to recurrent networks and uses stochastic gradient variational Bayes to train models for high-dimensional sequences.This proposal targets the limitations of factorised outputs while retaining stochastic-gradient training.
2 PRELIMINARIES
The preliminaries define deterministic RNN sequence modeling and SGVB variational inference. SGVB introduces latent variables, approximates their posterior with a recognition model, and optimizes a tractable upper bound on negative log-likelihood.
- 2 PRELIMINARIES: A simple RNN recursively maps an input sequence through hidden states to produce an output sequence.The input, hidden, and output sequences occupy dimensions κ, γ, and ω, respectively.
- 2 PRELIMINARIES: Network behavior is adapted by optimizing a loss over parameters with gradient-based methods, commonly using negative log-likelihood for sequence data.The data set consists of a finite collection of input sequences.
- 2 PRELIMINARIES: SGVB models p(x) by integrating a latent-variable model p(x|z)p(z), whose integral is generally intractable.Learning therefore uses a variational upper bound on negative log-likelihood.
- 2 PRELIMINARIES: The recognition model q(z|x) variationally approximates p(z|x), enabling fast approximate inference of latent variables from observed data.The generating model p(x|z) is generally unavailable in closed form for inversion.
- 2 PRELIMINARIES: Recognition and generating models may use arbitrary computational forms, provided they output probability distributions and support stochastic training.Their parameters are optimized jointly through stochastic optimization of the variational objective.
- 2 PRELIMINARIES: The model design selects a latent prior, a recognition model with tractable KL divergence, and a data-appropriate generating model.These choices determine the variational and generative components used during learning.
- 2 PRELIMINARIES: If p(x|z) is a universal function approximator, the overall latent-variable model is also a universal approximator of probability distributions.The argument constructs a transformation between x and z using inverse distribution functions.
3 METHODS
STORNs combine stochastic-gradient variational inference with recurrent networks by using latent variables in both recognition and generating models. The latent variables make the hidden-state sequence stochastic while retaining deterministic recurrent computation conditional on each latent sequence, and the model reduces to an sRNN when latent inputs are removed.
- Model construction: STORNs combine SGVB with sRNNs used as both the recognition model q(z_t|x_1:t−1) and generating model p(x_t|z_1:t).The generating sRNN receives latent variables as additional inputs.
- Comparison to RNNs: Removing latent variables, for example by setting W′g_in=0, reduces STORNs to sRNNs and establishes STORNs as a generalisation of deterministic recurrent networks.This preserves the sRNN as a special case of the proposed model.
- The generating model: Latent variables z_1:T are the only uncertain quantities in the hidden-state computation; conditional on x_1:T and z_1:T, each h_t is deterministic.The hidden-state distribution therefore follows a Dirac distribution before marginalising over the latent variables.
- Variational inference: A factorised prior p(z_1:T)=∏_t p(z_t) permits KL-divergence calculations independently across time steps and latent components.The paper restricts the prior to a standard Normal distribution.
- The generating model: Marginalising z_1:T makes the hidden-state sequence stochastic and allows it to model stochastic behaviour despite deterministic hidden-state updates.The resulting hidden-state sequence forms a Markov chain and can act as a universal approximator of sequence distributions.
- Variational inference: Training uses a variational upper bound on negative log-likelihood, samples z_t,k through reparameterisation, computes the KL term, and evaluates −log p(x_1:T|z_1:T).The recognition model supplies Gaussian statistics, while the generating model processes the sampled latent sequence.
4 EXPERIMENTS
The experiments evaluate STORN on polyphonic music and motion-capture sequences using stochastic variational training, likelihood-based selection, and reconstruction-based analyses. STORN improves over factorised-output models, while RNN-NADE remains competitive; motion-capture evaluation also reports strong MSE and negative log-likelihood results.
- Experimental setup: STORN was evaluated on four polyphonic music data sets and motion-capture data using recurrent recognition and generating models.The experiments used random hyperparameter searches and reported test negative log-likelihood for selected models.
- Polyphonic music generation: STORN improves over FD-RNN, sRNN, and Deep RNN on all evaluated cases, although RNN-NADE retains a competitive edge.The authors suggest noisy gradients from stochastic training as one possible explanation, but state that the results alone do not establish the reason.
- Motion capture data: Motion-capture likelihoods used a fixed Gaussian standard deviation during training and a post-training binary search for model selection.The recognition model was bidirectional, and the resulting validation negative log-likelihood estimate was used for selection.
- Motion capture data: 15.99 was the estimated negative log-likelihood for the motion-capture data.RNN-RBM, RTRBM, and cRBM lacked tractable log-likelihood estimates for direct probabilistic comparison.
- Motion capture data: Missing motion-capture intervals were initialized with random noise and reconstructed using a maximum a posteriori estimate of the latent variables.The procedure replaced time steps 30 to 40 across all 49 channels in randomly selected sequences.
5 DISCUSSION AND FUTURE WORK
The authors conclude that stochastic RNNs trained with SGVB improve over sRNNs that factorise output variables, strengthening sequence-model representation capabilities. They also identify the stochastic objective as a weakness and note that more powerful recurrent architectures remain an unshown extension.
- STORNs greatly improve over sRNNs that erroneously assume a factorisation of output variables.
- STORNs can be extended to LSTM or deep transition operators, but these architectures are not demonstrated in this work.
- Motion-capture samples show diverse outcomes while the learned distribution remains rather unimodal.The samples were generated after a 20-step stimulus prefix.
- The stochastic objective function is identified as an apparent weakness, though future optimization advances may benefit STORN.