Source-linked AI summary
Sequential Neural Models with Stochastic Layers
Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, Ole Winther
TL;DR
The paper addresses efficient uncertainty propagation in recurrent latent states by combining a deterministic RNN with a stochastic nonlinear SSM. Its structured variational inference follows the resulting posterior factorization, yielding state-of-the-art speech results and comparable polyphonic-music performance.
Problem
Complex sequential data can benefit from uncertainty in recurrent hidden states, but combining nonlinear recurrent dynamics with tractable inference over stochastic latent paths remains challenging.
Method
SRNN temporally interlocks a deterministic gated RNN with a nonlinear SSM and uses a structured variational inference network modeled on the posterior factorization.
Results
SRNN achieves state-of-the-art performance on Blizzard and TIMIT speech modeling and performs comparably to competing methods on polyphonic music modeling.
Takeaways & Limitations
Separating deterministic and stochastic layers supports inference using whole-sequence information and a residual parameterization for the non-stationary posterior.
Abstract
from arXiv · showhide
How can we efficiently propagate uncertainty in a latent state representation with recurrent neural networks? This paper introduces stochastic recurrent neural networks which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model. The clear separation of deterministic and stochastic layers allows a structured variational inference network to track the factorization of the model's posterior distribution. By retaining both the nonlinear recursive structure of a recurrent neural network and averaging over the uncertainty in a latent path, like a state space model, we improve the state of the art results on the Blizzard and TIMIT speech modeling data sets by a large margin, while achieving comparable performances to competing methods on polyphonic music modeling.
1 Introduction
The paper combines deterministic RNN dynamics with stochastic SSM states to model sequential data while enabling structured variational inference. This SRNN approach improves speech-modeling results substantially and matches competing methods on polyphonic music.
- 1 Introduction: The layered separation of deterministic and stochastic states creates independence properties that guide an efficient structured posterior approximation.The inference network uses whole-sequence information and is designed around the posterior factorization.
- 1 Introduction: SRNNs stack a gated RNN with a nonlinear SSM, combining deterministic hidden states with stochastic state transitions.The SSM transitions are neural-network parameterized and depend on the RNN hidden state, allowing use of long-term information.
- 1 Introduction: The inference network can learn each variational mean from the predictive prior mean, improving approximation of the non-stationary posterior.This design avoids requiring the network to learn the full variational approximation directly at every time step.
- 1 Introduction: SRNN improves state-of-the-art results on Blizzard and TIMIT speech modeling by a large margin.The evaluation covers speech and polyphonic music modeling tasks.
- 1 Introduction: SRNN performs comparably to competing models on polyphonic music modeling.
2 Recurrent Neural Networks and State Space Models
RNNs summarize sequence history with deterministic recurrent states, whereas SSMs represent latent states probabilistically and average over their uncertainty. The paper proposes stacking an SSM on an RNN to retain nonlinear gated dynamics while enabling structured posterior inference.
- 2 Recurrent Neural Networks and State Space Models: RNNs recursively propagate deterministic states, while SSMs treat latent states as random variables and average over the latent process.RNNs use nonlinear functions such as LSTM or GRU cells; SSM transitions and observations are often restricted for tractable inference.
- 2 Recurrent Neural Networks and State Space Models: The paper asks whether nonlinear stochastic SSM transitions can be combined with the gated activation mechanism of RNNs.
- 2 Recurrent Neural Networks and State Space Models: Stacking an SSM on top of an RNN creates a more expressive model whose posterior factorization guides a backward-recursive variational approximation.
3 Stochastic Recurrent Neural Networks
SRNNs interlock a deterministic RNN with a stochastic SSM, then use structured variational inference to approximate the latent-state posterior efficiently. The inference network exploits temporal structure, while residual parameterization addresses a training failure mode in posterior approximation.
- SRNN generative model: SRNNs define a generative model by temporally interlocking an SSM with an RNN, with deterministic states driven recursively and stochastic states following SSM transitions.The RNN state is computed from inputs, while the stochastic state directly depends on its preceding stochastic state and the deterministic RNN state.
- SRNN generative model: The model separates deterministic and stochastic components: the RNN remains unaffected by noisy samples, while the stochastic prior retains Markov dependence on zt−1.The stochastic transition is Gaussian with diagonal covariance, and its parameters depend on zt−1 and the deterministic state.
- Variational inference: Because z1:T cannot be integrated analytically, training maximizes an ELBO over sequences using a shared inference network and stochastic-gradient optimization.The ELBO combines an expected log likelihood with a KL divergence and is estimated using sampling and the reparameterization trick.
- Structured posterior: The variational posterior mirrors the model structure by treating the deterministic trajectory as the same delta-function state sequence and factorizing stochastic states conditionally over time.This cancels the matching deterministic factors and preserves posterior dependencies rather than using a mean-field approximation across time.
- Structured posterior: A backward recurrent network incorporates future observations and deterministic states, enabling smoothing; filtering instead uses only information available through time t.Shared parameters across time steps allow both formulations to handle variable-length sequences.
- Inference refinement: During training, the inference network can imitate the prior and produce a small KL term without using future information, so residual parameterization helps it track the changing posterior.The residual learns to correct predictive-prior dynamics using future deterministic states and observations.
4 Results
SRNN is evaluated on speech and polyphonic music, using conservative ELBO-based reporting alongside comparisons with competing models. It outperforms competing methods on Blizzard and TIMIT, performs comparably on four music datasets, and shows informative posterior behavior on Blizzard.
- Speech modeling: SRNN outperforms all competing methods for speech modeling on Blizzard and TIMIT.The comparison reports average log-likelihood per test sequence, while SRNN uses the more conservative evidence lower bound.
- Inference variants: Smoothing improves the variational approximation over filtering, and residual learning improves performance in 3 of 4 cases.These comparisons concern SRNN’s speech-modeling results.
- Posterior behavior: The average KL term increases at transitions in Blizzard raw audio, indicating that output information improves inference.Reconstructions of the output mean and log-variance are consistent with the original signal.
- Polyphonic music: SRNN performs comparably to other state-of-the-art methods on all four polyphonic music datasets.The reported music results use smoothing and residual learning, while filtering and direct mean learning perform similarly.
5 Related work
Related stochastic RNN models differ in how stochastic units depend across time, interact with deterministic units, and evaluate likelihoods. SRNN separates deterministic and stochastic layers while retaining nonlinear state transitions conditioned on recurrent state.
- Related stochastic RNNs: STORN and DRAW use isotropic Gaussian stochastic units that are independent across time and feed into deterministic RNN units.Both methods use the reparameterization trick to optimize an ELBO.
- Related stochastic RNNs: VRNN makes the latent prior depend on deterministic states, which recursively depend on previous deterministic and stochastic units.SRNN instead clearly separates its deterministic and stochastic parts.
- Other approaches: Other related approaches use sequential Monte Carlo with deep proposals or discrete stochastic units optimized with NVIL.These examples are discussed as extensions of RNNs with stochastic units.
6 Conclusion
The paper extends RNNs with nonlinear state-space models and designs structured variational inference around posterior independence properties. SRNN reaches state-of-the-art speech results and comparable polyphonic-music performance.
- Conclusion: SRNN combines nonlinear state-space models with RNNs to extend recurrent sequence-modeling capabilities.The conclusion frames this combination as the paper’s central modeling extension.
- Conclusion: The inference network uses whole-sequence information and the Resq parameterization to track a non-stationary latent-state posterior.Its design is motivated by independence properties of the intractable true posterior.
- Conclusion: SRNN achieves state-of-the-art performance on Blizzard and TIMIT speech modeling while performing comparably to competing methods on polyphonic music.This is the paper’s reported cross-task conclusion.
A.1 Blizzard and TIMIT
Blizzard and TIMIT use raw-audio sequence modeling with different dataset splits and test-sequence lengths. SRNN predicts the next audio vector using GRU-based deterministic states and 256-dimensional stochastic states.
- Preprocessing: Blizzard audio is chunked into 2-second segments and 200-sample vectors, producing 160 RNN steps per segment.Training uses backpropagation through time for 0.5 seconds, yielding four updates per 2-second segment.
- Model configuration: The model predicts the next vector from the current vector using a fully factorized Gaussian output distribution.The deterministic RNN uses GRUs, with 2048 units for Blizzard and 1024 for TIMIT.
- Data and evaluation: Blizzard uses 90% training, 5% validation, and 5% testing, while TIMIT uses its predefined test set and splits the remainder 95%/5%.Both use average log-likelihood evaluation under the reported testing setup.
- Model configuration: Both datasets use 256-dimensional stochastic states, while neural networks have two layers with dataset-specific widths.The two-layer networks use 1024 units for Blizzard and 512 for TIMIT.
A.2 Polyphonic music
The polyphonic music experiments use the speech-modeling architecture with Bernoulli output variables for active notes, while reducing model size.
- The model uses the same architecture as the speech modeling experiments, except for Bernoulli output variables modeling active notes.
- The GRU networks contain 300 deterministic hidden units and 100 stochastic units.
- The stochastic-unit distributions are parameterized by neural networks with one layer of 500 units.