PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamical Variational Autoencoders: A Comprehensive Review

Laurent Girin, Simon Leglaive, Xiaoyu Bie, Julien Diard, Thomas Hueber, Xavier Alameda-Pineda

arXiv:2008.12595v4cs.LGstat.ML

TL;DR

Temporal VAEs must model dependencies within sequential data rather than process vectors independently, but comparing DVAE models across papers is difficult. This paper unifies DVAE formulations, relates them to classical temporal models, and benchmarks selected models, finding robust speech resynthesis with fine-tuned SRNN but poorer 3D-motion performance for SRNN-GM.

  • Problem

    Independent processing of data and latent vectors is suboptimal for correlated temporal sequences, while cross-paper DVAE comparisons remain difficult because evaluations, notation, and experimental setups differ.

  • Method

    The paper organizes temporal VAE extensions into the DVAE class, aligns their presentation, relates them to classical models, and examines dependency-matched inference and sequential Monte Carlo approaches.

  • Results

    Fine-tuned SRNN resynthesis was reasonably robust for speech data, whereas SRNN-GM performed more poorly on 3D motion with limited gain over a vanilla VAE.

  • Takeaways & Limitations

    For temporal-data applications, VRNN or SRNN are relevant choices, but causal inference constraints can prevent using exact dependency structures at inference time.

  • Takeaways & Limitations

    The benchmark remains limited because an extended comparison across DVAE configurations and tasks is not yet available, and latent disentanglement is difficult to evaluate.

Abstract

from arXiv · show

Variational autoencoders (VAEs) are powerful deep generative models widely used to represent high-dimensional complex data through a low-dimensional latent space learned in an unsupervised manner. In the original VAE model, the input data vectors are processed independently. Recently, a series of papers have presented different extensions of the VAE to process sequential data, which model not only the latent space but also the temporal dependencies within a sequence of data vectors and corresponding latent vectors, relying on recurrent neural networks or state-space models. In this paper, we perform a literature review of these models. We introduce and discuss a general class of models, called dynamical variational autoencoders (DVAEs), which encompasses a large subset of these temporal VAE extensions. Then, we present in detail seven recently proposed DVAE models, with an aim to homogenize the notations and presentation lines, as well as to relate these models with existing classical temporal models. We have reimplemented those seven DVAE models and present the results of an experimental benchmark conducted on the speech analysis-resynthesis task (the PyTorch code is made publicly available). The paper concludes with a discussion on important issues concerning the DVAE class of models and future research guidelines.

1.1 Deep Dynamical Bayesian Networks

Deep dynamical Bayesian networks combine Bayesian dependency structures, temporal modeling, and deep neural networks for sequential data. The review frames dynamical variational autoencoders as VAEs augmented with temporal models and unifies their relationships, inference, training, and benchmark considerations.

  • Dynamical Bayesian networks extend Bayesian networks across time with repeated time-slice dependencies and cross-time dynamics, often implemented through RNN recursion or first-order Markov processes.
  • Deep Bayesian networks use neural networks to generate distribution parameters, combining probabilistic-model interpretability with deep neural networks’ high-dimensional and multimodal modeling capacity.
  • DDBNs combine deep Bayesian modeling with temporal dependencies, supporting sequence modeling but potentially increasing training complexity and cost.
  • Original VAEs process data vectors independently, making them suboptimal for correlated temporal sequences.
  • DVAEs are VAEs including temporal models for sequential data, trained by designing an inference model and maximizing the corresponding variational lower bound.
  • The review formally defines DVAEs, relates them to VAEs, RNNs, and SSMs, details seven selected models, and discusses inference dependencies and variational-bound computation.
  • The accompanying review and code aim to make different DVAE models, their classical connections, cross-connections, and unification more accessible for comparison.

Variational Autoencoders

VAEs extend autoencoders with probabilistic encoding and decoding, enabling latent-variable generation and approximate inference for high-dimensional data. Their training balances reconstruction accuracy with regularization while commonly using diagonal-covariance Gaussian likelihoods for computational tractability.

  • Autoencoder foundation: An autoencoder deterministically maps a high-dimensional input through a low-dimensional latent representation and reconstructs the input.The encoder produces z ∈ R^L with L ≪ F, while the decoder reconstructs x from z.
  • Probabilistic formulation: A VAE replaces deterministic encoding and decoding with probability distributions, allowing new data generation from latent values.The decoder outputs distribution parameters for x, while the encoding of z is also probabilistic.
  • Probabilistic formulation: The decoder models pθx(x|z) with parameters given by nonlinear functions of z implemented by a DNN.The standard VAE uses decoder parameters θx, with θz empty and θ = θx.
  • Model assumptions: Diagonal covariance matrices are assumed because full covariance parameters grow quadratically with dimension and often require matrix inversion.The review adopts Gaussian conditional distributions with diagonal covariance matrices for consistency and computational reasons.
  • Training and inference: VAE training minimizes divergence between the true data distribution and the model distribution, using Monte Carlo estimates when the marginal likelihood is intractable.The estimated parameters can subsequently be used to generate new data.
  • Training and inference: The variational lower bound combines reconstruction accuracy with regularization that encourages the approximate posterior to remain close to the prior.With an independent Gaussian prior, the regularization term encourages latent entries to encode different factors of variation.

Definition of Dynamical VAEs

Dynamical variational autoencoders extend VAEs to temporally correlated observed and latent sequences, using probabilistic dependencies implemented with neural networks or state-space mechanisms. The review emphasizes causal factorization, inference structure, and training-mode choices.

  • Definition of DVAEs: DVAEs are VAEs that model temporal dependencies in observed and latent sequences, with generative distributions parameterized by deep neural networks.They may also operate in driven mode, where an observed input sequence conditions the stochastic output and latent sequences.
  • Temporal factorization: The ordering chosen when applying the chain rule determines the factorization and the resulting sampling process.Causal generation alternates sampling latent and observed vectors over time, unlike factorizations requiring the complete latent sequence first.
  • Temporal factorization: Causal models restrict time-t distributions to past and present information, whereas anticausal and noncausal models use future or mixed temporal context.The review focuses primarily on causal DVAEs and identifies RVAE as the only known noncausal model in its surveyed literature.
  • Inference design: Approximate posteriors should ideally follow the exact posterior’s dependency structure, but early DVAE studies often selected inference models without that reference.D-separation provides a systematic way to identify dependencies and simplify posterior factorization.
  • Training and generation: Generation-mode training reduces train–test mismatch in autoregressive DVAEs and substantially improves performance, although teacher-forcing at both stages remains slightly better.Teacher-forcing uses ground-truth history, whereas generation mode reinjects previously generated outputs and is required for practical generation or compression.

Deep Kalman Filters

Deep Kalman Filters combine state-space models with neural networks, treating stochastic latent states as temporally evolving representations. Their inference models range from local or causal context to backward and bidirectional recurrent processing, with DKS matching the exact posterior structure more closely.

  • Model formulation: DKF combines a state-space model with neural networks and is also described as a deep SSM, Markovian VAE decoder, or fully stochastic RNN.The stochastic state replaces the deterministic recurrent state, while neural networks parameterize the observation and transition distributions.
  • Model development: Krishnan et al. (2017) retained the DKF generative model as DMM but clarified inference choices, including whether and how models depend on the previous latent state.The review keeps the DKF denomination for the shared generative model.
  • Model formulation: The DKF observation model is equivalent to a VAE decoder when the stochastic state has lower dimension than the observation.This links dimensional latent representation learning with temporally structured state-space modeling.
  • Posterior structure: The exact DKF posterior depends on the previous latent state and present-to-future observations and inputs.It is first-order Markovian in the latent state while being anticausal in the observed and input sequences.
  • Posterior structure: Early DKF inference variants used instantaneous, local-context, complete-past, or whole-sequence observations, with MLP, RNN, or bidirectional-RNN parameterizations.These choices differ in how much temporal observation context they provide to infer each latent state.
  • Inference implementations: DKS combines the previous latent state with present and future observations through a backward RNN and an additional combining network.A more general bidirectional model adds a forward RNN; removing it yields DKS.

Kalman Variational Autoencoders

Kalman Variational Autoencoders insert a learned feature sequence between observations and latent dynamics, separating nonlinear feature extraction from linear state-space modeling. Their inference combines a VAE encoder with a Kalman filter or smoother, while adaptive transition parameters address abrupt trajectory changes.

  • Generative model: KVAE inserts features a_t between observations x_t and latent states z_t, separating a deep feature extractor from a dynamical model.The feature sequence can be lower-dimensional than the observations and is intended to encode object properties, while z_t encodes their dynamics.
  • Generative model: The KVAE submodel on inputs, latent states, and features is a classical linear-Gaussian LDS, while the observation-to-feature mapping uses a deep neural network.This combines nonlinear representation learning with linear dynamical modeling.
  • Generative model: Given the latent state sequence, features are independent; given the feature sequence, observations are independent.These conditional independences support the model’s factorization.
  • Adaptive dynamics: To address LDS limitations with abrupt trajectory changes, KVAE can generate time-varying transition parameters as combinations of predefined parameter-bank elements.An LSTM estimates the combination coefficients from past features, and a related transition model was proposed independently for DVBF.
  • Inference model: The posterior over latent states has a closed-form Kalman filter or smoother solution when the feature sequence is known.The inference model therefore combines this Kalman solution with a neural approximation for the feature sequence.
  • Inference model: The combined inference solution benefits from Kalman efficiency, especially after reducing the observation-to-feature dimension.The review describes this dimension reduction as reducing the burden of the inverse-matrix calculations required by the Kalman solution.
  • Inference model: KVAE inference first samples features from qφ(a_t|x_t), then applies a standard Kalman smoother to sample the latent sequence.This procedure supports joint learning of the VAE parameters and the linear-Gaussian LDS parameters.

STOchastic Recurrent Networks

STORN combines deterministic recurrent dynamics with stochastic latent states, while its practical inference model omits dependencies present in the exact posterior.

  • Generative model: STORN combines an internal deterministic state h_t with an internal stochastic state z_t in a recurrent generative model.The model uses z_t as an additional input to the recurrent network.
  • Generative model: z_t is independently standard Gaussian, so temporal dependence arises through the recurrence on h_t and its use to generate x_t.There is no explicit temporal model on the prior distribution of z_t.
  • Model representations: The developed STORN representation includes deterministic variables, whereas the compact representation retains only random variables and can hide implementation-specific links.Optimization acts on parameters of the developed form, which is only a subgroup of models satisfying the compact form.
  • Inference: The exact posterior for z_t depends on past latent states and observed variables across the sequence, including future observations.This follows because z_t propagates through the recurrent state into later outputs.
  • Inference: STORN’s approximate posterior q_phi(z_t|z_1:t−1,x_1:T)=q_phi(z_t|x_1:t) omits past latent states and future observations.The resulting VLB uses analytic KL terms and Monte Carlo estimates for intractable expectations.

Variational Recurrent Neural Networks

VRNN combines a VAE with an RNN, using recurrent states, feature extractors, and a temporally structured latent distribution; its inference approximation remains incomplete relative to the exact posterior.

  • Generative model: VRNN combines a variational autoencoder with a recurrent neural network for sequential modeling.Its generative model uses recurrent internal states and latent variables at each time step.
  • Generative model: VRNN uses DNN feature extractors parameterized by tau, alongside recurrent networks for its observation and latent-state distributions.The feature extractors were described as important in practice.
  • Inference: The same internal state h_t is shared by the VRNN encoder and decoder, and inference depends on past outputs and past latent states.These choices make the approximate model closer to the exact posterior than STORN’s corresponding construction.
  • Generative model: VRNN makes no conditional-independence assumption in its generative model, making it the most general DVAE model described up to that point.Its variational lower bound therefore starts from the most general DVAE expression.
  • Inference: The approximate posterior still assumes q_phi(z_t|z_1:t−1,x_1:T)=q_phi(z_t|z_1:t−1,x_1:t), excluding future observations after time t.KL divergences are computed analytically and intractable expectations by Monte Carlo estimates.
  • Improved VRNN: An improved VRNN adds a backward RNN, future-information objectives, and removes the direct z_t-to-x_t link to improve reported performance.The authors reported better performance when latent variables were prevented from directly producing the next output.

Stochastic Recurrent Neural Networks

SRNN stacks a deterministic recurrent neural network with a state-space model, explicitly modeling first-order latent-state transitions and using smoothing inference aligned with the exact posterior.

  • Generative model: SRNN combines a deterministic recurrent neural network with a state-space model to form a stochastic sequential neural generative model.The review presents it in predictive mode by replacing the external input with x_t−1.
  • Generative model: The internal state h_t remains deterministic, while the latent state z_t is integrated at the decoder level.This provides a clearer separation of deterministic and stochastic layers than VRNN.
  • Latent dynamics: SRNN adds an explicit first-order Markov dependency from z_t−1 to z_t, combined with dependence on the recurrent state h_t.Unlike VRNN, this temporal dependency is explicit rather than mediated only through h_t.
  • Latent dynamics: The GRU recurrent state allows the state-space model to use long-term information captured by the RNN.The recurrent state is a deterministic function of past observations.
  • Inference: The exact posterior for z_t depends on the previous latent state and the whole observation sequence.SRNN’s approximate posterior uses a gated backward RNN and feed-forward network with the same dependency structure.
  • Inference: SRNN inference combines a forward pass through the shared recurrent state with a backward pass over the data sequence.The resulting smoothing process can alternatively be replaced by an instantaneous DNN filtering process.

Recurrent Variational Autoencoders

RVAE models speech sequences with recurrent latent representations in causal or noncausal forms, with the noncausal model using forward and backward recurrences over the latent sequence.

  • Model scope: RVAE was introduced for clean-speech representation in speech enhancement and was trained offline using VAE methodology.It was combined with a Gaussian noise model and nonnegative matrix factorization within a Bayesian framework.
  • Observation model: RVAE models sequences of complex-valued STFT vectors with a multivariate zero-mean circular complex Gaussian observation distribution.The review later uses a usual Gaussian distribution for real-valued observations to facilitate comparison.
  • Generative model: The model distinguishes instantaneous, causal, and noncausal observation dependencies through time-frame sets containing the current, past-and-present, or complete latent sequence.The review focuses on the causal and noncausal cases because the instantaneous case is equivalent to the original VAE.
  • Causal model: The causal RVAE recurrent state is updated from z_t and h_t−1, omitting the previous observation from the state update.This is a simplified version of the STORN or VRNN recurrent updates.
  • Noncausal model: The noncausal RVAE is the first noncausal DVAE identified in the review and combines forward and backward RNNs on z_t.Its recurrent representation h_t depends on the complete latent sequence z_1:T.
  • Inference: RVAE inference combines forward latent-state processing with backward observation processing, and the noncausal case extends the latent processing across the full sequence.The review describes the causal and noncausal VLB derivations as analogous, while detailing only the causal case.

Disentangled Sequential Autoencoders

DSAE augments frame-level latent dynamics with a sequence-level vector for content, using LSTM and neural-network components for generation and inference. Its proposed approximate posteriors do not fully match the exact posterior dependencies.

  • Model structure: DSAE adds a sequence-level latent vector v to frame-level latents z1:T, assigning v to sequence characteristics and zt to time-dependent features.The review illustrates this distinction with object characteristics versus object dynamics in video clips.
  • Model structure: For speech, DSAE uses an LSTM-parameterized Gaussian dynamical model pθz(zt|z1:t−1) and a feed-forward-DNN Gaussian observation model pθx(xt|zt, v).
  • Model structure: Compared with a DKF in undriven mode conditioned on v, DSAE replaces the first-order Markov latent process with a virtually infinite-order LSTM-based model.
  • Inference model: The posterior first estimates v from the entire observation sequence, then updates dynamics using v, present and future observations, and previous latent states.This decomposition follows from D-separation.
  • Inference model: The factorized inference model uses qφ(zt|xt) frame-wise and performs worse than the full inference model, which incorporates structured dependencies.
  • Inference model: Neither the factorized nor full approximation follows all dependencies of the exact posterior distribution, and the inference graphical model includes dependencies absent from that posterior.
  • Training: Computing the variational bound requires first computing qφv(v|x1:T), then evaluating latent posterior parameters and sampling prior latent states for successive KL terms.

Brief tour of other models

The review situates several related DVAE models by their latent structure, posterior approximation, dynamical parameterization, and sequence representation. These models include structured, switching, stochastic-transition, disentangled, recurrent, and segment-level approaches.

  • Models related to DKF: Latent LDS models use linear-Gaussian latent dynamics with DNN observation models; switching variants add a discrete latent variable selecting among dynamical models.
  • Models related to DKF: Structured VAEs combine structured mean-field inference with conjugate observation and latent models to make inference and training particularly efficient.
  • Models related to DKF: Black-box deep SSMs use multivariate Gaussian approximate posteriors with block-tridiagonal inverse covariance matrices and corresponding fast, scalable inference.
  • Models related to DKF: DVBFs use stochastic parameters in dynamical models, such as DNN-weighted combinations of linear-Gaussian transition matrices and vectors.
  • Disentangled models: DSSM separates sequence-level content from frame-level dynamics by conditioning the dynamical model on a sequence-level variable, unlike DSAE’s observation conditioning.
  • Recurrent models: VRAE encodes an entire sequence into one latent vector z, while its forward-RNN generator predicts observations autoregressively from z and past observations.
  • Segment-level models: FHVAE splits sequences into fixed-size segments and assigns segment-level z and sequence-level v variables to represent information at different temporal scales.

Experiments

Experiments show that DVAEs reconstruct and generate sequential signals better than a vanilla VAE, with autoregressive models generally strongest on human motion and SRNN/VRNN leading speech results. Training in generation mode exposes train/test mismatch, while scheduled sampling reduces but does not eliminate the resulting degradation.

  • Speech analysis-resynthesis: 6.9–11 dB SI-SDR: all tested DVAEs achieved fair speech reconstruction while compressing each 513-dimensional vector into a 16-dimensional latent vector.The models also use a 128-dimensional recurrent state encoding the past.
  • Speech analysis-resynthesis: All DVAE models outperformed the standard VAE on speech reconstruction, with SRNN performing best and VRNN second.The standard VAE reconstructed noisier and blurrier harmonics, illustrating the value of temporal modeling.
  • Speech generation: Generation-mode speech examples produced speech-like spectrograms for SRNN and VRNN, with VRNN giving the best result in the illustrated sentence; STORN quality was lower.Nonautoregressive DKF and DSAE also generated harmonic, smoothly evolving speech-like sounds.
  • Generation-mode training: Scheduled sampling reduced SRNN’s train/test mismatch in generation mode, but SRNN-GM remained below DKF and RVAE in the speech experiment.Teacher-forced SRNN tested in generation mode performed very poorly, motivating scheduled-sampling fine-tuning.

Discussion

The discussion emphasizes DVAEs’ flexibility in modeling sequential data, while identifying latent disentanglement, posterior collapse, and representation design as ongoing challenges.

  • Motivation: DVAEs combine dynamical models for sequential data with VAEs for latent variation, aiming to separate data dynamics from other variation factors.
  • Flexibility: The DVAE framework can support models ranging from linear to nonlinear dynamics and from first-order to much higher-order temporal dependencies.This flexibility is enabled by the VAE methodology and deep-learning toolkits, subject to computational cost.
  • Open challenges: Posterior collapse in DVAEs may reflect excessive latent-sequence capacity, while autoregressive components can still capture most information and ignore latent vectors.The review presents this as a conjecture and an open problem, supported by observations with autoregressive models.
  • Model perspective: DVAEs can be viewed as structured or hierarchical VAEs with temporally ordered latent and observed variables.
  • Open challenges: Disentangling and separately controlling speech-production factors remain largely open, alongside basic questions about latent and recurrent-state dimensions.The review specifically highlights uncertainty about reducing z_t while keeping h_t comparable to x_t in speech processing.
  • Open challenges: DVAEs provide a powerful framework for sequential representation learning, but substantial work remains on disentangling latent factors.

Marginalization of h1:T in STORN

The appendix explains how STORN's recurrent state sequence h1:T can be marginalized by treating each state as a deterministic recursive function of observed and latent variables. This converts the expanded formulation into a compact form and extends similarly to other models.

  • Marginalization of h1:T in STORN: The derivation treats h1:T as deterministic functions rather than free random variables.This distinction is maintained after simplifying the notation by identifying dt with ht.
  • Marginalization of h1:T in STORN: The appendix presents the derivation for STORN and notes that analogous derivations apply to the other models.
  • Marginalization of h1:T in STORN: STORN replaces each recurrent state ht with the deterministic function fh(xt−1, zt, ht−1).The state depends recursively on prior observations, the current latent variable, and the previous state.
  • Marginalization of h1:T in STORN: The sequence h1:T is represented as d1:T, where each dt depends only on x1:t−1 and z1:t through an unfolded recurrence.This recursively expands ht back to the initial recurrent state.
  • Marginalization of h1:T in STORN: Marginalizing over hT, hT−1, and subsequent earlier states replaces each state with its deterministic transition function.The procedure proceeds backward through the recurrent state sequence.

DVAE implementation with speech data

The speech-data implementation section establishes notation for the DVAE architectures used in the experiments. It defines a concise MLP notation covering inputs, layer dimensions, and activation functions.

  • DVAE implementation with speech data: The section provides complete specifications of the DVAE models used in the speech-data experiments.
  • DVAE implementation with speech data: Each DVAE model is described first through its generation network and then through its inference network.
  • DVAE implementation with speech data: MLP(y, n1, f1, …, nL, fL) denotes an L-layer MLP with input y, output dimensions nℓ, and element-wise activations fℓ.

166 DVAE implementation with speech data

The speech-data benchmark implementations instantiate the DVAE models with distinct recurrent architectures, feature extractors, priors, and inference networks. The configurations include STORN, VRNN, SRNN, RVAE, and DSAE-specific design choices.

  • STORN: STORN uses an i.i.d. standard Gaussian prior, a forward LSTM recurrent state, and MLP-based feature extraction for xt−1 and zt.Its generation network combines extracted observation and latent features before the recurrent transition.
  • VRNN: VRNN shares an internal recurrent state ht between inference and generation while explicitly using feature extractors for xt and zt.The extracted features feed the encoder and decoder modules.
  • SRNN: SRNN shares ht between generation and inference, using LSTM transitions and MLPs that combine recurrent, latent, and observation representations.
  • RVAE: RVAE uses an i.i.d. standard Gaussian prior and has causal and noncausal variants depending on whether xt generation uses future latents zt+1:T.The noncausal generation and inference configurations use bidirectional LSTMs.
  • DSAE: DSAE adds a sequence-level latent variable v with the same dimension as zt, while assuming a standard Gaussian prior for v.The total latent dimensionality remains comparable because z1:T has T times the dimension of zt.
  • DSAE: DSAE generates xt from [zt, v] and infers v with a bidirectional many-to-one LSTM, while zt inference uses a bidirectional LSTM and an RNN.The implementation omits optional dense layers before the BRNN because the authors report better experimental performance without them.

DVAE implementation with 3D human motion data

The human-motion implementations retain the speech-data DVAE architectures while reducing dimensions for the lower-dimensional pose vectors. Model-specific generation and inference networks are adapted mainly through smaller MLP and recurrent hidden dimensions.

  • Architecture adaptation: The motion-data implementations retain the general speech-data architectures because both inputs are sequences of 1D vectors.
  • Architecture adaptation: The latent variable zt and DSAE's v are reduced to dimension 10 because human pose vectors have dimension 96 versus 513 for speech vectors.
  • Model implementations: The gated-transition model reduces all MLP hidden dimensions to 10 and uses a two-hidden-layer MLP for generating xt.
  • RVAE: For RVAE, both causal and noncausal generation and inference retain the speech-data implementation apart from the LSTM hidden-state dimension.
  • DSAE: For DSAE, generation maps [zt, v] through a four-layer MLP, while the remaining components use the speech-data design with different LSTM hidden dimensions.
Loading 2008.12595v4…