Source-linked AI summary
Disentangled Sequential Autoencoder
Yingzhen Li, Stephan Mandt
TL;DR
The paper addresses unsupervised disentanglement in high-dimensional sequential data by separating static content from dynamic information. It proposes a VAE with a global content latent and stochastic time-local dynamics, enabling conditional generation and feature swapping. Experiments on video and speech support approximate disentanglement and show that stochastic transition models generally outperform deterministic ones.
Problem
Unsupervised sequence models have limited explicit separation of time-invariant content from time-varying dynamics, restricting controlled generation and feature swapping.
Method
The model uses a global latent variable for content, stochastic RNN dynamics with time-local latent variables, and standard amortized variational inference.
Results
Experiments on video and speech provide strong evidence of approximately disentangled representations supporting conditional generation and feature swapping, while stochastic transitions generally outperform deterministic ones.
Takeaways & Limitations
The architecture supports full and conditional generation and manipulations such as voice conversion and video content control.
Takeaways & Limitations
The paper does not test a discriminative objective because its assumptions may not always hold, and generated video attributes can drift over time.
Abstract
from arXiv · showhide
We present a VAE architecture for encoding and generating high dimensional sequential data, such as video or audio. Our deep generative model learns a latent representation of the data which is split into a static and dynamic part, allowing us to approximately disentangle latent time-dependent features (dynamics) from features which are preserved over time (content). This architecture gives us partial control over generating content and dynamics by conditioning on either one of these sets of features. In our experiments on artificially generated cartoon video clips and voice recordings, we show that we can convert the content of a given sequence into another one by such content swapping. For audio, this allows us to convert a male speaker into a female speaker and vice versa, while for video we can separately manipulate shapes and dynamics. Furthermore, we give empirical evidence for the hypothesis that stochastic RNNs as latent state models are more efficient at compressing and generating long sequences than deterministic ones, which may be relevant for applications in video compression.
1. Introduction
The paper targets unsupervised disentanglement in sequences by separating time-invariant content from time-varying dynamics. Its model enables controlled generation, feature swapping, efficient encoding, and evaluation of disentanglement, while experiments provide evidence that stochastic transitions generate more realistic long-sequence dynamics.
- Disentangled sequence representations should separate time-independent concepts, such as object identity, from time-varying information, such as pose and position.
- The proposed graphical-model design uses a global content variable and frame-associated dynamic variables to learn structured sequence representations without supervision.
- Conditioning on either content or dynamics enables controlled generation, including fixed-content random dynamics, fixed-dynamics random content, and identity or voice swapping.
- Factoring out content can make the representation more data efficient than frame-by-frame encoding by allowing smaller dynamical latent dimensions.
- The authors introduce a disentanglement metric based on the stability of an object classifier's predictions over time.
- For long sequences, experiments provide empirical evidence that stochastic transition models generate more realistic dynamics than deterministic ones.
2. The model
The model represents each high-dimensional sequence with a global latent content variable and time-local dynamic variables. Variational inference trains the model, which supports unconditional generation, conditional manipulation, and feature swapping by selectively fixing or resampling these latent variables.
- The generative model separates time-invariant concepts from time-varying concepts in a high-dimensional sequence x1:T.
- Generative model: The generation of frame xt depends on the corresponding dynamic latent zt and global content latent f.
- Generative model: A global latent variable f represents content, while latent variables zt represent dynamics associated with individual frames.
- Inference models: Variational inference with a VAE trains an approximate posterior over latent variables given the observed sequence.
- Inference models: The factorised q encoder assumes approximate independence between content and motion, conditioning content on the entire sequence and dynamics on individual frames.
- Inference models: The full q encoder conditions dynamic inference on f and the entire time series, reflecting possible dependence between content and motion.
- Generation: Unconditional generation samples latent variables from their priors and decodes them; fixing f or z1:T produces sequences sharing content or dynamics.
- Generation: Conditional generation and feature swapping infer content or dynamics from reference sequences, then randomise the complementary latent variables or combine latents from different sequences.
3. Related work
Prior work pursued disentanglement through objectives, architectural factorisation, or hierarchical priors, but sequence models had limited explicit separation of static and dynamic information. The paper positions its model as a minimalistic architecture that achieves disentanglement through probabilistic design.
- Earlier disentanglement research mainly modified VAE objectives or designed architectures with factored representations.
- Existing sequence VAEs generated realistic video or speech but did not explicitly disentangle time-invariant and time-dependent information.
- Because earlier sequence models lacked explicit factorisation, controlled generation and feature swapping were inconvenient for them.
- Prior video autoencoders used separate content and pose encoders, with differences in how pose information and training objectives were constructed.
- FHVAE addressed unsupervised disentanglement for speech with segment-level latent variables and hierarchical structure.
- Hierarchical priors in Structured and Factorised VAEs were used to learn more interpretable latent variables.
4. Experiments
Experiments on sprite videos and speech recordings provide evidence that the model approximately disentangles static content from dynamics and supports controlled generation and feature swapping. Additional experiments indicate that stochastic transition models better represent long, complex sequences than deterministic alternatives.
- 4.1. Video sequence: Sprites: The sprite dataset contains controllable character attributes and actions, enabling evaluation of whether static identity and dynamic motion are separately represented.The dataset includes 1,296 unique characters, with 1,000 used for training and validation and the remainder for testing.
- 4.1. Video sequence: Sprites: Generated sequences preserve either motion or character identity when the other latent component is sampled, demonstrating controlled manipulation of video content and dynamics.Using encoded dynamics with sampled content preserves motion patterns, while encoded content with sampled dynamics preserves characters but changes motion.
- 4.1. Video sequence: Sprites: Feature swapping combines dynamics inferred from one sequence with content encoded from another, producing a factored representation of content and pose.The resulting reconstructions support exchanging object identity while retaining pose information.
- 4.1. Video sequence: Sprites: 4% of 296 × 9 skin-color reconstruction pairs disagreed, while hairstyle disagreement was 0.06%, indicating strong preservation of static attributes.The reconstructed classifiers also had much smaller KL divergence than random predictions.
- 4.1. Video sequence: Sprites: Shared content latents yield similar static attributes but diverse actions, whereas shared dynamics latents yield mostly consistent actions across different characters.The model also generates multimodal actions and unseen turning sequences, suggesting generalization beyond the dataset.
- 4.2. Speech data: TIMIT: Speech conversion shifts harmonics and pitch between male and female outputs while preserving speech content, supporting disentangled speaker identity and dynamics.The proposed approach is competitive with or slightly better than FHVAE with α = 0, while the stronger FHVAE result uses an added discriminative objective.
- 4.2. Speech data: TIMIT: The speech disentanglement comparison is scoped by the model’s minimalistic, data-agnostic design and leaves improved unsupervised discriminative objectives for future work.The authors note that the tested discriminative objective relies on assumptions that may not always hold.
- 4.3. Comparing stochastic & deterministic dynamics: The stochastic transition model outperforms deterministic models for reconstructing sequences with missing frames, especially when the number of missing frames is small.It preserves ball shape and produces more physical trajectories, supporting better modeling of long, complex sequences.
5. Conclusions and outlook
The paper presents a minimalistic generative model that disentangles static content from dynamic features in high-dimensional time series, supporting generation and feature swapping across video and audio. It also finds that stochastic transition models outperform deterministic ones, while identifying more complex sequences and improved disentanglement as future directions.
- The model uses a global content latent variable and a stochastic RNN with time-local latent variables for dynamical features.
- It supports full and conditional generation and feature swapping, including voice conversion and video content manipulation.
- A stochastic transition model generally outperforms a deterministic one.
- Future work includes testing the architecture on more complex video and audio sequences and improving disentanglement with cross-entropy terms or discriminative training.
- Separating dynamic from static features may enable low-dimensional dynamics latents and applications in neural compression.
A. Computation graph for the full q inference network
The full inference network first infers a sequence-level content latent from the entire sequence, then uses it with the observations to infer time-local dynamical latents.
- A bidirectional LSTM computes the mean and variance of q(f|x1:T ) and samples the sequence-level content latent f.
- A second bidirectional LSTM receives each frame together with f and computes hidden representations for the z_t variables.
- A simple RNN uses the hidden representation at each time step to compute q(z1:T |x1:T , f).
B. Sound files for the speech conversion test
The speech-conversion demonstration provides sound files reconstructed from magnitude spectra, while leaving phase information unmodeled in the experiments.
- Sound files demonstrate conversion between female and male speech sequences.The files are provided as an audio demonstration of the speech conversion test.
- Waveforms are reconstructed from magnitude spectra using the Griffin-Lim algorithm.Griffin-Lim randomly initializes phase and iteratively refines it through SFFT and inverse SFFT transformations.
- Phase information is not modeled in these experiments or in the FHVAE tests.
- Modeling phase would require a von Mises rather than Gaussian distribution, but its Bessel-function normalizer complicates computation and optimization.
C. Network architecture
The architecture combines neural encoders and decoders with latent content and dynamics variables, using sequence-level or frame-level inference and dataset-specific likelihoods.
- Sprite: The Sprite decoder uses an LSTM-parameterized Gaussian dynamics prior and a deconvolutional network conditioned on z_t and f.The decoder first applies a one-hidden-layer MLP, then four deconvolutional layers with up-sampling.
- Sprite: The Sprite inference model extracts visual features convolutionally, then estimates sequence-level f and frame-level z_t with Gaussian encoders.The content encoder uses the entire sequence, while the factorized dynamics encoder uses each frame's visual feature.
- TIMIT: The TIMIT decoder uses a Gaussian likelihood whose mean and variance are produced by a two-hidden-layer MLP receiving z_t and f.For TIMIT, f and z_t are 64-dimensional unless otherwise stated, with hidden layers fixed at 256 units.
- Inference: The full q inference model uses bi-LSTMs over features extracted from x_t by a one-hidden-layer MLP.
- Bouncing ball: For bouncing-ball experiments, an RNN parameterizes the stochastic prior, with 16-dimensional z_t, 64-dimensional deterministic z, 64-dimensional f, and Bernoulli likelihoods.
- Bouncing ball: Deterministic-dynamics models use bi-LSTMs to infer q(f|x_1:T) and q(z|x_1:T), alongside convolutional visual-feature extraction.
- Shared components: All models share convolutional and deconvolutional components with symmetric three-layer visual networks and 512-unit hidden layers.The shared deconvolutional network uses 64 channels and up-sampling.