Source-linked AI summary
Unsupervised Learning of Disentangled Representations from Video
Remi Denton, Vighnesh Birodkar
TL;DR
Unsupervised video learning seeks representations that generalize across tasks, but DRNET targets explicit separation of stationary content from temporal pose. It uses adversarially trained encoders and latent-space prediction, producing coherent long-range generations and useful content- and pose-based classifications.
Problem
Unsupervised video learning aims to learn representations that generalize to tasks such as semantic classification, future-frame prediction, and activity classification without explicit labels.
Method
DRNET uses a predictive auto-encoder with adversarial training to factor each frame into time-independent content and time-varying pose, then predicts pose in latent space.
Results
DRNET generates convincing future frames hundreds of time steps ahead and supports object-identity and action classification using content and pose features, respectively.
Takeaways & Limitations
A factorized representation enables stable, coherent long-range prediction with a standard LSTM and supports multiple downstream video tasks.
Takeaways & Limitations
The factorization assumes that an object's pose is not distinctive to a particular clip.
Abstract
from arXiv · showhide
We present a new model DrNET that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can be used for a range of tasks. For example, applying a standard LSTM to the time-vary components enables prediction of future frames. We evaluate our approach on a range of synthetic and real videos, demonstrating the ability to coherently generate hundreds of steps into the future.
1 Introduction
The paper introduces DRNET, a predictive auto-encoder that uses adversarial training to separate each video frame into time-independent content and time-varying pose. This representation supports long-range frame prediction and classification across several tasks.
- Core approach: DRNET uses a novel adversarial loss to factor each frame into roughly time-independent content and time-varying pose components.The loss discourages pose features from carrying clip-identity information.
- Applications: The model applies a standard LSTM to pose features while conditioning on content from the last observed frame to predict future video frames.This separates temporal prediction from the stationary component of the representation.
- Applications: DRNET generates convincing predictions hundreds of time steps into the future, farther than existing approaches using real video data.The paper reports this result despite using a simple predictive model.
- Applications: Content features support object-identity classification, while pose features support action prediction.The two components are explored as task-specific representations.
2 Related work
Prior work learns video representations through self-supervision, auxiliary action labels, predictive models, or explicit disentanglement mechanisms. DRNET differs by using temporal structure and a factorized latent representation, predicting dynamics in latent space rather than directly generating pixels.
- Disentangled representations: Static-image methods explicitly separate factors such as what and where, whereas DRNET uses temporal structure to separate components in video.The comparison includes capsule and related auto-encoder approaches.
- Disentangled representations: Video factorization has also been studied with structure-from-motion, Slow Feature Analysis, and unsupervised content-motion models.These approaches differ in whether they use explicit projection models, temporal penalties, or related architectures.
- Video generation: Generative video models use approaches including discrete vector quantization, pixel-level LSTMs, pixel transformations, stacked auto-encoders, and optical-flow prediction.DRNET predicts a single pose vector rather than a spatial flow field.
- Video generation: Action-conditional models exploit known actions, whereas DRNET factors content from motion and applies prediction only to the latter in latent space.DRNET does not attempt to predict pixels directly.
- Video generation: Other work addresses future uncertainty with GAN-based sharpness or multiple-outcome models, while DRNET uses standard deterministic losses in its constrained latent-space setup.The paper attributes this choice to its model structure and non-pixel-space forward model.
3 Approach
DRNET separates each video frame into time-independent content and time-varying pose representations, using reconstruction, temporal similarity, and adversarial losses. The disentangled representation supports recurrent future-frame prediction and classification of semantic content or actions.
- The decoder concatenates content from one frame with pose from a future frame to reconstruct that future frame, while reconstruction alone does not ensure disentanglement.
- Two encoders represent each frame with content features capturing structure shared across time and pose features capturing content that varies over time.
- Temporal similarity penalizes differences between neighboring content vectors, encouraging the content representation to remain approximately time-independent.
- An adversarial scene discriminator classifies whether pose-vector pairs come from the same video, while the pose encoder makes this classification uncertain so pose features cannot carry object-content information.
- The training objective combines reconstruction, content similarity, and adversarial losses, alternating discriminator updates with updates to the encoders and decoder.
- 3.1 Forward Prediction: For prediction, an LSTM recurrently predicts future pose features while fixed content features from the last observed frame are passed to the decoder for pixel-space generation.
- The representation also supports classification by using content features for semantic image content and pose-feature sequences for video actions.
4 Experiments
Experiments evaluate DRNET on synthetic and real videos for disentanglement, long-range prediction, and classification. Across these settings, the model produces clean factorization, high-quality generations, and improved action classification, while the adversarial loss is important for separating semantic content from pose.
- Experimental setup: DRNET is evaluated on MNIST, NORB, SUNCG, and KTH videos across factorization, future prediction, and classification tasks.The experiments use both synthetic and real datasets, with architectures varying by dataset.
- MNIST: 500 time steps: MNIST generations remain crisp when only pose is predicted with an LSTM and content is fixed from the fifth frame.The model observes 5 initial frames for MNIST future prediction.
- NORB: DRNET cleanly factorizes held-out NORB content and pose, whereas removing the adversarial loss causes pose to retain content information and degrades classification.The adversarial-loss model also compares favorably with the factorization examples from Mathieu et al.
- SUNCG: On SUNCG, DRNET generates high-quality factorization examples and accurately rotates complex objects in held-out scenes.The figure also illustrates linear interpolation in pose space.
- KTH Action Dataset: On KTH, DRNET is compared with MCNet and AE-LSTM for held-out walking and running videos, using 10 conditioning frames to generate future frames.Additional generations extend to 100 time steps, and Inception scores decay more gracefully than MCNet’s.
- KTH Action Dataset: DRNET’s pose representation significantly boosts action-classification performance over an autoencoder baseline using few labeled examples.The evaluation extracts pose vectors from 24-frame sequences and trains a fully connected classifier.
5 Discussion
DRNET factors video into content and pose using paired encoders and adversarial training, enabling stable, coherent long-range prediction with a standard LSTM.
- DRNET uses paired encoders to factor video into content and pose representations.A novel adversarial loss achieves the separation during training.
- A standard LSTM applied to the pose representation supports stable and coherent long-range prediction.The model does not require GAN losses or probabilistic formulations used by other video-generation approaches.
- DRNET generations compare favorably with leading approaches despite the model’s simplicity.
A Further KTH generations
Additional KTH experiments compare long-range sequences generated by DRNET and MCNet under the same conditioning frames.
- Figure 11 presents additional long-range KTH sequences generated by DRNET and MCNet.
- Each paired set of rows uses the same conditioning frames, with DRNET shown above MCNet.
B Quantitative metrics for evaluating generations
Pixel-wise metrics can misrepresent long-range video quality because small velocity errors accumulate into large positional offsets. The paper therefore uses the Inception Score as its main alternative for evaluating generated sequences.
- PSNR and SSIM are misleading for long-range generations when positional alignment deteriorates.A person may remain sharp even as velocity errors accumulate into a significant position offset.
- The Inception Score evaluates generated sequences using label distributions from a classifier trained to recognize video actions.The classifier receives sequences of 10 concatenated frames and predicts action classes.
- Figure 12 compares DRNET and MCNet sequences, their differences from ground truth, and corresponding PSNR scores.
C KTH experimental settings
The KTH dataset contains real videos of 25 subjects performing six actions under varied scenarios, with subjects 1–16 for training and 17–25 for testing.
- KTH includes 25 subjects performing six actions against a static background.The actions are boxing, hand waving, hand clapping, jogging, running, and walking.
- Each subject performs every action in four scenarios with varied clothing and background conditions.
- The experiments train on subjects 1–16 and test on subjects 17–25, using 128×128-pixel frames.
D Details of classification experiments
The classification experiments used fully connected classifiers with task-specific hidden-layer sizes and regularization, while evaluating latent representations over fixed-length sequences. The KTH setup used a nonstandard train/test split.
- NORB object classification: NORB classification used a two-layer fully connected classifier with 256 hidden units, Leaky ReLUs, batch normalization, dropout, ADAM, and validation early stopping.These choices were used across the classifier layers, with early stopping intended to prevent overfitting.
- KTH action classification: KTH action classification used a two-layer fully connected classifier with 1200 hidden units and 24×24 inputs formed from sequences of length 24.Both DRNET and the autoencoder baseline produced 24-dimensional latent vectors.
- KTH action classification: The KTH experiment also evaluated an autoencoder baseline with a 128-dimensional latent space matching DRNET’s content-vector dimensionality.The passage begins describing this baseline but is truncated after stating the dimensionality match.
- KTH action classification: The KTH action-classification experiment did not use the standard train/test split.This is explicitly noted as an experimental assumption.