Source-linked AI summary
Unsupervised Learning of Video Representations using LSTMs
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov
TL;DR
The paper asks whether useful video representations can be learned without labelled videos, which matters because supervised video learning faces long-range structure and labelling challenges. It uses encoder-decoder LSTMs for reconstruction and future prediction, then evaluates the representations qualitatively and on action recognition. The representations improve action-recognition accuracy, especially with few labelled examples, while out-of-domain inputs reveal limitations of modelling whole frames in one pass.
Problem
Supervised video learning makes long-range credit assignment difficult and can require more labelled data or hand-engineered features, motivating unsupervised representation learning.
Method
Multilayer encoder-decoder LSTMs map video-frame sequences to fixed-length representations and decode them for reconstruction or long-horizon future prediction.
Results
Unsupervised representations improve action-recognition accuracy, especially with few labelled examples, including UCF-101 improvements from 29.6% to 34.3% with one labelled video and from 74.5% to 75.8% on the full dataset.
Takeaways & Limitations
Unlabelled video pretraining can provide representations that transfer to supervised action recognition, including when labelled training data is limited.
Takeaways & Limitations
On out-of-domain sequences, the model hallucinates missing digits or merges multiple digits into blobs, reflecting a drawback of modelling entire frames in one pass.
Abstract
from arXiv · showhide
We use multilayer Long Short Term Memory (LSTM) networks to learn representations of video sequences. Our model uses an encoder LSTM to map an input sequence into a fixed length representation. This representation is decoded using single or multiple decoder LSTMs to perform different tasks, such as reconstructing the input sequence, or predicting the future sequence. We experiment with two kinds of input sequences - patches of image pixels and high-level representations ("percepts") of video frames extracted using a pretrained convolutional net. We explore different design choices such as whether the decoder LSTMs should condition on the generated output. We analyze the outputs of the model qualitatively to see how well the model can extrapolate the learned video representation into the future and into the past. We try to visualize and interpret the learned features. We stress test the model by running it on longer time scales and on out-of-domain data. We further evaluate the representations by finetuning them for a supervised learning problem - human action recognition on the UCF-101 and HMDB-51 datasets. We show that the representations help improve classification accuracy, especially when there are only a few training examples. Even models pretrained on unrelated datasets (300 hours of YouTube videos) can help action recognition performance.
1. Introduction
The paper addresses unsupervised learning of useful video representations by extending sequence-to-sequence LSTMs to image sequences. It motivates this approach through video’s rich spatial and temporal structure, and evaluates representations through reconstruction, prediction, visualization, and action recognition.
- Approach: The authors extend sequence-to-sequence learning by encoding image sequences into fixed-length representations and decoding sequences from them using recurrent networks.Their setting uses unlabelled videos and applies the framework to image-sequence representations.
- Motivation: Video representations should capture objects, motion, camera movement, occlusion, and other spatial-temporal factors to support intelligent-machine tasks.The paper identifies action and gesture recognition as useful applications.
- Why Unsupervised Learning?: Supervised video learning faces difficult long-range credit assignment and often requires more labelled data or hand-engineered features.The paper presents these costs as motivation for unsupervised learning.
- Approach: The LSTM Encoder-Decoder framework applies the same operation at each time step, providing an inductive bias that reflects consistent world physics across states and times.The model is designed so information propagates through recurrent time steps.
- Approach: Inputs include image patches and high-level percepts extracted from an ImageNet-trained convolutional network.The paper uses natural image patches, moving MNIST digits, and convolutional hidden-state percepts.
- Evaluation: The learned representations are evaluated through qualitative reconstruction and prediction analysis and by initializing supervised action-recognition models.The paper reports that the classifier performs better, especially with few labelled examples.
2. Model Description
The model uses LSTM encoder–decoder networks to learn fixed-length video representations through reconstruction, future prediction, or a composite of both. LSTM memory and recurrent decoding are intended to preserve appearance and motion while limiting trivial memorization.
- Long Short Term Memory: The LSTM cell updates memory through input, forget, and output gates, enabling credit assignment over long sequences and discovery of long-range features.Its cell state sums activities over time, so error derivatives do not vanish quickly when propagated backward through time.
- LSTM Autoencoder Model: An LSTM encoder reads a sequence of image patches or features, and a decoder predicts the target sequence, optionally conditioned on its last generated frame.The autoencoder reverses the target sequence, while conditional and unconditioned decoder variants are both considered.
- LSTM Autoencoder Model: The autoencoder treats the encoder’s final hidden state as the video representation and requires it to retain object appearance, background, and motion for reconstruction.A fixed number of hidden units and recursive reuse of the same LSTM operation make arbitrary-length identity mappings less likely.
- LSTM Future Predictor Model: The Future Predictor uses the same encoder–decoder structure to predict a long sequence after the input, requiring information about objects, background, and motion for extrapolation.Unlike predicting only the next frame, this model predicts a sequence into the future and has conditional and unconditioned variants.
- Conditional Decoder: Conditional decoding can represent multiple target modes, but access to recent generated frames may also let the decoder exploit short-range video correlations during optimization.The multimodality argument is more relevant to future prediction because an input may admit multiple possible futures.
- Composite Model: The composite model asks one encoder state to reconstruct the input and predict future frames, discouraging both input memorization and forgetting early frames.Reconstruction counters the future predictor’s emphasis on recent frames, while future prediction makes trivial autoencoding representations less useful.
3. Experiments
The experiments evaluate qualitative behavior, temporal and domain generalization, unsupervised pretraining for action recognition, and comparisons among model variants and benchmarks. Across these tests, composite models improve future prediction and supervised recognition, though gains vary by input representation and setting.
- Experimental objectives: The experiments assess qualitative behavior, benefits of unsupervised initialization, model variants, and comparisons with action-recognition benchmarks.These objectives are evaluated using moving MNIST, natural image patches, UCF-101, and HMDB-51.
- Out-of-domain inputs: Out-of-domain tests showed good reconstruction and prediction for one moving digit, but the model hallucinated a second overlapping digit.Training used sequences containing two moving digits, while testing used one or three.
- Action recognition: Unsupervised pretraining improved action recognition across UCF-101 and HMDB-51, with larger gains when labelled data were scarce.For UCF-101, performance rose from 29.6% to 34.3% with one labelled video and from 74.5% to 75.8% on the full dataset; HMDB-51 rose from 14.4% to 19.1% and from 42.8% to 44.0%, respectively.
- Model and benchmark comparisons: The Composite Model consistently outperformed the Future Predictor for future prediction, and combining RGB and flow predictions reached 84.3 accuracy on UCF-101.Conditional models performed better for future prediction, while conditioning did not provide a clear supervised-learning advantage.