Source-linked AI summary
Spatio-temporal video autoencoder with differentiable memory
Viorica Patraucean, Ankur Handa, Roberto Cipolla
TL;DR
Video understanding is hindered by limited labelled video data and the difficulty of exploiting temporal redundancy. The paper introduces an end-to-end differentiable spatio-temporal autoencoder with convolutional LSTM visual memory for unsupervised motion prediction, and demonstrates its usefulness for weakly-supervised video segmentation.
Problem
Limited labelled video data and convnets’ inability to exploit temporal redundancy make transferring image-based capabilities to video tasks difficult.
Method
An end-to-end differentiable spatio-temporal video autoencoder uses convolutional LSTM visual memory, optical-flow prediction, and feedback to learn motion estimation without supervision.
Results
The convolutional LSTM module performs better than classic autoencoders and fully-connected LSTM implementations, while using fewer parameters, and supports weakly-supervised video semantic segmentation.
Takeaways & Limitations
The framework provides an unsupervised motion-prediction component that can propagate semantic-segmentation labels through video sequences.
Takeaways & Limitations
The authors hypothesise that overly downsampled visual memory can affect correct perception of visual stimuli, motivating investigation of memory size and resolution.
Abstract
from arXiv · showhide
We describe a new spatio-temporal video autoencoder, based on a classic spatial image autoencoder and a novel nested temporal autoencoder. The temporal encoder is represented by a differentiable visual memory composed of convolutional long short-term memory (LSTM) cells that integrate changes over time. Here we target motion changes and use as temporal decoder a robust optical flow prediction module together with an image sampler serving as built-in feedback loop. The architecture is end-to-end differentiable. At each time step, the system receives as input a video frame, predicts the optical flow based on the current observation and the LSTM memory state as a dense transformation map, and applies it to the current frame to generate the next frame. By minimising the reconstruction error between the predicted next frame and the corresponding ground truth next frame, we train the whole system to extract features useful for motion estimation without any supervision effort. We present one direct application of the proposed framework in weakly-supervised semantic segmentation of videos through label propagation using optical flow.
1 INTRODUCTION
The paper addresses limited labeled video data and the difficulty of exploiting temporal redundancy in videos. It proposes an unsupervised spatio-temporal autoencoder with differentiable visual short-term memory geared toward motion estimation and prediction.
- Limited labeled video data makes training high-level video models difficult.
- Convolutional networks designed for static images do not readily exploit temporal redundancy in videos.
- The proposed autoencoder integrates a differentiable short-term memory module trained without supervision for motion estimation and prediction.
- The architecture combines a classic convolutional image encoder-decoder with a nested convolutional LSTM memory module.
- The framework is applied to weakly supervised video semantic segmentation through optical-flow-based label propagation.
2 RELATED WORK
The related work spans LSTM video autoencoders, supervised optical-flow estimation, and approaches linking vision with motion. This paper instead uses spatial convolutional memory to predict dense motion without labeled training data.
- Earlier LSTM video autoencoders flatten frames or use fully connected operations, increasing parameters for image inputs.
- Supervised optical-flow methods establish frame correspondences from labeled data and apply variational smoothing.
- The proposed architecture integrates a smoothness penalty and requires no labeled training data for end-to-end flow prediction.
- Unlike global ego-motion estimates, the method predicts dense flow maps that accommodate different motion models for different objects.
3 ARCHITECTURE
The architecture nests a temporal autoencoder inside a spatial convolutional autoencoder. Convolutional LSTM memory integrates spatial features, while differentiable flow warping predicts the next frame and is trained with reconstruction and smoothness terms.
- At each time step, the nested autoencoder maps an input video frame to a predicted next frame of the same size.
- 3.1 SPATIAL AUTOENCODER E AND D: The spatial encoder-decoder uses convolution, tanh nonlinearities, max-pooling, and nearest-neighbor upsampling.
- The temporal autoencoder captures motion-related changes to predict the visual future from past and present information.
- 3.2.1 MEMORY MODULE LSTM: The memory module uses convolutional LSTM cells that preserve spatial locality and layout while updating memory from successive frame features.
- 3.2.2 OPTICAL FLOW PREDICTION MODULE: The optical-flow module predicts one two-dimensional displacement vector per pixel and uses it to warp the current frame toward the next frame.
- 3.2.4 LOSS FUNCTION: Training minimizes next-frame reconstruction error while adding a weighted Huber-based smoothness penalty on the flow map.
- The network contains 1,035,067 trainable parameters, including 64 memory filters and 15 × 15 flow-regression kernels.
4 TRAINING
Training uses RMSprop with a decaying learning rate and Xavier initialization for convolutional layers outside the memory module. Memory parameters use uniform initialization, with forget-gate biases set to one.
- RMSprop training starts with a learning rate of 10^-4 and decays it by 0.9 every five epochs.
- Convolutional layers outside the memory module are initialized with the Xavier method.
- Non-bias memory parameters are initialized from U(-0.08, 0.08), while forget-gate biases are set to 1 and other gate biases to 0.
- The Torch implementation extends the rnn package and training ran on an NVIDIA K40 GPU with 12G memory.
5 EXPERIMENTS
Experiments evaluate the architecture on synthetic and real videos, then apply it to weakly supervised semantic segmentation through optical-flow label propagation. Convolutional LSTM models outperform the convolutional baseline, while the real-video and segmentation results reveal accuracy and data limitations.
- Unsupervised experiments: The warping modules achieved an average per-pixel sampler error of 0.004 on Sintel, with boundary artifacts caused by occlusions.The test isolated the grid generator and sampler using ground-truth optical flow.
- Unsupervised experiments: Moving MNIST experiments compared AE-Conv, AE-fcLSTM, AE-ConvLSTM, and AE-ConvLSTM-flow using per-pixel prediction error and qualitative results.The dataset contains 20-frame 64 × 64 sequences with translated, potentially overlapping digits.
- Unsupervised experiments: All LSTM architectures report lower errors than AE-Conv, supporting the importance of temporal dependencies between frames.The convolutional LSTM architectures are more efficient than the fully connected LSTM, whose predictions are affected by blur.
- Unsupervised experiments: On real videos, the network identifies moving scene elements, but its flow accuracy is not comparable to supervised optical-flow setups.The model predicts flow from the observed frame to the unseen next frame, a more challenging setting than estimating flow between two observed frames.
- Application to weakly-supervised video semantic segmentation: SegNet-flow propagates segmentation predictions with learned optical flow and improves segmentation considerably for large classes, while performance diminishes for small thin structures.The authors attribute the limitation partly to the small training set and downsampled flow estimation.
- Application to weakly-supervised video semantic segmentation: The segmentation system jointly learns flow estimation and semantic segmentation end-to-end with about one million additional parameters over basic SegNet.The approach uses limited labelled video frames and propagates labels through recurrent merging and sampling modules.
6 CONCLUSION AND FUTURE WORK
The paper proposes an end-to-end differentiable spatio-temporal video autoencoder that uses convolutional LSTM cells as artificial visual short-term memory for unsupervised motion prediction. Its usefulness is demonstrated in weakly-supervised video semantic segmentation, while future work targets richer memory systems and possible memory ambiguities.
- 6 CONCLUSION AND FUTURE WORK: The proposed architecture performs unsupervised motion prediction through an end-to-end differentiable spatio-temporal video autoencoder.Its core component is a convolutional LSTM module functioning as artificial visual short-term memory.
- 6 CONCLUSION AND FUTURE WORK: Convolutional LSTM cells preserve spatial layout while outperforming classic autoencoders and fully-connected LSTM implementations with fewer parameters.The comparison is reported as part of the paper’s evaluation of the proposed memory module.
- 6 CONCLUSION AND FUTURE WORK: The overall memory-and-feedback-loop setup is useful for weakly-supervised video semantic segmentation.The application uses the learned motion representation to support segmentation of videos.
- 6 CONCLUSION AND FUTURE WORK: The authors hypothesize that overly downsampled memory and repeated spatial boundaries may produce geometric illusions or ambiguous motion activations.These hypotheses motivate investigating memory size, resolution, and downsampling.
- 6 CONCLUSION AND FUTURE WORK: Future development requires integrating the memory module with attention and long-term memory for a complete memory system suitable for supervised tasks.The paper identifies this integration as a necessary future development.