Source-linked AI summary
Unsupervised Learning of Long-Term Motion Dynamics for Videos
Zelun Luo, Boya Peng, De-An Huang, Alexandre Alahi, Li Fei-Fei
TL;DR
The paper targets the limited compact encoding of long-term motion dependencies in video representations. It learns to predict sequences of atomic 3D flows from frame pairs with an unsupervised RNN Encoder-Decoder, and reports effective activity recognition across modalities and datasets, including state-of-the-art performance on two RGB-D datasets.
Problem
Existing methods have difficulty compactly encoding long-term motion dependencies, while supervised video representation learning would require unrealistic amounts of labeled data.
Method
An unsupervised RNN-based Encoder-Decoder predicts sequences of low-dimensional atomic 3D flows from pairs of video frames across RGB, depth, and RGB-D inputs.
Results
The learned representations support activity recognition across multiple modalities and datasets, achieving state-of-the-art performance on two established RGB-D datasets.
Takeaways & Limitations
Predicting atomic 3D-flow sequences yields a representation reported as effective and discriminative for activity classification across modalities.
Abstract
from arXiv · showhide
We present an unsupervised representation learning approach that compactly encodes the motion dependencies in videos. Given a pair of images from a video clip, our framework learns to predict the long-term 3D motions. To reduce the complexity of the learning framework, we propose to describe the motion as a sequence of atomic 3D flows computed with RGB-D modality. We use a Recurrent Neural Network based Encoder-Decoder framework to predict these sequences of flows. We argue that in order for the decoder to reconstruct these sequences, the encoder must learn a robust video representation that captures long-term motion dependencies and spatial-temporal relations. We demonstrate the effectiveness of our learned temporal representations on activity classification across multiple modalities and datasets such as NTU RGB+D and MSR Daily Activity 3D. Our framework is generic to any input modality, i.e., RGB, Depth, and RGB-D videos.
1. Introduction
The paper addresses the difficulty of compactly encoding long-term motion dependencies for activity recognition. It proposes unsupervised learning of representations by predicting sequences of atomic 3D flows from video frames.
- Existing video representations often fail to compactly encode long-term motion dependencies needed for activity classification.
- Atomic 3D flows represent basic motions and are obtained by quantizing estimated dense 3D flows in space and time using RGB-D data.This motion description is intended to reduce output complexity relative to 2D motion representations.
- Unsupervised learning avoids the need for human-labeled video data, whose required scale is described as unrealistic to collect.
- The framework predicts a sequence of basic motions from a pair of video frames using an RNN-based Encoder-Decoder.The approach casts representation learning as sequence prediction.
- The method evaluates motion prediction across RGB, depth, and RGB-D input modalities.
- The learned representations support activity recognition across multiple modalities and datasets, outperforming prior unsupervised methods across RGB and depth modalities at introduction.
2. Related Work
Prior work learns video representations from temporal structure and classifies activities using handcrafted features or deep networks. The paper positions its approach as an unsupervised framework that predicts motion sequences with an LSTM Encoder-Decoder.
- Unsupervised video representation methods use objectives including future-frame prediction, temporal smoothness, and frame-order prediction.
- Figure 2 depicts an LSTM Encoder-Decoder that maps paired input frames through Conv and Deconv networks to atomic 3D flows.
- Existing learned representations may emphasize semantic features or use coarse supervision that is insufficiently discriminative for activity classification.
- RGB-based activity recognition includes handcrafted trajectory and BoVW methods as well as deep two-stream ConvNets combining semantic features with optical flow.
3. Method
The method represents long-term motion as sequences of quantized atomic 3D flows and learns to predict them from image pairs with an RNN Encoder-Decoder. The resulting temporal representation supports motion prediction and activity recognition while addressing spatial correlations and flow-class imbalance.
- Sequence of Atomic 3D Flows: Given a pair of images, the framework predicts a sequence of atomic 3D flows over T temporal steps.Inputs can be RGB, depth, or RGB-D, with each predicted flow represented over the image grid and three spatial directions.
- Learning Framework: The learned representation is reused as a motion feature for activity recognition by encoding frame pairs before training a classification layer.The representation’s weights are fixed during the activity-recognition classification stage.
- Sequence of Atomic 3D Flows: Atomic 3D flows are formed by quantizing estimated dense 3D flows in space and time, producing a lower-dimensional motion signal than unstructured flow representations.The sequence is intended to retain long-term dependencies while remaining easy to parameterize and discriminative for activity recognition.
- Learning Framework: The RNN Encoder-Decoder maps input-frame features to predicted atomic 3D-flow sequences, using convolutional downsampling, an LSTM temporal representation, and deconvolutional upsampling.The LSTM is chosen for long-term temporal dependencies, while the architecture addresses spatial correlations through the surrounding convolutional networks.
- Loss Function: The motion-prediction objective uses cross-entropy between ground-truth and predicted distributions over the atomic 3D-flow codebook.Class weighting rebalances the loss because small-magnitude background flows dominate the flow distribution and can otherwise encourage predicting the no-motion class.
4. Activity recognition
The learned encoder representation is used for activity recognition through a classification layer, with experiments examining training, fine-tuning, and fixed-encoder scenarios. The representation is compact and computationally effective, and the complete model is evaluated against supervised depth-based methods.
- The encoder output feeds a final classification layer for activity recognition.
- The evaluation compares random initialization, fine-tuning from the learned representation, and a fixed pretrained encoder with only the classifier tuned.
- The method does not combine its learned representation with an ImageNet-trained semantic representation.
- The representation runs in real time through an encoder forward pass and has compact size 7×7×32 for embedded-device implementation.
- The complete model is compared with supervised depth-map, skeleton-based, deep-learning, and unsupervised methods using mean AP on NTU-RGB+D.
5. Experiments
Experiments assess unsupervised motion prediction and transfer to activity recognition across RGB, depth, and RGB-D inputs. The learned representation is tested on NTU-RGB+D, MSRDailyActivity3D, and UCF-101, including cross-domain and modality comparisons.
- 5. Experiments: The experiments evaluate motion prediction with RGB, depth, and RGB-D inputs, then test the representations for activity classification across datasets and modalities.
- 5.1. Unsupervised Learning of Long-term Motion: The unsupervised framework is trained on NTU RGB+D, which contains 57K videos spanning 60 action classes, 40 subjects, and 80 viewpoints.
- 5.1. Unsupervised Learning of Long-term Motion: The architecture uses Conv downsampling, an LSTM encoder-decoder, and Deconv upsampling.
- 5.1. Unsupervised Learning of Long-term Motion: At eight timesteps, input modalities have very similar prediction errors, while the error increases over the first four frames and stabilizes over the final four.
- 5.1. Unsupervised Learning of Long-term Motion: The ablations compare 2D motion prediction with 3D motion and three-step prediction with the proposed eight-step prediction.
- 5.2.1 Depth-based Activity Recognition: On NTU-RGB+D, the learned representation outperforms state-of-the-art supervised methods without fine-tuning on classification.
- 5.2.1 Depth-based Activity Recognition: On MSRDailyActivity3D, the representation transfers across datasets, viewpoints, and activities and exceeds prior work based on 3D trajectories by 15%.
- 5.2.2 RGB-based Activity Recognition: UCF-101 evaluation compares supervised methods with unsupervised methods both without and with ImageNet semantics.
6. Conclusions
The paper presents a general framework for learning long-term temporal video representations across modalities. Its atomic 3D-flow supervision supports state-of-the-art activity recognition on two RGB-D datasets, while future work targets RGB datasets and tasks beyond activity recognition.
- The framework learns long-term temporal representations across different video modalities.
- Sequence supervision based on atomic 3D flows enables training on large numbers of unlabeled videos.
- The learned representation achieves state-of-the-art activity recognition performance on two established RGB-D datasets.
- Future work includes RGB datasets, supervised tasks beyond activity recognition, 3D scene interactions, and compact dense-trajectory representations.