Source-linked AI summary
Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics
Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, Wei Liu
TL;DR
The paper addresses the limited availability of human-annotated video labels and the need for spatio-temporal representations beyond frame-by-frame features. It trains a spatio-temporal network on unlabeled videos by regressing motion and appearance statistics, and reports improved C3D performance and state-of-the-art action-recognition results on UCF101 and HMDB51.
Problem
Human-annotated video data are laborious and expensive to obtain, while prior self-supervised video features are merely frame-by-frame and unsuitable for many spatio-temporal video tasks.
Method
The method trains a spatio-temporal representation network to regress motion and appearance statistics derived from spatial and temporal patterns in unlabeled videos.
Results
The approach significantly improves C3D over random initialization and achieves state-of-the-art performance on UCF101 and HMDB51 action recognition.
Takeaways & Limitations
Predicting statistical labels from motion and appearance provides a self-supervised route to learning spatio-temporal video representations with C3D.
Takeaways & Limitations
The approach assumes that predicting the proposed motion and appearance statistics leads to better spatio-temporal representations and benefits video understanding tasks.
Abstract
from arXiv · showhide
We address the problem of video representation learning without human-annotated labels. While previous efforts address the problem by designing novel self-supervised tasks using video data, the learned features are merely on a frame-by-frame basis, which are not applicable to many video analytic tasks where spatio-temporal features are prevailing. In this paper we propose a novel self-supervised approach to learn spatio-temporal features for video representation. Inspired by the success of two-stream approaches in video classification, we propose to learn visual features by regressing both motion and appearance statistics along spatial and temporal dimensions, given only the input video data. Specifically, we extract statistical concepts (fast-motion region and the corresponding dominant direction, spatio-temporal color diversity, dominant color, etc.) from simple patterns in both spatial and temporal domains. Unlike prior puzzles that are even hard for humans to solve, the proposed approach is consistent with human inherent visual habits and therefore easy to answer. We conduct extensive experiments with C3D to validate the effectiveness of our proposed approach. The experiments show that our approach can significantly improve the performance of C3D when applied to video classification tasks. Code is available at https://github.com/laura-wang/video_repres_mas.
1. Introduction
The paper targets self-supervised learning of spatio-temporal video representations, motivated by the cost of human annotation and limitations of frame-oriented approaches. It proposes predicting motion and appearance statistics from unlabeled videos as supervision for a spatio-temporal network.
- Human annotation makes learning from labeled video data laborious and expensive.
- The approach uses spatial regions and labels such as largest-motion location and direction, most divergent appearance region and color, and most stable appearance region and color.
- Existing unlabeled-video methods derive supervision from frame-order puzzles or computed flow and disparity maps, then transfer the learned representations to video tasks.
- Frame-oriented approaches do not directly provide features for tasks requiring spatio-temporal representations, motivating architectures such as C3D that process multiple frames with 3D convolutions.
- The proposed method predicts motion and appearance statistics from unlabeled videos to learn spatio-temporal video representations.
2. Related work
Related work uses unlabeled video to learn transferable representations, but many methods rely on frame-based inputs. The paper positions its approach among methods intended for spatio-temporal representation learning with 3D architectures.
- Self-supervised representation learning replaces expensive labeled data with unlabeled data and has been applied to image and video problems.
- Image self-supervised tasks include patch reordering, grayscale colorization, inpainting, primitive counting, rotation classification, and clustering-based label prediction.
- Video methods have used frame-order classification or predicted flow fields and disparity maps to learn transferable representations.
- Many earlier video approaches accept only one or two frames and therefore cannot be applied to architectures designed for spatio-temporal representations.
- The related methods most closely connected to this work include a video GAN with spatio-temporal 3D convolutions and a method for learning spatio-temporal representations.
3. Our Approach
The approach learns spatio-temporal video representations by training a CNN to predict motion and appearance statistics derived from unlabeled clips. These statistics encode motion locations and directions, color diversity, and dominant colors across spatial and temporal regions.
- 3.1. Statistical Concepts: The self-supervised task asks a CNN to predict motion and appearance statistics that reflect human visual habits.The queries concern largest motion location and direction, and regions with largest or smallest color diversity and their dominant colors.
- 3.2. Motion Statistics: Optical flow is converted into motion boundaries so constant or smoothly varying camera motion is suppressed.Horizontal and vertical flow components are differentiated, then motion boundaries are aggregated separately across frames.
- 3.2. Motion Statistics: Spatial-aware motion labels identify the block with the largest average motion magnitude and its dominant orientation using three partitioning patterns.The largest-motion locations are represented separately for horizontal and vertical optical-flow components.
- 3.2. Motion Statistics: Global motion labels complement local statistics by identifying the frame interval containing the largest motion.The largest motion is quantified separately from the aggregated horizontal and vertical motion boundaries.
- 3.4. Learning Framework: The method applies separate motion and appearance regression branches to C3D features for self-supervised pre-training.The architecture uses a 16-frame input with 14 motion outputs and 13 appearance outputs.
- 3.3. Appearance Statistics: Appearance labels quantify temporal color diversity with IoU and assign dominant colors to the most and least diverse regions and the whole video.The smallest IoU identifies largest color diversity, while the largest IoU identifies smallest color diversity; RGB-channel scores are averaged.
4. Experiments
Experiments evaluate motion and appearance statistics as self-supervised signals for C3D across action recognition, action similarity labeling, and dynamic scene recognition. The method improves recognition performance, transfers to ASLAN, and produces attention focused on motion-relevant or discriminative regions.
- Pattern ablation: All three motion-statistics patterns outperform training from scratch by around 8% on UCF101, while their similar performance suggests balanced pattern design.The comparison concerns different motion-statistics pattern designs for action recognition.
- Local versus global statistics: Global motion statistics improve action recognition by 3%, and combining all three motion patterns adds around 1.5% over a single pattern.The reported all-motion-statistics label accuracy is 57.8%, also exceeding training from scratch.
- Motion, RGB, and joint statistics: Motion and appearance statistics both provide useful supervision on UCF101 and HMDB51, while their combination improves HMDB51 by 3% but UCF101 by only 1%.The paper reports motion statistics as more powerful because temporal information is important for video understanding.
- Action recognition: Compared with prior self-supervised methods, the approach improves accuracy by 9.3% on HMDB51 and 2.5% on UCF101 over cited UCF101-pretrained methods.It also reports improvements of 0.6% on UCF101 and 5.1% on HMDB51 over a Kinetics-pretrained 3D CNN method using additional regularization.
- Visualization: Attention visualizations show C3D focusing on locations corresponding to the largest motion for similar actions and on discriminative locations for differently scaled motion.The visualization compares a video frame, an activation-based conv5 attention map, and motion boundaries Mu and Mv.
- Transfer evaluation: On ASLAN, the self-supervised C3D outperforms the C3D trained with human-annotated labels by 1.1% and also outperforms traditional and other self-supervised features.The transfer evaluation uses C3D features with a linear SVM for binary same-class versus different-class video classification.
5. Conclusions
The proposed method significantly improves C3D through self-supervised spatio-temporal representation learning from motion and appearance statistics, achieving state-of-the-art performance across the reported datasets and tasks.
- Pre-training C3D on unlabeled videos with this method significantly improves action-recognition performance over random initialization.
- The approach achieves state-of-the-art performance on UCF101 and HMDB51 compared with other self-supervised representation-learning approaches.
- The pre-trained C3D network also serves as a powerful feature extractor for action similarity labeling and dynamic scene recognition, achieving state-of-the-art performance on public datasets.