Source-linked AI summary
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
Longlong Jing, Xiaodong Yang, Jingen Liu, Yingli Tian
TL;DR
Labeled video data is expensive and difficult to scale, motivating self-supervised spatiotemporal representation learning from unlabeled videos. 3DRotNet predicts rotations applied to videos to learn semantic, spatial, and motion features, which transfer effectively to action recognition and outperform other self-supervised methods.
Problem
Large-scale supervised video learning requires millions of labeled video pairs, but collecting annotated datasets is laborious and expensive.
Method
3DRotNet applies rotations to unlabeled videos and trains a 3DCNN to predict them, learning spatial appearance and temporal motion features without annotations.
Results
3DRotNet achieves state-of-the-art performance among self-supervised methods on UCF101 and HMDB51 action recognition benchmarks.
Takeaways & Limitations
The learned representation can be transferred to video understanding tasks when only small labeled datasets are available.
Abstract
from arXiv · showhide
The success of deep neural networks generally requires a vast amount of training data to be labeled, which is expensive and unfeasible in scale, especially for video collections. To alleviate this problem, in this paper, we propose 3DRotNet: a fully self-supervised approach to learn spatiotemporal features from unlabeled videos. A set of rotations are applied to all videos, and a pretext task is defined as prediction of these rotations. When accomplishing this task, 3DRotNet is actually trained to understand the semantic concepts and motions in videos. In other words, it learns a spatiotemporal video representation, which can be transferred to improve video understanding tasks in small datasets. Our extensive experiments successfully demonstrate the effectiveness of the proposed framework on action recognition, leading to significant improvements over the state-of-the-art self-supervised methods. With the self-supervised pre-trained 3DRotNet from large datasets, the recognition accuracy is boosted up by 20.4% on UCF101 and 16.7% on HMDB51 respectively, compared to the models trained from scratch.
1. Introduction
3DRotNet addresses the cost and scale of labeled video data by learning spatiotemporal representations from unlabeled videos through rotation prediction. The learned features capture spatial appearance and temporal motion information and transfer effectively to action recognition.
- Motivation: 3DCNN-based supervised feature learning requires millions of labeled video pairs, while collecting large annotated datasets is laborious and expensive.Kinetics, for example, contains approximately 500,000 videos of 600 human actions.
- Method: 3DRotNet is a fully self-supervised framework that learns spatiotemporal features without human annotations.It uses video rotation transformation as an annotation-free pretext task.
- Method: Rotation prediction requires semantic understanding of persons, objects, locations, and motions, encouraging high-level semantic and motion representations.Attention maps illustrate that the model focuses on persons with the most significant motion.
- Method: The model predicts applied video rotations from RGB or frame-difference inputs, learning spatial cues such as location and shape alongside temporal motion and evolution.The proposed rotations include 0°, 90°, 180°, and 270°.
- Evaluation: 3DRotNet features transfer to action recognition and substantially outperform alternative self-supervised methods on UCF101 and HMDB51.The experiments pre-train on Kinetics and evaluate transferred features on both benchmarks.
- Evaluation: Kinetics-pre-trained 3DRotNet boosts recognition accuracy by 20.4% on UCF101 and 16.7% on HMDB51 versus models trained from scratch.The model is intended for video understanding tasks with small datasets.
2. Related Work
Prior self-supervised work learns visual representations from image structure or temporal relationships, but video methods often rely on 2DCNNs. 3DCNN-based self-supervised learning remains comparatively limited and underperforms supervised methods.
- Self-supervised learning: Self-supervised representation learning methods use image texture or temporal relationships between video frames as supervision signals.Examples include object boundaries, image context, patch similarity, frame order, and cross-modal correspondence.
- Image-based methods: Image-based approaches learn representations by solving spatial prediction tasks such as jigsaw puzzles or relative-position prediction.Other approaches predict image channels or use synthetic images for feature learning.
- Video-based methods: Most prior video methods apply 2DCNNs and use temporal information for tasks such as moving-object segmentation or frame-order verification.These methods primarily learn image representations despite operating on videos.
- 3DCNN-based methods: Although 3DCNNs jointly model spatial and temporal information, few studies had exploited them for self-supervised learning.Earlier examples include video generation and video colorization with 3DCNNs.
- 3DCNN-based methods: Existing 3DCNN self-supervised methods had substantially lower performance than supervised methods.
3. Method
3DRotNet learns spatiotemporal features without annotations by predicting discrete rotations applied to videos. The framework uses a 3D CNN to model spatial appearance and temporal information, with learned representations transferred to video understanding tasks.
- Rotation-prediction formulation: 3DRotNet uses a 3DCNN to learn spatiotemporal features by predicting pre-applied video rotation transformations.The transformation rotates every frame by a specified number of degrees.
- Rotation-prediction formulation: The rotation pretext task can be formulated as regression over continuous angles or classification over predefined discrete rotation categories.Regression uses a rotation parameter, whereas classification predicts one of K rotation categories.
- Proposed framework: The implemented framework applies four rotations—0°, 90°, 180°, and 270°—and trains 3DRotNet with cross-entropy loss to predict each rotation category.The model receives rotated videos and their corresponding categories as training examples.
- Network and evaluation design: The approach uses a 3D ResNet18 architecture and is designed to avoid complicated preprocessing and extra measures against trivial solutions.The architecture is selected for relatively few parameters and spatiotemporal learning capability.
- Network and evaluation design: Learned parameters are evaluated through transfer learning, including fine-tuning 3DRotNet for action recognition and comparison with state-of-the-art self-supervised methods.The evaluation also includes kernel, activation-map, and ablation analyses.
4. Experimental Results
The experiments pre-train 3DRotNet without action labels on large-scale video datasets, then fine-tune it for action recognition on smaller benchmarks. The setup targets feature quality and transfer performance across UCF101 and HMDB51.
- Experimental setup: The experiments evaluate the quality of learned spatiotemporal features for action recognition through extensive analyses and comparisons.The reported evaluation includes transfer performance against state-of-the-art self-supervised methods.
- Experimental setup: 3DRotNet is trained without action labels on Moment in Time and Kinetics, then supervised fine-tuned for action recognition on UCF101 and HMDB51.This evaluates whether self-supervised features transfer from large-scale datasets to relatively small benchmarks.
- Datasets: Moment in Time contains around 1 million videos across 339 action classes, and its training set is used for self-supervised learning without video labels.Videos last around 3 seconds on average.
- Datasets: Kinetics contains around 500,000 videos across 600 action classes, with about 480,000 videos used to train the self-supervised model without labels.Each video lasts around 10 seconds on average.
- Datasets: UCF101 contains 13,320 videos across 101 human action classes, while HMDB51 contains 6,770 videos across 51 actions.Both benchmarks are described as relatively small datasets for evaluating transferred action-recognition features.
4.2. Implementation Details
Training uses randomly sampled video clips resized and cropped into fixed spatiotemporal tensors, while transfer evaluation fine-tunes the pre-trained model for action recognition. Rotation recognition exceeds 89% on both UCF101 and HMDB51.
- Self-supervised learning: Training samples 16 consecutive frames, crops each to 112 × 112 pixels, and forms clips of size 3 channels × 16 frames × 112 × 112 pixels.Videos are downsampled and resized to 136 × 136 pixels before random cropping.
- Transfer learning: Transfer learning uses the same 16-frame, 112 × 112-pixel clip format and optimizes cross-entropy loss with stochastic gradient descent.The pre-trained model is fine-tuned for action recognition on UCF101 and HMDB51.
- Rotation-recognition evaluation: The four-rotation pretext task is tested after training on large-scale datasets, with class labels excluded during self-supervised training.The evaluated rotations are 0°, 90°, 180°, and 270°.
- Rotation-recognition evaluation: More than 89% accuracy is achieved for video-rotation recognition on both UCF101 and HMDB51.The cross-domain results demonstrate recognition of representative appearance cues, while spatiotemporal capture remains separately assessed.
4.4. Can 3DRotNet Learn Spatiotemporal Video Features?
Attention visualizations indicate that 3DRotNet captures spatial and temporal information, while transfer experiments show that its learned features improve action recognition after fine-tuning.
- Attention maps show 3DRotNet focusing on foreground persons and moving objects, including a crawling baby.These visualizations support capture of spatiotemporal information rather than reliance on trivial rotation cues.
- 3DRotNet features are transferred by fine-tuning pretrained models on UCF101 and HMDB51 action-recognition datasets.
- 20.4% improvement on UCF101 reaches 62.9%, while 16.7% improvement on HMDB51 reaches 33.7% over training from scratch.
- Freezing the first convolution block works best on UCF101, whereas freezing the first two works best on HMDB51; freezing all blocks performs worst.
- Action-recognition performance increases with rotation-recognition accuracy, supporting meaningful feature learning from rotation prediction.
4.6. Ablation Study of Impact of Rotations
The ablation study finds that four rotation classes provide the strongest transfer performance, while more finely divided rotation schemes perform worse under the tested setup.
- Four rotation degrees achieve the best UCF101 transfer performance, while two rotations achieve the worst.
- Using only two rotations yields 50.94% on UCF101, 11.96% below the network pretrained with four rotations.
- The authors attribute the two-versus-four rotation difference probably to the greater supervision signal from recognizing four rotations.
- The eight-rotation network reaches 57.0%, 5.9% below the four-rotation network, while the 360-rotation network reaches 60.9%, 2.0% below it.
- The authors suggest performance degradation may result from context loss caused by center cropping to remove empty areas after rotation.
4.7. Ablation Study of Impact of Data Amount
Transfer performance improves with more unlabeled pretraining videos, and self-supervised initialization remains beneficial when the labeled target dataset is extremely small.
- Transfer-learning performance increases as more Kinetics videos are used for self-supervised pretraining.
- The results indicate that large-scale data is needed for self-supervised learning and can further improve action recognition.
- With only 20 labeled UCF101 videos, the self-supervised pretrained model performs comparably to a model trained from scratch with nearly 10,000 labeled videos.
4.8. Learning Long-Term Temporal Information
Longer input clips improve transferred action-recognition performance, while difference-of-frames inputs provide stronger motion-related features than RGB clips.
- Longer input clips improve transfer-learning performance on UCF101.
- Increasing clip length from 16 to 64 frames improves UCF101 performance by 3.5%.
- Difference-of-frames inputs achieve higher performance than RGB clips, likely because they capture motion while remaining invariant to appearance.
4.9. Compare with Other Self-supervised Methods
3DRotNet outperforms prior self-supervised methods on UCF101 and HMDB51, with fusion further improving recognition. Its learned kernels are similar to those of a fully supervised model despite using no human-annotated category labels.
- Fusion improves performance by 2.3% on UCF101 and 4.5% on HMDB51 by combining RGB and DIF network scores.
- The fusion result is only 7.8% below the supervised result on UCF101.
- The self-supervised model learns kernels similar to those of the fully supervised model when both are trained on Kinetics.
5. Conclusion
The paper presents a straightforward 3DCNN-based method for learning spatiotemporal video features without human annotations. Experiments show that video rotation transformations provide information for learning spatial and temporal features and achieve state-of-the-art performance among self-supervised methods on two benchmarks.
- 3DRotNet learns spatiotemporal video features through a straightforward 3DCNN-based self-supervised approach.
- Video rotation transformations provide essential information for learning both spatial and temporal video features.
- 3DRotNet achieves state-of-the-art performance on two action-recognition benchmarks among existing self-supervised methods.