Source-linked AI summary
Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning
Dezhao Luo, Chang Liu, Yu Zhou, Dongbao Yang, Can Ma, Qixiang Ye, Weiping Wang
TL;DR
Video self-supervised learning needs representations that are less task-specific and more assessable without extensive annotation. VCP deletes video clips, replaces them with operation-transformed options, and predicts operation categories using 3D-CNNs. It supports proxy and target tasks, and experiments report improved action-recognition and video-retrieval performance with significant margins over state-of-the-art self-supervised models.
Problem
Video annotation is costly, while existing self-supervised methods can be task-specific and offer limited representation assessment.
Method
VCP withholds video clips, applies multiple spatio-temporal operations to create options, and predicts their operation categories with 3D-CNNs.
Results
VCP improves state-of-the-art self-supervised models on action recognition and video retrieval with significant margins.
Takeaways & Limitations
VCP provides flexible self-supervised representation learning and a uniform, interpretable approach for assessing learned models.
Abstract
from arXiv · showhide
We propose a novel self-supervised method, referred to as Video Cloze Procedure (VCP), to learn rich spatial-temporal representations. VCP first generates "blanks" by withholding video clips and then creates "options" by applying spatio-temporal operations on the withheld clips. Finally, it fills the blanks with "options" and learns representations by predicting the categories of operations applied on the clips. VCP can act as either a proxy task or a target task in self-supervised learning. As a proxy task, it converts rich self-supervised representations into video clip operations (options), which enhances the flexibility and reduces the complexity of representation learning. As a target task, it can assess learned representation models in a uniform and interpretable manner. With VCP, we train spatial-temporal representation models (3D-CNNs) and apply such models on action recognition and video retrieval tasks. Experiments on commonly used benchmarks show that the trained models outperform the state-of-the-art self-supervised models with significant margins.
1 Introduction
VCP addresses limitations in video self-supervised learning by using spatio-temporal operations to learn representations and by supporting interpretable model assessment.
- Large-scale video annotation is costly, motivating self-supervised learning from unlabeled data.
- Existing video methods often rely on domain knowledge, restrict operations, and lack effective model assessment.
- VCP withholds a video clip, applies multiple spatio-temporal operations, and trains a 3D-CNN to classify their categories.
- As a proxy task, VCP improves flexibility and reduces representation-learning complexity; as a target task, it enables interpretable assessment.
- VCP provides a simple framework for self-supervised spatio-temporal representation learning.
- Experiments apply VCP to 3D-CNNs for action recognition and video retrieval, improving state-of-the-art methods with significant margins.
2 Related work
Prior video self-supervised methods use annotation-free proxy tasks but often learn task-specific or frame-level features and provide limited assessment of representation quality.
- Self-supervised learning trains models using information from unlabeled data through annotation-free proxy tasks.
- Earlier approaches predicted image patch or video frame order, while other methods used motion statistics, odd-one-out detection, or space-time puzzles.
- Frame-by-frame features are poorly suited to video tasks requiring spatio-temporal representations.
- Existing methods are typically designed for specific target tasks, limiting learned-model capability and general applicability.
- Few proxy tasks assess feature representations, limiting the pertinence of self-supervised models.
- VCP is introduced to incorporate multiple self-supervised representations and improve learned-model generality.
3 Video Cloze Procedure
VCP forms a video cloze task by deleting a clip, generating spatial and temporal options, and predicting each option’s operation category with shared 3D-CNNs.
- VCP is motivated by Cloze Procedure, which evaluates completion after words are deleted from prose.
- Blank generation: Blank generation samples equal-length, non-overlapping clips and randomly deletes one clip from a sequence of m clips.
- Blank generation: Sampling clips at intervals weakens low-level cue relevance and encourages attention to middle- and high-level spatio-temporal features.
- Spatial operation: Spatial rotation and spatial permutation create options that preserve some spatial structure while promoting orientation-related learning.
- Temporal Operation: Temporal remote and adjacent shuffling generate options targeting temporal information while retaining temporal relevance.
- Cloze completion: Shared-parameter 3D-CNNs concatenate clip features and predict the operation category through a fully connected layer.
4 Self-supervised Representation Learning
VCP supports representation learning with labeled or additional unlabeled data and provides a uniform target task for evaluating self-supervised models.
- Representation learning: VCP can serve as a proxy task using only original labeled data or additional unlabeled data.
- Representation learning: Pretraining on unlabeled data provides warm-start representations that can be fine-tuned on annotated target datasets.
- Model assessment: As a target task, VCP freezes the backbone and fine-tunes a classification head to predict operation categories.
- Model assessment: Operation-classification accuracy serves as a uniform, interpretable metric for comparing self-supervised representation models.
- Complexity: Existing order-prediction methods have O(n!) computational complexity, whereas VCP has O(n) complexity.
- Flexibility: VCP can configure spatial or temporal operations for different target tasks and combine them flexibly.
5 Experiments
The experiments evaluate VCP's representation learning, model assessment, and applications to action recognition and video retrieval against state-of-the-art methods.
- The experiments evaluate VCP's option configurations, data strategies, model assessment, action recognition, and video retrieval.
5.1 Experiment Setting
Experiments use UCF101 and HMDB51 with C3D, R3D, and R(2+1)D backbones, standardized clip sampling, and operation-classification evaluations.
- Experiments use UCF101 and HMDB51, which contain 101 and 51 action categories, respectively.
- C3D, R3D, and R(2+1)D serve as VCP backbones for modeling spatial-temporal video information.
- Each cloze item contains three successive 16-frame clips sampled every 8 frames, with frames resized to 128 × 171 and cropped to 112 × 112.
- Operation-classification accuracy is reported for original clips and spatial or temporal transformations, while the ablation table reports action-recognition accuracy.
5.2 Representation Learning
VCP learns useful representations from flexible combinations of spatial and temporal operations, improving action recognition across option configurations and data strategies.
- 78.42% overall accuracy was achieved when classifying five VCP option categories on UCF101.
- 69.7% action-recognition accuracy was obtained by combining spatial and temporal operations, outperforming the random baseline by 7.7%.
- Combining spatial and temporal operations outperformed standalone operations, supporting flexible use of complementary options.
- VCP outperformed VCOP by 2.9% when pre-training and fine-tuning on UCF101, and slightly outperformed it under the corresponding HMDB51 strategy.
- VCP also outperformed VCOP when transferring between UCF101 and HMDB51, including by 2.6% when pre-training on HMDB51 and fine-tuning on UCF101.
5.3 Model Assessment
VCP provides an interpretable operation-classification assessment of learned models, revealing strong recognition of several operations and differing spatial-temporal capabilities.
- The assessment fine-tunes only a fully connected layer while keeping the self-supervised backbone fixed, producing operation-classification accuracy curves.
- Approximately 90% accuracy was reached for spatial rotation, spatial permutation, and temporal adjacent shuffling.
- Original clips and temporally remote-shuffled clips were difficult to classify, with their accuracies negatively correlated.
- ST-Puzzle and S-Puzzle outperform temporal-puzzle alternatives and VCOP for spatial classification but underperform them for temporal classification.
5.4 Action Recognition
VCP pre-training improves action-recognition accuracy across C3D and R3D backbones, outperforming random initialization and the self-supervised VCOP approach. The evaluation uses fine-tuned models on UCF101 and HMDB51.
- Evaluation procedure: VCP initializes the backbone, while randomly initialized fully connected layers are trained for action classification during fine-tuning.The procedures use 150 fine-tuning epochs, with features passed to fully connected classification layers.
- Action-recognition results: 6.7% and 7.8% gains over C3D random initialization are achieved on UCF101 and HMDB51, respectively.VCP also exceeds VCOP by 2.9% and 4.1% on the two datasets.
- Action-recognition results: 11.5% and 9.8% performance gains over R3D random initialization are reported for the evaluated benchmarks.VCP also outperforms VCOP with significant margins using the R3D backbone.
5.5 Video Retrieval
VCP is evaluated for nearest-neighbor video retrieval using pre-trained convolutional features. It significantly outperforms compared approaches across all reported accuracy metrics, with qualitative results also favoring VCP.
- Evaluation procedure: VCP extracts conv5 features from training and test videos for nearest-neighbor retrieval on the first UCF101 split.Each test video queries k nearest training videos under the stated retrieval protocol.
- Retrieval results: VCP significantly outperforms compared approaches on top-1, top-5, top-10, top-20, and top-50 accuracy.The result is reported for the video-retrieval evaluations summarized in Tables 5 and 6.
- Retrieval results: Qualitative retrieval results in Figure 5 also show VCP's superiority.The figure marks correct retrieval results in red.
6 Conclusion
The paper concludes that VCP learns rich spatio-temporal representations for 3D CNNs and improves their use in action recognition and video retrieval. It also presents VCP as a target task for assessing self-supervised models.
- Conclusion: VCP is proposed as a self-supervised method for learning rich spatial-temporal representations.The method trains spatial-temporal representation models using 3D CNNs.
- Conclusion: The trained models are applied to action recognition and video retrieval tasks.The conclusion reports validated improvements in representation capability across these applications.
- Conclusion: Designing VCP as a target task provides a model-assessment approach intended to improve the pertinence and interpretability of self-supervised learning.The paper presents this as a fresh insight based on simulating human language learning.