Source-linked AI summary
Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learning
Yuan Yao, Chang Liu, Dezhao Luo, Yu Zhou, Qixiang Ye
TL;DR
Self-supervised video representation learning has not fully explored temporal resolution and long-short term characteristics. PRP addresses this with dilated sampling plus collaborative discriminative and generative perception, and it outperforms state-of-the-art self-supervised models on action recognition and video retrieval.
Problem
Existing self-supervised video representations do not fully capture multiple temporal resolutions and long-short term characteristics.
Method
PRP uses dilated sampling to generate playback-rate supervision, then combines playback-rate classification with reconstruction and motion attention.
Results
PRP outperforms state-of-the-art self-supervised models with significant margins on action recognition and video retrieval.
Takeaways & Limitations
PRP provides a framework for self-supervised spatio-temporal representation learning that incorporates long-short term temporal characteristics.
Abstract
from arXiv · showhide
In self-supervised spatio-temporal representation learning, the temporal resolution and long-short term characteristics are not yet fully explored, which limits representation capabilities of learned models. In this paper, we propose a novel self-supervised method, referred to as video Playback Rate Perception (PRP), to learn spatio-temporal representation in a simple-yet-effective way. PRP roots in a dilated sampling strategy, which produces self-supervision signals about video playback rates for representation model learning. PRP is implemented with a feature encoder, a classification module, and a reconstructing decoder, to achieve spatio-temporal semantic retention in a collaborative discrimination-generation manner. The discriminative perception model follows a feature encoder to prefer perceiving low temporal resolution and long-term representation by classifying fast-forward rates. The generative perception model acts as a feature decoder to focus on comprehending high temporal resolution and short-term representation by introducing a motion-attention mechanism. PRP is applied on typical video target tasks including action recognition and video retrieval. Experiments show that PRP outperforms state-of-the-art self-supervised models with significant margins. Code is available at github.com/yuanyao366/PRP
1. Introduction
Existing self-supervised video methods do not adequately capture multiple temporal resolutions and long-short term characteristics. PRP addresses this gap by combining dilated sampling with discriminative and generative perception, improving downstream action recognition and video retrieval.
- Motivation: Video annotation is costly or impractical, motivating self-supervised learning from large-scale unlabelled data.Self-supervised methods construct annotation-free proxy tasks from intrinsic data correspondence.
- Motivation: Frame-based methods ignore temporal relations, while existing spatio-temporal approaches lack temporal-resolution perception needed for long-short term video representation.The paper identifies multiple temporal resolutions as essential for precise video understanding.
- Approach: PRP learns multiple temporal resolutions through dilated sampling, which creates videos with different playback rates and uses their content similarity as supervision.The original video simulates a high playback rate relative to frame-sampled videos.
- Approach: Discriminative and generative perception cooperatively retain spatio-temporal semantics by classifying playback rates and reconstructing low playback rate videos from high playback rate ones.The discriminative branch emphasizes low temporal resolution and long-term representation, while the generative branch emphasizes high temporal resolution and short-term representation.
- Approach: A motion-attention mechanism directs learned representations toward meaningful foreground regions.This mechanism is introduced within the generative perception model.
- Results: PRP improves state-of-the-art self-supervised models on action recognition and video retrieval with significant margins.The approach is applied to three kinds of 3D CNNs and two target tasks.
2. Related Work
Prior self-supervised video research uses spatial, temporal-order, motion, appearance, and reconstruction-based proxy tasks. The paper positions PRP as a response to the unresolved need for representations that capture multiple temporal resolutions and broader task generality.
- Self-supervised proxy tasks: Self-supervised methods use annotation-free proxy tasks based on discriminative transformations, generative reconstruction, and information from multiple sensory modalities.Examples include transformed-image classification, video-content classification, image inpainting, video reconstruction, ego-motion, sound, and cross-modal data.
- Spatial representation learning: Spatial representation methods learn from image transformations such as rotation, jigsaw completion, and withheld-region prediction.These approaches primarily construct supervision from spatial image structure.
- Spatio-temporal representation learning: Spatio-temporal methods use temporal continuity, frame or clip ordering, odd-clip identification, motion and appearance statistics, and space-time puzzle completion.These methods extend self-supervision beyond frame-level spatial content.
- Position of PRP: PRP combines dilated sampling with perception modeling to classify playback rates and reconstruct videos across temporal resolutions.Its perception modeling uses a feature encoder, discriminative module, and reconstructing decoder.
- Research gap: Existing methods largely ignore multiple temporal resolutions, limiting the representation capability of learned models for video-based tasks.The paper identifies temporal-resolution characteristics as essential for video analysis.
- Research gap: Video-classification pretraining can lack general applicability, producing suboptimal results when fine-tuned for action recognition and video retrieval.PRP is proposed to improve model generality by incorporating long-short term temporal representations.
3. Playback Rate Perception
Playback Rate Perception (PRP) learns multi-resolution spatio-temporal representations by combining dilated sampling with cooperative discriminative and generative perception. Its encoder classifies playback rates for long-term features, while its decoder reconstructs slow-down clips using motion attention for short-term details.
- Motivation: Fast-forward and slow-down playback motivate modeling both temporal resolution and long-short term characteristics in video understanding.Fast-forward supports rapid content comprehension, whereas slow-down playback exposes fine details.
- Dilated Sampling: Dilated sampling selects one frame every s frames to create videos with s× fast-forward rates while preserving content similarity across sampling intervals.Successive l frames from each sampled video form learning samples for 3D CNNs.
- Feature Encoder: The feature encoder uses C3D, R3D, or R(2+1)D blocks to extract spatial and temporal features from video clips.R3D uses residual connections, while R(2+1)D decomposes 3D convolution into spatial 2D and temporal 1D convolutions.
- Discriminative Perception: Discriminative perception classifies playback-rate classes, driving the encoder to capture low-resolution, long-term dynamics and subtle motion differences.The classifier predicts the input clip’s sampling-rate class using normalized probabilities and cross-entropy optimization.
- Generative Perception: Generative perception reconstructs r× slow-down clips with a feature decoder, using motion-attention regularized MSE to emphasize regions with significant motion.Motion attention is computed from frame differences, 3D pooling, activation, and 3D upsampling operations.
- Discriminative-Generative Fusion: Cooperative fusion combines long-term playback-rate discrimination with short-term reconstruction details to encode multi-resolution spatio-temporal characteristics.The framework is presented as distinct from VAE and GAN: it retains downstream-useful semantics through cooperative rather than adversarial discrimination-generation.
4. Experiments
Experiments evaluate PRP through ablations, activation-map analysis, action recognition, and video retrieval. Results show that its sampling, collaborative perception, and motion-attention design improve learned representations and downstream performance.
- Experimental setup: PRP is evaluated with C3D, R3D, and R(2+1)D backbones on action recognition and video retrieval tasks.The experiments include UCF101 and HMDB51 action recognition datasets and retrieval evaluation on UCF101 and HMDB51.
- Ablation study: Sampling intervals from {1,2} to {1,2,4,8} raise action-recognition accuracy from 68.3% to 69.9%, while {1,2,4,8,16} causes excessive content jumps.Discriminative-perception accuracy decreases from 88.3% to 60.1% as the sampling interval increases, motivating the default interval set {1,2,4,8}.
- Ablation study: The discriminative-generative model reaches 70.9% action-recognition accuracy, exceeding discriminative perception at 69.9% and generative perception at 68.4%.The ablation compares the branches against a 62.0% baseline.
- Ablation study: Motion attention increases action-recognition accuracy from 67.1% to 68.1% by focusing representation on meaningful foreground regions.Activation maps show complementary behavior: discriminative perception emphasizes major motion regions, while generative perception covers motion and broader context.
- Action recognition: With C3D, PRP improves over random initialization by 7.3% on UCF101 and 9.8% on HMDB51, and over VCOP by 3.5% and 6.1%, respectively.With R(2+1)D, PRP improves over random initialization by 16.3% on UCF101 and 13.0% on HMDB51.
- Video retrieval: PRP outperforms the state-of-the-art retrieval method across top-1, top-5, top-10, top-20, and top-50 metrics, with top-1 gains of 8.7–10.7% on UCF101.Qualitative retrieval examples also report more accurate and reasonable results than VCOP.
5. Conclusion
PRP uses dilated sampling with discriminative and generative models to learn long-short term spatio-temporal representations. Across action recognition and video retrieval, PRP-trained models outperform state-of-the-art self-supervised methods with significant margins.
- Conclusion: PRP augments videos into different temporal resolutions and learns long-short term characteristics through discriminative and generative models.The framework is applied to action recognition and video retrieval.