PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatiotemporal Contrastive Video Representation Learning

Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, Yin Cui

arXiv:2008.03800v4cs.CVcs.LG

TL;DR

Self-supervised video learning needs representations that capture both spatial and temporal information from unlabeled videos. CVRL learns them by contrastively aligning augmented clips from the same video while separating clips from different videos, achieving strong linear-evaluation results and improving with larger data and networks.

  • Problem

    Spatial self-supervision has been under-exploited in video representation learning, despite videos containing both spatial and temporal cues.

  • Method

    CVRL uses an InfoNCE contrastive framework with temporally consistent spatial augmentation and sampling-based temporal augmentation for paired clips from unlabeled videos.

  • Results

    22.9% top-1 accuracy improvement is achieved by combining the proposed spatial and temporal augmentations, while CVRL representations outperform prior baselines across linear, semi-supervised, and downstream evaluations.

  • Takeaways & Limitations

    CVRL benefits from larger datasets and networks, supporting its potential to scale spatiotemporal representation learning to larger unlabeled video collections.

Abstract

from arXiv · show

We present a self-supervised Contrastive Video Representation Learning (CVRL) method to learn spatiotemporal visual representations from unlabeled videos. Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away. We study what makes for good data augmentations for video self-supervised learning and find that both spatial and temporal information are crucial. We carefully design data augmentations involving spatial and temporal cues. Concretely, we propose a temporally consistent spatial augmentation method to impose strong spatial augmentations on each frame of the video while maintaining the temporal consistency across frames. We also propose a sampling-based temporal augmentation method to avoid overly enforcing invariance on clips that are distant in time. On Kinetics-600, a linear classifier trained on the representations learned by CVRL achieves 70.4% top-1 accuracy with a 3D-ResNet-50 (R3D-50) backbone, outperforming ImageNet supervised pre-training by 15.7% and SimCLR unsupervised pre-training by 18.8% using the same inflated R3D-50. The performance of CVRL can be further improved to 72.9% with a larger R3D-152 (2x filters) backbone, significantly closing the gap between unsupervised and supervised video representation learning. Our code and models will be available at https://github.com/tensorflow/models/tree/master/official/.

1. Introduction

CVRL learns spatiotemporal video representations by contrastively training on augmented clips, with augmentations designed to preserve temporal structure while exploiting spatial and temporal cues. It outperforms prior pre-training methods across linear, semi-supervised, and downstream evaluations, and benefits from more data and larger networks.

  • Motivation and approach: CVRL addresses the underuse of spatial self-supervision in videos by learning representations from augmented clips sampled from the same unlabeled video.The framework contrasts positive clips from one video against negative clips from different videos using InfoNCE.
  • Data augmentation: Temporally consistent spatial augmentation preserves natural motion, while decreasing-probability temporal sampling emphasizes nearby positive clips without discarding distant ones.Independent frame-wise spatial augmentation hurts learning because it breaks temporal consistency.
  • Ablations: 22.9% top-1 accuracy improvement results from combining the proposed spatial and temporal augmentations, whereas either augmentation alone performs relatively poorly.The combined design uses temporally consistent spatial augmentation and the proposed temporal sampling strategy.
  • Evaluation: CVRL representations outperform competing baselines by more than 15% in linear evaluation and improve over ImageNet pre-training by 12.6% on Kinetics-400.The representations also show advantages with limited labels and competitiveness with multimodal methods on downstream action classification.
  • Scalability: CVRL improves with 50% more training data and with wider or deeper networks, indicating scalability to larger unlabeled datasets and models.The evaluation controls training iterations when comparing different data amounts.

2. Related Work

Prior video self-supervised learning commonly exploits temporal structure, while image representation learning has advanced through spatial pretext and contrastive tasks. CVRL brings contrastive spatial and temporal augmentation principles into video representation learning.

  • Video self-supervised learning: Video self-supervised methods have used future prediction, altered sampling rates, frame or clip ordering, and combinations of temporal proxy tasks.These approaches exploit the temporal dimension of videos.
  • Image self-supervised learning: Image self-supervised learning has used auto-encoding, patch-location, jigsaw, rotation, and contrastive objectives on augmented images.Many image pretext tasks can be integrated into contrastive learning frameworks.
  • CVRL: CVRL combines spatial and temporal cues by contrasting augmented clips from the same video against clips from different videos.Its framework uses a 3D backbone and an InfoNCE contrastive loss.
  • Video as supervision: Video also provides supervision for image representations, visual correspondences, and robotic behaviors through object views, motion, and tracking cues.Conversely, video representations have also been learned by distillation from image representations.

3. Methodology

CVRL learns video representations by contrasting augmented clips with an InfoNCE loss, using a 3D-ResNet encoder and spatiotemporal augmentations. Its augmentation design preserves temporal structure while incorporating spatial variation and samples temporal intervals strategically.

  • Video Representation Learning Framework: CVRL samples N videos, creates 2N augmented clips, encodes them, and applies InfoNCE to contrast paired representations.The framework uses two augmented clips from each input video and organizes encoder, augmentation, and evaluation components around this loss.
  • Video Representation Learning Framework: The InfoNCE loss attracts each positive pair while repelling other encoded clips in the mini-batch.Similarity uses ℓ2-normalized vectors, excludes self-similarity from the denominator, and includes a temperature parameter τ.
  • Video Encoder: The video encoder uses 3D-ResNets with expanded 2D kernels to capture spatiotemporal information, producing 2048-dimensional backbone features and 128-dimensional projected vectors.Evaluation discards the projection head and uses the 2048-dimensional backbone representation; larger 2× and 4× backbones are also tested.
  • Data Augmentation: Temporal augmentation samples clip pairs according to temporal intervals rather than directly enforcing temporal invariance, accounting for video content changes over time.The strategy is motivated by the concern that direct temporal transformations could produce features invariant to temporal evolution.
  • Data Augmentation: Decreasing temporal-interval sampling distributions outperform uniform and increasing alternatives, with a linear decrease selected for simplicity and best performance.Figure 3 reports sampling probability against the temporal interval between clips and linear-evaluation accuracy after 200 pre-training epochs on Kinetics-400.
  • Data Augmentation: Temporally consistent spatial augmentation applies shared randomness across frames so strong spatial transformations do not break natural motion cues.The algorithm processes an M-frame clip into an augmented clip while fixing augmentation randomness across frames.
  • Evaluation: Representations are evaluated with frozen-backbone linear classification and with fine-tuning for semi-supervised learning, action classification, detection, and related downstream tasks.The encoder uses 16-frame stride-2 inputs during self-supervised pre-training and 32-frame stride-2 inputs for evaluation and downstream settings.

4. Experiments

Experiments evaluate CVRL on Kinetics-400 and Kinetics-600 using linear evaluation, semi-supervised learning, and downstream action tasks. Results compare CVRL with inflated ImageNet and SimCLR baselines and test augmentation, data, architecture, and optimization choices.

  • Evaluation setup: CVRL is evaluated primarily on Kinetics-400 and Kinetics-600, with linear evaluation as the main representation-quality metric.The study also assesses semi-supervised learning, downstream action classification, and action detection.
  • Linear evaluation: 70.4% versus 70.5% top-1 accuracy is achieved by CVRL and MMV on Kinetics-600, despite CVRL using 133× less pre-training data, 3× fewer parameters, and vision only.With R3D-101, CVRL reaches 71.6% versus MMV’s 70.5% using 60% of MMV’s parameters.
  • Semi-supervised learning: CVRL surpasses all other baselines across Kinetics-400 semi-supervised settings, especially with only 1% labeled data.The reported advantage is stronger when the labeled fraction is limited.
  • Ablation study: 63.8% top-1 accuracy results from adding temporal consistency to spatial and temporal augmentation, improving 52.3% by 11.5 percentage points.Using temporal augmentation only yields 33.0%, while spatial augmentation only yields 40.9%. Decreasing temporal interval distributions perform best, with a linear exponent selected.

5. Conclusion

CVRL learns spatiotemporal representations from unlabeled videos by leveraging both spatial and temporal cues. Evaluations across multiple settings show promising results, with future work targeting larger unlabeled datasets and additional modalities.

  • CVRL learns spatiotemporal representations from unlabeled videos using spatial and temporal cues.
  • Extensive evaluations cover linear evaluation, semi-supervised learning, downstream action classification, and detection.
  • Future work will apply CVRL to larger unlabeled video collections and incorporate additional modalities.

A. Details on Temporal Interval Sampling

Temporal intervals are sampled from a specified distribution using inverse transform sampling. Because the inverse CDF lacks a convenient closed form and intervals are integer-valued, binary search finds the sampled interval in O(log T) time.

  • Temporal interval t ∈[0, T] is sampled from a specified distribution P(t).
  • Inverse transform sampling generates v ∼U(0, 1) and targets t = F^-1(v), where F is the cumulative distribution function.
  • Because directly computing F^-1 is difficult, a monotonicity-based binary search finds the integer temporal interval t.
  • The temporal interval sampling algorithm has complexity O(log T).

B.1. Semi-Supervised Learning on Kinetics-600

On Kinetics-600, CVRL shows strong semi-supervised performance, especially when only 1% of videos per class are labeled.

  • CVRL shows strong semi-supervised performance on Kinetics-600, particularly with 1% labeled data.The evaluation samples balanced 1% and 10% subsets from each class while keeping the evaluation set unchanged.

B.2. Comparison with RandAugment

The proposed spatial augmentation outperforms RandAugment with temporal consistency in pre-training. RandAugment’s lower linear-evaluation accuracy suggests supervised image augmentations may not transfer directly to self-supervised video learning.

  • The experiments evaluate strong spatial augmentations, including RandAugment, for self-supervised video pre-training.RandAugment randomly selects two operators from a pool of 14.
  • The proposed temporally consistent spatial augmentation outperforms RandAugment with temporal consistency.
  • 54.2% top-1 accuracy is achieved by RandAugment with temporal consistency in linear evaluation.The result comes from 200-epoch pre-training on Kinetics-400.
  • Strong augmentations optimized for supervised image recognition do not necessarily perform as well in self-supervised video learning.

C.1. Pre-Training and Linear Evaluation

The section reports pre-training and linear-evaluation statistics for models trained on Kinetics-400, including four pre-training metrics and corresponding evaluation curves.

  • Four pre-training metrics are tracked: contrastive loss, regularization loss, entropy, and pre-training accuracy.The total loss is defined as the sum of contrastive loss and regularization loss.

C.2. Temporally Consistent Spatial Augmentation

Temporally consistent spatial augmentation strengthens each video clip spatially while preserving natural motion across frames. The accompanying figures track training statistics and linear-evaluation accuracy on Kinetics-400.

  • C.2. Temporally Consistent Spatial Augmentation: Independent frame-level spatial augmentation breaks motion cues across frames.The figure contrasts this detrimental middle-row treatment with the proposed temporally consistent approach.
  • C.2. Temporally Consistent Spatial Augmentation: The proposed augmentation changes the video clip’s spatial domain while maintaining its natural temporal motion changes.This preserves temporal consistency while applying spatial augmentation throughout the clip.
  • C.2. Temporally Consistent Spatial Augmentation: Figure 5 reports contrastive loss, regularization loss, entropy, and pre-training accuracy during model pre-training.The total loss combines the contrastive and regularization losses.
  • C.2. Temporally Consistent Spatial Augmentation: Figure 6 shows top-1 accuracy for linear-evaluation training and evaluation on Kinetics-400.Training is shown with dashed lines and evaluation with solid lines.
Loading 2008.03800v4…