PaperlayerView exact revision on arXiv

Source-linked AI summary

TCLR: Temporal Contrastive Learning for Video Representation

Ishan Dave, Rohit Gupta, Mamshad Nayeem Rizve, Mubarak Shah

arXiv:2101.07974v4cs.CV

TL;DR

Existing contrastive video methods had not explicitly examined whether features should be distinct across time. TCLR adds local-local and global-local temporal contrastive losses to standard instance discrimination, and reports state-of-the-art improvements across video understanding tasks while using RGB video alone.

  • Problem

    Prior contrastive video representation learning had not explored explicitly encouraging features to be distinct across the temporal dimension.

  • Method

    TCLR combines standard instance contrastive learning with local-local loss across non-overlapping same-video clips and global-local loss across aligned timesteps.

  • Results

    TCLR achieves state-of-the-art results across various downstream video understanding tasks and improves fine-grained action classification.

  • Takeaways & Limitations

    Explicitly modeling temporal variation extends contrastive learning beyond instance discrimination for video representation learning.

  • Takeaways & Limitations

    The work focuses on RGB video and does not use auxiliary modalities or additionally computed visual priors.

Abstract

from arXiv · show

Contrastive learning has nearly closed the gap between supervised and self-supervised learning of image representations, and has also been explored for videos. However, prior work on contrastive learning for video data has not explored the effect of explicitly encouraging the features to be distinct across the temporal dimension. We develop a new temporal contrastive learning framework consisting of two novel losses to improve upon existing contrastive self-supervised video representation learning methods. The local-local temporal contrastive loss adds the task of discriminating between non-overlapping clips from the same video, whereas the global-local temporal contrastive aims to discriminate between timesteps of the feature map of an input clip in order to increase the temporal diversity of the learned features. Our proposed temporal contrastive learning framework achieves significant improvement over the state-of-the-art results in various downstream video understanding tasks such as action recognition, limited-label action classification, and nearest-neighbor video retrieval on multiple video datasets and backbones. We also demonstrate significant improvement in fine-grained action classification for visually similar classes. With the commonly used 3D ResNet-18 architecture with UCF101 pretraining, we achieve 82.4\% (+5.1\% increase over the previous best) top-1 accuracy on UCF101 and 52.9\% (+5.4\% increase) on HMDB51 action classification, and 56.2\% (+11.7\% increase) Top-1 Recall on UCF101 nearest neighbor video retrieval. Code released at github.com/DAVEISHAN/TCLR.

1. Introduction

TCLR addresses the open question of whether video representations should remain temporally invariant or capture temporal variation by adding losses that explicitly encourage distinct features across time. Combined with instance contrastive learning, these losses improve downstream video understanding performance.

  • Motivation: Video actions often contain distinct temporal stages, yet instance contrastive learning encourages similar features for temporally distant clips from the same video.LongJump is described as progressing from running to jumping, while standard instance discrimination imposes temporal invariance.
  • Approach: TCLR explicitly learns temporally distinct video representations rather than relying only on temporal invariance.The framework targets temporal variation while retaining instance-level discrimination between videos.
  • Results: The temporal losses increase learned feature diversity and improve accuracy on downstream video understanding tasks.The paper reports gains across action recognition, retrieval, and limited-label classification settings.
  • Approach: The framework introduces local-local and global-local temporal contrastive losses alongside the standard instance contrastive loss.The local-local loss operates across non-overlapping clips, while the global-local loss operates across timesteps within a clip.
  • Contribution: TCLR is presented as the first contrastive framework to explicitly enforce within-instance temporal feature variation for video understanding.Experiments use multiple 3D CNN architectures and the UCF101 and HMDB51 datasets, with Diving48 proposed for fine-grained evaluation.

2. Related Work

Prior self-supervised video representation methods use pretext tasks, instance-level contrastive learning, temporal sampling, auxiliary modalities, or handcrafted visual priors. TCLR instead focuses on temporal variation using RGB video without additional modalities or computed priors.

  • Self-supervised video learning: Self-supervised video representation learning has been organized around pretext-task methods and contrastive-learning methods.Contrastive approaches commonly treat clips from the same video as positives and clips from different videos as negatives.
  • Contrastive approaches: Several contrastive methods address temporal behavior through sampling, temporal robustness, or temporal decay of representations.CVRL uses weighted temporal sampling, while VideoMoCo models temporal decay of keys.
  • Pretext-task combinations: Other methods combine contrastive learning with playback-rate, temporal-transform, frame-rotation, or frame-order pretext tasks.These approaches use auxiliary prediction objectives to learn temporally varying features.
  • Intra-video variance: Intra-video variance has also been studied with optical flow, frame manipulation, triplet losses, or spatial warping.These methods differ in how they generate intra-instance negatives or decouple scene and motion features.
  • Scope: Some self-supervised methods require cross-modal data or expensive handcrafted visual priors, whereas this work uses only RGB video.The stated scope excludes text, audio, optical flow, and dense trajectories as additional inputs or computed priors.

3. Method

TCLR combines instance discrimination with two within-instance temporal contrastive losses: local-local discrimination across non-overlapping clips and global-local discrimination across temporal feature-map representations.

  • Framework: TCLR combines instance discrimination with Local-Local and Global-Local losses to learn both instance-level and within-instance temporal distinctions.The framework applies both temporal losses within the same video instance alongside the standard instance contrastive loss.
  • Instance Contrastive Loss: Instance contrastive learning uses transformed clips from the same video as positive matches while distinguishing clips from other video instances.Video clips are generated from random spatio-temporal locations and transformed using spatial and temporal augmentations.
  • Local-Local Temporal Contrastive Loss: Local-Local loss brings randomly transformed versions of a clip together and separates non-overlapping clips from the same video instance.For each anchor clip, the other clips and their transformed versions provide 2 × Nclips − 2 negative pairs.
  • Global-Local Temporal Contrastive Loss: Global-Local loss matches local feature slices before temporal pooling with global representations of their corresponding local clips after pooling.A global clip is divided into four equal-length local clips, and the global clip is temporally downsampled to match each local clip.
  • Global-Local Temporal Contrastive Loss: The global-local objective explicitly encourages feature maps to represent temporal locality across the input clip’s temporal dimension.This complements Local-Local loss, which distinguishes separate clips, by promoting temporal variation within each clip.

4. Experiments

Experiments evaluate TCLR across multiple video datasets, backbones, downstream tasks, and ablations. Results show consistent gains from temporal contrastive losses, especially for retrieval, limited-label learning, and fine-grained class discrimination.

  • Experimental Setup: Experiments span UCF101, Kinetics400, HMDB51, and Diving48 using R3D-18, R(2+1)D-18, and C3D backbones.Evaluation covers action recognition, nearest-neighbor retrieval, limited-label training, and fine-grained action classification.
  • Action Recognition: TCLR consistently outperforms the state of the art across comparable backbone, pretraining-dataset, and fine-tuning-dataset combinations.The comparison uses supervised fine-tuning after self-supervised pretraining and excludes multimodal methods from the primary comparison.
  • Nearest-Neighbor Retrieval: TCLR outperforms prior methods in UCF101 nearest-neighbor retrieval by 12% to 30%, depending on the architecture.Retrieval uses self-supervised pretraining on UCF101 without supervised fine-tuning; test videos query a training-set gallery.
  • Limited-Label Learning: TCLR outperforms MotionFit, MemDPC, and RotNet3D across all tested limited-label settings on UCF101.It achieves competitive performance with MemDPC using only 10% of the labeled data.
  • Fine-Grained Recognition: TCLR pretraining outperforms random initialization and MiniKinetics supervised pretraining on Diving48 fine-grained action classification.Within-instance temporal discrimination helps represent fine-grained differences between parts of diving videos.
  • Ablation Study: The combined instance, local-local, and global-local losses produce the best ablation results across downstream tasks.Each temporal loss adds significant gains over instance contrastive and random-initialization baselines, while temporal losses alone provide a useful initialization prior for supervised fine-tuning.
  • Temporal Diversity: Increasing clips per video improves temporal-loss pretraining by about 7-8% for each individual loss and 14.67% for TCLR.Instance contrastive pretraining, which enforces temporal invariance, does not show a similar improvement.
  • Confusing Class Pairs: TCLR improves average recall for eight visually similar UCF101 classes from 42.5% without pretraining and 44.9% with instance contrastive pretraining to 74.8%.The confused pairs are mostly fine-grained action variants, such as BreastStroke and FrontCrawl.

5. Conclusion

The paper proposes temporal contrastive losses that improve self-supervised video representations beyond standard instance discrimination. Extensive experiments report state-of-the-art performance across diverse downstream video understanding tasks.

  • Conclusion: The work proposes two temporal contrastive losses to improve self-supervised video representations over standard instance discrimination.These losses target temporal variation within video instances.
  • Conclusion: Experiments provide extensive evidence across three diverse datasets and multiple downstream video understanding tasks.The reported tasks include action recognition, nearest-neighbor retrieval, and limited-label action classification.
  • Conclusion: The results underscore benefits of contrastive learning beyond instance discrimination for video representation learning.The conclusion attributes the broader gains to temporal contrastive objectives.

B. Datasets

The study uses three video datasets spanning common action-recognition benchmarks, a large-scale dataset, and a smaller action dataset.

  • UCF101 contains around 13,320 videos across 101 human-activity classes.
  • Kinetics400 contains 222,454 training videos across 400 action classes at the time of download.
  • HMDB51 contains around 7,000 videos covering 51 human actions.

C.1. Network Architecture

The network uses established 3D CNN backbones with modified temporal resolution, spatial pooling, an MLP projection head, and standard video augmentations.

  • The study uses 3D-ResNet-18, R(2+1)D18, and C3D implementations from prior work.
  • For 3D-ResNet and R(2+1)D, conv5 temporal stride is set to 1 and dilation to 2 to increase temporal feature resolution while preserving receptive field.
  • The two-layer MLP maps 512-dimensional features to 128 dimensions with ReLU and L2 normalization, and shares weights across losses.
  • Benchmark training uses appearance and geometry augmentations, including random scaling, cropping, cut-out, and horizontal flipping.

C.3. Self-supervised training

Self-supervised training combines instance discrimination with temporal contrastive objectives using global and local clips, augmented frames, and scheduled optimization.

  • The pretraining model combines a backbone and MLP projection head, using 112 × 112 clips of 16 frames.
  • Global clips sample 16 frames with skip rate 4 from 64 consecutive frames, while four local clips sample 16 frames with skip rate 1.
  • The total loss sums instance contrastive, local-local temporal contrastive, and global-local temporal contrastive terms over the mini-batch.
  • TCLR self-supervised pretraining runs for 400 epochs on UCF101 and 100 epochs for pretraining on the other stated dataset.
  • The augmentation algorithm takes frames as input and outputs transformed frames using random resized crops and contrast adjustments.
  • Training uses Adam with initial learning rate 0.001, plateau-based tenfold decay, and linear warm-up during the first 10 epochs.

C.4. Downstream Task Protocols

The pretrained encoder is evaluated on action recognition, limited-label classification, and nearest-neighbor retrieval using protocols adapted from prior work.

  • Action Recognition: Action recognition attaches a randomly initialized fully connected layer to the pretrained encoder and trains all model layers.
  • Label Efficiency/ Finetuning with limited data: Limited-label evaluation follows the action-recognition setting while training with restricted fractions of the available training data.
  • Nearest Neighbor Video Retrieval: Nearest-neighbor retrieval uses the pretrained encoder without supervised finetuning and averages features from 10 uniformly spaced clips per video.

D. Additional Ablations

Additional ablations examine the effects of temporal-loss components, augmentations, embedding size, timestep count, skip rates, and temperatures across downstream video tasks. The experiments identify settings that improve performance, reduce computation, or materially affect results.

  • Transformations: Removing Gaussian blur, shearing, and rotation reduces training time by as much as 30% while causing only a small downstream performance change.The reported augmentation ablation treats these transformations as computationally expensive.
  • Embedding size: Larger embedding sizes typically yield slightly better performance, while the main experiments use an embedding size of 128.
  • Number of timesteps: Using NT = 2 degrades performance, whereas higher timestep counts substantially increase GPU memory and computation requirements.The default setting is NT = 4; results are reported in Table 10.
  • Skip rates: Randomizing skip rates degrades performance by about 2.3% for instance contrastive loss and 2.6% for global-local temporal contrastive loss.The instance-loss experiment varies both clip skip rates, while the global-local experiment randomizes the global-clip skip rate and scales the local rate.
  • Temperatures: A temporal contrastive temperature of 0.1 is effective, and setting it lower than the instance-contrastive temperature leads to significantly poorer results.

E. Baseline Verification

The supplementary analyses verify the instance-contrastive and random-initialization baselines, then examine training duration, multi-clip evaluation, retrieval, visualization, and temporal feature diversity. Across these analyses, TCLR shows stronger retrieval and more separated representations than standard instance contrastive learning.

  • Baseline Verification: The instance-contrastive baseline achieves 71.3% under the paper’s matched architecture, input size, and augmentation setup.The baseline uses 112 × 112 input resolution and 16 frames.
  • Baseline Verification: The random-initialization UCF101 finetuning baseline reaches 62.3% top-1 classification accuracy, matching a prior reported result of 61.8%.
  • Training duration: TCLR outperforms the Vanilla IC baseline with 4-8× times longer pre-training on downstream UCF101 and HMDB51 action recognition.
  • Multi-clip evaluation: Increasing the number of clips improves TCLR’s linear evaluation on UCF101 and nearest-neighbor retrieval on HMDB51.The corresponding trends are shown in Figures 11 and 12.
  • Attention analysis: TCLR pretrained models focus significantly better on relevant video portions than the compared supervised baseline in qualitative attention visualizations.
  • Temporal diversity: Adding temporal contrastive losses increases temporal distinctiveness in feature maps, with LGL showing the maximum effect in the reported cosine-similarity analysis.The analysis compares cosine similarity between feature-map timesteps across loss configurations.
  • Representation analysis: TCLR produces more coherent and well-separated clusters than standard instance contrastive learning in UCF101 t-SNE visualizations, including visually similar action classes.The visualizations use self-supervised pretrained features without supervised fine-tuning.
Loading 2101.07974v4…