PaperlayerView exact revision on arXiv

Source-linked AI summary

TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning

Yang Liu, Keze Wang, Lingbo Liu, Haoyuan Lan, Liang Lin

arXiv:2112.03587v3cs.CV

TL;DR

Video self-supervised learning must exploit rich spatial-temporal information and generate supervisory signals from unlabeled data, but existing methods do not explicitly model multi-scale temporal dependencies. TCGL uses DCT-based motion enhancement, intra- and inter-snippet temporal contrastive graphs, and adaptive snippet order prediction, achieving state-of-the-art performance on action recognition and video retrieval benchmarks.

  • Problem

    Existing video self-supervised methods do not explicitly model multi-scale temporal dependencies or sufficiently increase temporal diversity in unlabeled videos.

  • Method

    TCGL combines DCT-based STKD motion-enhanced representations, intra- and inter-snippet Temporal Contrastive Graphs, and ASOP for adaptive snippet order prediction.

  • Results

    TCGL achieves state-of-the-art performance across three networks and two downstream tasks: large-scale action recognition and video retrieval.

  • Takeaways & Limitations

    The learned video representations demonstrate potential for downstream action recognition and video retrieval.

Abstract

from arXiv · show

Video self-supervised learning is a challenging task, which requires significant expressive power from the model to leverage rich spatial-temporal knowledge and generate effective supervisory signals from large amounts of unlabeled videos. However, existing methods fail to increase the temporal diversity of unlabeled videos and ignore elaborately modeling multi-scale temporal dependencies in an explicit way. To overcome these limitations, we take advantage of the multi-scale temporal dependencies within videos and proposes a novel video self-supervised learning framework named Temporal Contrastive Graph Learning (TCGL), which jointly models the inter-snippet and intra-snippet temporal dependencies for temporal representation learning with a hybrid graph contrastive learning strategy. Specifically, a Spatial-Temporal Knowledge Discovering (STKD) module is first introduced to extract motion-enhanced spatial-temporal representations from videos based on the frequency domain analysis of discrete cosine transform. To explicitly model multi-scale temporal dependencies of unlabeled videos, our TCGL integrates the prior knowledge about the frame and snippet orders into graph structures, i.e., the intra-/inter- snippet Temporal Contrastive Graphs (TCG). Then, specific contrastive learning modules are designed to maximize the agreement between nodes in different graph views. To generate supervisory signals for unlabeled videos, we introduce an Adaptive Snippet Order Prediction (ASOP) module which leverages the relational knowledge among video snippets to learn the global context representation and recalibrate the channel-wise features adaptively. Experimental results demonstrate the superiority of our TCGL over the state-of-the-art methods on large-scale action recognition and video retrieval benchmarks.The code is publicly available at https://github.com/YangLiu9208/TCGL.

I. INTRODUCTION

TCGL addresses video self-supervised learning challenges by explicitly modeling multi-scale temporal dependencies while discovering motion-enhanced spatial-temporal representations and generating supervisory signals from snippet order prediction.

  • Motivation: Existing video self-supervised methods use temporal information but generally model dependencies at only a single scale.The paper identifies multi-scale temporal modeling as a key challenge for exploiting unlabeled videos.
  • Motivation: Actions contain long-term inter-snippet dependencies and short-term intra-snippet dependencies whose internal relationships contribute to detailed semantic analysis.Handshaking combines walking, shaking hands, and hugging across snippets with periodic hand and foot movements within snippets.
  • Proposed approach: Video frames contain low-frequency scene information and high-frequency motion details, motivating frequency-domain extraction of both spatial and temporal knowledge.The STKD module uses discrete cosine transform analysis to highlight motion and discover discriminative spatial-temporal representations.
  • Proposed approach: TCGL constructs intra-snippet and inter-snippet Temporal Contrastive Graphs to increase temporal diversity among video frames and snippets through graph contrastive learning.The graphs explicitly integrate frame-set and snippet temporal dependencies.
  • Proposed approach: ASOP predicts snippet orders using relational knowledge, learning global context representations and adaptively recalibrating channel-wise snippet features.This module supplies supervisory signals for unlabeled videos.
  • Results: Experiments across three networks and two downstream tasks report state-of-the-art performance for the learned video representations.The evaluated tasks are large-scale action recognition and video retrieval.

II. RELATED WORK

Prior work spans supervised and self-supervised video representation learning, contrastive graph learning, and frequency-domain methods. TCGL combines these directions in a four-stage framework that extracts motion-enhanced features, models temporal graph structure, and predicts snippet order.

  • Supervised video representation learning: Supervised video methods use architectures such as two-stream CNNs, C3D, and Temporal Segment Networks to model spatial and temporal video information.These approaches include separate appearance and optical-flow processing, three-dimensional convolutions, and temporal segment sampling.
  • Self-supervised video representation learning: Existing self-supervised image models are difficult to apply directly to video because video processing requires substantially more computational resources.The paper therefore motivates computationally efficient modules for video self-supervision.
  • Self-supervised video representation learning: Self-supervised video methods generate supervisory signals through temporal pretext tasks including order verification, order prediction, odd-one-out detection, and speediness prediction.Contrastive learning methods and graph-based contrastive methods provide additional self-supervised approaches.
  • Frequency domain learning: Frequency-domain learning has been used for efficiency, compression, channel attention, dataset analysis, and video knowledge distillation.TCGL builds on this direction through DCT-based spatial-temporal representation discovery.
  • TCGL framework: TCGL samples and shuffles snippets, extracts motion-enhanced features with STKD and 3D CNNs, constructs temporal contrastive graphs, and predicts snippet orders.The graph module uses intra-snippet and inter-snippet dependencies, while order prediction outputs probabilities over possible snippet orders.

A. Sample and Shuffle

The method samples videos into shuffled snippets and frame-sets, then uses frequency-domain analysis to separate scene and motion information for discriminative spatial-temporal representations.

  • A. Sample and Shuffle: Videos are sampled into fixed-length snippets, shuffled, and each snippet is divided into equal-length frame-sets.Snippet order forms the global temporal structure, while frame-set order provides local temporal structure.
  • B. Discriminative Spatial-temporal Representation Extraction: The STKD module analyzes temporal frequency spectra to capture both scene appearance and motion information.Low frequencies retain scene structure, whereas high frequencies emphasize motion between neighboring frames.
  • B. Discriminative Spatial-temporal Representation Extraction: The frequency-domain construction sums components after excluding the low-frequency term to reduce scene-information influence during temporal-relation modeling.The formulation is motivated by the distinction between low-frequency scene representation and high-frequency motion representation.
  • B. Discriminative Spatial-temporal Representation Extraction: For a video snippet, the discriminative temporal representation subtracts the temporal average from the original snippet.This representation can be extracted without complex calculation and inserted into existing models as a plug-and-play module.

C. Temporal Contrastive Graph Learning Module

Temporal Contrastive Graph Learning explicitly represents inter-snippet and intra-snippet dependencies using order-informed graphs and contrastive views, combining their losses for temporal representation learning.

  • C. Temporal Contrastive Graph Learning Module: Graph convolutional networks model temporal correlations and capture dependencies through adaptive message propagation over the structured graphs.The graphs make prior temporal relationships explicit while supporting learned node interactions.
  • C. Temporal Contrastive Graph Learning Module: Inter-snippet and intra-snippet temporal graphs encode dependencies using snippet-order and frame-set-order knowledge.Known temporal order connects related nodes and disconnects temporally unrelated nodes; both graph types share structure but use different weights.
  • C. Temporal Contrastive Graph Learning Module: The method generates graph views by randomly removing edges and masking node features, jointly perturbing graph structure and attributes.Edge removal uses a Bernoulli mask, while feature masking zeros selected feature dimensions.
  • C. Temporal Contrastive Graph Learning Module: The contrastive loss compares matching nodes across graph views against both inter-view and intra-view negatives.This differs from NCE-style objectives that consider only inter-view negatives; similarity uses an L2-normalized projected embedding.
  • C. Temporal Contrastive Graph Learning Module: The overall temporal contrastive graph loss combines intra-snippet and inter-snippet graph losses with separate weighting coefficients.The coefficients α and β weight the intra-snippet and inter-snippet components, respectively.

D. Adaptive Order Prediction Module

ASOP predicts the original order of shuffled video snippets by aggregating their relational information into a global context and adaptively recalibrating snippet features.

  • Adaptive Snippet Order Prediction: ASOP concatenates features from different snippets to learn a global context embedding, which recalibrates each snippet's input features.The recalibration uses channel-wise excitation and gating while preserving correlations among snippets.
  • Adaptive Snippet Order Prediction: The order-prediction objective is formulated as classification, with cross-entropy measuring the correctness of predicted order probabilities.The overall TCGL loss combines graph contrastive and order-prediction losses, weighted by λg and λo.
  • Adaptive Snippet Order Prediction: The module's adaptive recalibration uses excitation signals and channel-wise products to let one snippet's features recalibrate another's.The design preserves correlations among different video snippets during feature refinement.
  • Adaptive Snippet Order Prediction: The refined snippet features are concatenated and classified with a two-layer perceptron and softmax to predict shuffled snippet orders.For three snippets, the task contains 6 possible order classes.

IV. EXPERIMENTS

The experiments evaluate TCGL through ablations, downstream action recognition and video retrieval, and visualization analyses.

  • Experiments: The experimental study covers settings, ablations of key components, downstream action recognition and video retrieval comparisons, and visualization results.The learned 3D CNNs are evaluated against state-of-the-art methods on the downstream tasks.

A. Experimental Setting

TCGL is evaluated across action-recognition and video-retrieval datasets using several 3D CNN backbones and a linear-probe protocol.

  • Datasets: The evaluation uses UCF101, HMDB51, Kinetics-400, Something-Something-V2, and ActivityNet for action recognition and video retrieval.Kinetics-400 provides approximately 240k training videos for pre-training.
  • Network Architecture: C3D, R3D-18, and R(2+1)D-18 serve as video-encoder backbones with 3D convolutional architectures.R(2+1)D-18 decomposes the 3D kernels used by the corresponding R3D-18 architecture.
  • Parameters: Training samples use 16-frame snippets, an interval of 8, three snippets per tuple, and four frame-sets within each snippet.These parameters define the temporal sampling configuration used in the experiments.
  • Evaluation Protocol: The linear-probe protocol freezes pretrained backbone layers while randomly initializing and updating only the downstream linear classifier.Backbones pretrained on UCF101 or Kinetics-400 are transferred to downstream datasets.

B. Ablation Study

Ablations show that TCGL benefits from both temporal graphs, STKD, ASOP, balanced loss weighting, and complementary topology- and feature-level graph views.

  • Hyperparameters: Snippet-order accuracy decreases as the number of snippets increases, motivating a three-snippet compromise between task complexity and prediction accuracy.More frame-sets also make intra-snippet temporal modeling harder, while additional GCN layers complicate convergence without improving performance comparably.
  • The intra-snippet and inter-snippet graphs: 83.0% snippet order prediction accuracy is achieved when α = β = 1, while weighting the intra- and inter-snippet graphs as 1 and 0.1 yields 80.2%.Exchanging those weights reduces accuracy to 54.9%, and removing either graph substantially degrades performance.
  • The adaptive snippet order prediction: Removing ASOP lowers both snippet order prediction and action recognition performance compared with the complete TCGL model.The ablation supports ASOP's use of relational information among snippets.
  • The spatial-temporal knowledge discovering: Removing STKD lowers action-recognition performance across all tested backbones and datasets.The comparison uses raw video frames instead of STKD-enhanced inputs.
  • The values of λg and λo: Performance drops without either TCG loss or ASOP loss and is best when λg = λo = 1.The result supports equal weighting of graph contrastive learning and snippet order prediction.
  • The methods of generating views: The fourth graph-view generation variant achieves the best performance, supporting collaborative use of feature-level and topology-level graph contrastive learning.Random node removal preserves the original temporal graph structure, whereas edge removal can alter prior temporal dependencies.

1) Performance on UCF101 and HMDB51:

TCGL performs favorably against existing self-supervised methods on UCF101 and HMDB51 across multiple backbones, with stable cross-backbone performance and competitive efficiency.

  • TCGL performs favorably against existing approaches on both UCF101 and HMDB51 with C3D, R3D-18, and R(2+1)D-18 backbones.
  • 78.0% MAP across three backbones on UCF101 compares with 76.9% for RTT, indicating more stable performance across backbones.
  • TCGL achieves significant improvement over random initialization on both UCF101 and HMDB51, demonstrating its utility for self-supervised video representation learning.
  • TCGL remains competitive with lightweight backbones, 16×112×112 clips, and batchsize 16, whereas stronger methods use larger backbones, clips, datasets, or batchsizes.

2) Performance on Something-Something-V2:

Across downstream recognition and linear evaluation settings, TCGL learns useful spatial-temporal representations without manual annotations, while larger pretraining resources can yield stronger results.

  • 48.5% and 44.6% accuracy with C3D and R3D-18, respectively, surpass the corresponding supervised pretrained results of 47.0% and 43.7% on Something-Something-V2.
  • TCGL outperforms random initialization and RSPNet under the same C3D and R3D-18 settings on Something-Something-V2.
  • TCGL improves over MoCo-ImageNet and ImageNet Pre-training on Kinetics400, although CORPf and SeCo benefit from larger batchsizes, longer training, or additional data.
  • 61.8% accuracy on ActivityNet is achieved when pretraining from scratch with small batchsize and no data augmentations, using fewer computational resources than stronger comparison settings.

D. Video Retrieval

TCGL produces strong retrieval representations without fine-tuning and highlights dominant motion regions, though difficult scenes can mislead its activation maps.

  • Video retrieval uses frozen backbone features and cosine nearest-neighbor search, so performance directly tests the representative capacity of self-supervised models.
  • TCGL outperforms state-of-the-art methods by substantial margins on nearly all UCF101 and HMDB51 retrieval metrics across backbones, including SpeedNet’s stronger S3D backbone.
  • Retrieved neighbors usually show strong semantic correlation with query videos, including complex interactive actions under significant camera motion.
  • Activation maps generally align with dominant movement, indicating that TCGL captures discriminative temporal regions in videos.
  • TCGL can misidentify motion in challenging scenes, activating mirror movement in salsaspin and water motion instead of swimmers in breaststroke.
Loading 2112.03587v3…