PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Co-training for Video Representation Learning

Tengda Han, Weidi Xie, Andrew Zisserman

arXiv:2010.09709v2cs.CV

TL;DR

The paper asks whether instance discrimination makes the best use of data for visual-only self-supervised video representation learning. It adds semantic positives as an oracle and proposes CoCLR to mine positives across RGB and optical flow, achieving strong downstream performance with substantially less pre-training data.

  • Problem

    The paper asks whether instance discrimination makes the best use of data in self-supervised video representation learning, focusing on neglected hard positives.

  • Method

    CoCLR improves InfoNCE’s sampling regime by using complementary RGB and optical-flow views to mine positive samples beyond individual instances.

  • Results

    CoCLR achieves state-of-the-art or comparable performance on action recognition and video retrieval while requiring less data for self-supervised pre-training.

  • Takeaways & Limitations

    Complementary views can bridge visually different instances of the same class and substantially improve video representations over instance-based InfoNCE training.

Abstract

from arXiv · show

The objective of this paper is visual-only self-supervised video representation learning. We make the following contributions: (i) we investigate the benefit of adding semantic-class positives to instance-based Info Noise Contrastive Estimation (InfoNCE) training, showing that this form of supervised contrastive learning leads to a clear improvement in performance; (ii) we propose a novel self-supervised co-training scheme to improve the popular infoNCE loss, exploiting the complementary information from different views, RGB streams and optical flow, of the same data source by using one view to obtain positive class samples for the other; (iii) we thoroughly evaluate the quality of the learnt representation on two different downstream tasks: action recognition and video retrieval. In both cases, the proposed approach demonstrates state-of-the-art or comparable performance with other self-supervised approaches, whilst being significantly more efficient to train, i.e. requiring far less training data to achieve similar performance.

1 Introduction

The paper argues that instance discrimination underuses semantic hard positives in self-supervised video learning. It introduces CoCLR, which mines positives across complementary RGB and optical-flow views to improve representation learning efficiently.

  • Motivation: Instance-based self-supervision may underuse hard positives from the same semantic class.The paper frames this as a limitation of instance discrimination and reports that including such positives improves representation quality.
  • Semantic positives: UberNCE adds same-class positives to InfoNCE and provides an oracle form of supervised contrastive learning.Its positive and negative sets are constructed using semantic class labels.
  • CoCLR: CoCLR mines positive samples from complementary RGB and optical-flow views instead of using semantic labels.Flow can bridge RGB clips of the same action, while RGB can link optical-flow clips.
  • CoCLR: CoCLR improves both RGB and Flow representations by constructing positive pairs beyond individual instances.The method also removes these positive samples from instance-level negatives, where they would otherwise be false negatives.
  • Evaluation and scope: The approach is evaluated on action recognition and video retrieval, with state-of-the-art or comparable performance and lower pre-training data requirements.The paper presents visual-only learning from RGB frames, optical flow, or both, and identifies broader applicability to other complementary views.

2 Related work

Related work spans visual-only self-supervised video learning, multimodal correspondence learning, paired-network co-training, and action recognition. These lines of research provide the context for CoCLR’s use of complementary views and flow-guided representation learning.

  • Visual-only Self-supervised Learning: Visual-only video learning uses spatio-temporal signals such as frame ordering, motion, co-occurrence, jigsaw tasks, rotation, speed, future prediction, and temporal coherence.
  • Multi-modal Self-supervised Learning: Multimodal self-supervised learning leverages correspondence between video and audio or video and textual narrations.Some approaches use contrastive loss, while another iteratively clusters and relabels video-audio data.
  • Co-training Paired Networks: Classical co-training assumes multiple complementary views, trains a classifier for each, and iteratively adds confident predictions from unlabeled data.The paper distinguishes its setting because it uses no labeled samples.
  • Co-training Paired Networks: Paired-network interaction also appears in generative adversarial networks and actor-critic reinforcement learning.
  • Video Action Recognition: Action-recognition research includes two-stream and single-stream RGB networks, with flow-stream distillation used to teach RGB-stream networks.This distillation goal is related at a high level to the paper’s use of flow alongside RGB.

3 InfoNCE, UberNCE and CoCLR

This section contrasts instance-based InfoNCE with semantic-positive UberNCE and introduces CoCLR, which mines positives across RGB and optical-flow views through alternating co-training.

  • InfoNCE: InfoNCE learns representations by comparing augmented views of each instance against augmented views from other instances.For each clip, augmentations form positives, while augmentations of different clips form negatives.
  • UberNCE: UberNCE adds same-class clips to the positive set and excludes other classes from it, extending instance-based contrastive learning with semantic labels.For a running clip, other running clips are positives and clips from other action classes are negatives.
  • UberNCE: A significant linear-probe performance gap between InfoNCE and UberNCE shows that semantic positives and the removal of false negatives improve representation learning.The sampling difference between the two methods is treating semantically related clips as positive pairs.
  • CoCLR: CoCLR improves sampling rather than introducing a new loss or pretext task, using complementary views to add same-class positives and remove false negatives from instance negatives.The paper applies this approach to RGB frames and unsupervised optical flow, while noting that other complementary views could play a similar role.
  • CoCLR: CoCLR mines hard positive pairs in one view to train the other, using optical-flow similarity for RGB representations and RGB similarity for optical-flow representations.The RGB positive set includes the clip’s augmentations and its top K nearest neighbours in optical-flow feature space; the reverse construction updates flow.
  • CoCLR: CoCLR alternates mining positives from the other network with independently minimizing each view’s loss after initializing both networks with InfoNCE.As representations strengthen, the method retrieves different and harder positives; ablations report K = 5 as optimal and more cycle alternations as beneficial.

4 Experiments

Experiments evaluate CoCLR through action classification and video retrieval on UCF101, Kinetics-400, and HMDB51. CoCLR improves representations over instance-based baselines and remains competitive with methods using much more data.

  • Datasets and protocols: Experiments use UCF101, Kinetics-400, and HMDB51 for self-supervised training and downstream evaluation.UCF101 and Kinetics-400 provide training videos; UCF101, Kinetics-400, and HMDB51 support downstream evaluation.
  • Datasets and protocols: The study evaluates frozen encoders with linear probes, end-to-end finetuning, and nearest-neighbour video retrieval.Retrieval directly uses extracted features, while classification uses either a frozen encoder or finetuned encoder.
  • UCF101 comparisons: 78.0 vs 46.8 linear-probe accuracy and 71.6 vs 33.1 retrieval R@1 show UberNCE outperforming InfoNCE on UCF101.CoCLR reaches 70.2 on the reported linear-probe comparison, approaching UberNCE’s 78.0.
  • UCF101 comparisons: Increasing co-training cycles improves R@1 for both RGB and Flow representations, indicating that same-class clips become closer in embedding space.Figure 2 tracks alternating optimization while one representation is fixed at each stage.
  • UCF101 comparisons: K = 5 mined samples is optimal in the reported ablation, while simultaneous RGB–Flow optimization performs slightly worse than alternating optimization.The authors attribute the simultaneous-optimization result to network weights being updated too quickly.
  • State-of-the-art comparisons: With finetuning, CoCLR reaches 81.4 on UCF101 and 52.1 on HMDB after UCF101 training, while two-stream CoCLR† reaches 90.6 and 62.9 after K400 training.CoCLR also compares favorably with methods trained on substantially larger visual or multimodal datasets.
  • State-of-the-art comparisons: CoCLR models further exceed InfoNCE models by a large margin on nearest-neighbour video retrieval across both benchmark datasets.The retrieval goal is to place clips from the same semantic category among the nearest neighbours.

5 Conclusion

The paper shows that complementary video views can mine positive samples across instances, substantially improving video representations over InfoNCE. It also suggests extending explicit positive mining to audio, text, other data, modalities, tasks, and learning methods.

  • Complementary video views bridge RGB clips from the same class and generate positive training sets that improve performance over InfoNCE.The paper demonstrates this role for optical flow and RGB views.
  • The authors conjecture that audio could similarly connect visually different clips, such as guitar videos linked by their sound.This proposed extension is not demonstrated in the paper.
  • Text can provide a bridge between visual instances in visual-text learning when narrations describe video content.
  • The authors expect explicit positive mining to extend to images, other modalities, tasks, and methods such as BYOL.

6 Broader Impact

The paper frames pretrained video representations as a way for future systems to benefit from prior training and potentially avoid the cost of retraining deep networks.

  • Pretrained representations may let future research reuse prior video representation learning instead of retraining systems from scratch.
  • Avoiding retraining could reduce the financial and environmental costs associated with computationally expensive end-to-end deep-network training.
  • The broader-impact discussion connects these potential savings to the data-hungry nature and computational demands of deep learning systems.

A.1 Encoder Architecture

At pretraining, the S3D feature encoder is followed by a two-layer non-linear projection head whose fully connected layers output 1024 and 128 dimensions.

  • S3D is followed by a non-linear projection head during pretraining.
  • The projection head is removed when evaluating downstream tasks.
  • The projection head contains two fully connected layers, designated FC-1024 and FC-128.These labels denote the output dimension of each fully connected layer.

A.2 Classifier Architecture

For action-classification evaluation, the pretrained representation is evaluated with a classifier whose fully connected output dimension equals the number of action classes.

  • During downstream action classification, the non-linear projection head is replaced by a single linear classification layer.
  • Table 5 describes the classifier architecture used to evaluate representations on action-classification tasks.
  • The classifier’s FC-num_class layer outputs one dimension for each action class.

A.3 Momentum-updated History Queue

CoCLR uses momentum-updated RGB and Flow history queues to cache features, identify Flow-space nearest neighbours, and form multiple positives for the InfoNCE objective.

  • A.3 Momentum-updated History Queue: The implementation maintains separate momentum-updated history queues for RGB and Flow features, with dataset-specific queue sizes.Both queues are updated by enqueuing the current minibatch and removing the earliest minibatch; queue sizes are 2048 for UCF101 and 16384 for K400.
  • A.3 Momentum-updated History Queue: CoCLR initializes query and key RGB encoders identically, while the Flow encoder is frozen and not updated by gradients.The key encoder is updated by momentum from the query encoder during training.
  • A.3 Momentum-updated History Queue: Each minibatch creates two randomly augmented RGB views, producing query and detached key features for RGB similarity computation.The RGB logits combine the current query-key similarity with similarities against the RGB history queue, then apply temperature scaling.
  • A.3 Momentum-updated History Queue: Flow features rank the Flow history queue, and the topK neighbours are converted into additional positive entries alongside the current paired sample.The resulting positive mask has one current positive plus the selected nearest neighbours for each sample.
  • A.3 Momentum-updated History Queue: The multi-instance InfoNCE loss sums softmax probabilities over all masked positives before averaging across the minibatch.This implements CoCLR's use of multiple Flow-derived positives in the contrastive objective.
Loading 2010.09709v2…