PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework

Li Tao, Xueting Wang, Toshihiko Yamasaki

arXiv:2008.02531v2cs.CV

TL;DR

Video representation learning can use unlabeled videos, while training action-classification networks usually requires properly trimmed clips that are costly to annotate. The paper proposes IIC, which adds same-video negatives created by breaking temporal relations to contrastive learning. It reports improvements over existing state-of-the-art methods on video retrieval and recognition, while leaving multiple models for different views to future work.

  • Problem

    Training action-classification networks usually requires properly trimmed clips, and annotating video datasets requires substantial resources despite the availability of unlabeled videos.

    Source: §1 · p. 1

  • Method

    IIC trains spatio-temporal convolutional networks with contrastive learning that adds intra-negative samples created by breaking temporal relations in anchor-video clips.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    IIC outperforms existing state-of-the-art methods on video retrieval and video recognition, with significant improvements reported using the same network architecture.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support using combined inter- and intra-sample learning as a general framework for self-supervised video representation.

    Source: §1 · p. 2

    Using multiple models for different views remains future work.

    Source: §5.1 · p. 6

Abstract

from arXiv · show

We propose a self-supervised method to learn feature representations from videos. A standard approach in traditional self-supervised methods uses positive-negative data pairs to train with contrastive learning strategy. In such a case, different modalities of the same video are treated as positives and video clips from a different video are treated as negatives. Because the spatio-temporal information is important for video representation, we extend the negative samples by introducing intra-negative samples, which are transformed from the same anchor video by breaking temporal relations in video clips. With the proposed Inter-Intra Contrastive (IIC) framework, we can train spatio-temporal convolutional networks to learn video representations. There are many flexible options in our IIC framework and we conduct experiments by using several different configurations. Evaluations are conducted on video retrieval and video recognition tasks using the learned video representation. Our proposed IIC outperforms current state-of-the-art results by a large margin, such as 16.7% and 9.5% points improvements in top-1 accuracy on UCF101 and HMDB51 datasets for video retrieval, respectively. For video recognition, improvements can also be obtained on these two benchmark datasets. Code is available at https://github.com/BestJuly/Inter-intra-video-contrastive-learning.

1 INTRODUCTION

Video representation learning needs temporal information, but labeling properly trimmed action clips requires substantial resources. IIC combines inter-sample contrastive learning with intra-negative samples formed by breaking temporal relations, and the paper reports improvements over prior state-of-the-art methods on video retrieval and recognition.

1 INTRODUCTION

  • Properly trimmed action clips are usually needed to train action-classification networks, making video annotation especially resource-intensive.The paper points to abundant unlabeled videos as a resource that self-supervised learning can use without labels.Source: §1 · p. 1
  • Prior video self-supervision often changes temporal information within a sample, while inter-sample learning treats same-video views as positives and different-video clips as negatives.The paper calls within-sample transformations intra-sample learning and gives frame shuffling as an example.Source: §1 · p. 2
  • IIC adds intra-negative samples by breaking temporal relations in an anchor video, so models learn spatial and temporal differences between samples.The framework adapts Contrastive Multiview Coding (CMC) [36] and introduces intra-negative sample learning.Source: §1 · p. 2
  • The authors report that IIC outperforms existing state-of-the-art methods on both video retrieval and video recognition.They describe the margins as notably large and report significant improvements with the same network architecture.Source: §1 · p. 2
  • The framework combines inter- and intra-sample learning for video representation and offers multiple configuration options.The paper presents this as a general self-supervised framework for video representation.Source: §1 · p. 2

2 RELATED WORKS

Related work covers within-sample pretext tasks, contrastive learning across samples, and video representation methods. The paper positions its approach as extending multiview contrastive learning with negatives created by breaking temporal relations in video clips.

2 RELATED WORKS

  • This section organizes prior work into intra-sample learning, inter-sample learning, and video representation techniques.The authors add video understanding techniques because the paper focuses on video representation.Source: §2 · p. 2

2.1 Intra-sample learning

  • Intra-sample methods train on transformations within a sample, using carefully designed tasks to help models learn features.Video methods target temporal information through order prediction, odd-clip identification, motion and appearance statistics, space-time puzzles, or transformation recognition.Source: §2.1 · p. 2

2.2 Inter-sample learning

  • Inter-sample methods bring features from the same sample together and separate different samples, with contrastive multiview coding using different views of one sample.The paper notes that these methods do not require specially designed tasks and describes IIC as extending CMC [36] with negatives formed by breaking temporal relationships.Source: §2.2 · pp. 2–3

2.3 Video representation

  • Some video self-supervised methods use image frames as inputs, while supervised video representations include 2D, two-stream, and spatio-temporal convolutional models.The paper also notes that 3D convolutional models have been used in self-supervised video learning.Source: §2.3 · p. 3

3 METHODS

IIC combines two video views with intra-negative clips that disrupt temporal relationships, using contrastive learning to learn video representations. It also uses one network to process different views and supports joint retrieval from their features.

3 METHODS

  • The method targets discriminative video representations that capture temporal information as well as appearance.Its inputs include two views of the same video, represented as clips of stacked continuous frames for a 3D convolutional network.Source: §3 · p. 3, §3.1 · p. 3

3.1 Multi-view and intra-negative inputs

  • Intra-negative clips are created by repeating a randomly selected frame or shuffling frames, disrupting temporal relationships while retaining much of the source clip’s content.In this work, they are generated from the original RGB view, with one generation method used per experiment.Source: §3.1 · p. 3

3.2 Contrastive learning

  • The contrastive framework uses different views of the same video as positives and extends negatives with intra-negative clips.Its framework uses three memory banks and non-parametric weights to optimize the network with contrastive loss.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Joint representation

  • A single network processes different views, whose extracted features can be concatenated for video retrieval.The input options include RGB clips, optical flow, and stacked frame differences, with RGB as the anchor view.Source: §3.3 · p. 4

4 EXPERIMENTS

The experiments evaluate IIC through video retrieval and video recognition, using UCF101 and HMDB51. They vary input views, backbone and intra-negative generation, and report training and evaluation procedures.

4 EXPERIMENTS

  • The experiments assess IIC configurations on retrieval and evaluate transferability by fine-tuning trained models for recognition.The trained models also serve as weight initialization for fine-tuning.Source: §4 · pp. 4–5

4.1 Datasets

  • UCF101 contains 13,320 videos across 101 action categories, while HMDB51 contains 7,000 videos across 51 classes.Unless otherwise stated, experiments use split 1 of each dataset.Source: §4.1 · p. 5

4.2 Evaluation tasks

  • The evaluation uses UCF101 and HMDB51 for video retrieval and recognition, with the model trained on UCF101 split 1 for retrieval on both datasets.For recognition, trained models are fine-tuned on both datasets; HMDB51 retrieval tests generalization from UCF101 training.Source: §4.2 · p. 5, §4.1 · p. 5

4.3 Options in our IIC framework

  • The study evaluates RGB clips paired with residual frames or optical flow, and tests frame repeating and temporal shuffling for intra-negative generation.R3D is the main backbone used in the experiments.Source: §4.3 · p. 5

4.4 Implementation details

  • Unsupervised training runs for 240 epochs with batch size 16, sampling 2k negatives from memory banks where k is 1,024.Retrieval uses K-nearest-neighbor search; transfer evaluation fine-tunes the trained model.Source: §4.4 · p. 5

5 RESULTS AND DISCUSSION

IIC adds temporally disrupted intra-negative samples to contrastive video learning, and the experiments evaluate its representations through retrieval and recognition. Across both tasks, the reported comparisons show gains over prior self-supervised methods.

5.1 Ablation study

  • Nearly 2 percentage points of top-1 retrieval accuracy were gained by concatenating two views processed by one model, and this joint retrieval strategy became the default.Across second-view modalities, retrieval accuracies were better with intra-negative samples; frame repeating worked best for residual frames, while optical flow outperformed residual frames.Source: §5.1 · pp. 5–6, Table 1 · p. 6
  • The optical-flow results varied by component and metric: temporal shuffling with v gave the best top-1 and top-5 accuracies, while frame shuffling with u was better for top-10, top-20, and top-50.The authors note that their setting used one model for both views and suggest using both optical-flow components with separate models as future work.Source: §5.1 · p. 6

5.2 Visualization: feature embedding

  • t-SNE visualizations of UCF101 features showed better clustering with intra-negative samples than the baseline without them.The visualization projected features from videos in the first ten action categories into two dimensions.Source: §5.2 · p. 6, Figure 5 · p. 6

5.3 Comparison: video retrieval

  • 42.4% top-1 retrieval accuracy was achieved on UCF101 with optical flow as the second view; HMDB51 retrieval reached 19.7%, over 12 percentage points above VCP.With residual frames as the second view, UCF101 top-1 retrieval accuracy was 36.5%, reported as 10.8 percentage points above the current state of the art.Source: Table 3 · p. 7, §5.3 · p. 7, Table 4 · p. 7

5.4 Comparison: video recognition

  • IIC reached 74.4% top-1 accuracy on UCF101 without optical flow and 38.3% on HMDB51 with residual frames, the best reported result there among methods without optical flow.Fine-tuning with the second-view modality outperformed VCP by up to 8.4 percentage points on UCF101; on HMDB51, optical-flow u data outperformed VCP by 5.3 points.Source: Table 6 · p. 8, §5.4 · p. 8

6 CONCLUSIONS

The paper presents IIC as a self-supervised video representation method designed to learn rich temporal features. It evaluates the learned representations on video retrieval and recognition, reporting stronger performance than other methods on retrieval.

6 CONCLUSIONS

  • IIC trains a spatio-temporal convolutional network with intra-negative samples generated by breaking temporal relations in video clips.The framework also experiments with different view-selection options.Source: §6 · p. 8
  • The learned representations are applied to video retrieval and video recognition tasks.For retrieval, the models use a joint strategy with one model handling different inputs.Source: §6 · p. 8
  • The authors report that IIC outperforms other methods by a large margin on video retrieval and helps models learn better video representations for recognition.Source: §6 · p. 8

A ALGORITHM

Algorithm 1 trains with inter-intra contrastive learning by generating intra-negative samples, forming weights from memory banks, computing a loss, and updating the network and banks.

A ALGORITHM

  • The training algorithm generates intra-negative samples from the input video views and fetches weights from the memory banks.Source: A ALGORITHM · p. 9
  • The fetched weights are concatenated with intra-negative weights to form the non-parametric weights for both views.Source: A ALGORITHM · p. 9
  • After the loss updates the network, the algorithm updates the memory banks with view features.Source: A ALGORITHM · p. 9
  • The stated main differences from baseline CMC [36] are in algorithm lines 3 and 6.Source: A ALGORITHM · p. 9

Related papers

Loading 2008.02531v2…