PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Supervised Action Recognition with Temporal Contrastive Learning

Ankit Singh, Omprakash Chakraborty, Ashutosh Varshney, Rameswar Panda, Rogerio Feris, Kate Saenko, Abir Das

arXiv:2102.02751v2cs.CV

TL;DR

Semi-supervised action recognition must learn from few labeled videos, and video speed provides temporal information beyond image-based views. TCL contrasts representations from unlabeled videos played at different speeds, also contrasting groups of videos with similar actions. It outperforms competing methods across four benchmarks, with fully supervised performance reached using 33% labeled data on Mini-Something-V2.

  • Problem

    Learning action recognition from only a handful of labeled videos is challenging because activity labels require tedious collection.

    Source: §1 · p. 1

  • Method

    TCL contrasts representations of the same unlabeled video at two speeds and adds group-level contrastive learning over videos with similar pseudo-labels.

    Source: §1 · pp. 1–2

  • Results

    TCL significantly outperforms competing methods on four standard benchmarks; on Mini-Something-V2, it reaches fully supervised performance with 33% labeled data.

    Source: §1 · p. 1, §5 · p. 8

  • Takeaways & Limitations

    TCL uses video time and action-similar neighborhoods as supervisory signals for semi-supervised action recognition.

    Source: §1 · p. 2, §5 · p. 8

    The setup assumes many unlabeled videos alongside a small labeled set.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Learning to recognize actions from only a handful of labeled videos is a challenging problem due to the scarcity of tediously collected activity labels. We approach this problem by learning a two-pathway temporal contrastive model using unlabeled videos at two different speeds leveraging the fact that changing video speed does not change an action. Specifically, we propose to maximize the similarity between encoded representations of the same video at two different speeds as well as minimize the similarity between different videos played at different speeds. This way we use the rich supervisory information in terms of `time' that is present in otherwise unsupervised pool of videos. With this simple yet effective strategy of manipulating video playback rates, we considerably outperform video extensions of sophisticated state-of-the-art semi-supervised image recognition methods across multiple diverse benchmark datasets and network architectures. Interestingly, our proposed approach benefits from out-of-domain unlabeled videos showing generalization and robustness. We also perform rigorous ablations and analysis to validate our approach. Project page: https://cvir.github.io/TCL/.

1. Introduction

Semi-supervised video action recognition remains underexplored, while video speed offers temporal information that image-based approaches do not use. TCL uses unlabeled videos at two playback speeds with instance- and group-level contrastive learning, and outperforms extended image-domain baselines across four datasets.

1. Introduction

  • TCL learns consistent representations of the same unlabeled video at different speeds, using temporal information for semi-supervised action recognition.The approach is motivated by the limited attention to semi-supervised video recognition and the success of differently paced videos in supervised action recognition.Source: §1 · p. 1
  • Instance-contrastive learning separates different videos but can miss shared action-class information, so TCL adds group-contrastive learning over pseudo-label groups.Groups are formed from unlabeled videos with the same pseudo-label, and their average representations are contrasted across pathways.Source: §1 · p. 2
  • 33% and 15% labeled data let TCL reach fully supervised performance on Mini-Something-V2 and Jester, respectively, while Pseudo-Label and FixMatch do not reach it at those proportions.The comparisons use the same ResNet-18 backbone.Source: §1 · pp. 1–2
  • The proposed group-contrastive loss couples discriminative motion representations with pace-invariance and is reported to improve semi-supervised action-recognition performance.Source: §1 · p. 2
  • TCL outperforms extended image-domain semi-supervised baselines across four datasets and benefits from unlabeled videos from a different domain.The introduction also reports ablation analysis of the framework's components.Source: §1 · p. 2

2. Related Work

Prior action-recognition systems rely on large labeled datasets, while semi-supervised learning has been studied much more extensively for images than for video. TCL uses video speed and group-wise contrastive learning to address temporal information that image-space augmentations do not capture.

Action Recognition

  • Deep action-recognition systems include two-stream CNNs, 3D CNNs, and temporal aggregation models, but depend on large datasets with costly human annotation.Source: §2 · p. 2

Semi-Supervised Learning

  • Semi-supervised image classification has advanced through generative models, pseudo-labeling, consistency regularization, and self-supervised signals, while video action recognition remains rarely addressed.Prior video work includes an Action Bank representation with an extreme learning machine and image-based FixMatch-style consistency.Source: §2 · p. 2
  • TCL's shared-weight base and auxiliary pathways process faster and slower video frames, respectively, and the base pathway alone recognizes actions at test time.The framework uses instance- and pseudo-label group-level contrastive objectives during training.Source: Figure 2 · p. 3
  • Unlike image-space augmentations, TCL models temporal variation by processing unlabeled videos at different speeds and adds group-wise to instance-wise contrastive learning.Source: §2 · p. 3

3. Methodology

TCL uses limited labeled videos to initialize a model, then learns from unlabeled videos sampled at different speeds. It combines instance- and group-level contrastive objectives, with optional self-supervised pretraining and pseudo-label finetuning.

3.1. Problem Setup

  • The labeled set is small relative to the unlabeled pool, whose videos are represented at fast and slow frame rates.The fast and slow versions are sampled from the same unlabeled video.Source: §3.1 · p. 3

3.2. Temporal Contrastive Learning

  • TCL processes fast and slow versions in base and auxiliary pathways that share a neural backbone.The framework uses the two speeds as corresponding views of the video.Source: §3.2 · p. 4

3.2.1 Supervised Training Stage

  • The backbone is first trained on labeled videos by minimizing supervised cross-entropy, using average frame logits or 3D-backbone logits as video representations.The representation depends on whether the backbone uses 2D or 3D convolutions.Source: §3.2.1 · p. 4

3.2.2 Instance-Contrastive Loss

  • Instance contrastive learning matches each video's fast and slow representations while pushing representations of different videos apart.The loss considers negative pairs drawn from either pathway and across both pathways.Source: §3.2.2 · p. 4

3.2.3 Group-Contrastive Loss

  • Group contrastive learning groups videos by predicted pseudo-label, averages their representations, and aligns same-label groups across pathways.Unlike instance-level contrast, grouping is intended to avoid pushing videos of the same action apart.Source: §3.2.3 · p. 4, Figure 3 · p. 4

3.3. TCL with Pretraining and Finetuning

  • Self-supervised pretraining uses instance contrast on labeled and unlabeled videos to initialize both pathways; pseudo-label finetuning can follow TCL.The paper reports that pretraining followed by TCL and finetuning benefits more than the same pretraining and finetuning after limited supervised training.Source: §3.3 · p. 5

4. Experiments

Experiments compare TCL with supervised and semi-supervised baselines across four datasets, including a domain-shift setting, and ablate its losses and hyperparameters. TCL improves on the compared methods across benchmarks, while the ablations examine the contributions and compute trade-offs of its design choices.

4.1. Experimental Setup

  • The experiments use Mini-Something-V2, Jester, Kinetics-400, and Charades-Ego, with first-person Charades-Ego videos used as unlabeled data in the domain-shift evaluation.Baselines include a supervised model and video extensions of image-domain semi-supervised methods, compared using the same backbone and settings.Source: §4.1 · p. 5

4.2. Large-scale Experiments and Comparisons

  • On Mini-Something-V2, TCL beats FixMatch by 1.75% with 1% labeled data and 8.14% with 5%, with the larger gains in the low-data setting.The comparison covers ResNet-18 and ResNet-50; the heavier backbone shows signs of overfitting at 1% and 5% labeling.Source: §4.2 · p. 6, Table 1 · p. 6
  • On Kinetics-400, TCL beats FixMatch by 1.31% with 1% labeled data and 4.63% with 5%.On Jester, TCL with pretraining and finetuning reaches 94.93% top-1 accuracy with 10% labels, 0.32% below the fully supervised baseline’s 95.25%.Source: Table 2 · p. 6, §4.2 · p. 7
  • On Charades-Ego, TCL outperforms the compared methods by around 1% mAP across all three proportions of target-domain unlabeled data.With all unlabeled data from the shifted domain, TCL also exceeds FixMatch using all target-domain unlabeled data.Source: §4.2 · p. 7, Table 3 · p. 7
  • On Jester and Kinetics-400 examples, TCL correctly predicts gestures and actions that the compared methods miss, including “climbing tree” and “zumba” on Kinetics-400.The qualitative examples use 1% labeled data for Jester and 5% for Kinetics-400.Source: Figure 5 · p. 7, §4.2 · p. 7

4.3. Ablation Studies

  • Removing group-contrastive loss lowers Mini-Something-V2 top-1 accuracy from 29.81% to 27.24%.The authors interpret this ablation as evidence that the loss helps capture high-level semantics.Source: §4.3 · p. 8, Table 4 · p. 8
  • On Mini-Something-V2, the contrastive loss gains around 6.21% top-1 accuracy over FixMatch’s pseudo-label consistency loss; TCL without group loss also beats the temporal-augmentation FixMatch variant by 2.66%.Replacing TCL’s temporal contrastive learning with O3N gives 24.58% accuracy versus TCL’s 29.81%.Source: §4.3 · p. 8, Table 4 · p. 8
  • Doubling both pathway frame rates raises top-1 accuracy from 29.81% to 31.31%, but the study retains 8- and 4-frame inputs because the compute and memory increase is heavy for little relative gain.Results are similar for unlabeled-to-labeled ratios µ of 3, 5, and 7, while setting instance-contrastive weight γ to 3 instead of 9 drops accuracy by more than 6%.Source: §4.3 · p. 8, Figure 6 · p. 8

5. Conclusion

The conclusion presents TCL as a temporal contrastive framework for semi-supervised action recognition. It uses video speed changes and action-similar video neighborhoods, and outperforms competing methods on four standard benchmarks.

  • TCL maximizes same-video representation similarity across two playback speeds while minimizing similarity between different videos run at different speeds.Source: §5 · p. 8
  • Its contrastive loss also groups videos with similar actions to capture high-level action semantics across distinct instances.Source: §5 · p. 8
  • Across four standard benchmark datasets, TCL significantly outperforms several competing methods.Source: §5 · p. 8

Appendix

The appendix outlines dataset and implementation details, additional results, and further analyses and examples.

  • Appendix sections A–C cover dataset details and implementation details for TCL and video extensions of image-based baselines.Source: Appendix · p. 11
  • Section D reports additional classwise improvements over S4L for 1% labeled data in Jester.Source: Appendix · p. 11
  • Sections E–F examine group contrastive loss on image datasets and provide additional qualitative examples from different datasets.Source: Appendix · p. 11

A. Dataset-Details

The appendix describes four datasets used in the experiments, including their composition and, for Charades-Ego, the evaluation protocol.

  • Mini-Something-V2 is a subset of Something-Something V2 with 81,663 training videos, 11,799 validation videos, and 87 basic object-interaction classes.Source: A. Dataset-Details · p. 11
  • Jester contains 148,092 videos across 27 hand-gesture classes, with 118,562 training clips and 14,787 validation clips.Source: A. Dataset-Details · p. 11
  • Kinetics-400 is a YouTube-video benchmark with around 300,000 videos across 400 diverse human-action classes.Source: A. Dataset-Details · p. 11
  • Charades-Ego pairs first- and third-person videos across 112 actors; overlapping activity classes make the dataset particularly challenging.Source: A. Dataset-Details · pp. 11–12
  • For Charades-Ego, experiments use 10% labeled third-person clips across 157 classes and evaluate mAP on the full third-person test set.Source: A. Dataset-Details · p. 12

B. Implementation Details

The implementation uses a 2D ResNet-18 with temporal shift modules, then specifies the training epoch and fine-tuning learning rate.

  • The video convolution uses 2D CNNs for lower computational complexity than 3D counterparts, with bidirectional temporal shifts to model temporal information.TSM hyperparameters follow [36].Source: B. Implementation Details · p. 12
  • A 2D ResNet-18 is the primary backbone, with temporal shift modules after every residual branch to avoid interfering with spatial feature learning.Source: B. Implementation Details · p. 12
  • One epoch is defined as one pass through labeled data, and fine-tuning uses a learning rate of 0.002.Source: B. Implementation Details · p. 12

C. Image-based Baseline Details

The appendix describes shared implementation choices and the configurations used for supervised and semi-supervised baselines. It also reports a classwise Jester comparison with S4L.

Shared implementation choices

  • For temporal consistency, every frame in a video receives the same augmentations; Random Horizontal Flip is omitted for highly spatially sensitive datasets, and baselines use initial lr 0.02 with cosine decay and 350 epochs unless otherwise specified.Source: C. Image-based Baseline Details · p. 12

Supervised

  • The supervised baseline uses the authors’ public code, the TCL initial learning rate, and dataset-specific hyperparameters from the cited work.It is trained with Lsup for 200 epochs.Source: C. Image-based Baseline Details · p. 12

MixMatch

  • MixMatch assigns unlabeled videos sharpened average predictions as targets, mixes labeled and unlabeled examples, and trains with cross-entropy and MSE losses.Cropping and flipping are the two augmentations; temperature is 0.5, and µ and γ are both 1.Source: C. Image-based Baseline Details · p. 12

S4L

  • S4L trains on video rotations of 0, 90, 180, and 270 degrees using rotation loss alongside Lsup for labeled videos.The experiment sets µ and γ to 5 and the initial learning rate to 0.1.Source: C. Image-based Baseline Details · p. 12

Pseudo-Label

  • Pseudo-Label first trains with Lsup for 50 epochs, then assigns model-predicted labels to unlabeled videos only when their highest activation exceeds 0.95.Training continues for 300 epochs using labeled and unlabeled videos; µ and γ are both 3.Source: C. Image-based Baseline Details · p. 12

Classwise Jester comparison

  • TCL improves over S4L on most Jester classes, including classes with fewer labeled videos.The figure compares classwise top-1 accuracy in the 1% labeled scenario and also shows labeled-video counts by class.Source: Figure 7 · p. 13

Mean Teacher

  • Mean Teacher uses a student and an EMA-weighted teacher with consistency between their unlabeled-video logits, while labeled data uses Lsup.γ rises from 0 to 1 over 50 epochs; µ and γ are both set to 1.Source: C. Image-based Baseline Details · pp. 12–13

FixMatch

  • Video FixMatch uses weakly augmented frames to produce pseudo-labels and strongly augmented frames to train consistency with those labels.Weak augmentation uses flips and shifts; strong augmentation uses RandAugment followed by CutOut. Experiments use 350 epochs, batch size 8, µ=3, and γ=9.Source: C. Image-based Baseline Details · p. 13

D. Classwise Improvements

The classwise Jester comparison reports that TCL improves on S4L for nearly all classes in the 1% labeled-data setting.

Classwise Improvements

  • Only 2 Jester classes show less top-1 accuracy improvement for TCL than S4L with 1% labeled data.The comparison is classwise and identifies S4L as the next-best method.Source: D. Classwise Improvements · p. 13

E. Group Contrastive Loss on Image Dataset

The image-dataset experiment finds that group contrastive loss improves semi-supervised classification performance on CIFAR-10.

Group Contrastive Loss on Image Dataset

  • 3.15% improvement (84.11% vs 87.26%) with group contrastive loss on CIFAR-10 using SimCLR, WideResNet-28-2, and 4 labeled samples per class.The passage presents this as evidence of effectiveness beyond video datasets.Source: E. Group Contrastive Loss on Image… · p. 13

F. Qualitative Examples

The supplementary examples compare TCL with supervised and semi-supervised baselines across Mini-Something V2, Jester, Kinetics-400, and Charades-Ego.

Mini-Something V2

  • On Mini-Something V2 with 5% labeled data and ResNet-18, TCL correctly recognizes actions where the supervised baseline and competing method are wrong in most examples.The figure compares top-1 predictions from the supervised baseline, FixMatch, and TCL.Source: F. Qualitative Examples · p. 13, Figure 8 · p. 14

Jester

  • On Jester with 1% labeled data and ResNet-18, TCL correctly recognizes hand gestures where the supervised baseline and S4L are wrong in most examples.The figure compares top-1 predictions from the supervised baseline, S4L, and TCL.Source: F. Qualitative Examples · p. 13, Figure 9 · p. 14

Kinetics-400

  • On Kinetics-400 with 5% labeled data and ResNet-18, TCL correctly recognizes actions where the supervised baseline and FixMatch are wrong in most examples.The figure compares top-1 predictions from the supervised baseline, FixMatch, and TCL.Source: F. Qualitative Examples · p. 13, Figure 10 · p. 15

Charades-Ego

  • On Charades-Ego, TCL correctly recognizes most relevant actions in the examples, while the supervised and competing methods miss all correct actions in most cases.The comparison uses top-K predictions on random frames, with K equal to the number of ground-truth classes for each sample.Source: F. Qualitative Examples · p. 13, Figure 11 · p. 15

Related papers

Loading 2102.02751v2…