PaperlayerView exact revision on arXiv

Source-linked AI summary

A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning

Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, Kaiming He

arXiv:2104.14558v1cs.CVcs.AIcs.LG

TL;DR

The paper studies how to learn useful spatiotemporal representations from unlabeled videos when evidence across unsupervised frameworks and evaluation settings is not yet unified. It adapts four image-based frameworks with a temporal-persistency objective over clips from the same video, finding strong performance across datasets, tasks, and architectures, including state-of-the-art results and some cases surpassing supervised pre-training.

  • Problem

    The paper addresses the need to study unsupervised video representation learning across multiple frameworks, datasets, downstream tasks, and architectures on common ground.

  • Method

    The method incorporates a simple objective encouraging similar representations for multiple temporally separated clips of the same video into MoCo, SimCLR, BYOL, and SwAV.

  • Results

    The resulting models achieve state-of-the-art unsupervised video representation learning results and can compete with or surpass supervised pre-training in several cases.

  • Takeaways & Limitations

    Longer positive timespans, momentum encoders, training duration, backbones, augmentation, and curation are important factors for unsupervised video representation performance.

  • Takeaways & Limitations

    Instagram videos in the relevant analysis are still sampled according to hashtags related to Kinetics-400 classes, so the conjecture might be biased.

Abstract

from arXiv · show

We present a large-scale study on unsupervised spatiotemporal representation learning from videos. With a unified perspective on four recent image-based frameworks, we study a simple objective that can easily generalize all these methods to space-time. Our objective encourages temporally-persistent features in the same video, and in spite of its simplicity, it works surprisingly well across: (i) different unsupervised frameworks, (ii) pre-training datasets, (iii) downstream datasets, and (iv) backbone architectures. We draw a series of intriguing observations from this study, e.g., we discover that encouraging long-spanned persistency can be effective even if the timespan is 60 seconds. In addition to state-of-the-art results in multiple benchmarks, we report a few promising cases in which unsupervised pre-training can outperform its supervised counterpart. Code is made available at https://github.com/facebookresearch/SlowFast

1. Introduction

This study generalizes image-based unsupervised representation learning to video by encouraging persistent representations across temporally separated clips. Across frameworks, datasets, tasks, and architectures, the study reports strong downstream performance, including state-of-the-art benchmark results and cases where unsupervised pre-training surpasses supervised pre-training.

  • Core findings: Longer timespans between positive clips, up to a minute, and multiple temporal samples improve downstream performance.The study treats videos as providing natural augmentations through motion, deformation, occlusion, and illumination changes.
  • Study scope: The work extends four image-based unsupervised frameworks into space-time using a simple objective that encourages similar representations for different clips of the same video.The frameworks are MoCo, SimCLR, BYOL, and SwAV.
  • Evaluation: The study evaluates representation quality across three pre-training datasets and six downstream datasets or tasks.The evaluations include classification and detection datasets such as Charades, Something-Something, and AVA.
  • Evaluation: Ablations examine temporal samples, objectives, momentum encoders, training duration, backbones, augmentation, curation, and video trimming.These experiments assess how implementation and data factors affect representation learning.
  • Results: The resulting methods achieve state-of-the-art unsupervised video representation learning results on UCF-101, HMDB51, and Kinetics-400.The paper also reports cases in which unsupervised pre-training surpasses supervised pre-training.

2. Related Work

The paper situates video representation learning alongside image-based unsupervised methods and prior video objectives, then compares four visual methodologies on common ground. It also emphasizes that video evaluation protocols and backbone choices remain less standardized than in image learning.

  • Image-based learning: Image-based unsupervised learning includes pretext tasks, instance discrimination with contrastive objectives, and methods based on positive pairs.The cited image methods use color, patch, augmentation, contrastive, or positive-pair objectives.
  • Video-based learning: Video unsupervised learning has used predictive tasks based on motion, color, and spatiotemporal ordering, as well as visual and audio-visual contrastive objectives.Later video methods also relate to image-based approaches and may transfer supervision through optical flow, audio, or text.
  • Positioning: This work focuses on purely visual unsupervised video learning and compares meta-methodologies on common ground.The compared approaches are adapted from image-based frameworks.
  • Evaluation gap: Unlike image-based evaluation, video understanding has not converged on common backbones, protocols, or downstream datasets.The paper addresses this variation by studying different encoders and six downstream benchmarks.

3. Approach

The approach learns space-time-persistent embeddings from multiple augmented clips of unlabeled videos and incorporates this objective into four unsupervised frameworks. The frameworks differ in their use of negative samples, momentum encoders, predictors, prototypes, and Sinkhorn-Knopp targets.

  • Persistent temporal feature learning: The study trains a spatiotemporal encoder to produce embeddings that persist across multiple temporally distant clips of the same video.The default encoder is a Slow-only 3D ResNet-50 followed by an MLP projection head.
  • Persistent temporal feature learning: The framework generalizes image-view invariance from crops and augmentations to temporally separated video clips.Its goal is to learn high-level categorical semantics by enforcing persistency over space-time.
  • Persistent temporal feature learning: Each minibatch samples ρ clips per video, using other clips from the same video as positive keys for the query clip.The loss is computed from similarities between the query and these positive clips.
  • Unsupervised learning frameworks: SimCLR and MoCo use contrastive losses with negatives, while BYOL and SwAV rely only on positive samples.SimCLR draws negatives from the minibatch; MoCo uses a queue and a momentum encoder, whereas BYOL uses a predictor and SwAV uses prototypes with a Sinkhorn-Knopp transform.
  • Implementation: The implementation uses a symmetric loss across all ρ clips, with sequential aggregation for MoCo and BYOL and parallel evaluation for SimCLR and SwAV.For MoCo and BYOL, memory consumption for ρ > 2 equals a single clip's forward and backward pass; for SimCLR and SwAV, it grows linearly with the number of clips.

4. Experiments

Experiments show that temporal persistence, multiple clips, and long training improve unsupervised video representations across frameworks and datasets, while performance depends on timespan, augmentation, and backbone choices. Long temporal spans remain effective up to 60 seconds, although uncurated videos mildly degrade beyond 36 seconds and dataset curation can bias this result.

  • Persistent temporal learning: Temporal persistence substantially improves representation learning, with multiple clips further increasing accuracy; BYOL reaches 68.9% K400 and 93.8% UCF101 at ρ=4.Sampling positives at different temporal locations is especially important for SimCLR and SwAV, whose performance drops from 60.5 to 36.1 and 61.6 to 38.6, respectively, when positives share a temporal instance.
  • Persistent temporal learning: Contrastive and non-contrastive methods show no clear performance difference, while momentum-encoder methods outperform non-momentum methods by approximately 4% on K400.The study interprets this pattern as evidence that within-video space-time persistence matters more than learning invariance across videos.
  • Training duration and datasets: Training longer generally improves performance, but BYOL can degrade under longer schedules, possibly because its hyperparameters are not adjusted across training durations.On web data, MoCo and SwAV gain roughly 3–4% from 50 to 200 epochs, while BYOL and SimCLR perform better on 10–16-second uncurated-short videos.
  • Timespan between positives: Long temporal spans remain effective: unrestricted positives on 60-second IG-Curated-1M videos achieve 64.1% top-1 K400 accuracy, whereas uncurated videos mildly decline beyond 36 seconds.On K400, performance is relatively robust up to 10 seconds; the curated-data conjecture may be biased because videos are hashtag-selected for K400-related classes.
  • Backbone architectures: Backbone architecture changes downstream outcomes: doubling Slow’s input frame-rate improves UCF101, while R2+1D shows 71.7% supervised versus 57.2% unsupervised Kinetics accuracy but 93.7% on UCF101.The authors suggest that UCF101’s small size may favor models with fewer parameters and easier overfitting.

5. Conclusion

The study finds that persistent spatiotemporal features provide strong unsupervised video representations across datasets and evaluation settings, while several implementation and scaling factors remain important.

  • The four studied unsupervised video methodologies benefit from longer positive timespans, with contrastive objectives less influential than momentum encoders.Training duration, backbones, augmentation, and data curation also materially affect performance.
  • Models learning persistent features across augmented space-time clips set a new state-of-the-art.
  • Linear readout on Kinetics is a good indicator of performance on other datasets.
  • Unsupervised pre-training competes with supervised pre-training on several datasets, although room for improvement remains.

A.1. Scaling “in-the-wild” data

Scaling in-the-wild Instagram data improves unsupervised performance, while curation affects accuracy and balancing has a smaller observed effect.

  • Clips versus training: Increasing clips to ρ=4 at 50 epochs yields 67.7% on K400 and 94.5% on UCF101 for BYOL.For BYOL, longer training to 200 epochs instead yields 60.2% on K400 and 92.7% on UCF101, compared with 64.1% and 93.5% at ρ=2 and 50 epochs.
  • Clips versus training: Increasing MoCo from ρ=2 to ρ=3 raises results by 1.6%/0.9% on K400 and 0.4%/1% on UCF101 for 100/200 epochs.Increasing ρ is reported as more accurate and faster than increasing epochs.
  • Curation: 65.8%, 63.2%, 63.1%, and 60.6% top-1 accuracy are obtained by K400, IG-Curated, IG-Curated-Unbalanced, and IG-Uncurated at 240K samples.The curation gaps from K400 to IG-Curated and from IG-Curated to IG-Uncurated are 2.6% and 2.5%, respectively.
  • Scale: Doubling the number of videos roughly linearly increases accuracy across all datasets.
  • Scale: 65.4% with 1M uncurated videos approaches the 65.8% obtained with K400 pre-training on K400 linear evaluation.The experiment uses 4× more videos for the uncurated setting.

A.2. Scaling Kinetics data

Larger Kinetics pre-training datasets improve unsupervised performance, with temporally persistent MoCo approaching supervised performance on UCF101 but retaining linear-evaluation gaps.

  • Results: 69.0%, 70.0%, 54.2%, and 93.6% are achieved on K400, K600, K700, and UCF101 by the best K400-pretrained MoCo model with ρ=4.Kinetics uses linear classification evaluation, while UCF101 uses finetuning of all weights.
  • Evaluation boundary: MoCo with ρ=4 approaches supervised pre-training on UCF101, but a gap remains for linear evaluation on K400, K600, and K700.
  • Results: MoCo trained on K600 with ρ=2 slightly outperforms the K400-pretrained variant on all datasets except UCF101.Using temporal persistency across ρ=4 clips increases accuracy by approximately 2% across all datasets.
  • Dataset scale: 2.7%, 3.2%, 3.9%, and 1.2% accuracy increases result from moving pre-training from K400 to K700 on K400, K600, K700, and UCF101.

A.3. Key implementation specifics

Performance depends strongly on implementation choices: momentum annealing and method-specific normalization help selectively, while deeper projection heads have different effects across frameworks and downstream protocols.

  • Momentum annealing: ∼1% accuracy is gained by cosine-annealing MoCo’s momentum, using mbase=0.994 as the default.Without annealing, MoCo achieves 64.5% accuracy.
  • Normalization and optimization: LARS boosts SimCLR and SwAV by around 10% but reduces performance in MoCo and BYOL.The comparison associates the optimizer effect with whether the method uses a momentum encoder.
  • Normalization and optimization: SyncBN and MLP-BN dramatically improve BYOL, whereas MLP batch normalization drastically degrades MoCo.MoCo shows no significant change from SyncBN alone.
  • Projection MLP: A 3-layer projection MLP improves MoCo by 1.2% on K400, with no further gain from a fourth layer.UCF101 performance remains stable for this MoCo modification.
  • Projection MLP: BYOL’s deeper predictor decreases performance by 2.2% on K400 and 2.5% on UCF101.Increasing projection depth alone leaves K400 unchanged and decreases UCF101 by 0.7%.
  • Projection MLP: A 3-layer projection improves SimCLR by 1.5% on K400 and 0.5% on UCF101, with no further gain from four layers.
  • Projection MLP: SwAV gains 1.3% from two to three projection layers and another 0.4% with four layers on K400, while UCF101 performance decays.
  • Overall implementation findings: K400 linear evaluation generally benefits from deeper projection heads, while fine-tuned UCF101 performance is relatively unchanged or decays.Deeper MLPs mostly leave training FLOPs unchanged.

B.1. Unsupervised pre-training

The study pre-trains video encoders using synchronized multi-GPU optimization, standardized augmentations, and method-specific implementations of MoCo, BYOL, SimCLR, and SwAV. The default backbone is an R-50 Slow model with 8×8 temporal sampling, while MLP depth and training settings are varied in ablations.

  • Encoder and training setup: Pre-training uses synchronized SGD on 64 GPUs with 8 clips per GPU, for a total mini-batch size of 512.Batch Normalization statistics are synchronized differently across methods, using 8 clips for MoCo and 64 clips for BYOL, SimCLR, and SwAV.
  • Ablations: The MLP-depth study evaluates training complexity through FLOPs and parameters alongside linear K400 and fine-tuned UCF101 accuracy.Accuracy is measured on the backbone without the MLPs, using K400 linear evaluation and UCF101 fine-tuning.
  • Optimization: The learning rate follows a half-period cosine schedule, with base rates ηMoCo = 0.4 and ηSimCLR = ηBYOL = ηSwAV = 4.8.LARS is used for BYOL, SimCLR, and SwAV except for bias and BN parameters, with trust coefficient 0.001.
  • Method-specific implementations: MoCo, BYOL, SimCLR, and SwAV each retain their characteristic projection, prediction, normalization, queue, prototype, or momentum mechanisms.MoCo uses 65,536 queued negatives; BYOL uses momentum annealing; SimCLR computes its loss over the synchronized full batch; SwAV uses Sinkhorn-Knopp normalization and frozen first-epoch prototypes.
  • Encoder and training setup: The default encoder is an R-50 Slow model with temporal dimension T and sample rate τ, using T×τ = 8×8 for ablations.The architecture is a ResNet-50-based Slow model, with temporal pooling only at the last layer.
  • Data augmentation: Video views are formed by random temporal clip sampling, spatial cropping or flipping, color distortion, and Gaussian blur.The spatial crop is 224×224 pixels after sampling the shorter side in [256, 320] pixels; color augmentation uses ColorJitter and RandomGrayscale by default.

B.2. Details: Kinetics Action Classification

Evaluation spans Kinetics action classification and multiple downstream video tasks, using frozen-feature linear evaluation, fine-tuning, detection, and standardized inference protocols. The protocols cover datasets ranging from large action collections to longer activities and human-object interactions.

  • Datasets: Kinetics-400, Kinetics-600, and Kinetics-700 provide approximately 240k, 392k, and 523k training videos, respectively, across hundreds of action classes.Their validation sets contain approximately 20k, 30k, and 35k videos, respectively.
  • Kinetics action classification: Linear evaluation freezes the pre-trained encoder and trains a classifier on last-layer features after unsupervised Kinetics pre-training.The classifier is trained for 60 epochs for the paper’s ablations, using cosine scheduling, warm-up, and zero weight decay.
  • Kinetics action classification: Kinetics validation accuracy uses 30-view testing: 10 uniformly sampled temporal clips, three 256×256 spatial crops per clip, and averaged softmax scores.This is reported as top-1 classification accuracy.
  • AVA action detection: AVA evaluates spatiotemporal action localization with mean Average Precision on 60 classes using 211k training and 57k validation video segments.The detector extends frame proposals into 3D RoIs, applies frame-wise RoIAlign and temporal global average pooling, then predicts with per-class sigmoid classifiers.
  • Charades: Charades contains approximately 9.8k training videos of longer activities averaging 30 seconds, and fine-tuning extends the input duration to T×τ = 16×8, increasing accuracy by approximately 3 mAP.Performance is measured in mAP for 157 multi-label classes, with spatiotemporal max-pooling during testing.
  • Something-Something V2: Something-Something V2 classifies human-object interactions into 174 classes, using segment-based sampling and disabling random flips because direction matters.The dataset contains 169k training and 25k validation videos, and evaluation uses single center-clip testing with top-1 accuracy.
  • Additional benchmarks: UCF101 and HMDB51 provide additional action-recognition evaluations, with UCF101 reported across three splits and HMDB51 following the UCF101 protocol.The paper notes that further hyperparameter tuning may improve HMDB51 downstream performance.
Loading 2104.14558v1…