PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning for Videos: A Survey

Madeline C. Schiappa, Yogesh S. Rawat, Mubarak Shah

arXiv:2207.00419v3cs.CVcs.MM

TL;DR

Expensive and biased video annotations motivate self-supervised representation learning that must account for temporal dynamics. The survey reviews video and multimodal methods through four learning categories, evaluates their reported landscape, and identifies limitations and future directions. It finds contrastive learning broadly strong for action-based tasks, while multimodal methods extend evaluation beyond action recognition.

  • Problem

    Video annotation is expensive, and human-generated labels can introduce bias, weak domain generalization, and limited robustness; video learning also must handle temporal dynamics.

  • Method

    The survey reviews video and multimodal self-supervised methods, categorizes them as pretext, generative, contrastive, or cross-modal agreement, and summarizes datasets and evaluation tasks.

  • Results

    Contrastive learning is reported as the best-performing objective across action-based downstream tasks, while multimodal approaches also support tasks such as text-to-video retrieval and action segmentation.

  • Takeaways & Limitations

    Temporal consistency strengthens image-based video methods, and multimodal signals can provide comparable performance without data augmentation in the surveyed settings.

  • Takeaways & Limitations

    Existing evaluations rely heavily on UCF101 and HMDB51, which lack temporal specificity, while detection tasks remain underexplored.

Abstract

from arXiv · show

The remarkable success of deep learning in various domains relies on the availability of large-scale annotated datasets. However, obtaining annotations is expensive and requires great effort, which is especially challenging for videos. Moreover, the use of human-generated annotations leads to models with biased learning and poor domain generalization and robustness. As an alternative, self-supervised learning provides a way for representation learning which does not require annotations and has shown promise in both image and video domains. Different from the image domain, learning video representations are more challenging due to the temporal dimension, bringing in motion and other environmental dynamics. This also provides opportunities for video-exclusive ideas that advance self-supervised learning in the video and multimodal domain. In this survey, we provide a review of existing approaches on self-supervised learning focusing on the video domain. We summarize these methods into four different categories based on their learning objectives: 1) pretext tasks, 2) generative learning, 3) contrastive learning, and 4) cross-modal agreement. We further introduce the commonly used datasets, downstream evaluation tasks, insights into the limitations of existing works, and the potential future directions in this area.

1 Introduction

Self-supervised learning addresses the cost and bias of large-scale video annotation, but video requires methods that account for temporal dynamics. This survey organizes video and multimodal approaches, evaluation resources, limitations, and future directions.

  • Large-scale labels are costly, while annotation bias, weak domain generalization, and limited robustness constrain supervised pre-training.
  • Video self-supervision cannot directly reuse image-based approaches because video representations must capture temporal information and motion.
  • Multimodal approaches use video, audio, and text, support tasks such as zero-shot learning and retrieval, and motivate broader organization of the field.
  • The survey reviews video and multimodal self-supervised learning and organizes methods into pretext, generative, contrastive, and cross-modal agreement categories.
  • It summarizes datasets, downstream tasks, evaluation metrics, and performance comparisons to support benchmarking future work.
  • The survey discusses limitations, open problems, and future directions in video self-supervised learning.

2 Preliminaries

The preliminaries introduce the building blocks needed to study self-supervised video learning, including downstream tasks, backbone architectures, and datasets.

  • The survey introduces downstream tasks, backbone network architectures used during training, and widely used datasets.

2.1 Downstream Evaluation

Learned video representations are evaluated through fine-tuning, linear probing, or zero-shot learning across recognition, localization, retrieval, segmentation, and captioning tasks.

  • Downstream evaluation uses fine-tuning, linear probing, or zero-shot learning to assess representations learned during pre-training.
  • Action recognition classifies each input video into a target action category using accuracy, precision, and recall.
  • Temporal action segmentation detects action start and end times, typically measured with frame accuracy.
  • Temporal action step localization assigns video segments to steps in complex activities, using frame-level recall against ground-truth intervals.
  • Video retrieval finds similar or near-duplicate videos, commonly reporting recall at top-1, top-5, and top-50 ranks.
  • Text-to-video retrieval returns videos matching textual queries and often evaluates text-video representations in zero-shot settings.
  • Video captioning generates textual descriptions from video clips and combines BLEU, METEOR, Rouge-L, and CIDEr metrics.

2.2 Building Blocks

Video representations are encoded with spatial, temporal, or combined architectures, while audio is transformed into Mel Spectrograms before CNN feature extraction and text is encoded with Word2Vec or BERT-based methods.

  • Building Blocks: The survey covers backbone architectures for encoding videos, text, and audio as building blocks for self-supervised learning.These encoders support representation learning across visual and multimodal settings.
  • Video Backbone Architectures: Video backbones use 2D CNNs for frame-level encoding, 3D CNNs for joint spatial-temporal encoding, or combinations of both.ResNet is popular for both 2D and 3D CNNs; transformer-based video encoders include TimeSformer, MViT, ViViT, and Swin.
  • Audio Backbone Architectures: Audio encoding converts waveforms into spectrograms, commonly Mel Spectrograms, which CNNs then use to extract audio features.The Mel scale replaces ordinary frequency values after Fourier analysis over signal bins.
  • Text Backbone Architectures: Text encoders commonly use Word2Vec or BERT, with BERT trained through self-attention and objectives such as masked language modeling or next-sentence prediction.Masked language modeling trains the model to predict masked word tokens.

2.3 Datasets

The survey catalogs datasets for self-supervised video pre-training, downstream evaluation, and multimodal learning, spanning action recognition, retrieval, captioning, localization, and related tasks.

  • Video-Based Datasets: Kinetics and HowTo100M are commonly used for pre-training, while UCF101 and HMDB51 are common downstream evaluation datasets.Kinetics is a large-scale human-action dataset, whereas HowTo100M contains instructional videos with captions.
  • Video-Based Datasets: Video datasets support downstream tasks including action recognition, retrieval, captioning, temporal localization, and video description.Instructional datasets such as YouCook2, COIN, and CrossTask provide temporal annotations for activities or task steps.
  • Video-Based Datasets: HowTo100M contains 136 million clips from 1.2 million YouTube videos, with captions spanning 23,000 activities.Its videos are untrimmed and often minutes long, and captions come from user-uploaded or automatically generated subtitles.
  • Video-Based Datasets: The benchmark collection includes datasets with varied scale, annotation density, activity coverage, and temporal structure.Examples range from UCF-101’s 13,320 clips and 101 categories to YouTube-8M’s 8 million videos and 4,716 classes.
  • Audio and Multimodal Datasets: Multimodal self-supervised learning uses video, text, and sometimes audio datasets for tasks such as event classification, retrieval, captioning, and summarization.Video datasets may include text without audio, so separate audio datasets can be needed for audio-inclusive approaches.

3 Self-Supervised Learning Approaches

The survey organizes video self-supervised learning methods into four categories according to their learning objectives: pretext, generative, contrastive, and cross-modal agreement.

  • Approach Categories: Video self-supervised learning approaches are divided into pretext, generative, contrastive, and cross-modal agreement categories.The categories structure the survey’s discussion of different learning objectives.

3.1 Pretext Learning

Pretext methods create supervisory signals from altered video appearance, playback speed, temporal order, or spatial-temporal patch arrangement. Among the reviewed approaches, playback-speed and temporal-consistency tasks typically perform better, although direct comparison is difficult across differing architectures and pre-training datasets.

  • Pretext Learning: Pretext tasks use predefined objectives that require models to solve input-derived problems to learn transferable representations.The approach assumes that solving a challenging task requiring high-level input understanding encourages more generalizable features.
  • Appearance Statistics: Appearance-statistics tasks apply spatial augmentations such as color changes, rotation, or noise and ask the model to predict the applied transformation.Video extensions may process separately rotated frames with a 3D CNN.
  • Playback Speed: Playback-speed tasks alter temporal sampling by selecting every p frames, then classify the playback rate or reconstruct the clip at its original speed.Some methods add transformation classification and reconstruction losses as additional training signals.
  • Temporal Order: Temporal-order tasks shuffle clips or frames and train models to identify the correctly ordered sequence, often through odd-one-out classification or sorting.Pairwise feature extraction can predict the actual order from shuffled high-motion clips.
  • Video Jigsaw: Jigsaw tasks shuffle image or video patches and train a classifier to predict the applied permutation, with video versions using three-dimensional patches.Video extensions face more possible permutations as the number of patches increases.
  • Discussion: Playback-speed or temporal-consistency pretext tasks typically perform better on downstream action recognition and video retrieval.The survey cautions that varied backbone architectures and pre-training datasets make direct comparisons difficult.

3.2 Generative Approaches

Generative approaches learn video representations by reconstructing, predicting, or masking visual and multimodal inputs. Across evaluations, temporal modeling and transformer-based masked generation are associated with stronger performance.

  • Overview: Generative learning synthesizes video or other signals using neural networks, extending image-domain GAN and masked-autoencoder techniques to video.Common video objectives include reconstruction, future-frame prediction, and masked-input generation.
  • Video-specific approaches: Video generative methods include adversarial synthesis, frame prediction, motion generation, recurrent path modeling, and masked-frame learning.These methods incorporate motion, frame sequence, or temporal masks to address video dynamics.
  • Masked modeling: Masked modeling extends from images to video and multimodal inputs by reconstructing masked patches, frames, or visual, textual, and audio tokens.Multimodal methods can combine tokens from several modalities during pre-training.
  • Action recognition: Transformer-based masked-input methods are the highest performers among generative approaches evaluated on UCF101 and HMDB51 action recognition.Among frame prediction and reconstruction methods, those using motion and frame sequence perform better.
  • Action recognition: Visual transformers and temporal-specific learning mechanisms show large performance increases on Something-Something and Kinetics400 action recognition.These datasets emphasize temporal understanding, especially Something-Something.
  • Video captioning: On YouCook2 caption generation, generative models perform better when pre-trained on HowTo100M, while multimodal masked modeling performs highly without target-dataset pre-training or fine-tuning.Masked modeling uses text and video, enabling caption-generation evaluation.

3.3 Contrastive Learning

Contrastive video learning brings positive pairs closer and separates negative pairs, with methods differing in how samples and embedding neighborhoods are constructed. Evaluations favor spatio-temporal augmentation, while multimodal clustering is another strong route.

  • Objective: Contrastive learning pulls positive input pairs together and pushes negative pairs apart in representation space.Positive and negative pairs may be formed from frames, clips, augmentations, or other modalities.
  • Approach categories: Video contrastive methods generate pairs through view, temporal, spatio-temporal, multimodal alignment, multimodal clustering, and cross-modal agreement strategies.These categories reflect different sources of invariance and correspondence.
  • Augmentation: View augmentation applies appearance transformations to each frame, whereas temporal augmentation changes clip ordering or interval boundaries.Temporal methods can maximize similarity between adjacent frames while contrasting frames from different videos.
  • Augmentation: Spatio-temporal methods combine spatial and temporal signals, including augmented RGB and optical-flow encoders trained with modified contrastive objectives.Some methods compare multiple positive samples or use local losses over non-overlapping clips.
  • Clustering: Clustering methods organize video representations using close and background neighbors, sometimes combining 3D visual features with motion trajectories.K-means clustering can identify neighborhood relationships used for representation learning.
  • Evaluation: Spatio-temporal augmentation achieves the highest action-recognition success, while image-derived methods perform well when temporal consistency is added.Audio-video pseudo-labeling is the strongest reported clustering approach and avoids data augmentation.

3.4 Cross-Modal Agreement

Cross-modal agreement learns from correspondence among video, text, and audio, primarily through contrastive objectives and, in some cases, clustering or joint embeddings. These approaches support several downstream tasks and often perform strongly in retrieval and action-related evaluation.

  • Overview: Cross-modal agreement primarily uses contrastive NCE losses to compare positive and negative pairs across video, text, and audio.The section focuses on methods using cross-modal agreement as the main self-supervised objective.
  • Video and text: MIL-NCE adapts noise-contrastive estimation to misaligned HowTo100M clip-narration pairs by constructing bags of positive candidate pairs.The method uses visual and text encoders, expanded temporal ranges, and negative video-narration pairs.
  • Video and text: Cross-modal methods enforce global and local representations, combine video-text contrastive and generative losses, or learn joint embeddings across modalities.Joint-embedding approaches concatenate or otherwise aggregate multimodal features before cross-modal prediction.
  • Video and audio: Audio-video learning uses transformations, synchronization objectives, or modality-specific encoders to align temporal information across signals.Some methods use two-branch audio and video networks followed by fusion or classification.
  • Video and audio: Audio and video can generate reciprocal pseudo-labels through separate clustering, while other methods cluster using both modalities jointly.Alwassel et al. use classification rather than contrastive learning for reciprocal pseudo-label generation.
  • Results and limitations: Many multimodal models rely on pre-extracted S3D-g visual features, motivating future study of raw-video input with transformer-based architectures.This is a stated limitation and future direction for understanding what these models learn.
  • Results and limitations: Multimodal contrastive approaches outperform masked modeling on video retrieval, and the strongest action-recognition result uses audio-video pseudo-labels.Fine-tuned VideoCLIP performs best on some downstream tasks, including text-to-video retrieval and temporal action-step localization.

4 Summary and Future Directions

The survey finds contrastive and masked-modeling approaches perform strongly, while highlighting limitations in temporal coverage, computational dependencies, interpretability, and benchmarking.

  • Summary and Future Directions: Contrastive learning is the best-performing objective across action-based downstream tasks, with spatio-temporal augmentations typically performing best.Positive and negative samples can be generated through view, temporal, spatio-temporal, clustering, or multimodal strategies.
  • Summary and Future Directions: Masked-modeling approaches achieve the strongest reported performance among generative methods.
  • Long-term video representation: Most approaches learn from short clips rather than whole videos, limiting long-term temporal representation learning.The survey recommends evaluating temporally relevant datasets such as Something-Something and NExT-QA.
  • Simple end-to-end approaches: Future work should develop simpler end-to-end models that reduce reliance on pretrained encoders and computational costs.The survey also identifies interpretability and explainability as open directions for understanding learned representations.
  • Interpretability and Explainability: Multimodal models can outperform MIL-NCE on text-to-video retrieval while producing distant text and video embeddings, motivating further interpretability work.The survey connects better interpretability with understanding modality interactions and temporal event relationships.
  • Standard Benchmarking: Comparisons remain difficult because backbone encoders and pre-training protocols vary, while video robustness research and standardized benchmarks remain limited.The survey calls for benchmarks covering both performance and robustness under distribution shifts.

5 Conclusion

The survey synthesizes self-supervised video learning methods and identifies recurring patterns in pretext, contrastive, and multimodal approaches. It concludes that standardized evaluation, long-term modeling, simpler systems, and interpretability are key future directions.

  • Conclusion: Playback-speed or temporal-consistency pretext tasks typically perform best among pretext approaches.
  • Conclusion: Image-based augmentations can work well for video contrastive learning when temporal elements are included.
  • Conclusion: Multimodal approaches can perform comparably to unimodal approaches without data augmentations.
  • Conclusion: The survey identifies standardized benchmarks, long-term video representation, simple end-to-end approaches, and interpretability as priority directions.
Loading 2207.00419v3…