PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey

Longlong Jing, Yingli Tian

arXiv:1902.06162v1cs.CV

TL;DR

Large-scale labeled data support visual feature learning but are costly to collect and annotate, motivating self-supervised learning from unlabeled images and videos. This paper surveys deep ConvNet-based methods, architectures, tasks, datasets, evaluations, and future directions. It reports strong image-feature results on some downstream tasks, while video-feature performance remains substantially behind supervised models.

  • Problem

    Large-scale labeled datasets improve deep visual feature learning, but collecting and annotating them is expensive, especially for videos.

  • Method

    The paper comprehensively surveys deep ConvNet-based self-supervised visual feature learning methods, architectures, pretext tasks, datasets, evaluations, and future directions.

  • Results

    Self-supervised image features approach supervised performance on some downstream tasks, whereas video methods remain substantially lower than supervised models.

  • Takeaways & Limitations

    Self-supervised learning provides a broad framework for learning transferable image and video features without human annotations, with especially small gaps on some image tasks.

  • Takeaways & Limitations

    Self-supervised video spatiotemporal feature learning with 3DConvNet remains insufficiently addressed and needs more effective video-specific pretext tasks.

Abstract

from arXiv · show

Large-scale labeled data are generally required to train deep neural networks in order to obtain better performance in visual feature learning from images or videos for computer vision applications. To avoid extensive cost of collecting and annotating large-scale datasets, as a subset of unsupervised learning methods, self-supervised learning methods are proposed to learn general image and video features from large-scale unlabeled data without using any human-annotated labels. This paper provides an extensive review of deep learning-based self-supervised general visual feature learning methods from images or videos. First, the motivation, general pipeline, and terminologies of this field are described. Then the common deep neural network architectures that used for self-supervised learning are summarized. Next, the main components and evaluation metrics of self-supervised learning methods are reviewed followed by the commonly used image and video datasets and the existing self-supervised visual feature learning methods. Finally, quantitative performance comparisons of the reviewed methods on benchmark datasets are summarized and discussed for both image and video feature learning. At last, this paper is concluded and lists a set of promising future directions for self-supervised visual feature learning.

1 INTRODUCTION

Self-supervised visual feature learning addresses the cost of large-scale human annotation by training ConvNets on automatically labeled pretext tasks using unlabeled images or videos. This survey defines the field, describes its pipeline and terminology, and reviews methods, datasets, quantitative comparisons, and future directions.

  • Motivation: Deep ConvNets and large-scale datasets improve computer-vision performance, but collecting and annotating such datasets is expensive and time-consuming.ImageNet contains about 1.3 million human-labeled images, while video annotation is even more costly because of the temporal dimension.
  • Core Concepts: Self-supervised learning trains ConvNets on pretext tasks using automatically generated pseudo labels rather than human annotations.Examples include colorizing grayscale images and other tasks whose labels derive from data attributes.
  • General Pipeline: The general pipeline learns features from a pretext task, transfers the learned model to downstream tasks by fine-tuning, and evaluates feature quality through downstream performance.Downstream tasks commonly use human-annotated labels, although some can match the pretext task without them.
  • Terminology: Pretext tasks are designed objectives for learning visual features, while downstream tasks evaluate those features in computer-vision applications.The survey distinguishes self-supervised, supervised, semi-supervised, weakly supervised, and unsupervised learning according to how labels are obtained.
  • Survey Scope: The survey presents a comprehensive review of self-supervised visual feature learning methods and datasets, quantitative performance comparisons, and possible future directions.Its stated contributions include an in-depth review, quantitative analysis, and guidance for future research.

2 FORMULATION OF DIFFERENT LEARNING SCHEMAS

The paper distinguishes supervised, semi-supervised, weakly supervised, and unsupervised learning by the types of labels used for visual feature learning. Self-supervised learning avoids human annotations by training networks with automatically generated pseudo labels and aims to produce transferable visual features.

  • 2.1 Supervised Learning Formulation: Supervised learning trains networks on data paired with accurate, fine-grained human-annotated labels.Its training dataset contains each input Xi and corresponding label Yi.
  • 2.1 Supervised Learning Formulation: Because human data collection and annotation are expensive and may require special skills, alternative learning schemas were proposed to reduce labeling costs.The paper motivates semi-supervised, weakly supervised, and unsupervised learning as alternatives to supervised learning.
  • 2.2 Semi-Supervised Learning Formulation: Semi-supervised learning combines a small labeled dataset with a large unlabeled dataset and models relations between unlabeled data and the labeled set.The formulation includes a task-specific function R(Zi, X) for representing these relations.
  • 2.3 Weakly Supervised Learning Formulation: Weakly supervised learning uses coarse-grained labels, whose lower acquisition cost makes large-scale datasets easier to obtain.Web-collected images labeled with hashtags have been used for image feature learning and reported good performance.
  • 2.4 Unsupervised Learning Formulation: Unsupervised learning does not require human-annotated labels and includes both fully unsupervised and self-supervised methods.Self-supervised methods explicitly train networks with automatically generated pseudo labels without human annotation.
  • 2.4.1 Self-supervised Learning: Self-supervised learning is defined by generating pseudo labels automatically without involving human annotations.The reviewed methods target visual feature learning whose features can transfer across multiple visual tasks and support learning new tasks from limited labeled data.
  • 2.4.1 Self-supervised Learning: The paper surveys self-supervised visual feature learning methods from multiple perspectives, including network architectures.Its scope centers on methods designed for transferable visual features rather than a single downstream task.

3 COMMON DEEP NETWORK ARCHITECTURES

The survey reviews common architectures for image and video feature learning, emphasizing how network designs extract hierarchical, spatial, and temporal information. It covers milestone 2DConvNets, video architectures, and their training implications.

  • Image feature learning: AlexNet, VGG, GoogLeNet, ResNet, and DenseNet are reviewed as milestone 2DConvNet architectures for image feature learning.These architectures differ in depth, width, skip connections, and dense connections.
  • Image feature learning: VGG uses deeper stacks of small 3×3 convolutions, whereas AlexNet uses larger kernels and convolution strides.The 16-layer VGG is highlighted for its moderate model size and performance.
  • Image feature learning: ResNet uses skip connections to address gradient vanishing and explosion, making very deep neural-network training feasible.Its smaller model size and performance also support its widespread use as a base network for other vision tasks.
  • Image feature learning: GoogLeNet builds wider networks with inception blocks containing parallel convolutions and 1 × 1 convolutions for dimension reduction.The 22-layer network achieved 93.3% top-5 test accuracy on ILSVRC-2014.
  • Image feature learning: DenseNet sends outputs from all preceding convolution blocks into the current block, separating low-level and task-specific feature learning across depths.This dense connectivity is designed to reduce the risk that deeper layers memorize low-level features.
  • Video feature learning: Video architectures include 2DConvNet-based, 3DConvNet-based, and LSTM-based methods that respectively fuse frame features, jointly extract spatial-temporal features, or model long-term dynamics.Two-stream networks use RGB inputs for spatial features and optical flow for temporal features; their fusion significantly improves action-recognition accuracy.
  • Video feature learning: C3D extracts appearance and temporal cues from 16-frame clips and achieved state-of-the-art results on several video-analysis tasks.Long-term temporal convolutions argue that 16 frames can be insufficient for long-lasting actions, motivating larger frame inputs.
  • Video feature learning: 3DResNet replaces ResNet’s 2D convolution layers with 3D convolutions and showed comparable state-of-the-art action-recognition performance on several datasets.This extends residual architectures to spatiotemporal video representation learning.

4 COMMONLY USED PRETEXT AND DOWNSTREAM TASKS

Self-supervised visual learning trains ConvNets on automatically generated pseudo labels from pretext tasks, then transfers the learned features to labeled downstream tasks. The survey organizes pretext tasks by data attributes and describes common evaluation tasks for images and videos.

  • Self-supervised pipeline: Self-supervised training minimizes prediction errors against automatically generated pseudo labels, avoiding human annotations throughout the pretext-task process.The resulting visual features can be transferred to downstream tasks as pretrained models.
  • Generation-based tasks: Image colorization converts grayscale images into RGB images, using the original RGB image as the pseudo label and requiring structural and contextual understanding.Training pairs can be generated in real time with negligible cost.
  • Pretext-task categories: Pretext tasks are categorized as generation-based, context-based, free semantic label-based, and cross modal-based according to the data attributes used.The categories cover image or video generation, contextual relations, automatically generated semantic labels, and correspondence across modalities.
  • Generation-based tasks: Generation-based methods learn features through image or video generation tasks, including colorization, super-resolution, inpainting, GAN generation, and video prediction.These tasks use generation as the learning objective rather than human-provided labels.
  • Context-based tasks: Context-based tasks exploit context similarity, spatial structure, or temporal structure, including clustering, jigsaw puzzles, geometric transformation recognition, and temporal-order verification.Temporal-order tasks use frame-sequence order as the supervision signal.
  • Other pretext tasks: Free semantic label-based tasks use semantic labels generated by hard-coded algorithms or game engines, while cross modal tasks verify correspondence between different input channels.Examples include moving-object segmentation, relative-depth prediction, visual-audio correspondence, and RGB-flow correspondence.
  • Downstream evaluation: Feature quality is evaluated by fine-tuning pretrained self-supervised models on downstream tasks such as image classification, segmentation, object detection, and action recognition.Transfer performance is used to assess the generalization ability of learned image or video features.
  • Downstream evaluation: FCN evaluates image features for semantic segmentation by replacing fully connected layers with transposed convolutions to produce dense pixel predictions.The pretrained parameters are fine-tuned on a semantic-segmentation dataset and compared with other self-supervised methods.

5 DATASETS

The survey summarizes image and video datasets used for self-supervised training and downstream evaluation. Human annotations can be discarded during pretraining, while labeled benchmarks are commonly used for fine-tuning and feature assessment.

  • Dataset usage: Datasets originally collected for supervised learning can support self-supervised training after human-annotated labels are discarded.Learned features are typically evaluated by fine-tuning on smaller labeled datasets for recognition, detection, or segmentation.
  • Image datasets: ImageNet contains 1.3 million images across 1,000 classes and is the most widely used dataset for self-supervised image feature learning.Each image is assigned one class label.
  • Image datasets: Places and Places365 provide large-scale scene datasets containing more than 2.5 million images across over 205 categories and more than 10 million images across over 400 classes, respectively.They support scene context and broader high-level visual-understanding tasks.
  • Synthetic datasets: SUNCG provides over 45,000 synthetic indoor scenes with realistic layouts, synthetic depth, object-level semantic labels, and volumetric ground truth.Its annotations support indoor-scene representation learning and evaluation.
  • Image datasets: STL-10 is designed for unsupervised feature learning and contains 100,000 unlabeled images alongside labeled training and testing sets.The dataset covers 10 object classes.
  • Video datasets: SceneNet RGB-D contains 5 million rendered RGB-D images from over 15,000 synthetic trajectories with annotations for scene and geometric understanding.Available labels include semantic and instance segmentation, object detection, optical flow, depth, and camera pose.
  • Video datasets: Kinetics contains around 500,000 video clips covering 600 human-action classes, with at least 600 clips per class.Each clip lasts around 10 seconds and has one action label.
  • Video datasets: UCF101 contains 13,370 clips spanning 101 action categories and is widely used to evaluate human-action recognition features after fine-tuning.The videos have 320 × 240 resolution and 25 FPS.

6 IMAGE FEATURE LEARNING

The survey reviews self-supervised image feature-learning methods in three groups: generation-based, context-based, and free semantic label-based methods. Cross modal methods are mainly discussed with video feature learning because many also apply to both modalities.

  • Image feature-learning methods: Image feature-learning methods are reviewed as generation-based, context-based, and free semantic label-based approaches.The survey points readers to a table summarizing methods by pretext-task category.

6.1 Generation-based Image Feature Learning

Generation-based methods learn visual features while reconstructing, transforming, or synthesizing images without human-annotated labels. Their objectives include realistic generation, inpainting, super-resolution, and colorization, though downstream feature evaluation remains limited for several tasks.

  • Overview: Generation-based methods use images themselves or automatically derived targets as pseudo-labels for image synthesis, reconstruction, or transformation without human annotations.The reviewed tasks include GAN generation, super-resolution, inpainting, and colorization.
  • Image Generation with GAN: GANs train a generator and discriminator adversarially so generated images become more realistic while the discriminator learns to distinguish real from generated data.The generator maps latent vectors to images, whereas the discriminator separates real data from generated data.
  • Evaluation Limitations: Generation-based methods often evaluate generated-image quality rather than transferable representation quality, and super-resolution transfer had not yet been tested in the reviewed work.Only a few random-variable generation methods evaluated discriminator features on high-level tasks.
  • Image Inpainting: Image inpainting uses an encoder-decoder generator to predict missing regions from image context, with adversarial training encouraging plausible hypotheses.The discriminator distinguishes generated inpainting outputs from real images.
  • Image Super-Resolution: SRGAN combines pixel-wise L2, content, and adversarial losses to recover fine high-resolution details that MSE-only models often lack.The adversarial loss pushes outputs toward the natural-image manifold, while the discriminator distinguishes generated from real high-resolution images.
  • Image Colorization: Image colorization predicts plausible colors from grayscale photographs, and Zhang et al.’s classification and class-rebalancing approach fooled humans in 32% of trials.Colorization requires object recognition and grouping pixels belonging to the same object part; some methods transfer the learned features to downstream tasks.

6.2 Context-Based Image Feature Learning

Context-based methods use spatial, similarity, and temporal relationships within images as automatically available supervision. Clustering and spatial-puzzle tasks train ConvNets to recover structure or assignments that require meaningful visual representations.

  • Overview: Context-based pretext tasks use image context similarity, spatial structure, or temporal structure as supervision signals for learning visual features.The network learns features while solving a task designed from contextual attributes rather than human labels.
  • Context Similarity: Clustering methods group similar images, then train ConvNets to predict cluster assignments or whether image pairs share a cluster.DeepCluster iteratively applies K-means and uses subsequent assignments to update network weights.
  • Spatial Context: Spatial-context methods train networks to infer relative patch positions or the correct ordering of shuffled patches from the same image.Solving these puzzles requires recognizing objects and relationships among their parts.
  • Spatial Context: A nine-patch jigsaw has 362,880 possible permutations, so methods select a subset with relatively large Hamming distances to constrain training.The reduction addresses ambiguity and makes the task more tractable.
  • Task Design: Puzzle tasks must balance difficulty: overly ambiguous tasks may prevent convergence, whereas overly easy tasks can invite trivial solutions.Reducing the search space is commonly used to control task difficulty.

6.3 Free Semantic Label-based Image Feature Learning

Free semantic label methods automatically generate labels such as depth, segmentation, contours, and optical flow using game engines or hard-coded programs. These labels can directly target semantic features, but synthetic-to-real domain gaps and label noise constrain applicability.

  • Overview: Free semantic labels are automatically generated without human annotation and can include segmentation masks, depth, optical flow, and surface normals.Such labels may be rendered by game engines or generated by hard-coded methods.
  • Game-Engine Labels: Game engines produce realistic synthetic images with accurate pixel-level labels at low cost, including depth, contours, surface normals, segmentation masks, and optical flow.These engines can generate large-scale datasets from object and environment models.
  • Game-Engine Labels: Networks trained only on synthetic images cannot be directly applied to real-world images because of the synthetic-to-real domain gap.The gap must be explicitly bridged for synthetic semantic supervision to transfer to real images.
  • Game-Engine Labels: Adversarial feature-space adaptation can reduce differences between real and synthetic domains while training on synthetic surface normals, depth, and instance contours.The approach uses a discriminator to minimize feature-space domain differences.
  • Hard-Coded Programs: Hard-coded detectors generate labels through a two-step process: label generation from images or videos, followed by ConvNet training on those labels.Examples include foreground segmentation, edge detection, and relative-depth prediction.
  • Hard-Coded Programs: Hard-coded semantic labels can directly drive semantic feature learning, but they are usually noisy and require specific handling.The supervision signal is semantically targeted, unlike many pretext tasks that impose semantic learning indirectly.

7 VIDEO FEATURE LEARNING

Video self-supervised learning uses generation, temporal context, free semantic labels, and cross-modal signals across 2DConvNets, 3DConvNets, and recurrent architectures. Video prediction and colorization exploit temporal dynamics and coherence, while temporal-order methods can require costly optical-flow-based preparation.

  • Overview: Self-supervised video methods are categorized into generation-based, context-based, free semantic label-based, and cross-modal approaches.Video features may be learned with 2DConvNets, 3DConvNets, or LSTM-based combinations, with 2DConvNets also extracting image features.
  • Generation-Based Methods: Video generation learns features without human labels by generating videos, colorizing frames, or predicting future frames from video data.In these tasks, the video itself commonly supplies the pseudo-training target.
  • Video Generation: VideoGAN uses separate streams to model static background regions and moving objects, while its discriminator distinguishes real-world from generated videos.The discriminator’s parameters can be transferred to downstream tasks after training on unlabeled videos.
  • Video Colorization: Video colorization uses temporal color coherence, either by copying colors from a reference RGB frame or by encoding and decoding grayscale video clips with 3D convolutions.The reference-frame approach learns connections between the RGB reference and grayscale target frame.
  • Open Limitations: Video color coherence is a strong supervision signal, but only a few studies had used it for self-supervised video feature learning.The survey identifies further study of color coherence as an open direction.
  • Video Prediction: Video prediction uses encoder-decoder models, often with LSTMs, to model spatial and temporal features and generate future frame sequences.MCnet uses content and motion encoders within an encoder-decoder convolutional architecture.
  • Video Prediction: Video prediction features can transfer to other tasks, but their generalization ability has not been studied; generated sequences are commonly evaluated with SSIM and PSNR.The reported evaluation compares generated frame sequences with ground-truth sequences.
  • Temporal Context: Temporal-order verification samples frames using optical-flow magnitude, shuffles them, and trains a network to identify whether the input order is correct.Optical-flow computation makes dataset preparation expensive and slow, motivating more efficient methods.

7.3 Cross Modal-based Learning

Cross-modal self-supervision learns video features from correspondences among visual, motion, audio, and egomotion signals. These tasks use automatically available relationships to train networks without human annotations.

  • RGB and Optical Flow: Optical flow supplies motion information complementary to RGB appearance for learning general video features.Pretext tasks include optical-flow estimation and RGB–optical-flow correspondence verification.
  • RGB and Optical Flow: RGB–optical-flow verification uses separate feature extractors and requires the network to capture mutual information between modalities.The inputs are judged as corresponding or non-corresponding.
  • Visual and Audio: Visual-audio correspondence learning uses vision and audio subnetworks, pairing same-video, same-time samples as positives and cross-video samples as negatives.The audio input is the Fast Fourier Transform of the video audio.
  • Visual and Audio: Jointly processing visual and audio inputs enabled the two networks to obtain good downstream performance.The pretext task trains both modalities together.
  • Visual and Egomotion: Egomotion-based methods exploit correspondence between visual signals and motor signals collected by self-driving cars.Two video frames are related using rotation and translation labels derived from odometry data.
  • Visual and Egomotion: Ego-motor signals provide accurate supervision for visual feature learning and can also support unsupervised depth and ego-motion learning.The resulting networks can be transferred to downstream tasks.

8 PERFORMANCE COMPARISON

The survey evaluates self-supervised features through downstream image and video tasks. Image features approach supervised performance on detection and segmentation, whereas video features remain substantially behind supervised models.

  • Image Feature Learning: Image classification is evaluated on ImageNet and Places using linear classifiers over frozen AlexNet convolutional features.Most methods pre-train on ImageNet without category labels.
  • Image Feature Learning: DeepCluster achieves the best self-supervised image-classification performance on both ImageNet and Places, while remaining below label-supervised models.Features from different layers benefit from self-supervised pretext-task training.
  • Image Feature Learning: Self-supervised models perform within less than 3% of supervised models on PASCAL VOC object detection and semantic segmentation.The comparison also includes image classification, using ImageNet-label supervision as the pre-training baseline.
  • Video Feature Learning: Video features are evaluated by fine-tuning for human action recognition on UCF101 and HMDB51.The comparison covers methods with varied pre-training datasets and network architectures.
  • Video Feature Learning: The best self-supervised UCF101 fine-tuning result is less than 66%, compared with more than 84% for a Kinetics-label supervised model.The survey therefore identifies a substantial video-learning performance gap.
  • Summary: The survey attributes lower 3DConvNet self-supervised performance partly to over-fitting from larger parameter counts and the complexity introduced by video’s temporal dimension.The best 3DConvNet result on UCF101 is more than 18% below the supervised model.
  • Summary: Downstream-task performance does not reveal what networks learn during self-supervised pre-training, motivating additional interpretability metrics.Network dissection is suggested as one such metric.

9 FUTURE DIRECTIONS

The survey highlights future directions for expanding self-supervised learning beyond current image- and video-focused approaches. These include synthetic and web data, richer sensor correspondences, spatiotemporal video learning, and multiple pretext tasks.

  • Learning Features from Synthetic Data: Synthetic data can provide millions of rendered images and videos with accurate pixel-level annotations for designing diverse pretext tasks.A remaining issue is bridging synthetic and real data.
  • Learning from Web Data: Web-collected images and videos offer large-scale raw data plus titles, keywords, reviews, and other associated information for training.Search engines can retrieve such data at negligible cost, but queries require careful curation.
  • Learning Spatiotemporal Features from Videos: More effective pretext tasks specifically designed for spatiotemporal video features are needed because 3DConvNet-based self-supervised learning remains insufficiently addressed.Image self-supervision is already close to supervised performance on some segmentation and detection tasks.
  • Learning with Data from Different Sensors: Correspondences among data from different sensors can provide additional supervision signals for self-supervised feature learning.Self-driving cars can collect large-scale multimodal data from cameras and other sensors.
  • Learning with Multiple Pretext Tasks: Combining multiple pretext tasks may produce more representative features because different tasks provide different supervision signals.Only a few studies had explored this direction at the time of the survey.

10 CONCLUSION

The survey reviews deep convolutional neural network methods for self-supervised image and video feature learning across architectures, tasks, algorithms, datasets, and evaluations. Its comparative tables summarize method properties and performance for the computer vision community.

  • Conclusion: Self-supervised image feature learning has brought self-supervised performance close to supervised performance on some downstream tasks.The paper presents this conclusion alongside its broad review and tabular comparisons.
  • Conclusion: The survey covers network architectures, pretext tasks, algorithms, datasets, performance comparisons, discussions, and future directions.Its tables organize the reviewed methods, datasets, and performance.
Loading 1902.06162v1…