Source-linked AI summary
A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends
Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, Dacheng Tao
TL;DR
SSL addresses the expense and scarcity of labeled data, while the relationships and evolution of its many variants remain insufficiently synthesized. This paper reviews SSL algorithms, applications, trends, and open questions, finding distinct strengths across contrastive and masked-image-modeling approaches. It concludes by organizing contemporary SSL research and identifying unresolved issues, including risks when model assumptions mismatch data structure.
Problem
Labeled data are expensive and time-consuming to obtain, while comprehensive studies connecting and tracing the evolution of SSL variants remain scarce.
Method
The paper conducts a survey of SSL algorithms, applications, research trends, and open questions, emphasizing mainstream visual methods and their connections with other learning paradigms.
Results
The review classifies mainstream visual SSL algorithms into context-based, generative, contrastive, and contrastive generative methods and compares their reported properties.
Takeaways & Limitations
The survey provides newcomers with an overall picture of contemporary SSL research and consolidates findings across algorithms, applications, and future directions.
Takeaways & Limitations
SSL performance can degrade when model assumptions fail to align with the underlying problem structure, and identifying such mismatches remains unresolved.
Abstract
from arXiv · showhide
Deep supervised learning algorithms typically require a large volume of labeled data to achieve satisfactory performance. However, the process of collecting and labeling such data can be expensive and time-consuming. Self-supervised learning (SSL), a subset of unsupervised learning, aims to learn discriminative features from unlabeled data without relying on human-annotated labels. SSL has garnered significant attention recently, leading to the development of numerous related algorithms. However, there is a dearth of comprehensive studies that elucidate the connections and evolution of different SSL variants. This paper presents a review of diverse SSL methods, encompassing algorithmic aspects, application domains, three key trends, and open research questions. Firstly, we provide a detailed introduction to the motivations behind most SSL algorithms and compare their commonalities and differences. Secondly, we explore representative applications of SSL in domains such as image processing, computer vision, and natural language processing. Lastly, we discuss the three primary trends observed in SSL research and highlight the open questions that remain. A curated collection of valuable resources can be accessed at https://github.com/guijiejie/SSL.
1 INTRODUCTION
The introduction motivates SSL as a response to costly labeling, surveys its growing research landscape, and positions this paper as a comprehensive review of algorithms, applications, and open questions.
- Motivation: Labeled data are often costly, arduous, or time-consuming to acquire because expert human annotation is required.The paper illustrates this challenge with web-user profiling and medical data.
- Motivation: SSL learns discriminative features from large quantities of unlabeled data using pseudo-labels generated without human annotations.After self-supervised pre-training, the acquired model can be transferred to downstream tasks.
- Motivation: SSL has demonstrated promising downstream results, narrowing the performance gap relative to supervised algorithms and producing generalizable features even from a single image.The cited example attributes this result to Asano et al.
- Research landscape: Approximately 18,900 SSL papers were published in 2021, motivating a timely survey to consolidate the rapidly expanding literature.The paper reports an average of 52 papers per day, or more than two per hour.
- Scope and contribution: The review differs from earlier application-specific, pre-2020, or contrastive-learning-focused surveys by emphasizing recent developments, especially in computer vision.It aims to address the need for a broader and more current account of SSL research.
2 ALGORITHMS
This section introduces SSL before explaining its pretext tasks and connections with other learning paradigms.
- Algorithms: The algorithms section begins with an introduction to SSL.
- Algorithms: It then explains the pretext tasks associated with SSL.
- Algorithms: The section also discusses how SSL integrates with other learning paradigms.
2.1 What is SSL?
SSL derives supervision from input data rather than manual labels, often by predicting hidden or transformed information through pretext tasks.
- Definition: SSL generates labels intrinsically from input examples by exploiting relationships between data components or different views.The derived labels come directly from the data examples.
- Definition: SSL uses freely derived labels as supervision instead of manually annotated labels.
- Definition: Autoencoders can be treated as SSL algorithms because their output labels correspond to the input data itself.The paper notes applications including dimensionality reduction and anomaly detection.
- Pretext tasks: A common SSL formulation predicts an unknown portion of the input from the available portions, including future, invisible, masked, or corrupted content.
- Pretext tasks: Pretext tasks are surrogate or proxy tasks that generate robust pre-trained models rather than directly solving the primary objective.Examples include rotation prediction and instance discrimination.
2.2 Pretext tasks
SSL pretext tasks derive training signals from data itself and are organized into context-based methods, contrastive learning, generative algorithms, and contrastive generative methods. These approaches include transformation recognition, instance discrimination, embedding decorrelation, variance preservation, and masked image modeling, with differing design choices and trade-offs.
- Overview: Pretext tasks derive supervised signals from the data itself while learning features that enable networks to solve the formulated task.The survey frames self-supervision as training on labels generated intrinsically from unlabeled examples.
- Overview: The survey groups SSL pretext tasks into four categories: context-based methods, contrastive learning, generative algorithms, and contrastive generative methods.Generative algorithms primarily refer to masked image modeling methods in this paper.
- Context-based methods: Context-based methods exploit spatial or local-global contextual relationships through tasks such as rotation recognition, jigsaw reconstruction, and colorization.Rotation predicts one of four geometric transformations; jigsaw reconstructs randomly rearranged image patches; colorization predicts ab channels from image lightness.
- Contrastive learning: Self-distillation and feature-decorrelation methods remove the need for negative examples while preserving positive-view consistency or reducing redundancy between feature components.BYOL uses online and target Siamese networks, while Barlow Twins encourages similar embeddings for distorted views and minimizes component redundancy.
- Contrastive learning: Negative-example contrastive learning treats views from the same instance as positives and views from different instances as negatives, encouraging proximity for positives and separation for negatives.MoCo stores momentum-encoder features in a queue, whereas SimCLR uses negatives from the current mini-batch.
- Generative and hybrid methods: VICReg explicitly preserves embedding variance to prevent collapse, while MIM predicts masked image content and naive CL–generative combinations may underperform generative baselines.VICReg encourages batch variance to meet a threshold for every dimension; MAE directly reconstructs original pixels, unlike BEiT’s token targets.
2.3 Combinations with other learning paradigms
The survey examines how SSL combines with semi-supervised, adversarial, multimodal, and other learning paradigms, while highlighting its evolving methods and applications. It also describes test-time training as an adaptation strategy under distribution shifts.
- Combinations with other learning paradigms: SSL has been combined with GANs by adding rotation prediction, incorporating rotation invariance into the generation process.SS-GAN combines the GAN objective with a rotation task during training.
- Combinations with other learning paradigms: S4L jointly optimizes categorization loss on labeled data and self-supervised loss on unlabeled data.The labeled and unlabeled objectives may use cross-entropy and a rotation task, respectively.
- Combinations with other learning paradigms: SSL can support semi-supervised learning through auxiliary tasks or pretraining on unlabeled data followed by fine-tuning on labeled data.SimCLR is cited as an example of the latter approach.
- Combinations with other learning paradigms: Multiview contrastive learning treats different views of one sample as positives and can train encoders using channel-split image views.The survey also describes adversarial unsupervised and semi-supervised variants with classifiers and cross-entropy loss.
- Combinations with other learning paradigms: Test-time training converts each unlabeled test example into a self-supervised problem, updating parameters before prediction.The survey states that TTT improves the bias-variance tradeoff under distribution shifts.
- Combinations with other learning paradigms: The survey presents SSL as dynamically developing through combinations with clustering, transfer learning, graph networks, reinforcement learning, and multimodal learning.CLIP is identified as an example combining contrastive learning with multimodal learning.
3 APPLICATIONS
SSL has expanded from early recognition and object-extraction settings to applications across computer vision, image processing, video, medical imaging, and remote sensing. Video methods exploit temporal structure such as frame order and playback properties.
- Applications: SSL has been applied across computer vision, natural language processing, medical image analysis, and remote sensing.The survey describes this expansion from early vowel-class recognition and object-extraction tasks.
- Applications: Image-processing and computer-vision applications include inpainting, human parsing, scene deocclusion, segmentation, monocular vision, re-identification, and visual odometry.The survey also lists scene-flow estimation, knowledge distillation, and optical-flow-related applications.
- Applications: SSL is widely used for video representation learning and video retrieval.The survey treats video as a distinct application domain within SSL.
- Applications: Video SSL can use frame order, playback direction, playback speed, and future-prediction information as temporal signals.These signals provide alternative forms of temporal information for self-supervised learning.
- Applications: Odd-one-out learning identifies a video subsequence with incorrect temporal frame order among related subsequences.The method is presented as a self-supervised CNN pre-training approach.
3.2 NLP
The survey describes NLP SSL as historically dominated by generative language-model objectives, while contrastive pretraining has shown benefits for zero-shot and few-shot performance. Automatic text augmentation remains difficult because small token changes can alter sentence meaning.
- NLP: Continuous bag-of-words and continuous skip-gram are generative self-supervised approaches based on language models and maximum likelihood estimation.The survey identifies them as pioneering SSL methods for word embeddings.
- NLP: Supervised contrastive pretraining enables zero-shot prediction of unseen text classes and enhances few-shot performance.The survey also reports that task-agnostic contrastive pretraining improves language modeling.
- NLP: Generative SSL continues to dominate NLP, including methods such as BERT, GPT, and recent trillion-scale language models.This dominance follows the field’s long reliance on generative language-model objectives.
- NLP: Automatic textual augmentation remains challenging because changing a single token can reverse a sentence’s meaning.This constraint complicates the use of contrastive methods in NLP.
- NLP: SSL is especially relevant in medicine and remote sensing because labeled data are limited while large quantities of unlabeled data exist.Applications include medical image segmentation, 3D medical image analysis, and remote sensing.
4 PERFORMANCE COMPARISON
The review compares SSL methods through downstream-task evaluation and summarizes distinct strengths of contrastive learning (CL) and masked image modeling (MIM), including accuracy, fine-tuning, and resource use.
- Evaluation setup: Downstream-task performance is the conventional measure of SSL feature quality, but it does not reveal what representations learned during pretraining.The review also mentions network dissection and other unsupervised metrics for interpretability.
- Comparative findings: Linear probe performance of CL models typically surpasses that of other SSL algorithms.The review attributes this to CL producing latent spaces that separate distinct categories and cluster similar ones.
- Comparative findings: MIM-pretrained models can usually be fine-tuned to achieve superior performance, whereas CL-based models provide comparatively limited downstream assistance.The review associates the discrepancy primarily with greater susceptibility of CL-based models to overfitting.
- Comparative findings: CL methods often require momentum encoders, memory queues, and multi-crop, increasing computing, storage, and communication demands.MIM methods use resources more efficiently, possibly because they do not require example interactions, and can therefore scale models more easily.
5 CONCLUSIONS, FUTURE TRENDS, AND OPEN QUESTIONS
The survey organizes contemporary SSL research around algorithms, applications, and future trends, while identifying unresolved questions about theory, pretext-task design, unlabeled data, modality integration, and method selection.
- Conclusions: The review classifies mainstream visual SSL algorithms into context-based, generative, contrastive, and contrastive generative methods.It also examines relationships between SSL and other learning paradigms and surveys applications and future directions.
- Main trends: SSL theory remains fragmented, motivating efforts to understand and unify different algorithms through representation distributions, spectral decomposition, and related analyses.The cited work includes analyses of negative samples, distillation-based non-collapse, and duality with covariance regularization.
- Main trends: Automatically designing an optimal pretext task for a fixed downstream task remains insufficiently resolved and requires further theoretical investigation.The review cites pixel-to-propagation consistency and dense contrastive learning as proposed approaches.
- Main trends: A unified SSL paradigm across modalities is suggested by progress in masked image modeling, masked language modeling, and transformer architectures spanning vision and language.The review notes that NLP has advanced further in leveraging SSL models and may offer approaches for computer vision.
- Open problems: Open questions include whether SSL can consistently benefit from vast amounts of unlabeled data and how to identify its theoretical inflection point.The review also asks how SSL and multimodality learning can be integrated into a robust, comprehensive model.
- Open problems: No universally optimal SSL algorithm exists because suitable selection depends on the specific problem structure.The review proposes investigating a checklist to help users choose methods for particular circumstances.
- Open problems: The assumption that unlabeled data invariably improves outcomes requires scrutiny because performance can degrade when model assumptions do not match the underlying problem structure.The review gives heavily overlapping Cauchy distributions as an example and states that identifying such mismatches remains unresolved.
APPENDIX
The appendix connects contrastive learning to established analytical frameworks and discusses its relationship with supervised learning, computational complexity, and SSL research resources.
- Connections to Existing Methods: Contrastive learning can be formulated as a max-min problem whose maximization component is equivalent to PCA in deep linear networks.The max function increases contrast between feature representations, while the min function weights similarly represented pairs.
- Connections to Existing Methods: Contrastive-learning representations correspond to embeddings of a positive-pair graph in spectral clustering.The population augmentation graph links augmented data when they originate from the same original example.
- Connections to Existing Methods: Spectral decomposition constructs the representation matrix, and a linear transformation recovers the corresponding feature extractor by minimizing an unconventional contrastive loss.Each matrix row represents an example before the feature extractor is retrieved.
- Connections to Existing Methods: When representation dimensionality exceeds the maximum number of disjoint subgraphs, linear classification using the learned representations is guaranteed to yield minimal error.The result applies to representations derived from the spectral-clustering connection.
- Connections to Supervised Learning: Contrastive pretraining is effective for categorization tasks, but its effectiveness may vary across other task domains, motivating further study alongside supervised learning.The cited discussion asks whether contrastive pretraining can surpass traditional supervised-learning accuracy.
- Computational Complexity: The paper notes that computational complexity and memory consumption primarily come from the neural network rather than SSL components, including cross-correlation computation in Barlow Twins.SSL methods are categorized by architecture for complexity analysis.