PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Supervised and Unsupervised Deep Visual Learning: A Survey

Yanbei Chen, Massimiliano Mancini, Xiatian Zhu, Zeynep Akata

arXiv:2208.11296v1cs.CVcs.AIcs.LG

TL;DR

Costly and incomplete labeling motivates learning visual representations from abundant unlabeled data. The survey unifies SSL and UL through a taxonomy and analysis of representative algorithms, applications, and emerging trends, concluding that these paradigms support visual representation learning while leaving cross-distribution benefits of unsupervised pretraining open.

  • Problem

    Large-scale manual labeling is costly and may weaken generalization, motivating scalable visual learning from unlabeled data.

  • Method

    The survey reviews representative deep SSL and UL algorithms, organizes them with a unified taxonomy, and connects their rationales across applications and learning scenarios.

  • Results

    The survey provides a comprehensive synthesis spanning SSL and UL methods, visual applications, self-supervised learning trends, and open challenges.

  • Takeaways & Limitations

    SSL and UL offer complementary paradigms for leveraging unlabeled visual data, with self-supervision enabling task-agnostic training without task-specific labels.

  • Takeaways & Limitations

    How unsupervised pretraining on differently distributed unlabeled data benefits specific downstream tasks remains an open research question.

Abstract

from arXiv · show

State-of-the-art deep learning models are often trained with a large amount of costly labeled training data. However, requiring exhaustive manual annotations may degrade the model's generalizability in the limited-label regime. Semi-supervised learning and unsupervised learning offer promising paradigms to learn from an abundance of unlabeled visual data. Recent progress in these paradigms has indicated the strong benefits of leveraging unlabeled data to improve model generalization and provide better model initialization. In this survey, we review the recent advanced deep learning algorithms on semi-supervised learning (SSL) and unsupervised learning (UL) for visual recognition from a unified perspective. To offer a holistic understanding of the state-of-the-art in these areas, we propose a unified taxonomy. We categorize existing representative SSL and UL with comprehensive and insightful analysis to highlight their design rationales in different learning scenarios and applications in different computer vision tasks. Lastly, we discuss the emerging trends and open challenges in SSL and UL to shed light on future critical research directions.

1 INTRODUCTION

Supervised vision models depend on costly labels and may generalize poorly to unseen distributions. This survey examines semi-supervised and unsupervised learning as unified ways to exploit abundant unlabeled visual data.

  • Manual labeling is expensive, time-consuming, and sometimes restricted, while annotator bias can weaken supervised models’ generalization.
  • SSL and UL leverage abundant unlabeled visual data to address visual variation that large labeled sets may not cover.
  • SSL combines sparsely labeled data with auxiliary unlabeled data, whereas UL learns from unlabeled data without task-relevant label supervision.
  • The survey organizes SSL and UL around unified objectives involving supervised and unsupervised loss terms, with UL using only the unsupervised term during pretraining.
  • It proposes a comprehensive taxonomy covering representative algorithms, applications, design connections, emerging trends, and open challenges.

2 SEMI-SUPERVISED LEARNING (SSL)

Semi-supervised learning exploits large unlabeled datasets alongside sparse labels and has been applied across visual recognition and other application domains. The survey defines SSL and provides a taxonomy of its deep learning methods.

  • SSL exploits large unlabeled datasets together with sparsely labeled data.
  • SSL supports visual tasks including image classification, retrieval, detection, semantic segmentation, and pose estimation.
  • The survey defines the SSL problem setup and analyzes existing semi-supervised deep learning methods through a dedicated taxonomy.

2.1 The Problem Setting of SSL

SSL trains a deep neural network with limited labeled samples and abundant unlabeled samples. Its evaluation compares against supervised baselines and examines whether additional unlabeled data increases the improvement margin.

  • SSL uses limited labeled samples, each assigned to one of K class labels, together with a large unlabeled sample set.
  • The SSL objective combines supervised and unsupervised losses, often increasing the unsupervised weight through a training-iteration ramp-up.
  • SSL evaluation requires outperforming a labeled-only supervised baseline and increasing the improvement margin as the unlabeled-data proportion grows.
  • SSL commonly relies on smoothness and manifold assumptions about the underlying data structure formed by clusters or graphs.

2.2 Taxonomy on SSL Algorithms

The survey organizes semi-supervised deep learning into five representative families, distinguished by how they exploit unlabeled data. These families impose consistency, infer pseudo labels, propagate graph structure, model data distributions, or use self-supervised objectives.

  • Taxonomy on SSL Algorithms: Semi-supervised deep learning is divided into consistency regularization, self-training, graph-based regularization, deep generative models, and self-supervised learning.The taxonomy covers five representative families of methods.
  • Graph-based regularization: Graph-based regularization represents samples as graph nodes and propagates labels or imposes smoothness using relationships among neighboring samples.Regularization can operate in feature or prediction space.
  • Consistency regularization: Consistency regularization enforces similar outputs under variations in the input space or model space.Variations may arise from data transformations, noise, perturbations, model-weight changes, or ensembling.
  • Self-training: Self-training learns from unlabeled samples by assigning or deriving targets from confident predictions, interacting classifiers, or teacher models.Its representative lines are entropy minimization, co-training, and distillation.
  • Deep generative models: Deep generative models integrate unsupervised distribution modeling with supervised learning, chiefly through variational auto-encoders and generative adversarial networks.GAN-based SSL uses a discriminator to classify labeled samples and generated samples in an auxiliary class.
  • Self-supervised learning: Self-supervised SSL avoids task-specific label supervision during training and can serve as unsupervised pre-training or an auxiliary task alongside supervised learning.This gives self-supervision a task-agnostic role within SSL.

3 UNSUPERVISED LEARNING (UL)

Unsupervised learning learns visual representations from unlabeled data, with models evaluated through feature classification or transfer to downstream tasks. The survey organizes methods into pretext tasks, discriminative models, and generative models.

  • 3.1 The Problem Setting of UL: UL learns representations from only unlabeled data, then supports fine-tuning for generalization on downstream tasks.Its motivation includes costly labels, distribution gaps across tasks, and the need for strong model initialization.
  • 3.1 The Problem Setting of UL: UL methods are evaluated using linear classification on frozen features or fine-tuning for downstream objectives.The second protocol initializes a model for tasks such as object detection before task-specific training.
  • 3.2 Taxonomy on UL Algorithms: The survey groups unsupervised deep learning into pretext tasks, discriminative models, and generative models.Pretext and discriminative models construct proxy supervision, whereas generative models explicitly model the data distribution.
  • 3.2.1 Pretext Tasks: Pretext tasks learn representations by predicting task-agnostic properties through pixel-level reconstruction or instance-level transformation prediction.Pixel-level tasks include inpainting, denoising, and colorization; instance-level tasks include rotation and patch-order prediction.
  • 3.2.1 Pretext Tasks: Pretext-task self-supervision can produce transferable initialization for unseen tasks and act as a regularizer in semi-supervised learning or domain generalization.The surveyed objectives capture spatial context, structured image correlations, or semantic content without directly matching downstream objectives.
  • 3.2.2 Discriminative Models: Discriminative models enforce invariance to task-irrelevant variation through instance discrimination or group-level deep clustering.Contrastive learning pulls positive pairs together and pushes negative pairs apart, while clustering memberships serve as pseudo labels.

4 DISCUSSION ON SSL AND UL

The survey connects SSL and UL through shared learning assumptions and discusses their applications across computer vision tasks.

  • 4 DISCUSSION ON SSL AND UL: SSL and UL are connected through common assumptions and their use across diverse computer vision applications.The discussion covers shared learning assumptions and applications in different vision tasks.

4.1 The learning assumptions shared by SSL and UL

SSL and UL can be unified through smoothness assumptions that impose consistency across transformed or neighboring samples, or derive targets from global structure. Their shared rationales distinguish methods while training protocols and distribution relationships remain important differences.

  • Local smoothness: Local smoothness assumes transformed versions of a sample or nearby latent-space samples share the same class.Unsupervised losses compare model outputs using metrics such as mean square error or contrastive loss.
  • Local smoothness: Consistency regularization enforces predictive agreement across different views of the same sample, while graph-based methods propagate labels from neighboring samples.These mechanisms adapt local smoothness to SSL and UL settings.
  • Global smoothness: Global smoothness assigns unlabeled samples targets from global structures and decision boundaries expected to lie in low-density regions.Targets can include cluster membership or the most confident predicted class, with inconsistency penalized by losses such as cross-entropy.
  • Global smoothness: Self-training techniques generally derive unlabeled targets from class decision boundaries discovered during model training.Examples include confidence-based pseudo-labeling, co-training, and distillation.
  • Shared rationale: SSL and UL methods broadly impose consistency across transformations, smoothness between neighbors, or learning targets derived from global structure.The survey presents these rationales as a common taxonomy for representative methods.
  • Training protocols: UL commonly separates unlabeled pre-training from labeled downstream fine-tuning, while SSL jointly trains with labeled and unlabeled data.UL can use datasets with different distributions, but how this affects downstream generalization remains open.

4.2 Applied SSL and UL in Visual Recognition

The survey extends SSL and UL learning rationales beyond image classification to segmentation, detection, and domain adaptation. These applications adapt consistency, self-training, contrastive learning, and generative techniques to task-specific outputs and distribution shifts.

  • Semantic segmentation: Semi-supervised semantic segmentation reduces pixel-wise annotation requirements by combining a small labeled set with unlabeled training images.Methods adapt consistency regularization, self-training, and contrastive learning to pixel-level representations or predictions.
  • Semantic segmentation: Pixel-level consistency methods compare predictions for clean and perturbed images or feature representations.CutOut, CutMix, and activation noise provide perturbations while preserving consistency on corresponding regions.
  • Semantic segmentation: Self-training for segmentation propagates pseudo segmentation maps on unlabeled images using pre-trained or co-trained teacher models.A representative procedure assigns pixel-wise pseudo-labels and then retrains the model.
  • Object detection: Semi-supervised object detection trains on labeled images and completely unlabeled images to improve generalization without bounding-box or class-label annotations.Its main SSL formulations are consistency regularization and self-training.
  • Object detection: Detection consistency methods enforce agreement for both classification and localization across augmented views.They may need to re-estimate bounding-box locations after transformations such as flipping.
  • Object detection: Self-training for detection generates pseudo bounding boxes and class labels with a teacher model, but pseudo-annotations may be noisy.Interactive self-training progressively refines pseudo-label quality.
  • Unsupervised domain adaptation: UDA addresses labeled source and unlabeled target data from different domains using consistency, self-training, distribution alignment, and pixel-level translation.Generative models provide a complementary way to mitigate domain discrepancy through cross-domain image-to-image translation.

5 EMERGING TRENDS AND OPEN CHALLENGES

Emerging SSL and UL research addresses open-set, incremental, and multimodal learning from unlabeled data. The central challenges concern irrelevant or changing data and assumptions about semantic compatibility across modalities.

  • Open-set learning: Open-set SSL relaxes the closed-set assumption because unlabeled data may contain unseen or task-irrelevant OOD classes.Such samples can cause significant performance degradation through catastrophic error propagation.
  • Open-set learning: Open-world SSL uses sample-specific selection strategies to discount harmful unlabeled OOD samples.The setup recognizes that unlabeled samples are not uniformly beneficial.
  • Incremental learning: Incremental learning from unlabeled data updates existing models without access to previous training data.Regularization, parameter-importance estimation, and knowledge distillation are used to reduce catastrophic forgetting.
  • Multi-modal learning: Multimodal unsupervised learning combines vision with text or audio to create self-supervision signals through joint modeling, correspondence prediction, or representation alignment.Some approaches align modalities in shared or hierarchical latent spaces.
  • Multi-modal learning: Multimodal learning often assumes that modalities are semantically correlated, but semantic gaps can degrade model performance.Learning from multimodal unlabeled data with cross-modal semantic gaps remains an open challenge.

6 CONCLUSION

The survey positions SSL and UL as scalable approaches for visual representation learning with limited or no manual supervision. It unifies their definitions and taxonomies, relates methods across scenarios and tasks, and identifies unresolved research challenges.

  • Conclusion: The survey introduces unified problem definitions and taxonomies for recent SSL and UL deep learning methodologies.It correlates methods across learning scenarios and visual classification applications.
  • Conclusion: Its discussions of emerging trends and open challenges aim to clarify the status quo and inspire solutions for the limited-label regime.The stated scope centers on visual representation learning with unlabeled data.
Loading 2208.11296v1…