PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Semi-supervised Learning

Xiangli Yang, Zixing Song, Irwin King, Zenglin Xu

arXiv:2103.00550v2cs.LG

TL;DR

Obtaining labeled data is difficult, expensive, or time-consuming, motivating methods that use both labeled and unlabeled data. The paper surveys deep semi-supervised learning through a five-category taxonomy and reviews representative methods, while highlighting open challenges including class imbalance and unsafe use of unlabeled data.

  • Problem

    Labeled samples often require substantial expert effort, while deep semi-supervised learning needs a comprehensive account of its growing methods and open problems.

  • Method

    The survey taxonomizes DSSL into five categories, reviews representative methods and model designs, and discusses challenges and future directions.

  • Results

    The survey provides a taxonomy of existing DSSL methods, groups them into five categories, and compares approaches within categories using illustrative figures.

  • Takeaways & Limitations

    The survey offers a structured overview of DSSL methods and identifies future research directions for this promising field.

  • Takeaways & Limitations

    Class imbalance remains an open challenge because common SSL frameworks often assume uniformly distributed class labels.

Abstract

from arXiv · show

Deep semi-supervised learning is a fast-growing field with a range of practical applications. This paper provides a comprehensive survey on both fundamentals and recent advances in deep semi-supervised learning methods from perspectives of model design and unsupervised loss functions. We first present a taxonomy for deep semi-supervised learning that categorizes existing methods, including deep generative methods, consistency regularization methods, graph-based methods, pseudo-labeling methods, and hybrid methods. Then we provide a comprehensive review of 52 representative methods and offer a detailed comparison of these methods in terms of the type of losses, contributions, and architecture differences. In addition to the progress in the past few years, we further discuss some shortcomings of existing methods and provide some tentative heuristic solutions for solving these open problems.

1 INTRODUCTION

Deep semi-supervised learning addresses the difficulty of obtaining labeled data by combining labeled and unlabeled data through deep neural networks. This survey organizes the field into five method categories, reviews representative approaches, and identifies open problems and future directions.

  • Motivation: Labeled samples are often difficult, expensive, or time-consuming to obtain because labeling may require expert effort.Medical tasks may require expensive machinery and analysis by multiple human experts.
  • Motivation: Deep semi-supervised learning uses deep neural networks to effectively utilize both labeled and unlabeled data.It extends the classic semi-supervised learning framework to deep learning settings.
  • Taxonomy: The survey classifies DSSL methods into generative, consistency regularization, graph-based, pseudo-labeling, and hybrid categories.The taxonomy is based on distinctive features of semi-supervised loss functions and model designs.
  • Survey scope: It provides detailed reviews of methods, background knowledge, model variants, standardized descriptions, and unified sketch maps.These materials are intended to help readers quickly grasp frontier ideas in DSSL.
  • Challenges: The paper identifies open problems and discusses future directions for deep semi-supervised learning.Its final section discusses challenges and tentative heuristic solutions.

2 BACKGROUND

The background defines semi-supervised learning, its settings and assumptions, and the optimization framework underlying methods that combine labeled and unlabeled data. It then introduces representative classical approaches and related learning paradigms.

  • Formulation: Semi-supervised learning trains models with both a small labeled subset and a large unlabeled subset, generally assuming L ≪ U.The notation distinguishes labeled data XL and unlabeled data XU.
  • Formulation: The SSL objective combines supervised loss, unsupervised loss, and per-example regularization.Ls may be cross-entropy, Lu is an unsupervised loss, and R may be a consistency or designed regularization term.
  • Learning settings: Transductive learning predicts the unlabeled training samples, whereas inductive learning generalizes to data beyond those samples.The distinction depends on whether test data are wholly available during training.
  • Assumptions: SSL relies on assumptions including self-training, co-training, generative-model, cluster, low-density-separation, and manifold assumptions.These assumptions specify when unlabeled data can provide useful information about labels or decision boundaries.
  • Assumptions: Under the cluster assumption, same-cluster points share labels and decision boundaries should lie in low-density regions.Unlabeled data can therefore help adjust the classification boundary.
  • Classical methods: Classical SSL methods include self-learning, co-training, generative models, transductive SVMs, and graph-based methods.These approaches respectively exploit confident predictions, multiple views, distributional models, low-density separation, or data geometry.

3 GENERATIVE METHODS

Deep generative semi-supervised methods use GANs and VAEs to exploit unlabeled data through distribution modeling, feature learning, generated samples, and latent-variable inference. The survey organizes GAN approaches by how they support SSL and reviews representative architectures and objectives.

  • GAN-based methods: GAN-based SSL methods use discriminator features, generated samples, inference models, or GAN-produced data to support classification.These four themes cover the principal ways GANs are adapted to semi-supervised learning.
  • GAN-based methods: Semi-supervised GAN objectives combine supervised classification with unsupervised losses from unlabeled and generated data.The classifier includes a supervised cross-entropy term alongside two unsupervised terms.
  • GAN-based methods: CatGAN replaces binary discrimination with K-category prediction and incorporates mutual information between examples and predicted class distributions.Its discriminator assigns samples to K categories rather than producing only a binary discriminator value.
  • GAN-based methods: CCGAN uses context-conditioned image in-painting, sharing discriminator representations with a classifier trained using cross-entropy.A binary mask removes part of an image, and the generator fills the missing region before discriminator-based feature sharing.
  • GAN-based methods: GoodBadGAN deliberately generates visually unrealistic samples to push the discriminator boundary between category data manifolds.It adds a penalty affecting high-density generated samples while leaving low-density samples unaffected.
  • VAE-based methods: VAE-based methods use latent representations, latent class variables, auxiliary variables, and skip connections to model data and infer missing labels.SSVAEs include M1, M2, and M1+M2; ADGM adds auxiliary variables, while SDGM uses skip connections and joint end-to-end training.

4 CONSISTENCY REGULARIZATION

Consistency regularization trains models to produce similar outputs under perturbations, using assumptions about smoothness or data manifolds. The survey reviews architectures that vary perturbations, target construction, averaging, and the level at which consistency is imposed.

  • Foundations: Consistency regularization assumes realistic perturbations should not change model outputs, encouraging a smooth manifold for the data.The approach is grounded in manifold or smoothness assumptions and uses unlabeled data in the consistency objective.
  • Foundations: Teacher-Student structures are common, but self-generated targets can create confirmation bias when target predictions are incorrect.Methods differ in how they construct the consistency target and can improve target quality through perturbation selection or other strategies.
  • Architectures: Ladder Network injects noise into hidden layers and applies denoising consistency losses across latent representations using skip connections.The clean and corrupted encoder paths share mappings, while denoising functions reconstruct representations at multiple layers.
  • Architectures: The Π Model compares predictions from two randomly perturbed passes of the same sample and minimizes their difference.Perturbations arise from data augmentation, dropout, and random maxpooling.
  • Architectures: Temporal Ensembling reduces Π Model’s repeated-forward overhead by accumulating predictions across epochs with an Exponential Moving Average.Mean Teacher instead averages model weights and applies consistency between student and teacher predictions.
  • Perturbation and constraint levels: Consistency methods impose constraints at input, network, or training-process levels, including adversarial dropout, additive or DropConnect perturbations, SWA, and EMA.VAdD compares random and adversarial dropout masks, while WCP strengthens vulnerable weights and connections against worst-case perturbations.

5 GRAPH-BASED METHODS

Graph-based methods construct similarity graphs to propagate labels from labeled to unlabeled samples while incorporating topology and node features. Deep variants use richer encoders, especially autoencoders and GNNs, to produce embeddings for downstream semi-supervised tasks.

  • Graph embedding: Graph embedding maps each node v to a lower-dimensional vector z_v while retaining measurements of graph proximity.The embedding function f_z maps nodes into R^d with d much smaller than the number of nodes.
  • Deep graph embeddings: Deep graph embedding methods comprise AutoEncoder-based and GNN-based approaches, replacing shallow node-ID lookup encoders with models that use node attributes.Recent GSSL research emphasizes deep models for the embedding term, whereas shallow methods include DeepWalk, LINE, and node2vec.
  • AutoEncoder-based methods: AutoEncoder-based methods reconstruct neighborhood vectors, with SDNE jointly preserving first- and second-order proximities through unsupervised and supervised components.SDNE uses an autoencoder to rebuild each node’s neighborhood and a Laplacian Eigenmaps penalty to keep related vertices close.
  • AutoEncoder-based methods: DNGR combines random surfing, PPMI calculation, and stacked denoising autoencoders to obtain node embeddings.Random surfing creates a stochastic similarity matrix, which is transformed into a PPMI matrix before encoding.
  • GNN-based methods: GNN message passing aggregates neighbor messages, combines them with prior hidden states, and applies nonlinear activation before producing final node embeddings for classification.GNN layers can be stacked, and the last layer’s output is used as the node embedding for downstream SSL tasks.
  • Graph-based methods: Graph-based methods represent samples as nodes and similarities as edges, enabling label inference from labeled nodes to unlabeled nodes.Their central goal is to exploit graph geometry and propagate label information through the constructed similarity graph.
  • GNN-based methods: GNN-based graph models propagate labels using both topological and feature knowledge while deep encoders generate more discriminative representations.A classifier is trained on labeled nodes and then applied to unlabeled nodes using the model’s final hidden states.

6 PSEUDO-LABELING METHODS

Pseudo-labeling methods exploit unlabeled data through confident predicted labels, disagreement among multiple learners, or self-training objectives. The section covers co-training variants, Tri-Net, pseudo-label training, entropy minimization, Noisy Student, and self-supervised auxiliary tasks.

  • Pseudo-labeling methods: Pseudo-labeling methods add high-confidence predictions for unlabeled samples to the training set, unlike consistency methods that rely on transformed-data constraints.Their two main patterns use disagreement across views or networks, and confident predictions to expand labeled data.
  • Disagreement-based models: Co-training assumes two different, complementary views, trains one classifier per view, and iteratively transfers the most confident labels between classifiers.Its key requirement is that each view is sufficient for training a good classifier while remaining complementary to the other.
  • Disagreement-based models: Deep Co-training adds a View Difference Constraint because standard co-training losses ensure prediction consistency but do not enforce sufficiently different views.The constraint uses adversarial examples generated by a generative model and minimizes cross-entropy between the resulting predictions.
  • Disagreement-based models: Tri-Net creates three labeled datasets through output smearing and trains three modules that pseudo-label unlabeled samples using maximum posterior probability.A shared module supplies features to three different modules, which are initialized using standard softmax cross-entropy losses.
  • Disagreement-based models: Disagreement-based methods train multiple learners that label unlabeled samples for one another and exploit their prediction differences.Two or three networks may be trained simultaneously to exchange labels during learning.
  • Self-training and related methods: Entropy Minimization encourages low-entropy predictions on unlabeled data, while Noisy Student trains a larger student from labeled and teacher-pseudo-labeled examples with augmentation.S4L instead uses self-supervised tasks such as predicting image rotations and exemplars to learn representations from image databases.
  • Self-training and related methods: Pseudo-label trains on labeled data with cross-entropy while using the model’s maximum-confidence predictions as labels for unlabeled samples.The objective balances supervised and unsupervised losses with a coefficient α(t).

7 HYBRID METHODS

Hybrid methods combine pseudo-labeling, consistency regularization, entropy minimization, and Mixup-style augmentation in diverse architectures. Representative methods illustrate distinct combinations for interpolation consistency, pseudo-label refinement, distribution alignment, noisy-label handling, and weak–strong augmentation.

  • Hybrid methods combine pseudo-labeling, consistency regularization, entropy minimization, and Mixup to improve semi-supervised learning performance.
  • Mixup-based methods: Mixup constructs virtual examples by linearly combining paired samples and their labels, enforcing corresponding linear label interpolations.
  • Mixup-based methods: ICT enforces prediction consistency between interpolated unlabeled inputs and interpolated predictions, motivated by low-density separation.
  • Representative hybrid methods: MixMatch combines consistency regularization and entropy minimization by generating, sharpening, and Mixup-augmenting pseudo-labels for unlabeled data.
  • Representative hybrid methods: ReMixMatch extends MixMatch with distribution alignment and augmentation anchoring, using weak and strong augmentations in its procedure.
  • Representative hybrid methods: DivideMix uses two networks and dynamic Gaussian Mixture Models to separate noisy training data into labeled and unlabeled sets before subsequent SSL training.
  • Representative hybrid methods: FixMatch combines consistency regularization with pseudo-labeling, accepting weak-augmentation predictions as labels only when confidence is high.

8 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies unresolved challenges involving theory, domain shift, noisy labels, imbalance, robustness, and the possibility that unlabeled data can harm performance. It proposes future work addressing these limitations through better analysis, domain knowledge, specialized learning methods, and safer SSL.

  • Theoretical analysis: Existing DSSL methods lack clear theoretical explanations for how augmentations, training procedures, and loss functions interact.A single supervised–unsupervised loss weight also treats all unlabeled instances equally despite differing suitability.
  • Incorporation of domain knowledge: When labeled and unlabeled distributions differ or model assumptions fail, using more unlabeled data can worsen performance.
  • Learning with noisy labels: Common SSL formulations assume accurate labeled data, leaving noisy-label learning as an open problem addressed by consistency, graph, and co-training approaches.
  • Imbalanced semi-supervised learning: Many SSL methods assume uniformly distributed class labels, although severe imbalance can bias models toward majority classes or ignore minority classes.
  • Robust semi-supervised learning: Robust SSL commonly uses consistency training with augmented unlabeled data, including adversarial perturbations and advanced augmentation strategies.
  • Safe semi-supervised learning: Empirical studies report that unlabeled data can degrade generalization below supervised-only performance, motivating methods that avoid significant degradation.

9 CONCLUSION

The survey organizes deep semi-supervised learning into five categories, compares approaches within categories using illustrative figures, and identifies challenges and future research directions.

  • The survey groups DSSL methods into generative, consistency regularization, graph-based, pseudo-labeling, and hybrid models.
  • It uses illustrative figures to compare differences among approaches within each category and discusses challenges requiring further study.
Loading 2103.00550v2…