PaperlayerView exact revision on arXiv

Source-linked AI summary

An Overview of Deep Semi-Supervised Learning

Yassine Ouali, Céline Hudelot, Myriam Tami

arXiv:2006.05278v2cs.LGstat.ML

TL;DR

Deep semi-supervised learning addresses the resource-intensive creation of large labeled datasets. This paper reviews the field’s assumptions, evaluation considerations, and dominant deep-learning approaches, concluding that deep SSL may approach fully supervised performance while evaluation and training limitations remain important.

  • Problem

    Large labeled datasets require substantial resources, time, and effort, motivating methods that reduce the amount of annotation needed for deep learning.

  • Method

    The paper provides a comprehensive overview of deep SSL, covering its assumptions, evaluation considerations, and dominant approaches across consistency, generative, graph-based, and holistic methods.

  • Results

    The review concludes that deep SSL methods could approach fully supervised performance and have broad applications across systems and learning paradigms.

  • Takeaways & Limitations

    Deep SSL offers a data-efficient framework for using unlabeled data, provided that relevant structural assumptions hold and evaluation reflects real-world conditions.

  • Takeaways & Limitations

    SSL evaluation needs to address class-distribution mismatch because unlabeled data can have different class distributions from labeled data in real-world applications.

Abstract

from arXiv · show

Deep neural networks demonstrated their ability to provide remarkable performances on a wide range of supervised learning tasks (e.g., image classification) when trained on extensive collections of labeled data (e.g., ImageNet). However, creating such large datasets requires a considerable amount of resources, time, and effort. Such resources may not be available in many practical cases, limiting the adoption and the application of many deep learning methods. In a search for more data-efficient deep learning methods to overcome the need for large annotated datasets, there is a rising research interest in semi-supervised learning and its applications to deep neural networks to reduce the amount of labeled data required, by either developing novel methods or adopting existing semi-supervised learning frameworks for a deep learning setting. In this paper, we provide a comprehensive overview of deep semi-supervised learning, starting with an introduction to the field, followed by a summarization of the dominant semi-supervised approaches in deep learning.

1 Introduction

Semi-supervised learning combines a small labeled set with abundant unlabeled data to improve deep models, but depends on assumptions about data structure and careful evaluation. The review organizes SSL methods into broad methodological categories and discusses their learning paradigms and assumptions.

  • Main assumptions: SSL is effective only under assumptions that align the problem structure with the model, including smoothness, clustering, and low-dimensional manifold structure.A mismatch between assumptions and problem structure can degrade classification performance.
  • Semi-supervised learning: SSL uses labeled and unlabeled examples, typically with far fewer labeled instances, to train models that outperform training on labeled data alone.The labeled portion may comprise only 1–10% of the dataset.
  • Semi-supervised learning: Unlabeled data can reveal distribution structure, such as low-density separation between classes, reducing ambiguity in the learned decision boundary.The two-moons example illustrates how unlabeled points can inform the boundary between classes.
  • SSL methods: The review groups deep SSL approaches into consistency regularization, proxy-label methods, generative models, and graph-based methods, alongside entropy minimization.Consistency methods enforce similar predictions under realistic perturbations, whereas graph methods propagate labels through node similarity.
  • Learning paradigms: SSL algorithms follow transductive or inductive paradigms, depending on whether they target only unlabeled instances observed during training or generalize to unseen instances.The transductive setting is mainly associated with graph-based algorithms in the supplied passage.
  • Evaluation: Evaluation should address class-distribution mismatch because labeled and unlabeled data may differ in real-world applications even when standard datasets do not expose that discrepancy.The review identifies this mismatch as relevant to real-world adoption of SSL.

2 Consistency Regularization

Consistency regularization trains models to produce similar predictions for related versions of unlabeled data, encouraging decision boundaries away from data-dense regions. Deep implementations include Ladder Networks and the simpler Π-Model, which combine unsupervised consistency or reconstruction losses with supervised loss when labels are available.

  • Consistency regularization: Consistency regularization favors functions with consistent predictions for similar data points and pushes decision boundaries away from unlabeled examples.The regularized model minimizes cost over a manifold around each data point rather than only at the points themselves.
  • Ladder Networks: Ladder Networks use clean and corrupted encoders plus a denoising decoder to reconstruct clean activations from corrupted representations.Gaussian noise is injected after batch normalization in the corrupted encoder.
  • Ladder Networks: The Ladder Network unsupervised loss is a weighted sum of layer-wise mean-squared errors between clean and reconstructed activations.The reconstruction errors are computed across layers from the input to the final layer.
  • Ladder Networks: For labeled inputs, Ladder Networks add a supervised cross-entropy term to the unsupervised reconstruction loss.The supervised term uses the provided label for the input example.
  • Ladder Networks: The Γ-Model reduces Ladder Network computation by omitting the decoder and comparing the two outputs directly with mean-squared error.This variant sets λ_l = 0 for layers below the final layer and approximately avoids the tripled computation of the full Ladder Network.
  • Π-Model: The Π-Model uses stochastic regularization, such as augmentation and dropout, to produce two predictions from the same network and minimize their distance.A supervised cross-entropy loss is also computed when the input is labeled.

2.2 Pi-Model

The Π-Model enforces prediction consistency across stochastic evaluations of the same input, combining unsupervised consistency loss with supervised loss for labeled examples. Temporal Ensembling stabilizes these targets by aggregating prior predictions, while trading memory for faster training and more stable results.

  • Π-Model: The Π-Model uses one network to produce predictions for corrupted and uncorrupted versions of an input, minimizing their distance and adding supervised cross-entropy for labeled data.Dropout and data augmentation provide stochasticity while typically preserving the desired prediction.
  • Temporal Ensembling: Temporal Ensembling replaces single-evaluation targets with an aggregation of previous predictions to reduce target instability and extract more training signal from unlabeled examples.The aggregated target is updated through an exponentially moving average.
  • Temporal Ensembling: The temporal-ensembling unsupervised loss is the MSE between the current prediction and aggregated target, with supervised loss added for labeled inputs.A weighting function gradually increases the unsupervised-loss contribution from zero to a fixed value.
  • Temporal Ensembling: Temporal Ensembling needs one forward pass for current predictions while maintaining an EMA of each example’s predictions, making training faster and targets more stable.The method requires storing aggregate predictions for all training examples, which can become memory-intensive for large datasets and dense tasks.

2.4 Mean teachers

Mean Teacher separates target generation from student learning by using a teacher whose weights are an EMA of the student’s weights. This design aims to incorporate learned signals faster and avoid confirmation bias while retaining supervised and consistency losses.

  • Motivation: Mean Teacher is proposed to incorporate learned information faster and avoid confirmation bias associated with using the same model as both teacher and student.Earlier approaches can reuse potentially misclassified targets, especially when unsupervised loss dominates supervised loss.
  • Mean Teacher: Mean Teacher uses a separate teacher model to generate consistency-training targets, while the student minimizes supervised loss on labeled examples and consistency loss on unlabeled examples.Both models receive injected noise during each training iteration.
  • Mean Teacher: Unlike the Π-Model and Temporal Ensembling, Mean Teacher defines teacher weights as an EMA of successive student weights.The Π-Model uses identical student and teacher models, whereas Temporal Ensembling averages successive predictions.
  • Loss: The teacher’s targets are used in the sum of supervised and unsupervised losses for each input.The teacher supplies the targets specifically for the unsupervised component.

2.5 Dual Students

Dual Students trains two independently initialized student models and selects the less stable one for additional consistency training. The surrounding discussion also motivates weight averaging, with fast-SWA collecting more averaging points within a cycle than SWA.

  • Limitations: Mean Teacher can converge toward the student after many iterations, carrying biased and unstable predictions back into student learning.This is identified as a drawback motivating alternatives such as Dual Students.
  • Dual Students: Dual Students simultaneously trains two student models with different initializations, letting one provide targets for the other at each iteration.The target provider is selected using prediction-stability conditions.
  • Stability conditions: Stability requires matching predictions for clean and perturbed inputs and confident outputs far from the decision boundary.Confidence can be tested against a threshold such as ϵ = 0.1.
  • Losses: Each student minimizes supervised and unsupervised losses, with hyperparameters λ1 and λ2 controlling their contributions.The less stable student also receives the additional consistency loss.
  • Update rule: When both students are stable, the model with the largest prediction variation E_i = ∥f_i(x) − f_i(˜x)∥_2 is updated with an additional loss.If one model is unstable, that model is updated instead.
  • Fast-SWA: SWA averages SGD-traversed weights late in training using a cyclic learning rate, while fast-SWA averages networks at many points within the same cycle.Fast-SWA is proposed to produce a better final model and faster ensembling with fewer additional epochs.

2.7 Virtual Adversarial Training

Virtual Adversarial Training (VAT) smooths predictions around unlabeled examples by identifying and penalizing changes along the most adversarial input direction. Related adversarial-dropout methods instead optimize dropout masks, including channel-wise masks for convolutional layers.

  • Virtual Adversarial Training: VAT targets the input direction where the model’s label probability is most sensitive, improving robustness to local perturbations without requiring labels.The adversarial perturbation is approximated from prediction discrepancies rather than true labels.
  • Virtual Adversarial Training: VAT estimates the adversarial perturbation by adding Gaussian noise, computing a KL-divergence gradient, and selectively smoothing the output distribution along that direction.The method uses a perturbation that maximizes prediction change before applying consistency regularization.
  • Virtual Adversarial Training: Because perturbation estimation requires extra forward and backward passes, VAT commonly uses a single power iteration as a computational compromise.The resulting unsupervised loss is the MSE between predictions with and without the adversarial perturbation.
  • Adversarial Dropout: Adversarial Dropout optimizes dropout masks to alter predictions, inducing sparse structural changes rather than directly adding noise to inputs.Element-wise adversarial dropout is designed for fully connected networks.
  • Adversarial Dropout: Channel-wise Adversarial Dropout extends adversarial masking to convolutional layers by dropping entire feature maps instead of individual activations.This addresses the ineffectiveness of standard or element-wise dropout under strong spatial correlations within feature maps.

2.9 Interpolation Consistency Training

Interpolation Consistency Training (ICT) enforces consistent predictions for interpolations of unlabeled examples, using teacher-generated targets mixed alongside the inputs. Its interpolation strategy is motivated by the possibility of crossing low-density regions between examples from different clusters.

  • Interpolation Consistency Training: ICT trains a student to predict consistently on mixed unlabeled inputs and the corresponding mixture of teacher outputs.The teacher is an exponential moving average of the student model.
  • Interpolation Consistency Training: MixUp forms interpolated inputs as Mixλ(a, b) = λ · a + (1 −λ) · b, with λ sampled from Beta(α, α).ICT applies this operation to pairs of unlabeled examples and uses the same interpolation principle for teacher predictions.
  • Interpolation Consistency Training: Compared with random perturbations, interpolation can move an unlabeled example toward another cluster and potentially into a low-density region.This motivation assumes many classes with similarly distributed examples and pairs that likely belong to different classes.

2.10 Unsupervised Data Augmentation

Unsupervised Data Augmentation uses realistic, advanced transformations as perturbations for consistency-based semi-supervised learning. Training combines supervised correctness on labeled examples with consistency between original and augmented unlabeled examples.

  • Unsupervised Data Augmentation: Unsupervised Data Augmentation applies advanced transformations as consistency-training perturbations for unlabeled data.Examples include AutoAugment, RandAugment, and back-translation.
  • Unsupervised Data Augmentation: RandAugment samples uniformly from a fixed set of PIL transformations without requiring labeled data to search for an augmentation policy.It is described for image classification.
  • Unsupervised Data Augmentation: Back-translation augments text by translating an example from language A to language B and back to language A.The procedure is described for text classification.
  • Unsupervised Data Augmentation: The training objective combines correct predictions on labeled data with consistent predictions for original and augmented unlabeled examples.The procedure is presented as straightforward after the augmentations are defined.

3 Entropy Minimization

Entropy minimization encourages confident predictions on unlabeled data as an alternative way to push decision boundaries toward low-density regions. However, high-capacity neural networks can achieve confidence by producing large logits, limiting the method when used alone.

  • Entropy Minimization: Entropy minimization encourages low-entropy predictions on unlabeled examples regardless of the predicted class.This provides an alternative to consistency regularization under the cluster assumption.
  • Entropy Minimization: High-capacity neural networks can overfit low-confidence data by producing large logits and consequently very confident predictions.This is identified as a failure mode of entropy minimization.
  • Entropy Minimization: Entropy minimization does not produce competitive results on its own but can achieve state-of-the-art results when combined with other approaches.The passage characterizes combination with other methods as important for its strongest performance.

4 Proxy-label Methods

Proxy-label methods generate labels for unlabeled examples and use them as additional training targets. The section covers self-training, student–teacher refinement, multi-view strategies, and cross-view consistency.

  • Proxy-label Methods: Proxy-label methods create labels for unlabeled data using the prediction function or a variant, despite often noisy labels.These proxy labels supplement labeled examples with additional training information.
  • Self-training: Self-training adds an unlabeled example when its most likely class probability exceeds a threshold τ.The model is first trained on labeled data, then iteratively assigns pseudo-labels to unlabeled examples.
  • Self-training: Self-training can amplify model mistakes into confident but erroneous proxy labels.Biased or incorrect predictions may be reinforced during iterative labeling.
  • Student–Teacher Methods: Student–teacher methods separate proxy-label generation from student training, while Meta Pseudo Labels updates the teacher using the student’s validation performance.MPL has a student-learning phase and a teacher-update phase based on held-out validation loss.
  • Student–Teacher Methods: MPL requires held-out validation data because student performance alone may not provide a sufficient signal for training the teacher.The student may enter a poor parameter region before the teacher produces meaningful target distributions.
  • Multi-view and Cross-view Methods: Multi-view methods exploit distinct or restricted input views, with co-training and related variants using agreement among models to label unlabeled examples.Cross-view training instead encourages consistent predictions across different input views.

5 Holistic Methods

Holistic SSL methods combine dominant techniques into unified training frameworks. MixMatch and its extensions use augmentation, guessed labels, sharpening, distribution alignment, and consistency-based losses, while FixMatch combines confidence-filtered pseudo-labeling with strong augmentation.

  • Holistic Methods: Holistic SSL approaches unify dominant semi-supervised paradigms in a single framework to improve performance.The motivation is to combine complementary ideas rather than rely on one method family.
  • MixMatch: MixMatch augments labeled and unlabeled batches, averages predictions across unlabeled augmentations, sharpens guessed labels, and applies MixUp.The resulting batches support supervised and unsupervised SSL losses.
  • ReMixMatch: ReMixMatch adds distribution alignment and augmentation anchoring to MixMatch.Distribution alignment matches aggregate unlabeled predictions to the labeled class distribution, while anchoring uses weak-view predictions as targets for strong augmentations.
  • ReMixMatch: ReMixMatch also adds a self-supervised rotation-prediction loss alongside supervised and unsupervised losses.Unlabeled examples are rotated by angles sampled from {0, 90, 180, 270}.
  • FixMatch: FixMatch accepts a weak-view pseudo-label only when its highest class probability exceeds τ, then applies it to strongly augmented versions.Both supervised and unsupervised objectives use cross-entropy loss.
  • FixMatch: FixMatch uses standard weak augmentations and RandAugment or CTAugment for strong augmentation.Weak augmentation includes horizontal flips and random translations, while strong augmentation randomly applies transformations such as color inversion or contrast adjustment.

6 Generative Models

Generative SSL methods use learned data representations or generative models to improve classification with limited labels. The section covers VAEs, GAN-based approaches, multi-class discriminators, and the finding that effective SSL may require a complementary rather than realistic generator.

  • Generative Models: Generative models can learn transferable features from the data distribution for downstream supervised tasks.The generative perspective is used to improve classification beyond training on labeled data alone.
  • Variational Autoencoders: VAE-based SSL ranges from unsupervised pretraining to models that incorporate labels as observed or latent variables.The M1 model learns latent representations before supervised classification, while M2 jointly models labels, latent variables, and inputs.
  • Variational Autoencoders: Auxiliary-variable VAEs enrich inference by introducing latent auxiliary variables and, in SSL, an additional class variable y.The resulting model uses auxiliary inference, latent inference, classification, and generative networks.
  • Generative Adversarial Networks: GAN-based SSL alternates discriminator and generator training, with the discriminator distinguishing real data from generated samples.The generator maps latent variables into the input space, while the discriminator estimates whether inputs are real or generated.
  • Variational Autoencoders: Infinite VAE addresses the fixed latent dimension and fixed generative-model capacity of standard VAEs by allowing an expandable mixture of autoencoders.Its capacity can grow with data complexity rather than being chosen entirely in advance.
  • GAN-based SSL: Semi-Supervised GANs jointly learn a generative model and classifier, improving classification performance, generated-sample quality, and training time over using learned representations afterward.The approach exploits feedback between discriminator and classifier improvements.
  • GAN-based SSL: Feature-matching GANs use a (C + 1)-class discriminator, but feature matching favors classification whereas minibatch discrimination favors realistic image generation.The passage reports strong empirical results for the discriminator objective but identifies this task trade-off.
  • GAN-based SSL: For (C + 1)-class GAN-based SSL, good semi-supervised learning may require a bad generator that acts as a complement generator rather than matching the true data distribution.Its role is to help the discriminator obtain correct decision boundaries in high-density feature-space regions.

7 Graph-Based SSL

Graph-based SSL represents data as nodes connected by similarity or externally supplied edges, then exploits graph structure through label propagation, embeddings, or graph neural networks.

  • Graph Construction: Each labeled or unlabeled data point becomes a graph node, while edge weights encode pairwise similarity or externally supplied relations.The adjacency matrix records graph structure, with zero entries for unconnected nodes.
  • Graph Construction: A useful graph should reflect domain knowledge, making graph construction a practitioner responsibility; sparse alternatives include kNN and ϵNN graphs.Fully connected graphs support differentiable edge-weight updates but have high computational cost.
  • Label Propagation: Label propagation assumes points on the same manifold tend to share labels and transfers labeled-node information according to manifold structure and node similarity.Its objective combines smoothness across nearby nodes with fitting to initial labeled values.
  • Label Propagation: Label propagation can use iterative propagation instead of matrix inversion, but the standard iterative method does not produce the optimal solution’s labeling results.The optimal computation involves matrix inversion, which is computationally heavy for large graphs.
  • Graph Embedding: Node embeddings encode local graph positions in low-dimensional vectors; random-walk methods make co-occurring nodes similar and can support classifiers on unlabeled nodes.DeepWalk uses hierarchical softmax, whereas node2vec uses negative sampling to approximate the expensive loss.

8 Self-Supervision for SSL

Self-supervised learning creates supervisory signals from unlabeled data through pretext tasks, then adapts the learned representations to labeled examples for SSL.

  • Self-Supervised Learning: Self-supervised learning trains models with supervised losses on pretext tasks whose supervision comes from the data itself, targeting transferable downstream features.The pretext task is not intended to maximize final task performance directly.
  • Pretext Tasks: Exemplar-CNN treats transformed patches from an image as separate classes and predicts the class of an input patch.The task generates N transformed patches for each image.
  • Pretext Tasks: Rotation prediction classifies which of four 90-degree-multiple rotations was applied to an input image.The possible rotations are 0°, 90°, 180°, and 270°.
  • Pretext Tasks: Patch-based pretext tasks predict the relative position of neighboring image patches, while jigsaw variants predict a random permutation of nine patches.The original task uses a center patch and eight neighboring or non-overlapping patches.
  • Pretext Tasks: Colorization predicts missing luminance or color information after converting an image from RGB to Lab space.The objective can be framed as regression or classification after quantizing Lab space.
  • Pretext Tasks: Contrastive predictive coding distinguishes positive image or patch pairs from randomly sampled negative examples using a contrastive loss.Positive pairs may be transformed views or neighboring patches.
  • Adapting to SSL: Pretext-trained models support SSL by adapting to labeled data either simultaneously with the pretext task or iteratively through fine-tuning.The adaptation uses standard cross-entropy loss on the labeled set.

9 Conclusion

The review introduces deep semi-supervised learning and organizes its techniques into four broad approach categories. It concludes that growing data-efficiency research may bring deep SSL closer to fully supervised performance across applications.

  • Conclusion: The review covers SSL techniques within deep learning, including their main approaches and assumptions.It frames deep SSL as a response to limited labeled data.
  • Conclusion: The approaches are organized into consistency regularization, generative models, graph-based methods, and holistic approaches.These categories summarize the dominant approaches discussed in the review.
  • Conclusion: The paper foresees deep SSL potentially approaching fully supervised performance and being integrated into different systems and learning paradigms.This is presented as a forward-looking conclusion tied to growing interest in data-efficient deep learning.
Loading 2006.05278v2…