PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Training: A Survey

Massih-Reza Amini, Vasilii Feofanov, Loic Pauletto, Lies Hadjadj, Emilie Devijver, Yury Maximov

arXiv:2202.12040v6cs.LG

TL;DR

Semi-supervised learning seeks to use abundant unlabeled data when labeled examples are scarce, but its value depends on compatibility between the data distribution and target function. This survey synthesizes self-training methods and related approaches, reviews theory, applications, and benchmark impacts, and identifies pseudo-label quality and initial-label dependence as limitations.

  • Problem

    Unlabeled data do not always improve learning, and their value depends on a relationship between the marginal data distribution and the target function.

  • Method

    The survey reviews self-training methods, variants, related approaches, theoretical advances, applications, and the impact of self-training features on benchmark datasets.

  • Results

    The survey presents self-training as an increasingly prominent approach and synthesizes its methods for binary and multi-class classification alongside consistency-based and transductive approaches.

  • Takeaways & Limitations

    The survey identifies margin distributions as important for selecting unlabeled samples for pseudo-labeling and outlines integration with consistency regularization and multi-view training as a future direction.

  • Takeaways & Limitations

    Self-training is sensitive to incorrect pseudo-labels and to an initial labeled dataset that is unrepresentative or too small.

Abstract

from arXiv · show

Semi-supervised algorithms aim to learn prediction functions from a small set of labeled observations and a large set of unlabeled observations. Because this framework is relevant in many applications, they have received a lot of interest in both academia and industry. Among the existing techniques, self-training methods have undoubtedly attracted greater attention in recent years. These models are designed to find the decision boundary on low density regions without making additional assumptions about the data distribution, and use the unsigned output score of a learned classifier, or its margin, as an indicator of confidence. The working principle of self-training algorithms is to learn a classifier iteratively by assigning pseudo-labels to the set of unlabeled training samples with a margin greater than a certain threshold. The pseudo-labeled examples are then used to enrich the labeled training data and to train a new classifier in conjunction with the labeled training set. In this paper, we present self-training methods for binary and multi-class classification; as well as their variants and two related approaches, namely consistency-based approaches and transductive learning. We examine the impact of significant self-training features on various methods, using different general and image classification benchmarks, and we discuss our ideas for future research in self-training. To the best of our knowledge, this is the first thorough and complete survey on this subject.

1 Introduction

Self-training addresses scarce labeled data by iteratively leveraging abundant unlabeled examples, while semi-supervised learning relies on assumptions about data structure and label compatibility. This survey organizes self-training and related approaches, reviews applications and theory, and identifies limitations and future directions.

  • Motivation: Self-training uses unlabeled data alongside scarce labeled examples, supporting inference when annotation is costly or time-consuming.The approach has applications including computer vision, natural language processing, and speech recognition.
  • Semi-supervised learning assumptions: Semi-supervised learning commonly adapts the smoothness hypothesis through cluster, low-density separation, and manifold assumptions.These assumptions respectively relate labels to groups, decision boundaries to low-density regions, and high-dimensional data to low-dimensional structures.
  • Compatibility: Unlabeled data benefit training only when a relationship exists between the marginal data distribution and the target function.Some studies report no performance effect from unlabeled data, while stronger assumptions can be required for guarantees beyond supervised learning.
  • Survey scope: The survey focuses on self-training, whose discriminant strategy assigns pseudo-labels to high-confidence unlabeled examples and incorporates them into learning.It distinguishes this focus from broader semi-supervised surveys and situates self-training among related approaches.
  • Survey organization: The paper reviews self-training methods, variants, related approaches, theoretical studies, applications, benchmark impacts, and future prospects.Its organization covers the self-training framework, transductive and consistency-based methods, applications, and concluding perspectives.

2 Self-Training

The semi-supervised framework combines a small labeled set with a larger unlabeled set to learn a hypothesis intended to generalize better than one trained only on labeled data. Its scoring functions, margins, and unsigned margins connect classification confidence with low-density decision boundaries.

  • Framework and notation: The classic semi-supervised setting has m ≪u, with labeled examples drawn from a joint distribution and unlabeled examples from its marginal distribution.With no labeled data the problem becomes unsupervised; with no unlabeled data it becomes supervised.
  • Framework and notation: A learner maps inputs to labels and is assumed to produce lower generalization error using labeled and unlabeled data than using labeled data alone.The framework defines generalization error through the expected indicator loss under the data distribution.
  • Scoring and margins: Classifiers use a scoring function f, and the classification function selects the class with the highest score.The scoring function is defined over input-class pairs.
  • Scoring and margins: Margins quantify the score of a function for an example and class, with separate unsigned-margin definitions for binary and multi-class classification.The survey introduces both binary and multi-class unsigned margins as confidence-related quantities.
  • Low-density separation: Maximizing the unsigned margin tends to place the decision boundary in low-density regions, following the low-density separation assumption.This links the classifier’s confidence-related margin to a central semi-supervised learning hypothesis.

2.2 Self-training: the idea

Self-training is an iterative wrapper that trains on labeled data, pseudo-labels selected unlabeled examples, and retrains using the enlarged labeled set. The process uses a pseudo-labeling strategy and can continue until no unlabeled examples remain or none qualify for labeling.

  • Core idea: Self-training, also called decision-directed or self-taught learning machine, is an early semi-supervised approach that has grown in popularity.The paper presents it as the central algorithmic approach discussed in this section.
  • Pseudo-labeling: A pseudo-labeler assigns labels to selected examples from the unlabeled set to support finding a decision boundary in low-density regions.The pseudo-label of an unlabeled example is denoted ỹ, and the pseudo-labeled set is denoted X̄_U.
  • Iterative procedure: The algorithm starts with a supervised classifier on S, then selects part of XU and assigns pseudo-labels using current predictions at each iteration.This creates the pseudo-labeled subset used in subsequent training rounds.
  • Iterative procedure: Pseudo-labeled examples are removed from XU, and a new classifier is trained on S ∪ X̄_U as additional labeled data.The training objective is a regularized empirical loss over the original and pseudo-labeled examples.
  • Training objective: The pseudo-labeled data contribution is controlled by γ, while λ controls regularization; the instantaneous loss is often cross-entropy.These hyperparameters determine how pseudo-labeled examples and regularization enter learning.
  • Stopping condition: Iterations stop when XU is empty or no pseudo-labels are produced, returning the classifier and remaining data sets.Algorithm 1 explicitly records the updated unlabeled and pseudo-labeled sets at each round.

2.3 Pseudo-labeling strategies

Pseudo-labeling strategies select unlabeled examples using confidence, thresholds, proportions, or curriculum schedules, then add selected examples to training. The section also covers threshold selection through majority-vote risk bounds and highlights errors arising from poorly calibrated or tuned confidence criteria.

  • Pseudo-labeling selects a subset of unlabeled examples to avoid overfitting the initial classifier, using proposed criteria with different strengths and weaknesses.
  • Threshold-Based Methods: Threshold-based methods pseudo-label only sufficiently confident observations, leaving examples that fail the confidence conditions unlabeled.The classical low-density assumption places most classifier errors near the decision boundary.
  • Threshold-Based Methods: High thresholds reduce wrong-label risk but can overtrust confidence estimates biased by small labeled samples, with the optimal threshold potentially changing across iterations.
  • Proportion-Based Methods: Proportion-based methods pseudo-label a fixed fraction of the most confident examples and increase that fraction across iterations, but require careful tuning.One described schedule increases the proportion until p = 0.5.
  • Curriculum-Based Methods: Curriculum-based methods begin with easy-to-learn examples and progressively admit more complex pseudo-labeled data, subject to tuning α_k and prediction-distribution assumptions.
  • Majority Vote Classifiers: Majority-vote methods can choose θ by minimizing an upper bound on conditional unlabeled risk, trading off pseudo-label quantity against induced error.The bound is reported to be tight when errors concentrate in low-margin regions, and this technique outperforms fixed-threshold strategies on different multi-class problems.
  • Variants: Two-head approaches separate pseudo-label creation from pseudo-label use to reduce training errors, while adaptive thresholding also adds class-fairness regularization.Adaptive thresholding adjusts confidence thresholds according to model learning status but adds computational and training complexity.

2.4 Self-training with two classifiers

Two-classifier self-training methods let models exchange pseudo-labels, extending single-model confidence-based self-training through consensus and complementary predictions. The section situates these methods in co-training, cross pseudo supervision, theoretical learnability, and noisy-label analysis.

  • Two-classifier methods train each model using the other model’s outputs, commonly relying on consensus between predictions.
  • Co-training: Co-training assumes two comparable but not entirely correlated views whose complementary information can support classifier learning.
  • Co-training: Both co-training and self-training iteratively assign pseudo-labels and use them to improve models, but co-training relies on multiple models rather than one confidence-based model.
  • Cross Pseudo Supervision: Cross Pseudo Supervision uses two neural networks receiving the same images, with each network treating the other’s pseudo-labels as ground truth at each mini-batch.
  • Theory and Noise: Co-training learnability has been studied under PAC theory, including noise in unlabeled class labels introduced during pseudo-labeling.
  • Summary: Table 1 summarizes principal self-training algorithms organized around pseudo-labeling with one or two classifiers.

2.5 Self-training under Domain Shift

Self-training is extended to distribution-shift settings such as unsupervised domain adaptation, where pseudo-labeled target examples are progressively incorporated into source training. Large source–target discrepancies make confidence reliability and calibration central concerns.

  • In unsupervised domain adaptation, self-training transfers knowledge from a labeled source domain to an unlabeled target domain by progressively adding pseudo-labeled target examples.
  • The approach is especially relevant to gradual domain adaptation when unlabeled instances from intermediate domains are available.
  • Without intermediate domains, methods must ensure target pseudo-labels are reliable and not biased toward source data, using policies such as tri-training or alternating gradient steps.
  • Large source–target discrepancies can bias prediction confidence so it fails to distinguish correct from incorrect pseudo-labels, motivating calibration and uncertainty-estimation methods.

2.6 Theoretical studies

Theoretical studies analyze when self-training can achieve strong classification and generalization guarantees under specific model, distribution, noise, and initialization assumptions.

  • Under expansion properties and suitable neighbor-based class separation, self-training admits distributional guarantees and finite-sample bounds for deep neural networks.
  • With an initial pseudo-labeling error below C_err, gradient-based self-training with linear models can achieve Bayes-optimal classification error up to ϵ.
  • O(d) labeled examples suffice for gradient descent to learn a pseudo-labeling strategy whose classification error is no more than C_err.
  • For two-layer neural networks trained on isotropic Gaussian data, self-training can converge to the ground truth with fewer observations than supervised learning under stated assumptions.
  • Under Massart noise, unlabeled data in the proposed half-space self-training algorithm does not degrade the performance of the initial labeled-data classifier.
  • Theoretical work also studies identifiability through inverse propensity weighting and error behavior under distribution shift, including gradual shifts.

3 Related and unrelated approaches

The survey distinguishes self-training from related semi-supervised paradigms and reviews transductive, consistency-based, self-supervised, and reinforced extensions.

  • Consistency-based approaches: Consistency learning constrains predictions for similar unlabeled examples without assigning pseudo-labels, whereas self-training uses pseudo-labels.
  • Self-supervised learning: Self-supervised learning is separate from self-training: it learns from unlabeled data through pretext tasks whose representations transfer to labeled downstream tasks.
  • Transductive learning: Transductive learning assigns labels only to finite unlabeled training samples, often using unsigned-margin distributions and function classes structured by prior knowledge.
  • Transductive learning: Multi-class transductive neural methods jointly optimize unlabeled class labels and network parameters using cross-entropy over labeled and unlabeled data.
  • Transductive learning: A parameter-free Fisher-Rao transductive regularizer uses unlabeled data under gated APIs with compute-cost and data-privacy constraints.
  • Consistency-based approaches: Mean Teacher uses teacher and student neural networks based on the assumption that close models should make the same prediction for the same input.
  • Consistency-based approaches: Contrastive semi-supervised language modeling combines prompt-based pseudo-labeling, contrastive learning, and mask consistency to improve generalization in few-shot text classification.

4 Applications

Applications span web and language processing, vision, speech, and other domains, with methods adapting pseudo-label generation, augmentation, and thresholding to task-specific challenges.

  • Natural language processing: Co-training for web-page classification uses two word-based views—page content and hyperlinks—to train classifiers from complementary information.
  • Natural language processing: Randomly splitting bag-of-words into two views works for text representations but does not generalize to sequential base classifiers.
  • Natural language processing: NLP methods use teacher predictions, partial annotation, and pseudo-labels to address named-entity recognition and structured label spaces.
  • Image classification: Image-classification systems commonly use student-teacher neural networks, learned image representations, and augmentation or noise-control strategies.
  • Image classification: FixMatch combines consistency regularization with confidence-based pseudo-label selection using weak and strong augmentations of the same image.
  • Image classification: Fixed thresholds across classes disregard differing learning conditions, motivating curriculum methods with dynamically adapted class-specific thresholds.
  • Speech recognition: Domain-specific filters and language models validate speech pseudo-labels using phonetic and linguistic information.
  • Speech recognition: Punjabi low-resource speech self-training generated highly accurate pseudo-labels and significantly improved word error rate over state-of-the-art approaches.

5 Conclusion, Limitations and Perspectives

The survey synthesizes self-training methods, their variants, related approaches, theoretical developments, applications, and benchmark effects. It also identifies persistent limitations and directions for more robust, efficient, and broader self-training research.

  • The survey reviews strategies for selecting unlabeled samples, variants of self-training, related approaches, theory, applications, and benchmark effects.
  • Limitations: Incorrect or noisy pseudo-labels can propagate errors through training, causing suboptimal performance.
  • Limitations: Self-training effectiveness depends on the initial labeled dataset being sufficiently representative and large enough to support improvement.
  • Limitations: Iterative pseudo-labeling and retraining can be time-consuming and resource-intensive, particularly for large-scale datasets.
  • Future Work: Future work includes adaptive thresholds, uncertainty quantification, integration with consistency or multi-view methods, and applications in medical and industrial data.
  • Future Work: Most current studies focus on perturbation-based deep learning for visual, text, and audio applications, leaving other domains less explored.
  • Future Work: Theoretical research could connect pseudo-label uncertainty with performance and address final-classifier training under noisy labels.

A Empirical Study

The empirical study evaluates self-training under both sufficient and severely limited labeled-data regimes. It focuses on pseudo-label noise and threshold selection as central factors affecting effectiveness.

  • The study evaluates self-training performance under sufficient labeled data and severely limited labeled data.
  • It focuses on how pseudo-label noise and threshold selection affect self-training effectiveness.
  • With sufficient labels, the setup permits training an initial supervised complex model before self-training.

A.1 Noise Account

The noise-account experiment compares FlexMatch with and without debiased self-training on CIFAR-10 and CIFAR-100. Accounting for pseudo-label noise improves performance, especially on CIFAR-100, although the difference narrows with more labeled data.

  • The experiment uses CIFAR-10 and CIFAR-100, each with 50000 training and 10000 test images, to study pseudo-label noise and dynamic thresholds.
  • DST trains a dedicated head on pseudo-labeled examples to capture and account for noise in pseudo-labels, combined with FlexMatch thresholding.
  • FlexMatch uses Wide ResNet as the base classifier, while experiments compare FM, FM+DST, and fully supervised WRN models across five random seeds.
  • In both datasets, accounting for pseudo-label noise improves performance, with a larger improvement on CIFAR-100.
  • Figure 2 reports Accuracy on CIFAR-10 and CIFAR-100 for 4, 10, 20, or 50 initial labeled samples per class, with Supervised using m = 50000 and u = 0.
  • In CIFAR-100, noise handling aids class differentiation and generalization in the task’s 20-superclass, 100-class structure.
  • With more initial labeled examples, the performance gap between FM and FM+DST narrows as pseudo-labeling errors decrease.

A.2 The impact of threshold selection

Experiments on 9 publicly available datasets compare supervised Random Forest with self-training using automatically selected or fixed pseudo-labeling thresholds. Performance depends strongly on threshold choice: automatic selection is competitive, whereas thresholds that are too low or arbitrarily fixed can degrade results.

  • Experimental setup: The experiments use 9 publicly available datasets and compare supervised Random Forest (RF) with self-training using automatic pseudo-labeling (PL∗).The classifier uses the scikit-learn Random Forest implementation with 200 trees; experiments randomly select labeled examples and repeat results 20 times.
  • Threshold selection: The automatic strategy selects the threshold that minimizes the Random Forest error bound over unlabeled training samples.
  • Automatic thresholds: Self-training with an automatically determined threshold can perform competitively and may improve results over supervised RF.
  • Fixed thresholds: Fixed thresholds tend to produce inferior results compared with supervised learning, indicating that arbitrary threshold choices may yield suboptimal pseudo-labeling.
  • Threshold extremes: Low thresholds θ ∈ {0.5, 0.7} likely introduce label noise, while θ = 0.9 is competitive on Isolet and MNIST but yields too few pseudo-labeled examples for efficient learning.
  • Conclusion: The findings emphasize using a dynamic and adaptive mechanism to select the pseudo-labeling threshold.
Loading 2202.12040v6…