PaperlayerView exact revision on arXiv

Source-linked AI summary

Debiased Self-Training for Semi-Supervised Learning

Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, Mingsheng Long

arXiv:2202.07136v5cs.LGcs.CV

TL;DR

Semi-supervised self-training reduces labeling demands but suffers from bias and instability when incorrect pseudo labels are reused. DST decouples pseudo-label generation from utilization and optimizes representations against worst-case bias, achieving state-of-the-art results across 13 tasks and improving stability and class balance.

  • Problem

    Self-training can be unreliable because data bias and incorrect pseudo-label training accumulate bias, causing instability and imbalanced class performance.

  • Method

    DST uses parameter-independent classifier heads to decouple pseudo-label generation from utilization and adversarially optimizes representations against worst-case bias.

  • Results

    DST achieves state-of-the-art performance on 13 semi-supervised learning tasks, including a 6.3% average improvement over state-of-the-art methods and 18.9% over FixMatch.

  • Takeaways & Limitations

    DST can serve as a universal add-on for existing self-training methods while improving training stability and performance balance across classes.

Abstract

from arXiv · show

Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets. Yet these datasets are time-consuming and labor-exhaustive to obtain on realistic tasks. To mitigate the requirement for labeled data, self-training is widely used in semi-supervised learning by iteratively assigning pseudo labels to unlabeled samples. Despite its popularity, self-training is well-believed to be unreliable and often leads to training instability. Our experimental studies further reveal that the bias in semi-supervised learning arises from both the problem itself and the inappropriate training with potentially incorrect pseudo labels, which accumulates the error in the iterative self-training process. To reduce the above bias, we propose Debiased Self-Training (DST). First, the generation and utilization of pseudo labels are decoupled by two parameter-independent classifier heads to avoid direct error accumulation. Second, we estimate the worst case of self-training bias, where the pseudo labeling function is accurate on labeled samples, yet makes as many mistakes as possible on unlabeled samples. We then adversarially optimize the representations to improve the quality of pseudo labels by avoiding the worst case. Extensive experiments justify that DST achieves an average improvement of 6.3% against state-of-the-art methods on standard semi-supervised learning benchmark datasets and 18.9%$ against FixMatch on 13 diverse tasks. Furthermore, DST can be seamlessly adapted to other self-training methods and help stabilize their training and balance performance across classes in both cases of training from scratch and finetuning from pre-trained models.

1 Introduction

Semi-supervised self-training reduces labeling demands by assigning pseudo labels to unlabeled data, but can be unstable and amplify bias. DST separates pseudo-label generation from utilization and optimizes representations against worst-case bias.

  • Self-training iteratively assigns model-generated pseudo labels to unlabeled samples and trains on them as labeled examples.
  • Training can fluctuate sharply, and FixMatch can create a Matthew effect in which well-behaved categories improve while poorly-behaved categories collapse.
  • DST distinguishes data bias from training bias caused by incorrect pseudo-label training.
  • DST decouples pseudo-label generation and utilization with separate classifier heads, then optimizes representations to reduce worst-case bias.
  • 6.3% average improvement over state-of-the-art methods and 18.9% over FixMatch were reported across standard benchmarks and 13 diverse tasks.

2 Related Work

Related work addresses pseudo-label quality and tolerance to inaccurate pseudo labels through confidence methods, label propagation, teacher models, and multi-model training. Other approaches use self-supervision or adversarial training, with stated feasibility and task-specificity limitations.

  • DST defines bias as deviation between pseudo-labeling and ground-truth labeling functions, rather than only category-wise quantity imbalance.
  • Self-training methods generate pseudo labels through prediction averaging, confidence thresholds, adaptive curricula, or local-neighborhood density.
  • Methods such as Mean Teacher, Noisy Student, Co-training, and DivideMix separate pseudo-label generation from utilization to reduce confirmation bias.
  • Self-supervised methods can use unlabeled data but typically require large datasets and heavy computation, while learning task-specific information remains difficult.
  • Adversarial semi-supervised methods construct generated or perturbed samples, including input-noise approaches such as VAT.

3 Analysis of Bias in Self-Training

The analysis separates self-training bias into data bias inherent to the task and training bias added by pseudo-label optimization. Sampling, representations, and aggressive pseudo-label training can produce category-specific disparities that self-training may enlarge.

  • Bias formulation: Self-training bias measures class-conditional fractions of incorrectly pseudo-labeled samples through the deviation between learned and true decision hyperplanes.
  • Sources of bias: Different labeled-data samples can yield dramatically different accuracy for the same category because supporting points have different distances to true decision hyperplanes.
  • Sources of bias: Different pre-trained representations produce different category biases even with identical pre-training and downstream datasets.
  • Self-training effects: Aggressive pseudo-label training enlarges category gaps: some accuracies rise from 60% to 80%, while others fall from 15% to 0%.
  • Self-training effects: The resulting Matthew effect improves already well-behaved categories while further degrading poorly-behaved ones.
  • Bias types: Data bias arises from biased labeled-data sampling and parameter initialization, while training bias is the increment introduced by a self-training strategy.

4 Debiased Self-Training

DST addresses two sources of self-training bias: direct accumulation from incorrect pseudo-label utilization and data bias that makes pseudo labels unreliable. It decouples pseudo-label generation from utilization and adversarially optimizes representations against worst-case decision hyperplanes.

  • Reduce training bias: FixMatch generates and uses pseudo labels with the same model, allowing model errors to amplify during self-training.Confidence thresholding does not eliminate unreliable pseudo labeling when labeled data are extremely scarce.
  • Reduce training bias: DST trains the task-specific head only on clean labels and uses a parameter-independent pseudo head to learn representations from pseudo labels.The independent pseudo head prevents incorrect pseudo labels from directly accumulating bias in the task-specific head.
  • Reduce generation of erroneous pseudo labels: Limited labeled data can shift learned decision hyperplanes from the true hyperplanes, causing incorrect pseudo labels near biased boundaries.This data bias is especially problematic when the labeled sample size is very small.
  • Reduce generation of erroneous pseudo labels: DST estimates worst-case bias using a head that classifies all labeled samples correctly while making as many mistakes as possible on unlabeled data.The discrepancy between this worst-case head and the current pseudo-labeling function estimates its unlabeled mistakes.
  • Reduce generation of erroneous pseudo labels: DST adversarially optimizes feature representations so unlabeled samples remain correctly distinguished even by the worst hyperplanes.The objective pushes unlabeled features away from current hyperplanes to reduce data bias.

5 Experiments

Experiments show that DST improves accuracy, training stability, pseudo-label quality, and class balance across standard benchmarks and pre-trained settings. Ablations and analyses support its independent heads and worst-case estimation as key design choices.

  • Main results: DST yields consistent improvement on all evaluated tasks, including 8.3% and 10.7% gains for FixMatch and FlexMatch on CIFAR-100 and STL-10.On CIFAR-100, DST (FixMatch) also surpasses FixMatch’s best accuracy by 10.9% relatively.
  • Transfer from a pre-trained model: DST surpasses FixMatch by 19.9% across datasets with supervised pre-trained models and relatively outperforms it by 23.5% on average with unsupervised pre-training.DST improves on all datasets in the unsupervised pre-training setting.
  • Ablation studies: Independent classifier-head mechanisms outperform mutual learning, nonlinear pseudo heads outperform linear ones, and worst-case estimation improves performance by large margins.These findings come from the CIFAR-100 ablation study.
  • Analysis: DST improves pseudo-label quantity and quality, while FixMatch produces over 70% pseudo labels whose accuracy falls below 60%.DST pseudo-label accuracy rises above 70% throughout training and generates more pseudo labels later in training.
  • Analysis: For the worst 20 classes, FixMatch reaches 1.0% average accuracy, versus 28.5% with training-bias reduction and 34.5% with worst-case estimation.FixMatch’s class imbalance ratio reaches infinity after 5000 iterations, indicating that poorly learned classes are ignored.

6 Conclusion

The paper concludes that DST addresses confirmation bias in pseudo-label-based semi-supervised learning and reaches state-of-the-art performance across 13 tasks. Its released implementation includes the method and baseline code.

  • Conclusion: DST achieves state-of-the-art performance on 13 semi-supervised learning tasks and serves as a universal add-on for existing self-training methods.The conclusion frames DST as a method for decreasing bias in self-training.
  • Conclusion: The released implementation includes code for DST and all baselines.The code is based on PyTorch.
  • Experimental setup: The experiments use specified architectures, optimization settings, augmentation policies, and algorithm-specific hyperparameter searches across scratch and pre-trained configurations.Reported settings include Wide ResNet variants for scratch experiments and ResNet50 with SGD for pre-trained experiments.

A.3 DST as a general add-on to previous self-training methods

DST can be integrated into several self-training methods by separating pseudo-label generation from utilization while retaining method-specific generation mechanisms. The main head is trained only on clean labeled data.

  • Debiased FixMatch: Debiased FixMatch generates unlabeled pseudo labels with the main head and uses them for training with the pseudo head.The main head is trained only on clean labeled samples.
  • Debiased FlexMatch: Debiased FlexMatch follows the Debiased FixMatch design, with the main head estimating each category’s learning status.
  • Debiased Mean Teacher: Debiased Mean Teacher generates pseudo labels from the main head’s exponential moving average and utilizes them with the pseudo head.The main head remains trained only on clean labeled data.
  • Debiased Noisy Student: Debiased Noisy Student uses the previous round’s head to generate pseudo labels and the pseudo head to utilize them.The current main head is trained only on clean labeled data.

B.1 Experiments on training stability

Stability experiments identify irreversible degradation and severe failure cases for FixMatch with pre-trained models. Noise in pseudo labels can drive severe performance decline, while DST is intended to improve stability in such settings.

  • Failure cases: When pre-trained-model performance declines, it cannot be recovered later across tested confidence thresholds.Figures 12(a) and 12(b) show similar behavior for thresholds 0.7, 0.8, 0.9, and 0.95.
  • Failure cases: With unsupervised pre-training and confidence threshold 0.7, noisy pseudo labels cause FixMatch performance to decline severely.This case is presented to show DST’s training-stability benefit and is not the Table 2 entry for that dataset.

B.2 Experiments on performance balance between categories

DST improves the performance of poorly-behaved categories on CIFAR-100 under both supervised pre-training and training from scratch.

  • DST improves the performance of poorly-behaved categories on CIFAR-100 with supervised pre-trained models and when training from scratch.Figures 13 and 14 use 400 labeled samples and unlabeled data.

B.3 Experiments with varying amounts of labeled data

Across varying pre-training and labeled-data settings, DST outperforms comparison methods, is less sensitive to labeled-data quantity, and reduces pseudo-label bias.

  • DST outperforms FlexMatch and DebiasMatch on CIFAR-100 with 1000 labels under different pre-trained models.
  • DST is less sensitive to the amount of labeled data than FixMatch and yields consistent improvement as labeled samples per class vary from 1 to 25.The comparison uses CIFAR-100 with supervised pre-trained models.
  • DST reduces pseudo-label bias by improving pseudo-label quantity, accuracy, and class-balance behavior in experiments with unsupervised pre-trained models.The analysis uses CIFAR-100 with 400 labels and defines I as max_c N(c)/min_c′ N(c′).
  • DST outperforms baselines that use dynamic thresholds or multi-view training, while Figure 17 reports improved pseudo-label quality.

B.5 Ablation study on nonlinear main classifier head

The ablation indicates that a nonlinear pseudo head improves performance, whereas nonlinear and linear main heads perform similarly on average.

  • A nonlinear pseudo head improves performance in the experiments.
  • Nonlinear and linear main heads achieve similar average performance.The authors conjecture that a nonlinear main head is more likely to over-fit with few labeled samples.
  • DST is presented as improving performance and training stability across existing self-training methods, especially when labeled data is scarce.The paper also reports improved performance balance across classes.
Loading 2202.07136v5…