PaperlayerView exact revision on arXiv

Source-linked AI summary

Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning

Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, Kevin McGuinness

arXiv:1908.02983v5cs.CV

TL;DR

The paper addresses semi-supervised image classification with limited labeled data, where consistency regularization dominates recent approaches. It uses soft pseudo-labels with mixup and labeled-sample oversampling to reduce confirmation bias, achieving state-of-the-art performance across four datasets and outperforming consistency-regularization methods.

  • Problem

    Semi-supervised image classification seeks to learn from sparse labeled data and abundant unlabeled data, but incorrect pseudo-labels can accumulate through confirmation bias.

  • Method

    The approach uses network predictions as soft pseudo-labels for unlabeled samples, together with mixup, a minimum labeled-sample count per mini-batch, dropout, and data augmentation.

  • Results

    The approach achieves state-of-the-art performance across four datasets and outperforms consistency-regularization methods while remaining conceptually simple.

  • Takeaways & Limitations

    Pseudo-labeling is a suitable alternative to consistency regularization for semi-supervised image classification, and can outperform it without adding consistency regularization strategies.

Abstract

from arXiv · show

Semi-supervised learning, i.e. jointly learning from labeled and unlabeled samples, is an active research topic due to its key role on relaxing human supervision. In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different perturbations of unlabeled samples. We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions. We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art results in CIFAR-10/100, SVHN, and Mini-ImageNet despite being much simpler than other methods. These results demonstrate that pseudo-labeling alone can outperform consistency regularization methods, while the opposite was supposed in previous work. Source code is available at https://git.io/fjQsC.

I. INTRODUCTION

Semi-supervised learning addresses the scarcity of labeled data by exploiting abundant unlabeled samples, especially for image classification. This paper focuses on pseudo-labeling as an alternative to consistency regularization and targets confirmation bias with simple modifications.

  • Semi-supervised learning targets settings with sparse labeled data and extensive unlabeled data, including image classification.
  • Recent image-classification methods mainly use consistency regularization, while pseudo-labeling directly generates labels for unlabeled samples.
  • Confirmation bias arises when incorrect unlabeled-sample predictions are reused for training, increasing confidence in errors and resistance to change.
  • The paper studies network-prediction pseudo-labeling and reports state-of-the-art performance after simple modifications that address confirmation bias without consistency regularization.
  • Mixup regularization is proposed to calibrate predictions and alleviate confirmation bias in pseudo-labeling.

II. RELATED WORK

Prior deep semi-supervised learning methods primarily differ in how they exploit unlabeled data: consistency regularization enforces prediction agreement, whereas pseudo-labeling generates labels to guide learning.

  • Deep semi-supervised methods commonly use mini-batch optimization and cross-entropy or similar loss on labeled data.
  • Consistency regularization: Consistency regularization requires the same sample under different perturbations to produce the same output.
  • Consistency regularization: Teacher-student consistency methods separate target generation from learning, including teachers updated through exponential moving averages of student weights.
  • Consistency regularization: Co-training uses multiple networks to agree on predictions while disagreeing on errors, with adversarial attacks defining differing predictions.
  • Pseudo-labeling: Pseudo-labeling generates labels for unlabeled samples, with prior methods using hard predictions, uncertainty weighting, consistency terms, or graph-based label propagation.

III. PSEUDO-LABELING

The method formulates semi-supervised learning by combining labeled targets with soft pseudo-labels for unlabeled data. It updates pseudo-labels from network predictions and adds regularization to improve convergence and reduce degenerate solutions.

  • Semi-supervised training splits the dataset into labeled and unlabeled sets and learns a CNN model hθ(x).
  • The method combines true labels for labeled samples with pseudo-labels for unlabeled samples in a unified training set.
  • Categorical cross-entropy is optimized using the model’s softmax probabilities for the combined labeled and pseudo-labeled data.
  • Soft pseudo-labels are updated after each epoch from stored mini-batch softmax predictions, with a 10-epoch labeled-data warm-up supplying the first epoch’s predictions.
  • A class-balance regularizer discourages assigning all samples to one class by matching mean predicted probabilities to a uniform prior.
  • Entropy regularization concentrates each soft pseudo-label on a single class, and the total loss combines data and regularization terms.
  • The adapted pseudo-labeling baseline is not state of the art; the proposed mechanisms make pseudo-labeling a suitable alternative.

A. Confirmation bias

Incorrect network predictions used as pseudo-labels can produce confirmation bias, while mixup and labeled-sample oversampling regularize pseudo-labeling against this failure.

  • Confirmation bias occurs when incorrect predictions become pseudo-labels, increasing confidence in errors and making the model resist later changes.
  • Mixup combines sample pairs and their labels, regularizing the network toward linear behavior between training samples.The mixing coefficient is randomly sampled from a beta distribution.
  • Mixup reduces prediction confidence and improves calibration through label smoothing, limiting overfitting to soft pseudo-labels.This is especially relevant for unlabeled samples whose predictions serve as soft labels.
  • When labeled data are scarce, the unlabeled loss can dominate, so oversampling labeled examples provides more frequent updates toward fitting clean labels.The method sets a minimum number of labeled samples per mini-batch.
  • Experiments report that mixup, a minimum labeled-sample count per mini-batch, dropout, and data augmentation reduce confirmation bias and make pseudo-labeling an effective alternative to consistency regularization.

A. Datasets and training

The approach is evaluated across four image-classification datasets using varied labeled-data regimes, architectures, augmentations, and training schedules.

  • Experiments use CIFAR-10, CIFAR-100, SVHN, and Mini-ImageNet, with labeled and unlabeled subsets of the training images.
  • CIFAR-10, CIFAR-100, and SVHN provide 32×32 color images, while Mini-ImageNet uses 84 × 84 color images across 100 classes.
  • The labeled-set sizes range from 0.25K to 4K for CIFAR-10, 0.25K to 10K for CIFAR-100, 0.25K to 1K for SVHN, and 4K to 10K for Mini-ImageNet.
  • Experiments use the 13-CNN architecture and additionally test Wide ResNet-28-2 and PreAct ResNet-18 for architectural generalization.
  • Training applies normalization, random flips, translations, SGD with momentum 0.9, weight decay of 10−4, scheduled learning rates, and dataset-specific warm-up periods.

B. Effect of mixup on confirmation bias

Regularizing pseudo-labeling with mixup and a minimum number of labeled samples reduces confirmation bias. Combining both techniques improves semi-supervised performance, especially when labels are scarce.

  • Mixup alleviates confirmation bias and produces a smoother semi-supervised decision boundary than naive pseudo-labeling.Combining mixup with a minimum labeled-sample count further improves the boundary.
  • 11.40/48.54 validation error with cross-entropy fell to 7.16/41.80 with mixup for CIFAR-10/100 using 4000 labels.
  • 32.10 validation error with 500 CIFAR-10 labels remained high under mixup alone, motivating a minimum labeled-sample count per mini-batch.
  • Mixup and minimum k reduce rt, the average certainty of incorrect predictions during training.The measure averages the cross-entropy of incorrect samples’ softmax outputs with a uniform distribution.

C. Extended hyperparameters study

The hyperparameter study finds that the adopted configuration is close to the best observed performance, while stronger mixup provides little additional benefit alongside other regularization.

  • α = 4 and α = 8 may improve results over α = 1, but final experiments found only marginal differences.With the final configuration, α = 4 achieved 8.54 versus 8.80 ± 0.45 for α = 1.
  • The adopted λA = 0.8 and λH = 0.4 configuration was very close to the best performance in the hyperparameter experiment.The authors report only marginal improvements from more careful tuning.
  • Stronger mixup regularization might not be additive to dropout and extra data augmentation in this setting.

D. Generalization to different architectures

The approach generalizes across architectures when dropout and data augmentation are used. Implementation choices also protect pseudo-label quality during their computation.

  • Dropout and data augmentation help achieve good performance across all evaluated architectures.
  • Dropout p = 0.1 and p = 0.3 improve convergence in CIFAR-10, while color jitter further reduces error.
  • Pseudo-label quality requires disabling dropout and data augmentation during the second forward pass used to compute pseudo-labels.This configuration is used for state-of-the-art comparisons.

E. Comparison with the state-of-the-art

The proposed pseudo-labeling approach outperforms consistency regularization and other pseudo-labeling methods across several image-classification benchmarks, including settings with few labels and higher-resolution images.

  • The approach clearly outperforms consistency regularization methods, purely pseudo-labeling approaches, and their combinations on CIFAR-10/100.
  • In SVHN, the approach outperforms most state-of-the-art methods, especially when very few labels are available.
  • The results demonstrate better generalization than methods that fail when the number of labels decreases.
  • The approach obtains an over 10 point margin over the best related work on Mini-ImageNet.
  • For CIFAR-10 with few labels, performance depends on the number of labels, dataset complexity, and architecture type; the 13-CNN converges robustly with 250 labels and obtains 9.37 test error.
  • A straightforward confirmation-bias modification yields a competitive semi-supervised approach without consistency regularization.

V. CONCLUSIONS

The paper presents a simple pseudo-labeling approach for image-classification SSL that combines network predictions with regularization and outperforms related work across four datasets.

  • The approach uses network predictions as soft pseudo-labels for unlabeled data, together with mixup, a minimum labeled-sample count per mini-batch, dropout, and data augmentation.
  • The conceptually simple approach outperforms related work in four datasets, positioning pseudo-labeling as an alternative to consistency regularization.
  • Future work should examine SSL on class-unbalanced and large-scale datasets and the synergies between pseudo-labeling and consistency regularization.
Loading 1908.02983v5…