PaperlayerView exact revision on arXiv

Source-linked AI summary

In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning

Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, Mubarak Shah

arXiv:2101.06329v3cs.LGcs.CV

TL;DR

Pseudo-labeling is general but suffers from incorrect high-confidence labels caused by poor calibration, while consistency methods may require domain-specific augmentations. The paper proposes UPS, which combines confidence with uncertainty and supports negative pseudo-labels, achieving strong results across image, video, and multi-label benchmarks.

  • Problem

    Pseudo-labeling performs relatively poorly because poorly calibrated models can produce incorrect high-confidence pseudo-labels, while consistency methods rely on domain-specific augmentations.

  • Method

    UPS guides pseudo-label selection with prediction confidence and uncertainty, generalizing pseudo-labeling to negative labels for single-label and multi-label learning.

  • Results

    UPS achieves strong performance across CIFAR-10, CIFAR-100, UCF-101, and Pascal VOC, including a 0.45% improvement over MixMatch on CIFAR-10 with 4000 labels.

  • Takeaways & Limitations

    UPS preserves pseudo-labeling’s simplicity and modality flexibility while reducing calibration-related training noise and extending its use beyond single-label classification.

  • Takeaways & Limitations

    UPS does not inherently rely on domain-specific augmentations, which are not straightforward to extend from images to modalities such as video, text, and speech.

Abstract

from arXiv · show

The recent research in semi-supervised learning (SSL) is mostly dominated by consistency regularization based methods which achieve strong performance. However, they heavily rely on domain-specific data augmentations, which are not easy to generate for all data modalities. Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models; these predictions generate many incorrect pseudo-labels, leading to noisy training. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process. Furthermore, UPS generalizes the pseudo-labeling process, allowing for the creation of negative pseudo-labels; these negative pseudo-labels can be used for multi-label classification as well as negative learning to improve the single-label classification. We achieve strong performance when compared to recent SSL methods on the CIFAR-10 and CIFAR-100 datasets. Also, we demonstrate the versatility of our method on the video dataset UCF-101 and the multi-label dataset Pascal VOC.

1 INTRODUCTION

SSL reduces reliance on large labeled datasets, but consistency regularization can depend on modality-specific augmentations. UPS revisits pseudo-labeling by using uncertainty to reduce calibration-induced label noise and extends it to negative labels.

  • SSL addresses the costly and often infeasible construction of large labeled datasets.
  • Consistency regularization methods often rely on rich, domain-specific augmentations that are less effective for modalities such as video and medical images.
  • Poorly calibrated networks can assign high confidence to incorrect predictions, producing noisy pseudo-labels that harm training and generalization.
  • UPS uses prediction uncertainty to guide pseudo-label selection while retaining pseudo-labeling’s simplicity and modality generality.
  • UPS generalizes pseudo-labeling by creating negative labels for negative learning and multi-label classification.
  • UPS achieves strong performance on CIFAR-10, CIFAR-100, UCF-101, and Pascal VOC.

2 RELATED WORKS

Related work situates UPS among semi-supervised, pseudo-labeling, consistency-regularization, and uncertainty-based approaches. The paper distinguishes its use of uncertainty for pseudo-label selection and negative learning from prior methods.

  • SSL-related approaches include transductive, graph-based, generative, self-supervised, pseudo-labeling, and consistency-regularization methods.
  • Pseudo-labeling: Pseudo-labeling and self-training generate labels for unlabeled samples using models trained on labeled data or neighborhood information.
  • Consistency Regularization: Consistency regularization seeks perturbation- or augmentation-invariant output distributions through input perturbations, temporal ensembling, or teacher-student training.
  • Uncertainty and Calibration: Prior work studies uncertainty for segmentation and downstream pseudo-label selection, but this paper targets poor calibration in the pseudo-labeling process.
  • Uncertainty and Calibration: For hard samples, this work uses negative learning to reduce noise rather than selecting them for positive learning.

3 PROPOSED METHOD

The method generates hard pseudo-labels from prediction probabilities, selects confident positive or negative labels, and adds uncertainty thresholds to reduce calibration-related noise. Selected labels train networks iteratively, with negative learning supporting single-label and multi-label settings.

  • 3.1 PSEUDO-LABELING FOR SEMI-SUPERVISED LEARNING: Pseudo-labeling generates labels for unlabeled samples from the probability outputs of a trained network.
  • 3.1 PSEUDO-LABELING FOR SEMI-SUPERVISED LEARNING: A threshold γ ∈ (0, 1) produces hard labels, while γ = 0.5 yields binary pseudo-labels for multi-label classification.
  • 3.2 PSEUDO-LABEL SELECTION: Confidence-based selection chooses positive labels when p(i)_c ≥ τp and negative labels when p(i)_c ≤ τn.
  • 3.2 PSEUDO-LABEL SELECTION: Selected positive pseudo-labels use cross-entropy, while samples without selected positive labels use negative cross-entropy for single-label classification.
  • 3.3 UNCERTAINTY-AWARE PSEUDO-LABEL SELECTION: High-confidence pseudo-label selection removes training noise and improves performance over traditional pseudo-labeling.
  • 3.3 UNCERTAINTY-AWARE PSEUDO-LABEL SELECTION: UPS combines confidence and prediction uncertainty, requiring uncertainty thresholds κp and κn before selecting labels.
  • 3.3 UNCERTAINTY-AWARE PSEUDO-LABEL SELECTION: The uncertainty-aware selection process increases pseudo-label accuracy relative to traditional and confidence-based selection.
  • 3.3 UNCERTAINTY-AWARE PSEUDO-LABEL SELECTION: The procedure repeatedly trains networks on labeled data and selected pseudo-labels until the number of selected pseudo-labels converges.

4 EXPERIMENTAL EVALUATION

UPS is evaluated across image, video, and multi-label datasets, with comparisons to SSL baselines, backbone variants, and component ablations. Results show competitive or improved performance across these settings, while uncertainty-aware selection and negative learning contribute to CIFAR-10 gains.

  • Datasets and settings: UPS is evaluated on CIFAR-10, CIFAR-100, Pascal VOC2007, and UCF-101 using modality-appropriate architectures and labeled-set sizes.The experiments use CNN-13 for CIFAR, ResNet-50 for Pascal VOC2007, and 3D ResNet-18 for UCF-101.
  • CIFAR-10 and CIFAR-100: 0.45% improvement over MixMatch is reported for CIFAR-10 with 4000 labels, while UPS outperforms prior CNN-13 methods on CIFAR-10 with 1000 labels and on CIFAR-100.Table 1 reports mean and standard deviation across three random splits.
  • Backbone analysis: UPS remains effective with Wide ResNet-28-2 and Shake-Shake backbones, achieving further performance improvements with stronger backbones.The comparison reports CIFAR-10 error rates across the two backbone types.
  • UCF-101: UPS outperforms all reported SSL methods on UCF-101, where both UPS and standard pseudo-labeling achieve strong results relative to Mean Teacher.The evaluation uses 20% and 50% of the training data as labeled sets.
  • Pascal VOC2007: UPS improves over competing methods by 1.67% with 10% labeled data and 0.72% with 20% on Pascal VOC2007.The reported metric is mAP, and the method is designed to support multi-label classification.
  • Ablations: 5.54% and 1.73% further error-rate reductions follow uncertainty-aware selection for 1000 and 4000 labels, while negative learning adds 1.32% and 0.32%.Confidence-based selection first reduces error by 6.1% and 3.71% in the two labeled-data settings; uncertainty-aware selection then provides additional gains.
  • Robustness analysis: κp < 0.1 produces comparable test error on CIFAR-10 with 1000 labels, whereas larger uncertainty thresholds cause predictable performance drops.The thresholds are selected using a CIFAR-10 validation set of 1000 samples, without extensive hyperparameter tuning.

5 DISCUSSION

UPS is designed as an uncertainty-agnostic framework: it can incorporate multiple uncertainty estimators and avoid inherent dependence on domain-specific augmentations, supporting SSL beyond image data.

  • Uncertainty Estimation: UPS does not depend on a particular uncertainty measure and can incorporate multiple uncertainty estimation methods.Experiments use MC-Dropout, MC-SpatialDropout, MC-DropBlock, and random data augmentation without uncertainty-estimation-specific hyperparameters.
  • Data Augmentation in SSL: UPS avoids inherent reliance on domain-specific augmentations, unlike many consistency-regularization methods.This matters for modalities such as video, text, and speech, where rich augmentation sets may be difficult to construct.
  • Data Augmentation in SSL: On UCF-101, standard pseudo-labeling and UPS outperform MT, demonstrating an advantage in a video setting.Video augmentations such as spatial cropping and temporal jittering are difficult to divide into strong and weak categories.

6 CONCLUSION

UPS reduces pseudo-label noise through uncertainty-aware selection while preserving pseudo-labeling’s simplicity and generality. The framework supports multiple modalities and both single-label and multi-label classification, with strong reported results on several datasets.

  • 6 CONCLUSION: UPS uses prediction uncertainty to reduce noise from incorrect pseudo-labels caused by poor neural-network calibration.The framework retains pseudo-labeling’s simplicity, generality, and ease of implementation.
  • 6 CONCLUSION: UPS extends pseudo-labeling to negative labels, enabling negative learning and multi-label classification.This generalizes pseudo-labeling beyond prior single-label classification settings.
  • 6 CONCLUSION: UPS achieves strong performance across multiple benchmark datasets and can be applied to multiple data modalities.The supplied results include Pascal VOC2007, UCF-101, CIFAR-10, and CIFAR-100 contexts.
  • 6 CONCLUSION: UPS obtains 40.34 mAP with 20% and 34.22 mAP with 10% of Pascal VOC2007 training-validation data available.Using all labeled data, a fully supervised ResNet-50 reaches 52.62% mAP.
  • 6 CONCLUSION: 50.2% accuracy on UCF-101 with 50% labeled data is similar to the 50.4% fully supervised accuracy.The comparison uses a fully supervised 3D ResNet-18 as the reference.

C.2 CIFAR-10 RESULTS

On CIFAR-10 with very few labeled examples, UPS achieves results close to MixMatch in the reported CNN-13 experiment.

  • C.2 CIFAR-10 RESULTS: Within 1.5% of MixMatch is achieved by UPS on CIFAR-10 with 250 labels using CNN-13.The experiments also use 500 labeled examples, although the supplied passage gives the explicit comparison for 250 labels.
  • C.2 CIFAR-10 RESULTS: The CIFAR-10 experiments evaluate settings with 250 and 500 labeled examples.The results are presented in Table 8, which reports error rates for very few training samples.

D UNCERTAINTY ESTIMATION: IMPLEMENTATION DETAILS

UPS can use different uncertainty estimators; the implementation details describe Monte Carlo sampling with SpatialDropout and DropBlock and stochastic inference settings.

  • D UNCERTAINTY ESTIMATION: IMPLEMENTATION DETAILS: MC-SpatialDropout and MC-DropBlock estimate uncertainty through Monte Carlo sampling during inference.For MC-SpatialDropout, the reported setting uses dropout rate 0.3 and 10 stochastic forward passes, with uncertainty from output-probability standard deviation.

E DATA AUGMENTATION

UPS is evaluated with standard augmentations and without input augmentations, while class imbalance in early pseudo-label selection is addressed for CIFAR-10.

  • Experimental augmentations: RandAugment is used for CIFAR-10, CIFAR-100, and Pascal VOC2007, while UCF-101 uses random crop and temporal jittering.Mixup is additionally tested as a possible augmentation for UPS.
  • No input augmentations: 28.14% and 14.98% error rates are achieved on CIFAR-10 with no input augmentations using 1000 and 4000 labels, respectively.These results improve upon Π model and Mean Teacher under the same no-augmentation setting.
  • Class balancing: With limited labels, pseudo-label selection becomes class-imbalanced because the network favors easy classes, especially during initial iterations.For CIFAR-10 with 1000 labels, selected pseudo-labels range from 1065 for cat to 3734 for automobile, producing an imbalance ratio of 3.5.
  • Class balancing: UPS uses class-balanced pseudo-label selection for CIFAR-10 during the first 10 pseudo-labeling iterations.The adjustment targets the early class imbalance observed with limited labeled data.

G EXPECTED CALIBRATION ERROR (ECE) COMPUTATION

The paper measures calibration with Expected Calibration Error and notes that standard single-label label sharpening cannot be applied to multi-label predictions.

  • ECE computation: Expected Calibration Error measures calibration by averaging the discrepancy between confidence and accuracy across 15 equally spaced confidence bins.The paper computes ECE on a selected subset of unlabeled samples rather than conventionally over an entire test set.
  • Multi-label limitation: Label sharpening for single-label predictions cannot be applied to multi-label predictions because multi-label outputs assume class independence.This constraint affects the use of MixMatch for Pascal VOC2007.
  • Multi-label limitation: For multi-label outputs, sharpening can instead be applied independently to each output by dividing logits by temperature T and applying a sigmoid.The experiments found no significant change from this operation and therefore reported MixMatch results without label sharpening.

I HYPERPARAMETER SELECTION

UPS thresholds are selected from CIFAR-10 validation distributions and then reused across datasets, while uncertainty-aware selection is favored over confidence-only thresholding.

  • Threshold selection: The validation distributions are skewed toward confidence 1 and uncertainty 0, guiding thresholds that retain most pseudo-labels while maintaining accuracy.The distributions are examined on a 1000-sample CIFAR-10 validation set.
  • Threshold selection: A confidence threshold τp = 0.7 and uncertainty threshold κp = 0.05 select 531 CIFAR-10 validation labels with 92.28% accuracy.Once κp is fixed, varying the confidence threshold produces similar selection counts and accuracy under the reported setting.
  • Cross-dataset reuse: Fixed hyperparameters obtained from CIFAR-10 yield strong performance on CIFAR-100 and UCF-101 without dataset-specific hyperparameter tuning.The paper still notes that dataset-specific tuning could find better hyperparameters.
  • Calibration and threshold selection: Confidence-only thresholding can work at some threshold, but finding a threshold that transfers across datasets is difficult; UPS provides robust confidence and uncertainty thresholds.The comparison reports that calibrated confidence thresholding remains insufficient to outperform uncertainty-aware selection even with adjusted thresholds.
  • Negative labels: Negative labels in this work incorporate more unlabeled samples and extend pseudo-labeling to multi-label classification.Their negative labels are selected through an uncertainty-aware process rather than solely through confidence-based selection.

L QUALITATIVE RESULTS

UPS filters incorrect high-confidence CIFAR-10 pseudo-labels by using prediction uncertainty, addressing failures that confidence-based selection cannot reliably remove.

  • Incorrect high-confidence predictions: All incorrect CIFAR-10 predictions shown have confidence greater than 0.9, so no reasonable confidence-based criterion would filter them out.The examples come from a network trained with 1000 labeled samples.
  • Uncertainty-aware filtering: UPS filters these incorrect pseudo-labels by leveraging prediction uncertainty.The figure illustrates the adverse effect of poor calibration and the benefit of uncertainty-aware selection.
Loading 2101.06329v3…