PaperlayerView exact revision on arXiv

Source-linked AI summary

FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, Takahiro Shinozaki

arXiv:2110.08263v3cs.LGcs.CV

TL;DR

FixMatch and related SSL methods use a constant confidence threshold across classes, limiting how unlabeled data are selected despite differing class learning status and difficulty. The paper proposes Curriculum Pseudo Labeling, which dynamically adjusts class-specific thresholds without extra parameters or computation, and applies it to FixMatch as FlexMatch. FlexMatch improves benchmark performance and convergence, with especially strong gains under scarce labels or challenging tasks.

  • Problem

    Fixed thresholds in modern SSL methods treat all classes equally and can exclude useful unlabeled data while ignoring different class learning difficulties.

  • Method

    Curriculum Pseudo Labeling dynamically adjusts each class’s threshold according to current learning status without adding parameters or forward or backward computation.

  • Results

    FlexMatch achieves state-of-the-art performance on most SSL image-classification benchmarks, converges faster, and reaches final accuracy in less than 1/5 of FixMatch’s training time.

  • Takeaways & Limitations

    CPL can be integrated into other modern SSL algorithms and improves their accuracy and convergence speed.

  • Takeaways & Limitations

    Performance can be less favorable on unbalanced datasets such as SVHN, where low class thresholds may admit noisy pseudo-labeled samples.

Abstract

from arXiv · show

The recently proposed FixMatch achieved state-of-the-art results on most semi-supervised learning (SSL) benchmarks. However, like other modern SSL algorithms, FixMatch uses a pre-defined constant threshold for all classes to select unlabeled data that contribute to the training, thus failing to consider different learning status and learning difficulties of different classes. To address this issue, we propose Curriculum Pseudo Labeling (CPL), a curriculum learning approach to leverage unlabeled data according to the model's learning status. The core of CPL is to flexibly adjust thresholds for different classes at each time step to let pass informative unlabeled data and their pseudo labels. CPL does not introduce additional parameters or computations (forward or backward propagation). We apply CPL to FixMatch and call our improved algorithm FlexMatch. FlexMatch achieves state-of-the-art performance on a variety of SSL benchmarks, with especially strong performances when the labeled data are extremely limited or when the task is challenging. For example, FlexMatch achieves 13.96% and 18.96% error rate reduction over FixMatch on CIFAR-100 and STL-10 datasets respectively, when there are only 4 labels per class. CPL also significantly boosts the convergence speed, e.g., FlexMatch can use only 1/5 training time of FixMatch to achieve even better performance. Furthermore, we show that CPL can be easily adapted to other SSL algorithms and remarkably improve their performances. We open-source our code at https://github.com/TorchSSL/TorchSSL.

1 Introduction

Modern SSL methods use unlabeled data through confidence-based pseudo-labeling, but fixed thresholds treat classes uniformly and can overlook useful samples. CPL addresses this by dynamically adjusting class-specific thresholds; combined with FixMatch as FlexMatch, it improves accuracy and convergence while remaining efficient.

  • Motivation: Fixed confidence thresholds ignore class-specific learning difficulties and exclude many unlabeled samples, especially early in training.FixMatch and related methods use only samples above a fixed confidence threshold for the unsupervised loss.
  • Curriculum Pseudo Labeling: CPL dynamically adjusts a threshold for each class according to its current learning status without adding parameters or forward or backward computation.The method replaces pre-defined thresholds with flexible class-specific thresholds and is applied to FixMatch as FlexMatch.
  • Results: FlexMatch achieves state-of-the-art performance on most SSL image-classification benchmarks and converges faster while retaining FixMatch-level training efficiency.With CPL, FlexMatch reaches final accuracy in less than 1/5 of FixMatch’s training time, and adapting CPL to other SSL algorithms also improves accuracy and convergence speed.
  • Results: 18.96% relative performance improvement over FixMatch is reported on STL-10 with 400 labels.The reported improvements over FixMatch on STL-10 are 18.96%, 16.11%, and 7.68% for label amounts of 400, 2500, and 10000, respectively.
  • Contributions: CPL is presented as an almost cost-free approach that can be integrated into other SSL methods.The paper also releases TorchSSL, a unified PyTorch-based codebase for studying SSL algorithms.

2 Background

SSL commonly combines consistency regularization with pseudo-labeling, using confidence thresholds to mask noisy unlabeled data. CPL extends this framework with curriculum learning because classes can differ in learning difficulty and a constant threshold may be suboptimal.

  • Consistency Regularization: Consistency regularization encourages similar model outputs under different augmentations and is used with pseudo-labeling in modern SSL methods.The basic consistency loss can use an ℓ-2 objective, while pseudo-labeling converts consistency regularization into entropy minimization for classification.
  • Curriculum Pseudo Labeling: CPL estimates class-wise learning effects from above-threshold unlabeled predictions and uses them to adjust flexible thresholds.These estimated effects can decrease when later predictions shift toward other classes.
  • Pseudo Labeling: A confidence threshold masks low-confidence unlabeled samples from the unsupervised loss to reduce noisy pseudo-labels.The thresholded loss uses cross-entropy with either hard one-hot or sharpened soft pseudo-labels.
  • FixMatch: FixMatch generates pseudo-labels from weakly augmented unlabeled data and uses them as targets for strongly augmented data.FixMatch combines this procedure with strong augmentation and consistency regularization.
  • Curriculum Learning: Curriculum learning motivates replacing a constant threshold because different classes may have different learning difficulties.The goal is to introduce samples according to the model’s learning process so that training remains appropriately challenging.

3 FlexMatch

FlexMatch applies Curriculum Pseudo Labeling to adjust pseudo-label thresholds by class and training step, using estimated learning effects without extra validation data or inference. Its warm-up, normalization, and nonlinear mapping mechanisms adapt data selection as classes become learned.

  • Curriculum Pseudo Labeling: CPL replaces fixed pseudo-label thresholds with class-specific thresholds dynamically adjusted according to each class’s learning status.Lower estimated learning effects produce lower thresholds, while better-learned classes retain stricter filtering.
  • Curriculum Pseudo Labeling: Estimating learning status from unlabeled-data prediction behavior avoids an extra validation set and additional inference processes.A validation-based alternative would require scarce labeled data and repeated evaluations that slow training.
  • Curriculum Pseudo Labeling: CPL normalizes each class’s estimated learning effect into β_t(c), which scales the fixed threshold τ through T_t(c) = β_t(c) · τ.The best-learned class has β_t(c) = 1 and keeps threshold τ, while harder classes receive lower thresholds.
  • FlexMatch algorithm: FlexMatch combines supervised and unsupervised losses while updating flexible thresholds at each iteration without additional forward propagation or new parameters.CPL reuses prediction-confidence marking performed when the consistency loss is computed.
  • Curriculum Pseudo Labeling: A warm-up process addresses unreliable early estimates caused by prediction bias toward a particular class.The warm-up duration depends on the unlabeled-data amount and the dataset’s learning difficulty.
  • Non-linear mapping function: The nonlinear mapping M is monotone increasing, ranges from 0 to 1, and constrains flexible thresholds to the interval from 0 to τ.The chosen convex function M(x) = x 2−x makes thresholds more sensitive as β_t(c) becomes larger.

4 Experiments

Experiments show that FlexMatch improves SSL performance and convergence while preserving FixMatch’s training efficiency. Gains are strongest with scarce labels or challenging datasets, although class imbalance can limit performance.

  • Main results: FlexMatch achieves state-of-the-art performance on most benchmark datasets, except SVHN, where Flex-UDA or UDA performs best on specific splits.Experiments cover CIFAR-10/100, STL-10, SVHN, and ImageNet.
  • Main results: 39.94% average error rate versus FixMatch’s 46.42% is achieved on CIFAR-100 with 400 labels.This split provides only 4 labeled samples per class.
  • Main results: CPL reduces UDA’s error rate from 37.4% to 29.53% on the STL-10 40-label split without adding computational burden.The authors also report that CPL can improve other threshold-based SSL algorithms.
  • Main results: 29.15% error rate versus FixMatch’s 35.97% is achieved on STL-10 with 40 labels, a relative improvement of 18.96%.STL-10 contains unlabeled images from a broader distribution, making it more challenging than its labeled set.
  • Limitations: FlexMatch performs less favorably on SVHN because class imbalance can keep thresholds low for minority classes, allowing noisy pseudo-labels to persist.The authors note that FixMatch’s fixed high threshold is less problematic on this relatively simple dataset.
  • ImageNet: On ImageNet, CPL still improves performance despite class imbalance, but results after 220 iterations do not represent fully converged or fully tuned performance.The ImageNet comparison uses 100K labeled examples and shared hyperparameters.
  • Convergence: FlexMatch converges faster than FixMatch: its loss decreases faster and more smoothly, and its CIFAR-10 accuracy reaches 94.29% by iteration 200K.FixMatch reaches 56.35% at 200K iterations, while FlexMatch exceeds FixMatch’s final accuracy after 1M iterations.

5 Related Work

Prior work combines curriculum learning with semi-supervised learning and explores dynamic thresholds in related tasks, but uses varied mechanisms and settings.

  • Pseudo-Labeling converts model predictions into hard artificial labels for semi-supervised learning.
  • UDA and FixMatch use confidence-based thresholding with strong augmentations, while UDA uses sharpened soft labels and FixMatch uses one-hot hard labels.
  • Curriculum learning has been applied to optimize unlabeled-image selection, facilitate segmentation, and identify in-distribution samples.
  • Dynamic thresholds have been studied in sentiment analysis and semantic segmentation, including gradual threshold reduction and automated thresholding.

6 Conclusion and Future Work

The paper presents CPL as a simple, low-cost approach that improves the performance and convergence speed of threshold-based SSL algorithms. FlexMatch, combining CPL with FixMatch, achieves state-of-the-art performance across diverse SSL benchmarks, while long-tail settings remain future work.

  • CPL improves the performance and convergence speed of threshold-based SSL algorithms without additional parameters or substantial computational cost.
  • FlexMatch, the CPL-enhanced version of FixMatch, achieves state-of-the-art performance on a variety of SSL benchmarks.
  • The authors identify extremely imbalanced unlabeled classes in long-tail scenarios as a direction for future improvement.

Broader Impact

CPL addresses class-dependent learning difficulty in modern SSL by using the model’s learning status to guide unlabeled-data utilization. The paper reports improvements in both convergence speed and final accuracy and encourages further study of this approach.

  • CPL considers the inherent learning difficulties of different classes during SSL training.
  • Using class-dependent learning status is associated with improved convergence speed and final accuracy.
  • The authors encourage future work on utilizing unlabeled data according to model learning status and per-class learning difficulty.

Funding Disclosure

The paper reports computing-resource support and partial foundation support, and directs readers to tables containing algorithm-dependent and algorithm-independent hyperparameters.

  • Computing resources were granted by Tokyo Institute of Technology and Microsoft Research Asia, with partial support from the Toray Science Foundation.
  • Detailed hyperparameter settings are provided separately for algorithm-dependent and algorithm-independent parameters.
  • Table 5 lists class-wise accuracy comparisons on the CIFAR-10 40-label split.

A.3 Median error rates

Median error rates over the last 20 checkpoints show that CPL improves existing SSL algorithms, with FlexMatch achieving the best accuracy under matched iteration counts.

  • CPL dramatically improves the performance of existing SSL algorithms in median error rates over the last 20 checkpoints.Methods ran the same number of iterations, with 1000 iterations between checkpoints.
  • FlexMatch achieves the best accuracy among the evaluated methods.
  • The median-error-rate findings are consistent with the main-text results and support CPL's effectiveness.

A.4 Detailed results

On CIFAR-10, CPL provides the best reported precision, recall, F1 score, and AUC alongside reduced error rates. The accompanying TorchSSL implementation includes eight popular SSL methods and uses a BatchNorm Controller to address instability during separate labeled and unlabeled forwards.

  • A.4 Detailed results: CPL achieves the best precision, recall, F1 score, and AUC performance on CIFAR-10, in addition to reduced error rates.AUC denotes area under the curve.
  • A.4 Detailed results: TorchSSL is an open-source PyTorch SSL toolbox containing eight popular semi-supervised learning methods.The listed methods include Π-Model, Pseudo-Labeling, VAT, Mean Teacher, MixMatch, ReMixMatch, UDA, and FixMatch.
  • A.4 Detailed results: The BatchNorm Controller updates BatchNorm only on labeled data when labeled and unlabeled data are forwarded separately to address observed instability.The procedure records BatchNorm statistics before unlabeled-data propagation and restores them afterward.

B.2 Benchmark results

TorchSSL benchmarks its algorithms across CIFAR-10, CIFAR-100, SVHN, and STL-10, reporting best error rates as reference results for the toolbox.

  • B.2 Benchmark results: TorchSSL evaluates all implemented algorithms on four common SSL datasets: CIFAR-10, CIFAR-100, SVHN, and STL-10.
  • B.2 Benchmark results: The benchmark section reports best error rates for the evaluated methods across the four datasets.Results are presented in Tables 8, 9, 10, and 11, respectively.
  • B.2 Benchmark results: These benchmark results provide a reference for using the TorchSSL toolbox.
Loading 2110.08263v3…