PaperlayerView exact revision on arXiv

Source-linked AI summary

FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning

Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, Xing Xie

arXiv:2205.07246v3cs.LGcs.CV

TL;DR

Existing SSL thresholding methods rely on fixed or ad-hoc thresholds, limiting effective use of unlabeled data. FreeMatch uses self-adaptive thresholding and class-fairness regularization, and it outperforms strong competitors across SSL benchmarks, especially with extremely limited labels.

  • Problem

    Existing threshold-based SSL methods require fixed thresholds or ad-hoc adjustment schemes to select confident unlabeled samples.

  • Method

    FreeMatch combines self-adaptive thresholding, which estimates global and class-specific thresholds from model predictions during training, with class-fairness regularization encouraging diverse predictions.

  • Results

    FreeMatch achieves superior performance across varied SSL benchmarks, especially in barely supervised settings, including a 5.78% error reduction on CIFAR-10 with 10 labels.

  • Takeaways & Limitations

    FreeMatch is particularly effective when labeled data are extremely limited, with large improvements reported on CIFAR-10 and STL-10.

  • Takeaways & Limitations

    The motivating analysis considers a binary classification problem with an even mixture of two Gaussians, limiting that analysis's demonstrated setting.

Abstract

from arXiv · show

Semi-supervised Learning (SSL) has witnessed great success owing to the impressive performances brought by various methods based on pseudo labeling and consistency regularization. However, we argue that existing methods might fail to utilize the unlabeled data more effectively since they either use a pre-defined / fixed threshold or an ad-hoc threshold adjusting scheme, resulting in inferior performance and slow convergence. We first analyze a motivating example to obtain intuitions on the relationship between the desirable threshold and model's learning status. Based on the analysis, we hence propose FreeMatch to adjust the confidence threshold in a self-adaptive manner according to the model's learning status. We further introduce a self-adaptive class fairness regularization penalty to encourage the model for diverse predictions during the early training stage. Extensive experiments indicate the superiority of FreeMatch especially when the labeled data are extremely rare. FreeMatch achieves 5.78%, 13.59%, and 1.28% error rate reduction over the latest state-of-the-art method FlexMatch on CIFAR-10 with 1 label per class, STL-10 with 4 labels per class, and ImageNet with 100 labels per class, respectively. Moreover, FreeMatch can also boost the performance of imbalanced SSL. The codes can be found at https://github.com/microsoft/Semi-supervised-learning.

1 INTRODUCTION

FreeMatch addresses the limited and poorly adapted confidence thresholding used in SSL by adjusting global and class-specific thresholds to model learning status, while adding fairness regularization. It reports strongest gains when labeled data are scarce.

  • SSL methods use fixed or ad-hoc confidence thresholds, limiting unlabeled-data utilization and potentially slowing convergence.
  • FreeMatch uses self-adaptive thresholding to adjust global and local confidence thresholds according to the model’s learning status.The approach is motivated by estimating learning status from model predictions.
  • FreeMatch combines Self-Adaptive Thresholding with Self-Adaptive class Fairness regularization to encourage diverse predictions.The fairness component is intended to address barely supervised settings.
  • 5.78% error reduction is reported on CIFAR-10 with 1 labeled sample per class compared with the latest state-of-the-art method.

2 A MOTIVATING EXAMPLE

The motivating analysis shows that confidence thresholds should adapt to model learning status and differ across classes. Low early thresholds improve utilization and convergence, whereas later training requires higher thresholds to limit confirmation bias.

  • The motivating example studies thresholding in a binary classification problem with an even mixture of two Gaussian classes.The analysis simplifies the actual model and training process to derive threshold-setting implications.
  • The parameter β represents model learning status and is expected to grow as the model becomes more confident.
  • Larger thresholds reduce unlabeled-data utilization and can slow early training when model confidence is still low.
  • Larger thresholds can produce imbalanced pseudo labels, distorting the decision boundary through pseudo-label bias.Class-specific thresholds are suggested as a remedy for this imbalance.
  • When classes become more similar, a moderate threshold balances sampling because many samples occupy low-confidence regions.
  • Thresholds should rise with β: low early values encourage utilization and diverse pseudo labels, while higher later values reduce confirmation bias.

3 PRELIMINARIES

The SSL formulation combines supervised cross-entropy on labeled data with confidence-thresholded pseudo-label training on weakly and strongly augmented unlabeled data. A fairness objective encourages balanced class predictions, but a fixed uniform prior has limitations.

  • SSL uses labeled and unlabeled datasets, denoted D_L and D_U, respectively.
  • The supervised objective applies cross-entropy between labeled targets and model probabilities under stochastic augmentation.
  • The unsupervised objective uses confidence-thresholded hard pseudo labels with weak and strong augmentations.Weak augmentation uses random crop and flip; strong augmentation uses RandAugment.
  • A fairness objective encourages each class to be predicted at the same frequency using a uniform prior distribution.
  • A uniform prior may not generalize to non-uniform data and ignores mini-batch pseudo-label imbalance caused by sampling.

4 FREEMATCH

FreeMatch’s SAT automatically sets global and class-specific confidence thresholds from EMA prediction statistics, increasing selectivity as learning progresses. Its SAF objective additionally promotes diverse predictions, especially when labeled data are scarce.

  • Self-adaptive Global Threshold: SAT estimates a global threshold from EMA confidence on unlabeled data and adjusts it over training to reflect overall learning status.The EMA avoids repeatedly computing confidence across the full unlabeled dataset.
  • Threshold Adaptation: At training start, thresholds remain low to accept more potentially correct samples; as confidence rises, they increase to filter incorrect pseudo labels and reduce confirmation bias.This schedule targets faster early convergence while becoming more selective later.
  • Self-adaptive Local Threshold: SAT modulates the global threshold with class-specific thresholds estimated from EMA class predictions to account for intra-class diversity and class adjacency.The final threshold is τt(c) = MaxNorm(˜pt(c)) · τt.
  • Self-adaptive Fairness: SAF normalizes EMA prediction expectations by pseudo-label histogram distributions to counter imbalance while encouraging diverse predictions.The objective is intended to produce meaningful self-adaptive thresholds in barely supervised settings.
  • Overall Objective: FreeMatch combines the unsupervised loss and SAF with separate weights in its overall training objective.The paper states that the combined objectives maximize mutual information between outputs and inputs.

5 EXPERIMENTS

Experiments evaluate FreeMatch across standard SSL benchmarks, with additional qualitative and ablation analyses of its adaptive thresholding and fairness components. FreeMatch achieves especially strong results when labeled data are extremely limited.

  • 5.1 SETUP: FreeMatch is evaluated on CIFAR-10/100, SVHN, STL-10, and ImageNet with varying amounts of labeled data.
  • 5.1 SETUP: Experiments use a unified TorchSSL codebase with shared backbones and hyperparameters for fair comparison.
  • 5.2 QUANTITATIVE RESULTS: FreeMatch achieves the best performance on CIFAR-10, STL-10, and ImageNet, closely matches the best competitor on SVHN, and beats ReMixMatch on CIFAR-100 with 400 labels.
  • 5.2 QUANTITATIVE RESULTS: On ImageNet with 100 labels per class, FreeMatch outperforms FlexMatch by 1.28% and has faster computation because FlexMatch requires heavy sample-indexing operations.
  • 5.2 QUANTITATIVE RESULTS: 5.78% error reduction occurs on CIFAR-10 with 10 labels, 1.96% on CIFAR-100 with 400 labels, and 13.59% on STL-10 with 40 labels.
  • 5.3 QUALITATIVE ANALYSIS: On STL-10 with 40 labels, FreeMatch starts with lower thresholds, increases them as confidence grows, accelerates convergence, and produces better class-wise accuracy.
  • 5.3 QUALITATIVE ANALYSIS: FreeMatch’s class-specific thresholds better reflect learning status than FlexMatch’s fixed 0.95 threshold for confused classes, retaining 0.87 versus 0.84 for similar classes.
  • 5.4 ABLATION STUDY: SAT performs best among evaluated threshold schemes, while its global and local threshold components individually achieve comparable results.

6 RELATED WORK

Related work covers confidence thresholding for pseudo-label quality and entropy-based regularization for prediction confidence or fairness in SSL.

  • Class fairness objectives are compared experimentally, with SAF outperforming FlexMatch in both tested settings and remaining effective when combined with FlexMatch.
  • Confidence-based thresholding retains unlabeled samples whose confidences exceed a threshold to reduce confirmation bias in pseudo labeling.
  • Prior entropy-based fairness methods encourage predictions across classes to occur at similar frequencies, often assuming a uniform class prior.

7 CONCLUSION

FreeMatch combines self-adaptive thresholding with class-fairness regularization for SSL and outperforms strong competitors across benchmarks, especially when supervision is scarce.

  • FreeMatch uses self-adaptive thresholding and class-fairness regularization for SSL.
  • FreeMatch outperforms strong competitors across varied SSL benchmarks, particularly in barely-supervised settings.
  • The authors identify prediction-based adaptiveness as a potential limitation and call for research on optimal thresholding.

A EXPERIMENTAL DETAILS OF THE “TWO-MOON” DATASET.

The two-moon experiment uses two labeled points and 1,000 unlabeled points to compare samples selected by FreeMatch and FixMatch thresholds.

  • The experiment generates one labeled sample per class and 1,000 unlabeled samples in two-dimensional space.
  • A 3-layer MLP with 64 neurons per layer and ReLU activation is trained for 2,000 iterations.
  • Red samples have confidence above FreeMatch’s threshold but below FixMatch’s threshold.
  • The sampling rate is computed on unlabeled data as PNU b 1(max(qb) > τ)/NU.

B PROOF OF THEOREM 2.1

The proof analyzes pseudo-label probabilities in a binary Gaussian-mixture setting and establishes how class-separation changes affect the probability of an unassigned pseudo-label.

  • The theorem concerns a binary classification problem and gives the probability distribution of the pseudo-label Yp.
  • P(Yp = 0) increases as µ2 − µ1 becomes smaller.
  • The proof defines pseudo-label assignments through threshold-based conditions on x for labels 1 and −1.
  • Conditional pseudo-label probabilities are obtained by integrating over x.
  • The proof establishes monotonic increase of the relevant Gaussian-CDF expression using its derivative and the standard-normal density.
  • The FreeMatch algorithm updates global and local thresholds from unlabeled batches and computes corresponding histograms during each training step.
  • The algorithm computes the unlabeled loss using confidence-based thresholding and returns Ls + wu · Lu + wf · Lf.

D HYPERPARAMETER SETTING

The appendix documents reproduction settings, significance testing, and experimental limitations for the reported benchmark results.

  • The appendix provides detailed algorithm-dependent and algorithm-independent hyperparameter settings for reproducing FreeMatch experiments.
  • ImageNet uses batch size 128, while FixMatch uses batch size 1024 and a different optimizer.
  • The Friedman test reports F value τF = 3.56, exceeding both the α = 0.05 and α = 0.1 thresholds.
  • The significance analysis indicates differences between all evaluated algorithms.

E.3 DETAILED RESULTS

FreeMatch maintains strong classification performance across multiple metrics and datasets, while thresholding and fairness ablations support its design choices. Its performance is also relatively insensitive to EMA decay, whereas fixed-threshold methods are sensitive to threshold selection.

  • FreeMatch achieves the best precision, recall, F1 score, and AUC across the evaluated CIFAR-10/100, SVHN, and STL-10 settings.
  • Fixed-threshold FixMatch and FlexMatch are quite sensitive to changes in the predefined threshold τ.
  • Different EMA decay values produce close results on CIFAR-10 with 40 labels, indicating limited sensitivity to this hyper-parameter.
  • FreeMatch outperforms FlexMatch across both compared class-fairness settings, and SAF remains effective when combined with FlexMatch.

E.7 ABLATION OF IMBALANCED SSL

FreeMatch is evaluated for imbalanced SSL and related diagnostics, achieving the strongest reported performance across imbalance settings while producing less confusing feature clusters and supporting more accurate pseudo-labeling. The experiments also examine thresholding, fairness, and confusion patterns under limited or imbalanced supervision.

  • Imbalanced SSL results: FreeMatch achieves the best performance across all reported imbalanced SSL settings and outperforms the second-best method by 2.4% on CIFAR-10 at imbalance ratio 150.
  • Imbalanced SSL results: When combined with another imbalanced SSL method, FreeMatch still attains the best performance in most settings.
  • Feature visualization: FreeMatch produces better STL-10 feature representations than FlexMatch, with less confusing clusters in the T-SNE visualization.
  • Pseudo-label analysis: Using a high fixed threshold can prevent unlabeled samples from training, causing overfitting on labeled data and a small amount of unlabeled data.
  • Pseudo-label analysis: Introducing appropriate unlabeled data during training can avoid overfitting and produce more accurate pseudo labels.
  • Confusion analysis: With the least prototypical labeled data, FreeMatch still obtains good CIFAR-10 results while other SSL methods fail to separate unlabeled data into distinct clusters.
Loading 2205.07246v3…