PaperlayerView exact revision on arXiv

Source-linked AI summary

SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning

Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, Marios Savvides

arXiv:2301.10921v2cs.LGcs.AIcs.CV

TL;DR

SSL must exploit abundant unlabeled data without allowing unreliable pseudo-labels to undermine learning. SoftMatch replaces hard thresholding with truncated-Gaussian confidence weighting and adds Uniform Alignment; experiments report improvements across image, text, and imbalanced classification benchmarks.

  • Problem

    Threshold-based pseudo-labeling faces an inherent trade-off: high thresholds preserve quality but discard correct samples, while low thresholds admit more potentially erroneous labels.

  • Method

    SoftMatch uses a dynamically estimated truncated Gaussian to weight pseudo-labels by confidence and Uniform Alignment to address class imbalance in pseudo-labels.

  • Results

    SoftMatch outperforms previous methods across varied image and text settings and achieves the best test error across the reported long-tailed settings.

  • Takeaways & Limitations

    Maintaining both high pseudo-label quantity and quality can improve utilization of unlabeled data across balanced, text, and imbalanced classification settings.

  • Takeaways & Limitations

    The weighting approach assumes the marginal prediction distribution follows a dynamic truncated Gaussian distribution.

Abstract

from arXiv · show

The critical challenge of Semi-Supervised Learning (SSL) is how to effectively leverage the limited labeled data and massive unlabeled data to improve the model's generalization performance. In this paper, we first revisit the popular pseudo-labeling methods via a unified sample weighting formulation and demonstrate the inherent quantity-quality trade-off problem of pseudo-labeling with thresholding, which may prohibit learning. To this end, we propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training, effectively exploiting the unlabeled data. We derive a truncated Gaussian function to weight samples based on their confidence, which can be viewed as a soft version of the confidence threshold. We further enhance the utilization of weakly-learned classes by proposing a uniform alignment approach. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.

1 INTRODUCTION

SSL seeks to improve generalization by learning from limited labeled data and abundant unlabeled data, but threshold-based pseudo-labeling faces a quantity-quality trade-off. SoftMatch addresses this trade-off with confidence-based weighting and Uniform Alignment, and is reported to outperform prior methods across image and text settings.

  • SSL aims to exploit large unlabeled datasets to improve models trained with few labeled examples.
  • Hard confidence thresholds preserve pseudo-label quality but discard many potentially correct samples, limiting unlabeled-data utilization.In the cited Two-Moon example, around 71% of correct pseudo-labels are excluded from training.
  • SoftMatch uses a truncated Gaussian weighting function to retain unconfident yet potentially correct pseudo-labels while maintaining pseudo-label quality.
  • Uniform Alignment addresses pseudo-label imbalance caused by different learning difficulties across classes.
  • Experiments report that SoftMatch outperforms previous methods across varied image and text evaluation settings.

2 REVISIT QUANTITY-QUALITY TRADE-OFF OF SSL

The paper reframes pseudo-labeling through a unified sample-weighting function, defining pseudo-label quantity and quality and explaining their trade-off. It argues that thresholding and other weighting schemes impose oversimplified assumptions on prediction distributions, motivating more structured weighting.

  • 2.1 PROBLEM STATEMENT: The unsupervised SSL objective can be formulated as weighted cross-entropy, with a sample-weighting function determining how unlabeled examples contribute.The weighting function λ(p) assigns weights in [0, λmax] to unlabeled samples.
  • 2.2 QUANTITY-QUALITY TRADE-OFF FROM SAMPLE WEIGHTING PERSPECTIVE: Pseudo-label quantity is defined as the expected sample weight over unlabeled data, while quality is the expected weighted 0/1 pseudo-label error.
  • 2.2 QUANTITY-QUALITY TRADE-OFF FROM SAMPLE WEIGHTING PERSPECTIVE: The trade-off arises because the weighting-induced PMF oversimplifies the actual distribution of model predictions.Ideally, λ(p) should reflect the true prediction distribution to achieve both high quantity and high quality.
  • 2.2 QUANTITY-QUALITY TRADE-OFF FROM SAMPLE WEIGHTING PERSPECTIVE: Naive pseudo-labeling maximizes quantity by assigning equal weight to every pseudo-label, but this can admit erroneous labels and reduce quality.
  • 2.2 QUANTITY-QUALITY TRADE-OFF FROM SAMPLE WEIGHTING PERSPECTIVE: Confidence thresholding uses a step-function weight that includes samples above τ and discards the rest, creating a quality-quantity trade-off.Higher τ improves quality but reduces enrolled samples; lower τ increases quantity while admitting more potentially erroneous pseudo-labels.

3 SOFTMATCH

SoftMatch adaptively weights pseudo-labels with a truncated Gaussian fitted to confidence distributions, while Uniform Alignment adjusts predictions toward less-represented classes. Together, these mechanisms target the quantity-quality trade-off and class imbalance in pseudo-labeling.

  • Gaussian Function for Sample Weighting: SoftMatch assumes confidence follows a dynamic truncated Gaussian with mean µt and variance σt, using confidence deviation from µt as a correctness proxy.The Gaussian parameters are adaptively estimated during training rather than fixed in advance.
  • Gaussian Function for Sample Weighting: Exponential Moving Average aggregates batch statistics to stabilize the adaptive estimation of Gaussian parameters.The method uses momentum m over previous batches and plugs the estimates into the sample-weighting function.
  • Gaussian Function for Sample Weighting: SoftMatch guarantees at least λmax/2 quantity during training while improving pseudo-label quality as confidence distributions become tighter.Highly unconfident erroneous pseudo-labels receive lower weights, while samples near ˆµt are utilized more efficiently.
  • Uniform Alignment for Fair Quantity: Uniform Alignment normalizes predictions using the ratio between a uniform class distribution and the EMA-estimated pseudo-label distribution.This adjustment compensates for class-level differences in learning difficulty and pseudo-label frequency.
  • Evaluation: Table 2 reports Top-1 error rate (%) for CIFAR-10, CIFAR-100, STL-10, and SVHN across three random seeds, with bold values marking the best results.Numbers marked with ∗ come from the original papers.
  • Uniform Alignment for Fair Quantity: Unlike Distribution Alignment, Uniform Alignment is integrated into pseudo-label weighting, where normalization can otherwise create erroneous pseudo-labels and damage quality.The distinction arises because Distribution Alignment uses normalized predictions as soft targets, whereas pseudo-labeling uses them to form hard-label weights.

4 EXPERIMENTS

SoftMatch is evaluated across balanced and long-tailed image, text, and ablation settings, generally outperforming prior methods. Results indicate improved pseudo-label utilization and robustness to imbalance and design choices.

  • Classic image classification: SoftMatch obtains state-of-the-art results on almost all classic image settings, with comparable performance on the listed exceptions.It surpasses FixMatch by 1.48% on SVHN with 40 labels and exceeds FlexMatch by 7.73%, 2.84%, and 1.33% on CIFAR-100, STL-10, and ImageNet settings, respectively.
  • Long-tailed image classification: SoftMatch achieves the best test error across all evaluated long-tailed settings.At γ = 150 on CIFAR-10-LT, it outperforms the second-best method by 2.4%.
  • Text classification: SoftMatch consistently outperforms other methods on text datasets, especially topic classification tasks.It reports 12.68% error on AG News with 40 labels and 1.68% on DBpedia with 70 labels, surpassing the second best by 2.81% and 0.5%.
  • Qualitative analysis: On CIFAR-10 with 250 labels, SoftMatch maintains the highest pseudo-label quantity and quality across training, including the best and worst learned classes.The reported behavior is associated with better convergence and error rate, especially during the first 50k iterations.
  • Ablation study: The truncated Gaussian weighting function is more robust than linear, quadratic, and truncated Laplacian alternatives.Adaptive Gaussian parameter estimation outperforms fixed parameters, while per-class estimation is comparable at 250 labels.
  • Ablation study: Uniform alignment improves performance on imbalanced SSL and remains robust to different target distributions on balanced settings.Removing UA causes a slight performance drop, while per-class estimation is significantly inferior on SVHN.

5 RELATED WORK

Related SSL work develops pseudo-labeling, consistency regularization, loss weighting, and confidence-based sample selection. These methods address data utilization and training ambiguity through weighting, ramp-up, and thresholding strategies.

  • Pseudo-labeling and consistency regularization: Pseudo-labeling generates artificial labels for unlabeled data, while consistency regularization encourages consistent predictions for similar inputs.Subsequent work improves these paradigms through loss weighting and related mechanisms.
  • Loss weighting: Loss-weight ramp-up progressively increases the unlabeled-data loss weight to avoid involving too much ambiguous data early in training.Per-sample loss weighting is another strategy for exploiting unlabeled data.
  • Confidence thresholding: Confidence-thresholding methods enroll high-confidence unlabeled samples, but fixed thresholds can limit utilization and produce imbalanced pseudo-label distributions.Dash gradually increases its threshold during training to improve unlabeled-data utilization.

6 CONCLUSION

The paper revisits pseudo-labeling through unified sample weighting and proposes SoftMatch with truncated Gaussian weighting and Uniform Alignment. Across varied tasks, the method maintains high pseudo-label quantity and quality and demonstrates broad effectiveness.

  • Conclusion: SoftMatch combines truncated Gaussian weighting with Uniform Alignment to address pseudo-label quantity-quality trade-offs.The paper frames improved weighting-function design as a direction for future work.
  • Conclusion: Extensive experiments demonstrate SoftMatch’s effectiveness across image, text, classic, and long-tailed classification tasks.The reported conclusion emphasizes high pseudo-label quantity and quality during training.

A.1 QUANTITY-QUALITY TRADE-OFF

The paper formulates pseudo-label quantity and quality through a unified sample-weighting perspective. The weighting function λ(p) is directly connected to the implicit confidence distribution assumed for pseudo-labels.

  • Unified formulation: The weighting function λ(p) ∈ [0, λmax] is directly related to the implicit probability mass function over pseudo-label confidence p.This connection provides the basis for analyzing quantity and quality across existing methods and SoftMatch.

A.1.1 QUANTITY AND QUALITY

The paper defines pseudo-label quantity as how much unlabeled data is enrolled through sample weights and quality as how much of that enrolled data is correctly pseudo-labeled.

  • Quantity: Quantity f(p) is the percentage of unlabeled data enrolled in the weighted unsupervised loss.It equals the average sample weight over unlabeled data.
  • Quality: Quality g(p) is the percentage of enrolled pseudo-labels that are correct, assuming unlabeled ground-truth labels are known.Correctness is determined using the pseudo-label argmax and the ground-truth label.
  • Unified weighting: The PMF of prediction confidence describes how sample weights are distributed across unlabeled examples.The PMF is nonnegative and sums to one.
  • Unified weighting: Choosing a sample-weighting function implicitly imposes an assumption on the PMF of prediction confidence.This connection enables theoretical analysis even though ground-truth labels are unknown in practice.

A.1.2 NAIVE PSEUDO-LABELING

Naive pseudo-labeling assigns the same maximum weight to every unlabeled example, fully enrolling pseudo-labels but exposing training to low-quality predictions and confirmation bias.

  • Naive pseudo-labeling: Naive pseudo-labeling directly uses model-generated pseudo-labels for training with a fixed maximum sample weight.The fixed weight λmax applies regardless of prediction confidence.
  • Naive pseudo-labeling: Full enrollment maximizes pseudo-label quantity but can produce low quality when the model is poorly learned.Early incorrect pseudo-labels may lead the model to memorize them and become confident in them.

A.1.3 LOSS WEIGHT RAMP UP

Loss-weight ramp-up gradually increases the common weight on unlabeled samples, delaying full enrollment so early training can focus on labeled data.

  • Loss weight ramp up: Ramp-up makes the sample weight a function of training iteration and increases it linearly to λmax after T warm-up iterations.The schedule is intended to avoid involving too many erroneous pseudo-labels early.
  • Loss weight ramp up: Because every unlabeled sample receives the same weight, ramp-up retains the same uniform PMF assumption and quality function as naive self-training.Changing the weight over time does not distinguish samples by confidence.

A.1.4 FIXED CONFIDENCE THRESHOLDING

Fixed confidence thresholding fully weights predictions above τ and ignores the rest, yielding a quantity-quality trade-off. SoftMatch replaces this hard filter with a dynamic truncated-Gaussian weighting scheme and improves utilization of unlabeled data.

  • Fixed confidence thresholding: Confidence thresholding assigns λmax to examples with max(p) ≥ τ and zero weight to the others.It therefore filters unlabeled data using a step-function sample weight.
  • Fixed confidence thresholding: A high threshold can improve pseudo-label quality but reduces quantity by excluding unconfident examples, producing an inherent trade-off.The uniform confidence assumption within [τ, 1] also fails to reflect the actual confidence distribution during training.
  • SoftMatch: SoftMatch models confidence with a truncated Gaussian below µt and a uniform distribution above it, estimating parameters from training predictions.The resulting weighting function is a soft alternative to fixed confidence thresholding.
  • SoftMatch: SoftMatch guarantees at least half of the possible contribution to the final loss, improving utilization of unlabeled data.Estimating σt from confidence makes the quantity lower bound tighter as the model becomes more confident.
  • SoftMatch: Lower-confidence pseudo-labels are down-weighted according to their deviation from µt, while higher-confidence examples retain threshold-like quality.As training improves, increased confidence and reduced variance strengthen this weighting behavior.
  • Ablation: Using the variance directly causes a slight performance drop, while 2σ and 3σ produce similar results.The main experiments use the 2σ range to control the softness of the truncated Gaussian.
  • Uniform Alignment: Uniform Alignment produces more balanced class-wise sample weights, helping the model learn more equally across classes.Unlike Distribution Alignment, it uses original predictions for pseudo-label computation to avoid creating more erroneous pseudo-labels after normalization.
Loading 2301.10921v2…