PaperlayerView exact revision on arXiv

Source-linked AI summary

CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning

Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, Fan Yang

arXiv:2102.09559v2cs.CV

TL;DR

Class-imbalanced SSL is under-studied, and missing labels make it difficult to rebalance unlabeled data even though minority-class pseudo-labels can have high precision. CReST iteratively expands the labeled set with preferentially selected minority pseudo-labels, while CReST+ progressively strengthens distribution alignment. Across long-tailed CIFAR and ImageNet127 experiments, the methods improve baseline SSL and minority-class recall, though very high imbalance can leave minority training data insufficient.

  • Problem

    Class-imbalanced SSL is under-studied, and missing label information prevents direct rebalancing of the unlabeled set.

  • Method

    CReST iteratively retrains a baseline SSL model by adding pseudo-labeled samples selected more frequently from predicted minority classes; CReST+ progressively increases distribution-alignment strength.

  • Results

    CReST and CReST+ improve baseline SSL by up to 11.8% accuracy on CIFAR-LT, up to 4.0% over DARP, and 7.9% recall on ImageNet127.

  • Takeaways & Limitations

    The framework reduces majority-class bias by exploiting high-quality minority pseudo-labels and improving recall on minority classes across imbalanced SSL settings.

  • Takeaways & Limitations

    When the imbalance ratio is very high, such as γ = 200, performance is constrained by insufficient minority-class training samples.

Abstract

from arXiv · show

Semi-supervised learning on class-imbalanced data, although a realistic problem, has been under studied. While existing semi-supervised learning (SSL) methods are known to perform poorly on minority classes, we find that they still generate high precision pseudo-labels on minority classes. By exploiting this property, in this work, we propose Class-Rebalancing Self-Training (CReST), a simple yet effective framework to improve existing SSL methods on class-imbalanced data. CReST iteratively retrains a baseline SSL model with a labeled set expanded by adding pseudo-labeled samples from an unlabeled set, where pseudo-labeled samples from minority classes are selected more frequently according to an estimated class distribution. We also propose a progressive distribution alignment to adaptively adjust the rebalancing strength dubbed CReST+. We show that CReST and CReST+ improve state-of-the-art SSL algorithms on various class-imbalanced datasets and consistently outperform other popular rebalancing methods. Code has been made available at https://github.com/google-research/crest.

1. Introduction

Imbalanced SSL is under-studied and difficult because missing labels prevent direct rebalancing, while biased pseudo-labels can intensify majority-class bias. CReST addresses this by preferentially adding high-precision minority pseudo-labels across generations, improving balanced-test performance.

  • Motivation: Imbalanced SSL remains under-studied because missing labels prevent rebalancing the unlabeled set and biased pseudo-labels can intensify model bias.Existing SSL algorithms have not been thoroughly evaluated on imbalanced distributions.
  • Motivation: Existing SSL models often achieve high recall on majority classes but low recall on minority classes, which harms accuracy on balanced test sets.FixMatch recognizes only 8.4% of the most-minority CIFAR10-LT samples correctly, versus 98.5% and 99.7% recall for the two largest classes.
  • Motivation: Minority classes nevertheless yield surprisingly high pseudo-label precision, despite their low recall.This property motivates using minority predictions as relatively reliable additions to the labeled set.
  • CReST framework: CReST retrains a baseline SSL model over generations, adding pseudo-labeled samples with higher selection probability for predicted minority classes.Selection probabilities are based on the class distribution estimated from the labeled set.
  • Results: CReST and CReST+ improve baseline SSL performance by up to 11.8% on CIFAR-LT, up to 4.0% over DARP, and by 7.9% recall on ImageNet127.The method particularly improves recall on minority classes.

2. Related work

Related work covers mainstream SSL techniques and established approaches for class-imbalanced supervised learning. Class-imbalanced SSL has received comparatively limited attention, with recent methods targeting pseudo-label refinement or minority-class consistency.

  • Semi-supervised learning: SSL methods commonly use entropy minimization, pseudo-labeling, predictive probabilities with temperature scaling, or consistency regularization.These techniques form the shared foundation of many recent SSL approaches.
  • Class-imbalanced learning: Class-imbalanced supervised learning includes re-sampling, re-weighting, instance weighting, knowledge transfer, and representation-classifier decoupling.These methods rebalance class contributions or transfer information toward minority classes.
  • Class-imbalanced SSL: Class-imbalanced SSL is under-explored, with recent work studying unlabeled-data benefits, suppressed minority-class consistency loss, and DARP pseudo-label refinement.DARP refines raw pseudo-labels through convex optimization.
  • Class-imbalanced SSL: FixMatch exhibits high recall but low precision on majority classes, and low recall but high precision on minority classes.This behavior motivates approaches that exploit minority-class pseudo-label precision rather than only correcting majority bias.

3. Class-Imbalanced SSL

Class-imbalanced SSL is under-studied and suffers from majority-class bias, especially poor minority recall despite high minority precision. CReST exploits this precision through class-rebalancing self-training, while CReST+ progressively strengthens distribution alignment.

  • Problem setup and baselines: Class-imbalanced SSL is under-studied, and skewed data can bias pseudo-labels toward majority classes, worsening minority-class performance.Existing SSL methods can amplify initial bias because subsequent training uses pseudo-labels generated by a classifier trained on imbalanced data.
  • A closer look at the model bias: FixMatch reaches 98.5% and 99.7% recall on the two largest CIFAR10-LT classes but only 8.4% recall on the most minority class.On the balanced test set, this minority recall deficit lowers recall averaged over classes, also called accuracy.
  • A closer look at the model bias: Minority classes retain 97.7% and 98.3% precision on CIFAR10-LT, despite their poor recall, motivating selective use of minority pseudo-labels.The model predicts many minority samples as majority classes, producing high minority precision but low minority recall.
  • Class-rebalancing self-training: CReST alternates baseline SSL training with expansion of the labeled set using a selected subset of pseudo-labeled unlabeled samples.Less frequent classes receive higher sampling probabilities, and the most confident pseudo-labels within each class are selected.
  • Class-rebalancing self-training: CReST’s rebalancing uses estimated labeled-set class frequencies, selecting minority-class pseudo-labels more often while retaining conventional self-training when α = 0.Adding minority samples makes the labeled set more balanced and can yield a less biased classifier for subsequent online pseudo-labeling.
  • Progressive distribution alignment: CReST+ adds progressive distribution alignment, decreasing the temperature over generations to strengthen class rebalancing later in training.The schedule is intended to preserve high early-generation pseudo-label precision while enabling stronger late-generation rebalancing and fewer generations.

4. Experiments

Experiments show that CReST and CReST+ improve SSL accuracy across long-tailed CIFAR and ImageNet127 settings, while ablations clarify how rebalancing strength and distribution alignment affect performance.

  • CIFAR-LT results: CReST improves baseline FixMatch accuracy by 3.0%–11.8% across evaluated CIFAR-LT settings, with CReST+ providing the strongest gains.CReST alone reaches as much as 9.6% absolute gain; progressive distribution alignment adds further improvement.
  • CIFAR-LT results: CReST improves performance as labeled data increases and better utilizes labeled samples to reduce model bias under imbalanced class distributions.The method achieves 11.8% and 6.1% gains at γ = 100 with 10% and 30% labeled data, respectively.
  • CIFAR-LT results: At γ = 200, CReST is constrained by insufficient minority-class training samples, limiting its ability to improve performance.The authors attribute particularly strong gains at moderate imbalance to finding more correctly pseudo-labeled minority samples.
  • CIFAR-LT results: CReST consistently outperforms DARP by up to 4.0% accuracy on MixMatch and 2.4% on FixMatch across three imbalance ratios.Both methods are drop-in additions to standard SSL algorithms under the same datasets and protocol.
  • Ablation study: Class-rebalancing sampling improves iterative retraining over conventional self-training, but excessive strength reverses the imbalance and degrades majority-class performance.With α = 1, minority recall rises from 55.0% to 71.1%, while seven other classes suffer severe recall degradation and balanced accuracy drops 3.0%.
  • Ablation study: CReST and CReST+ trade small majority-class recall losses for substantial minority-class gains, improving recall across all classes overall.The same pattern appears on both the balanced test set and the imbalanced unlabeled set, supporting improved pseudo-label quality.

5. Conclusion

CReST is a class-rebalancing self-training framework for imbalanced semi-supervised learning, while CReST+ progressively increases distribution-alignment strength. Experiments show both methods substantially improve baseline SSL algorithms and outperform state-of-the-art rebalancing methods.

  • CReST refines a baseline SSL model by repeatedly adding high-quality pseudo-labels, with minority classes updated more aggressively than majority classes.
  • CReST+ extends CReST by progressively increasing distribution-alignment strength across self-training generations.
  • CReST and CReST+ improve baseline SSL algorithms by a large margin on long-tailed CIFAR datasets and ImageNet127.
  • Both methods consistently outperform state-of-the-art rebalancing methods in the reported experiments.

A.1. Details on learning rate schedule

The methods use cosine learning-rate decay during each training generation, with decay settings differing between FixMatch- and MixMatch-based models.

  • Cosine learning-rate decay sets the learning rate using the initial rate, current step, total generation steps, and decay-rate hyper-parameter.
  • CReST and CReST+ with a FixMatch base use l = 1, whereas CReST models with a MixMatch base use l = 5.

A.2. Details on datasets

The appendix evaluates CReST on class-imbalanced CIFAR-derived datasets and the naturally imbalanced ImageNet127 dataset, using class-balanced recall for ImageNet127. It also describes skewed class distributions and per-class precision–recall behavior across SSL algorithms.

  • ImageNet127: Because ImageNet127’s test set is not balanced, evaluation uses averaged class recall rather than accuracy.
  • ImageNet127: ImageNet127’s skewed distribution and adequate minority-class sample counts make it a test bed for imbalanced semi-supervised learning.

A.3. Precision and recall of other SSL algorithms

Across three SSL algorithms, majority classes tend to have high recall and low precision, whereas minority classes tend to have low recall and surprisingly high precision.

  • Pseudo-Labeling, Mean Teacher, and MixMatch exhibit the same class-dependent precision–recall pattern on CIFAR10-LT.
  • Majority classes obtain high recall but low precision, while minority classes suffer low recall but achieve surprisingly high precision.
Loading 2102.09559v2…