Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning

Jaehyung Kim; Youngbum Hur; Sejun Park; Eunho Yang; Sung Ju Hwang; Jinwoo Shin

Source-linked AI summary

Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning

Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, Jinwoo Shin

arXiv:2007.08844v2Jul 17, 2020cs.LGstat.ML

TL;DR

Class imbalance biases SSL pseudo-labels toward majority classes, undermining balanced-test performance and minority-class accuracy. DARP softly refines those pseudo-labels through a convex distribution-matching optimization while preserving original information, and improves state-of-the-art SSL methods across tested imbalanced scenarios. Its unlabeled-distribution estimation relies on a labeled/unlabeled confusion-matrix similarity assumption when class counts are otherwise unknown.

Problem
Class-imbalanced SSL can produce pseudo-labels more biased toward majority classes than the underlying data, creating a gap for balanced testing and minority-class performance.
Method
DARP solves a convex optimization that matches refined pseudo-labels to the true unlabeled class distribution while minimizing distortion from the original pseudo-labels.
Results
DARP improves MixMatch, ReMixMatch, and FixMatch by up to 77.2%, 31.4%, and 53.1% relative reductions in balanced test error, respectively, across tested scenarios.
Takeaways & Limitations
DARP is presented as a simple method compatible with existing pseudo-label-based SSL algorithms and as a strong baseline for class-imbalanced SSL.
Takeaways & Limitations
Estimating the unlabeled class distribution from labeled data assumes that labeled and unlabeled confusion matrices are similar, which holds when their input distributions match.

Abstract

from arXiv · show

While semi-supervised learning (SSL) has proven to be a promising way for leveraging unlabeled data when labeled data is scarce, the existing SSL algorithms typically assume that training class distributions are balanced. However, these SSL algorithms trained under imbalanced class distributions can severely suffer when generalizing to a balanced testing criterion, since they utilize biased pseudo-labels of unlabeled data toward majority classes. To alleviate this issue, we formulate a convex optimization problem to softly refine the pseudo-labels generated from the biased model, and develop a simple algorithm, named Distribution Aligning Refinery of Pseudo-label (DARP) that solves it provably and efficiently. Under various class-imbalanced semi-supervised scenarios, we demonstrate the effectiveness of DARP and its compatibility with state-of-the-art SSL schemes.

1 Introduction

Class-imbalanced training biases SSL predictions toward majority classes, making pseudo-labels even more imbalanced and potentially harming minority-class performance. DARP addresses this by softly aligning pseudo-label distributions with the unlabeled data while preserving original pseudo-label information, and improves several SSL methods across tested scenarios.

Imbalanced training distributions bias DNN predictions toward majority classes and make balanced-test generalization difficult.
SSL can amplify this problem because pseudo-labels generated from biased predictions become more imbalanced than the underlying class distribution.For MixMatch, the pseudo-label imbalance ratio reaches γ = 1046 versus the true ratio γ = 150.
Recent SSL classifiers can degrade minority-class performance relative to a vanilla model trained only on labeled data.The paper identifies class-imbalanced SSL as an important but under-explored research problem.
DARP refines biased pseudo-labels by matching their distribution to the true unlabeled-data distribution while constraining distortion from the originals.The resulting formulation minimizes distortion while enforcing distribution matching.
DARP efficiently solves the convex refinement problem with a provable guarantee and optionally removes small, noisy pseudo-label entries.
DARP improves MixMatch, ReMixMatch, and FixMatch by up to 77.2%, 31.4%, and 53.1% relative reductions in balanced test error, respectively.The method is reported to improve recent state-of-the-art SSL algorithms in all tested cases and becomes more effective as distribution mismatch increases.

2 Related works

Related work covers class-imbalance methods that rebalance supervised objectives and SSL methods that leverage unlabeled data through pseudo-labels. The paper emphasizes that prediction-derived pseudo-labeling is vulnerable when class imbalance biases predictions toward majority classes.

Long-tailed class distributions make standard DNN training harder to generalize, motivating re-weighting and re-sampling approaches.
Rebalancing objectives can overfit minority classes, leading later work to explore effective-number weighting and delayed rebalancing.
SSL methods commonly generate pseudo-labels from model predictions and train on them, including pseudo-labeling, consistency regularization, and combined schemes.
Prediction-derived pseudo-labels can be inefficient or harmful for minority classes when class imbalance biases the model toward majority classes.

3 Handling imbalanced semi-supervised learning

DARP formulates pseudo-label refinement as a convex optimization that matches the unlabeled class distribution while preserving original predictions, then solves it with an efficient dual procedure. It also removes small entries and estimates unknown unlabeled distributions through labeled-data confusion statistics under an explicit similarity assumption.

3.1 Pseudo-label under imbalanced semi-supervised learning: The setup considers K-class classification with labeled and unlabeled datasets whose class counts can be highly imbalanced.
3.1 Pseudo-label under imbalanced semi-supervised learning: SSL methods infer unlabeled labels from model predictions, but imbalance can make pseudo-labels more imbalanced than the true distribution and harm minority classes.
3.2 Distribution aligning refinery of pseudo-label: DARP accepts pseudo-labels from any SSL algorithm and refines them using the true class distribution of unlabeled data.
3.2 Distribution aligning refinery of pseudo-label: The convex objective minimizes KL divergence from original pseudo-labels while enforcing class-count constraints that match the unlabeled distribution.
3.2 Distribution aligning refinery of pseudo-label: Confidence weights favor preserving information from high-confidence pseudo-labels, assigning larger weights to lower-entropy examples.
3.2 Distribution aligning refinery of pseudo-label: DARP solves the optimization with an efficient coordinate-ascent procedure on its Lagrangian dual, converging to the unique solution as iterations increase.The paper reports that T = 10 is empirically sufficient for convergence.
3.2 Distribution aligning refinery of pseudo-label: Removing small pseudo-label entries concentrates refinement on confident entries by clipping relatively small class probabilities to zero.
3.2 Distribution aligning refinery of pseudo-label: DARP adds at most 20% of the vanilla SSL algorithm’s running time in the reported experiments.

4 Experiments

Experiments evaluate DARP across imbalanced CIFAR-10, CIFAR-100, and STL-10 settings, including mismatched labeled and unlabeled distributions. DARP consistently improves applied SSL methods, with larger gains when distribution mismatch is severe.

Experimental setup: Experiments vary labeled and unlabeled imbalance using synthetically long-tailed CIFAR-10, CIFAR-100, and STL-10 datasets.Imbalance is controlled by γl and γu; larger ratios indicate more imbalanced class distributions.
Experimental setup: DARP is applied to MixMatch, ReMixMatch, and FixMatch alongside re-balancing and semi-supervised baselines.The evaluation measures gains from adding DARP to recent SSL algorithms under multiple imbalance settings.
CIFAR-10 with matched imbalance: 22.4%/20.3% relative error reductions in bACC/GM are achieved by DARP with FixMatch when γ = 150.DARP improves all applied SSL algorithms in this setting and outperforms the baselines.
CIFAR-10 with mismatched imbalance: DARP consistently improves baselines when labeled and unlabeled imbalance differ, with gains increasing as γu decreases and the overall distribution becomes more balanced.The reported explanation is that DARP refines pseudo-labels to approximately follow the true unlabeled class distribution.
Reversed unlabeled distribution: 19.5%/22.5% relative error reductions in bACC/GM are obtained over ReMixMatch* under reversed unlabeled imbalance with γu = 100.In this scenario, SSL methods provide wrong pseudo-labels for most unlabeled data, whereas DARP refines them.
CIFAR-100 and STL-10: DARP consistently improves SSL algorithms on CIFAR-100 and STL-10, with especially significant gains on STL-10 under stronger labeled–unlabeled distribution mismatch.The STL-10 result is attributed to its unlabeled data being close to uniformly distributed.

5 Conclusion

The conclusion presents DARP as a simple method for refining biased pseudo-labels in class-imbalanced SSL. It matches the unlabeled class distribution while preserving original pseudo-label information and can be adapted to SSL algorithms.

Conclusion: DARP refines biased pseudo-labels so their distribution matches the true unlabeled class distribution while preserving original information.The method can additionally remove noisy entries from the original pseudo-labels.
Conclusion: DARP is presented as an easy-to-use method adaptable to any SSL algorithm using pseudo-labels.The conclusion identifies class-imbalanced SSL as under-explored and positions the method as guidance for future work.

Broader Impact

The paper identifies class imbalance as a vulnerability of recent SSL algorithms and situates DARP as a possible contribution to broader bias-related concerns. It also notes that the study focuses on ordinary classification under class-imbalanced distributions.

The paper identifies that state-of-the-art SSL algorithms can suffer under class-imbalanced training distributions because predictions become biased toward majority classes.
The study focuses on ordinary classification under class-imbalanced distributions rather than broader forms of DNN bias.
The authors suggest that DARP may contribute to addressing undesirable DNN bias beyond the setting studied in this paper.
The paper connects its findings to prior concerns about out-of-distribution unlabeled samples as another limitation of current SSL algorithms.

Distribution Aligning Refinery of Pseudo-label

The section is titled “Distribution Aligning Refinery of Pseudo-label.”

The section names the proposed method Distribution Aligning Refinery of Pseudo-label.
The title identifies pseudo-label refinement as the section’s subject.
The title specifies that the refinement is distribution aligning.

A Proof of Theorem 1

The proof interprets DARP as coordinate ascent on the Lagrangian dual of a convex optimization problem. Strong duality and the dual formulation connect the algorithm’s updates to the unique primal optimum under the stated convergence condition.

The proof frames DARP as a coordinate-ascent algorithm for the Lagrangian dual of its original convex optimization problem.
The optimization is an I-projection when all weights w_i equal 1, linking the formulation to an information-theoretic problem.
Strong duality makes the optimal objective values of the primal and dual formulations equivalent.
DARP’s coordinate updates optimize the dual and recover the unique optimum M once the algorithm converges.
Correct convergence is guaranteed when the dual function has a unique critical point satisfying the stated equalities.
The desired matrix M minimizes the primal objective at a critical point satisfying the coordinate-wise optimality condition for each M_ij.

C Experimental results on SUN397

The SUN397 experiment evaluates DARP on a constructed real-world class-imbalanced dataset using several training baselines. The reported comparison shows that DARP achieves the best performance among the evaluated algorithms.

Table 5 reports balanced accuracy and geometric mean for classification performance on SUN397.
SUN397 contains 108,754 RGB images across 397 classes, with 50 samples per class held out for testing.
The experiment compares vanilla, classifier retraining, FixMatch, and FixMatch+DARP on the constructed dataset.
DARP surpasses all baseline algorithms in the SUN397 comparison, supporting its extensibility to a real-world dataset.
The training setup uses randomly cropped and rescaled 224 × 224 images, a ResNet-34 model, and 300 epochs of unlabeled-data training.

D Combination of re-balancing and semi-supervised learning

DARP can be combined with classifier re-training to improve performance under class imbalance. With cRT applied, DARP still outperforms ReMixMatch on balanced-accuracy and geometric-mean evaluation.

D Combination of re-balancing and semi-supervised learning: Table 6 compares CIFAR-10 classification performance using bACC/GM across three class-imbalance ratios γ = γ_l = γ_u.The table reports relative test error gains from DARP against the corresponding baseline SSL algorithms.
D Combination of re-balancing and semi-supervised learning: Combining cRT significantly improves both ReMixMatch and DARP under class-imbalanced training.The comparison evaluates ReMixMatch with and without DARP after classifier re-training.
D Combination of re-balancing and semi-supervised learning: DARP still outperforms ReMixMatch when both methods are combined with cRT.This shows that DARP retains an advantage after applying the same re-balancing procedure.
D Combination of re-balancing and semi-supervised learning: 11.3%/12.2% relative error gains on bACC/GM are reported for DARP versus ReMixMatch with cRT when γ = γ_l = γ_u = 100.These gains are reported for the balanced-accuracy and geometric-mean metrics, respectively.

E Implementation details

The experiments use a fixed Wide ResNet-28-2 training setup and apply DARP periodically with fixed hyperparameters. Convergence is assessed through distribution mismatch, with T = 10 reported as sufficient and DARP superior to ReMixMatch’s distribution-alignment term.

E Implementation details: All experiments use Wide ResNet-28-2, batch size 64, and 2.5 × 10^5 training iterations except in Section C.SSL training uses Adam with learning rate 2 × 10^-3 and default β_1, β_2, and ε settings.
E Implementation details: DARP is applied every 10 iterations with fixed hyperparameters δ = 2 and T = 10.The paper reports these settings as empirically sufficient for DARP convergence.
E Implementation details: Convergence is measured by the distribution mismatch between refined pseudo-labels and the target class distribution.The mismatch is evaluated for each iteration and compared with ReMixMatch’s distribution-alignment term.
E Implementation details: T = 10 is sufficient for convergence, and DARP is superior to distribution alignment in the reported cases.The comparison uses MixMatch-trained models under three class-distribution settings.
E Implementation details: FixMatch uses µ = 2, λ_u = 1, and τ = 0.95, while ReMixMatch uses K = 2 and RandAugment.Other method-specific hyperparameters follow the corresponding original implementations.
E Implementation details: cRT reinitializes and retrains only the linear classifier for 1.0 × 10^4 iterations while freezing the remaining network parameters.The re-balanced objective uses loss re-weighting rather than re-sampling, and EMA is not used for re-balancing evaluations.