Rethinking the Value of Labels for Improving Class-Imbalanced Learning

Yuzhe Yang; Zhi Xu

Source-linked AI summary

Rethinking the Value of Labels for Improving Class-Imbalanced Learning

Yuzhe Yang, Zhi Xu

arXiv:2006.07529v2Jun 13, 2020cs.LGcs.CVstat.ML

TL;DR

Class imbalance creates a dilemma because labels improve supervision yet can bias classifiers toward majority classes. The paper analyzes both facets theoretically and empirically, using unlabeled data semi-supervisedly and self-supervised pre-training without extra data. Both strategies produce strong gains, motivating a rethink of how imbalanced labels are used in long-tailed learning.

Problem
Long-tailed data challenges recognition, while imbalanced labels simultaneously improve supervision and introduce majority-class label bias.
Method
The paper develops theoretically grounded semi-supervised and self-supervised strategies to exploit both facets of imbalanced labels.
Results
Both strategies achieve superior performance on large-scale imbalanced datasets, with self-supervised pre-training consistently outperforming corresponding baselines.
Takeaways & Limitations
Imbalanced labels should be rethought as resources whose positive and negative facets can both improve class-imbalanced learning.
Takeaways & Limitations
The strategies are extensively tested on academic datasets and evaluated mainly with standard accuracy, while fairness, privacy, and other ethical issues receive limited attention.

Abstract

from arXiv · show

Real-world data often exhibits long-tailed distributions with heavy class imbalance, posing great challenges for deep recognition models. We identify a persisting dilemma on the value of labels in the context of imbalanced learning: on the one hand, supervision from labels typically leads to better results than its unsupervised counterparts; on the other hand, heavily imbalanced data naturally incurs "label bias" in the classifier, where the decision boundary can be drastically altered by the majority classes. In this work, we systematically investigate these two facets of labels. We demonstrate, theoretically and empirically, that class-imbalanced learning can significantly benefit in both semi-supervised and self-supervised manners. Specifically, we confirm that (1) positively, imbalanced labels are valuable: given more unlabeled data, the original labels can be leveraged with the extra data to reduce label bias in a semi-supervised manner, which greatly improves the final classifier; (2) negatively however, we argue that imbalanced labels are not useful always: classifiers that are first pre-trained in a self-supervised manner consistently outperform their corresponding baselines. Extensive experiments on large-scale imbalanced datasets verify our theoretically grounded strategies, showing superior performance over previous state-of-the-arts. Our intriguing findings highlight the need to rethink the usage of imbalanced labels in realistic long-tailed tasks. Code is available at https://github.com/YyzHarry/imbalanced-semi-self.

1 Introduction

The paper frames imbalanced labels as a double-edged sword: they provide useful supervision but can bias decision boundaries toward majority classes. It systematically exploits both facets through semi-supervised learning with unlabeled data and self-supervised pre-training, achieving strong improvements in imbalanced recognition.

1 Introduction: Long-tailed class distributions are common in real-world and safety-critical data, where extreme imbalance remains challenging despite specialized techniques.The paper highlights autonomous driving and medical diagnosis as examples and notes persistent performance drops under extreme imbalance.
1 Introduction: Imbalanced labels have a double-edged role: supervision typically improves classifiers, while majority classes can significantly drive the decision boundary.This tension motivates the paper’s central question of how to maximally exploit labels for imbalanced learning.
1 Introduction: The paper decomposes label value into positive and negative facets and argues that both can be exploited through semi-supervised and self-supervised learning.The two viewpoints are presented as complementary directions for improving class-imbalanced learning.
1 Introduction: Extra unlabeled data can reduce label bias when combined with original imbalanced labels through pseudo-labeling, improving classifiers across varied labeled and unlabeled imbalance settings.The paper supports this strategy with theoretical and empirical evidence, including a Gaussian-model analysis.
1 Introduction: Self-supervised pre-training can compensate for biased label information, with classifiers consistently outperforming corresponding baselines across settings and base training techniques.The paper reports theoretical support and significant improvements on large-scale imbalanced datasets.
1 Introduction: The contributions include systematic analysis of imbalanced labels, semi-supervised use of unlabeled data, and self-supervised pre-training without extra data.The strategies are presented with theoretical interpretations and state-of-the-art results on large-scale imbalanced benchmarks.

2 Imbalanced Learning with Unlabeled Data

This section studies how extra unlabeled data can mitigate label bias in class-imbalanced learning through theoretical analysis and semi-supervised experiments. The proposed pseudo-labeling framework consistently improves performance, especially for tail classes and extreme imbalance.

2.1 Theoretical Motivation: More unlabeled data improves estimation with high probability, although training imbalance affects accuracy while unlabeled-data imbalance affects the probability of success.Balanced unlabeled data maximizes the relevant class-count term, but additional unlabeled data remains helpful even when it is imbalanced.
2.2 Semi-Supervised Imbalanced Learning Framework: The semi-supervised framework generates pseudo-labels with an intermediate classifier, then jointly trains a final model on labeled and pseudo-labeled data.Its objective is L(DL, θ) + ωL(DU, θ), and the procedure seeks better class boundaries, particularly for tail classes.
2.2.1 Main Results: +10% on CIFAR-10-LT and +6% on SVHN-LT under ρ = 100, showing that unlabeled data substantially improves existing imbalanced-learning techniques.The gains are consistent across different unlabeled imbalance ratios and learning strategies.
2.2.1 Main Results: More balanced unlabeled data generally yields larger gains, while minority classes tend to benefit more and improvements gradually diminish as labeled or unlabeled data grows.Visualizations associate the gains with clearer tail-class boundaries and improved class separation.

3 A Closer Look at Unlabeled Data under Class Imbalance

Semi-supervised learning can substantially improve imbalanced classification, but its value depends strongly on the relevance and class balance of the unlabeled data. Mismatched or more heavily imbalanced unlabeled data can reduce performance, limiting the approach in applications where meaningful unlabeled data are difficult to construct.

Unlabeled data can substantially improve imbalanced learning, confirming the value of imbalanced labels when extra data are available.
Adding unlabeled data from mismatched classes can hurt performance; relevance must reach 60% to be effective in the tested setting.
With 60% relevance fixed, higher ρU for relevant unlabeled data produces higher test error, and ρU must not exceed 50 to help.
Unlabeled-data imbalance creates an additional challenge, unlike the traditional setting where unlabeled-data balance is not emphasized.
Semi-supervised learning is especially powerful when relevant unlabeled data from each class are readily available, but other techniques are needed otherwise.

4 Imbalanced Learning from Self-Supervision

Self-supervised pre-training is presented as a way to reduce label bias in class-imbalanced learning by first learning label-agnostic representations, then training a classifier. Theory and experiments show consistent benefits across imbalance settings, datasets, and base techniques.

Theoretical Motivation: Standard training cannot achieve accuracy ≥3/4 in the stated Gaussian model, whereas self-supervision on the resulting representations provides a better classifier.The comparison motivates replacing raw-input training with training on self-supervised features.
Theoretical Motivation: Self-supervised classification has an error probability that decays exponentially with dimension d, with success probability depending exponentially on d and the amount of data.The theorem’s probability guarantee can be weaker under extreme imbalance, but the exponential dependence still permits a satisfying classifier.
Method: Self-supervision first learns label-agnostic representations, after which standard training produces the final classifier.The proposed pipeline abandons label information during pre-training and then initializes ordinary classifier training with the learned representation.
Main Results: Adding SSP consistently outperforms competing imbalanced-learning techniques on long-tailed CIFAR, with larger benefits as the imbalance ratio ρ increases.The comparison covers multiple losses and training strategies across different ρ values.
Main Results: SSP sets new state-of-the-art results on ImageNet-LT and iNaturalist 2018, with 4% absolute performance gains over current techniques.The consistent gains support applying self-supervision in large-scale imbalanced-learning settings.
Further Analysis and Ablation Studies: SSP reduces tail-class leakage and preserves clearer representation separation, while improving all classes with especially evident gains for tail classes.Across four SSP methods, Rotation performs better on smaller datasets and MoCo performs better on larger datasets; step-imbalance experiments also show consistent improvements.

5 Related Work

Related work addresses imbalanced learning through sampling and weighting strategies, semi-supervised learning through unlabeled-data methods, and self-supervised learning through pretext and contrastive tasks.

Imbalanced Learning & Long-tailed Recognition: Imbalanced-learning methods include oversampling minority classes, undersampling frequent classes, and cost-sensitive reweighting during training.These approaches target class-frequency differences through data selection or class- and sample-specific weights.
Semi-Supervised Learning: Semi-supervised learning combines labeled and unlabeled samples using entropy minimization, pseudo-labeling, generative models, or consistency-based regularization.Consistency methods use unlabeled data to encourage decision boundaries in low-density regions.
Self-Supervised Learning: Self-supervised learning uses pretext tasks such as exemplar classification, rotation prediction, colorization, jigsaw puzzles, clustering, and contrastive losses.The literature has expanded from manually designed pretext tasks to contrastive-learning approaches.

6 Conclusion

The paper proposes two theoretically grounded strategies that leverage imbalanced labels in semi-supervised and self-supervised manners, demonstrating sound guarantees and superior performance on large-scale imbalanced datasets.

The study systematically analyzes both the positive and negative facets of imbalanced labels in class-imbalanced learning.
The proposed strategies leverage imbalanced labels through semi-supervised and self-supervised learning.
The strategies achieve sound theoretical guarantees and superior performance on large-scale imbalanced datasets.
The findings highlight the need to rethink how inherently biased labels are leveraged for imbalanced learning.

Broader Impact

The paper addresses long-tailed data through unlabeled-data use and self-supervision, while cautioning that its evaluation is mainly academic and does not cover broader high-stakes constraints and ethical concerns.

The paper targets skewed long-tailed distributions rather than uniform class distributions.
One perspective uses unlabeled data without requiring additional human labeling.
A second perspective explores intrinsic data properties through self-supervision.
The strategies are presented as generic frameworks for improving imbalanced learning tasks and benefiting practitioners handling heavily imbalanced data.
The strategies are extensively tested on academic datasets, while fairness, privacy, and other ethical issues in minor classes are largely ignored.
The authors recommend rigorous validation before applying the methods in critical, high-stakes applications.

Appendices

The appendix excerpts outline a proof that bounds pseudo-label-related terms using Gaussian concentration, union bounds, and the triangle inequality before concluding the argument.

The proof rewrites the pseudo-labeled variable using a Bernoulli correctness model.
Each term in Eq. (1) is bounded separately.
Standard Gaussian concentration provides bounds for intermediate quantities.
A union bound combines concentration inequalities from Eqs. (2), (3), and (4) into a lower bound on event E.
The final inequality follows by combining the preceding equation with the triangle inequality.
The appendix concludes the proof after establishing the preceding bounds.

B Proof of Theorem 2

The proof analyzes a self-supervised classifier under Gaussian assumptions using Gaussian and χ2 concentration, then bounds its intercept and error probability through conditional inequalities.

For X ∼ N(0, σ2I_d), projecting with θ gives θ^T X ∼ N(0, ||θ||2σ2).
The raw classifier has error probability at least 1/4.
The proof invokes a standard concentration inequality for sub-exponential random variables.
Because squared standard Gaussians are sub-exponential, χ2 concentration is used for the self-supervised analysis.
The self-supervised classifier is represented as f_ss(X) = sign(−Z + b), with Z derived from a χ2-distributed quantity.
A union bound establishes a high-probability bound for the classifier's intercept b.
Conditioning on the intercept bound, the proof separately analyzes negative and positive classes before substituting the inequalities into the error probability.

D.1 Imbalanced Dataset Details

The experiments use five long-tailed datasets and construct unlabeled data with controlled relevance and imbalance. Semi-supervised learning consistently improves imbalanced recognition, with advanced methods generally producing larger gains.

Unlabeled data sourcing: Unlabeled CIFAR-10 data are mined from 80 Million Tiny Images using an 11-class classifier to rank class-relevant samples.SVHN instead uses an additional 531.1K labeled samples, whose labels are discarded during experiments.
Unlabeled data construction: Irrelevant unlabeled data are selected as samples receiving high confidence for the extra non-CIFAR class and mixed at varying proportions.This creates unlabeled datasets with different degrees of data relevance.
Unlabeled data construction: The unlabeled set keeps its total size fixed while varying its imbalance ratio from uniform to double the labeled imbalance.The settings are ρU = 1, ρ/2, ρ, and 2ρ, with an example size of 5x the labeled set.
Semi-supervised methods: VAT enforces local prediction robustness through consistency regularization, whereas Mean Teacher matches student outputs to an EMA-updated teacher.Both methods can be incorporated into the imbalanced learning framework.
Semi-supervised methods: Across datasets and imbalance ratios, adding unlabeled data consistently benefits imbalanced learning through semi-supervised learning.Advanced semi-supervised techniques generally obtain larger improvements than the simple pseudo-label strategy.

E.2 Class-wise Generalization Results

Class-wise analyses show that unlabeled data primarily improve tail-class generalization without materially harming head-class performance. Gains increase with more unlabeled data and remain consistent across self-supervised pre-training methods.

Semi-supervised generalization: Unlabeled data consistently and substantially improve tail-class generalization across base training techniques.The class-wise error analysis evaluates CIFAR-10-LT and SVHN-LT at ρ = 50.
Semi-supervised generalization: Head-class performance remains almost unaffected while tail-class errors improve with unlabeled data.Confusion matrices show that unlabeled data largely eliminate leakage from tail classes into head classes.
Unlabeled data scale: Larger unlabeled datasets consistently produce higher gains, with substantial improvement already appearing when unlabeled data equal 0.5x the labeled-set size.The experiment varies DU from 0.5x to 10x of DL while fixing ρ = ρU = 50.
Labeled data scale: With less labeled data, vanilla cross-entropy errors increase sharply, whereas adding unlabeled data maintains sufficiently low errors.The labeled-set size varies from 0.5x to 1x while DU remains 5x the labeled data and ρ = ρU = 50.
Self-supervised methods: All evaluated self-supervised pre-training methods benefit imbalanced learning across datasets, although their gains vary by technique.Pretext-task methods generally outperform contrastive methods on the smaller CIFAR-LT datasets.

F.2 Class-wise Generalization Results

Self-supervised pre-training improves class-wise generalization across CIFAR-10-LT and ImageNet-LT, including many-, medium-, and few-shot classes. It also reduces tail-to-head prediction leakage and remains effective under step imbalance.

Class-wise generalization: Self-supervised pre-training consistently improves class-wise generalization on both CIFAR-10-LT and ImageNet-LT.ImageNet-LT evaluation separates classes into Many-shot, Medium-shot, and Few-shot subsets.
Confusion matrices: Self-supervised pre-training greatly compensates tail-to-head leakage in CIFAR-10-LT confusion matrices, improving minority-category performance.Vanilla cross-entropy exhibits large leakage from tail classes to head classes.
Step imbalance: Under step imbalance, self-supervised pre-training consistently benefits different imbalanced learning techniques across datasets.The reported results confirm improvements for CIFAR-10 and CIFAR-100 with varying imbalance ratios.
Step imbalance: Higher imbalance ratios tend to produce larger performance gains from self-supervised pre-training.This pattern demonstrates the value of self-supervision under more extreme class imbalance.