PaperlayerView exact revision on arXiv

Source-linked AI summary

Debiased Learning from Naturally Imbalanced Pseudo-Labels

Xudong Wang, Zhirong Wu, Long Lian, Stella X. Yu

arXiv:2201.01490v2cs.LGcs.CLcs.CV

TL;DR

Pseudo-labels can be imbalanced and bias adaptation even when labeled and target data are balanced, motivating a method tailored to this previously unknown problem. The paper combines counterfactual debiasing with pseudo-label-aware margins and reports gains on ImageNet-1K across SSL and ZSL. Its scope includes robustness to domain shift and long-tailed settings, while confidence-threshold selection retains an imbalance–precision/recall trade-off.

  • Problem

    Pseudo-labels can be naturally imbalanced and create biased learning even when source and target data are balanced.

  • Method

    The method combines adaptive counterfactual debiasing of classifier responses with adaptive class margins based on pseudo-label imbalance.

  • Results

    26% improvement for SSL with 0.2% annotations and 9% improvement for ZSL were reported on ImageNet-1K, with gains across pseudo-labeling methods.

  • Takeaways & Limitations

    DebiasPL is presented as a universal add-on for pseudo-labeling models, improving performance under domain shift and long-tailed settings.

  • Takeaways & Limitations

    Zero-shot confidence thresholding involves a trade-off between pseudo-label imbalance and precision/recall.

Abstract

from arXiv · show

Pseudo-labels are confident predictions made on unlabeled target data by a classifier trained on labeled source data. They are widely used for adapting a model to unlabeled data, e.g., in a semi-supervised learning setting. Our key insight is that pseudo-labels are naturally imbalanced due to intrinsic data similarity, even when a model is trained on balanced source data and evaluated on balanced target data. If we address this previously unknown imbalanced classification problem arising from pseudo-labels instead of ground-truth training labels, we could remove model biases towards false majorities created by pseudo-labels. We propose a novel and effective debiased learning method with pseudo-labels, based on counterfactual reasoning and adaptive margins: The former removes the classifier response bias, whereas the latter adjusts the margin of each class according to the imbalance of pseudo-labels. Validated by extensive experimentation, our simple debiased learning delivers significant accuracy gains over the state-of-the-art on ImageNet-1K: 26% for semi-supervised learning with 0.2% annotations and 9% for zero-shot learning. Our code is available at: https://github.com/frank-xwang/debiased-pseudo-labeling.

1. Introduction

Pseudo-labels can be naturally imbalanced and bias learning even when source and target data are balanced. The paper proposes debiasing with counterfactual reasoning and adaptive margins, achieving large gains across SSL and ZSL.

  • Motivation: Pseudo-labels produced by machine-learning models can be naturally imbalanced, creating biased learning beyond imbalance in human-provided labels.This issue appears even when source and target data are class-balanced or sampled from the same domain.
  • Pseudo-labeling: Pseudo-labels are confident teacher-model predictions on unlabeled data that supervise student-model adaptation.The framework is used in semi-supervised learning and zero-shot transfer learning.
  • Evidence of bias: More than 3500 ImageNet-1K instances were predicted as class 0, three times the actual number of class-0 samples.The imbalance occurs with CLIP predictions despite approximately balanced CLIP training data.
  • Method: The proposed method combines adaptive debiasing through counterfactual reasoning with adaptive margins based on pseudo-label imbalance.The debiasing module removes classifier response bias, while the marginal loss adjusts each class’s margin.
  • Results: 26% improvement for SSL with 0.2% annotations and 9% improvement for ZSL were reported on ImageNet-1K.The method also functions as a universal add-on to pseudo-labeling models and improves long-tailed SSL.
  • Contribution: The work establishes a debiased pseudo-labeling pipeline for SSL and ZSL, including vision-language models such as CLIP.The authors report state-of-the-art gains and broad applicability across pseudo-labeling models.

2. Related Work

Prior work addresses SSL, zero-shot classification, and long-tailed recognition, but the paper identifies biased pseudo-labeling as a previously unexamined source of imbalance. Its method differs by requiring no prior class-distribution knowledge.

  • Semi-supervised learning: SSL methods include consistency regularization, pseudo-labeling, and transfer learning from large unlabeled datasets.Pseudo-labeling expands training data with confidently labeled unlabeled instances.
  • SSL debiasing: CReST rebalances SSL sampling using the labeled-data distribution but does not work when labeled data are balanced.The paper’s approach instead does not assume a prior distribution for the labeled set.
  • Zero-shot classification: Prior zero-shot classification strategies use attributes, pretrained semantic embeddings, knowledge graphs, or vision-language models.These approaches transfer information to novel classes not seen during training.
  • Long-tailed recognition: Long-tailed recognition methods commonly rebalance or reweight classes, use label-distribution-aware margins, or adjust predictions post hoc.These methods generally address ground-truth class imbalance rather than biased pseudo-labels.
  • Research gap: The paper targets an implicitly biased pseudo-labeling problem that previous SSL literature had not thoroughly analyzed.The proposed debiasing module focuses on this issue without requiring prior class-distribution knowledge.

3. Pseudo-Labels are Naturally Imbalanced

Pseudo-labels can become strongly class-imbalanced even when source and target data are balanced, because inter-class similarities produce systematic confusions. These biases can persist or amplify during training, while DebiasPL dynamically alleviates them.

  • Background: Pseudo-labeling transfers a teacher’s confident predictions on unlabeled data into student training targets.FixMatch generates pseudo-labels from weakly augmented samples and trains on strongly augmented versions, discarding predictions below a confidence threshold.
  • Pseudo-label imbalance: Balanced labeled and unlabeled data can still yield highly class-imbalanced pseudo-labels throughout FixMatch training.The imbalance is especially pronounced early and persists as training progresses.
  • Bias amplification: Pseudo-label errors can reinforce themselves because students inherit biased targets and feed increasingly biased predictions back into training.The paper describes confusing samples as difficult to self-correct once wrongly pseudo-labeled.
  • DebiasPL: DebiasPL dynamically alleviates pseudo-label imbalance, eventually producing an almost balanced distribution.The method addresses the bias through dynamic debiasing rather than relying on balanced source or target data alone.
  • Zero-shot learning: Pseudo-label bias is widespread: CLIP produces highly biased zero-shot predictions on ImageNet and multiple other benchmarks.The reported datasets include EuroSAT, MNIST, CIFAR10, CIFAR100, and Food101.
  • Bias analysis: Inter-class confounding provides a mechanism for pseudo-label bias, with visually similar classes attracting systematic misclassifications.Low-frequency ImageNet classes often have strong correlations with confusing classes, and FixMatch frequently maps ships to planes.

4. Debiased Pseudo-Labeling

DebiasPL adds adaptive debiasing and adaptive marginal loss to pseudo-labeling pipelines such as FixMatch. Counterfactual reasoning removes classifier response bias, while class-specific margins adapt to pseudo-label imbalance without requiring true class-distribution knowledge.

  • Method: DebiasPL is a universal add-on that integrates adaptive debiasing and adaptive marginal loss into pseudo-labeling methods.The method is designed for various networks and benchmarks, including FixMatch-based semi-supervised learning.
  • Adaptive Debiasing: Counterfactual reasoning targets the direct causal effect of the input on predictions while holding the mediator at a fixed level.The controlled direct effect contrasts outcomes under the observed input and a counterfactual input intervention.
  • Adaptive Debiasing: Momentum-updated counterfactual outcomes approximate computationally expensive controlled direct effects for producing debiased pseudo-labels.The approximation assumes that model bias does not change drastically over time.
  • Adaptive Debiasing: The debiasing factor controls the strength of the indirect-effect correction, with excessive or insufficient values harming fitting or bias removal.The method uses probability distributions rather than raw logits because logit scale is unstable, especially early in training.
  • Adaptive Marginal Loss: Adaptive marginal loss uses dynamic class-specific margins to counteract inter-class confusion and prevent dominant-class scores from overwhelming other classes.Unlike fixed-margin alternatives, the proposed adjustment is dynamically adaptive to pseudo-label bias.
  • Distinctions and Connections: Distribution alignment is limited when the true marginal class distribution is unavailable or source and target data differ, whereas DebiasPL does not require that prior knowledge.The comparison concerns methods that explicitly use true class-distribution information.

5. Experiment

Experiments evaluate DebiasPL across balanced, long-tailed, cross-distribution, semi-supervised, and zero-shot settings. The method achieves state-of-the-art results, transfers across pseudo-labeling methods, and is especially robust under domain shift.

  • Overview: DebiasPL delivers state-of-the-art results across semi-supervised and zero-shot learning benchmarks and improves balanced, long-tailed, and hybrid data settings.The experiments also assess its use as a universal add-on and its robustness to domain shifts.
  • Experimental Setup: DebiasPL is evaluated on CIFAR10, long-tailed CIFAR10, and ImageNet-1K with especially low labeled-data fractions.The ImageNet-1K settings include 1% and 0.2% labeled data.
  • Semi-supervised Learning: DebiasPL surpasses prior methods on balanced and long-tailed SSL benchmarks using a unified framework and the same hyper-parameters.The comparison includes settings where competing methods are designed specifically for balanced or long-tailed data.
  • Semi-supervised Learning: 11.4% gain is reported over the baseline when labeled and unlabeled data follow different distributions.The result is reported for scenarios with distribution differences between labeled and unlabeled data.
  • Semi-supervised Learning: DebiasPL produces consistent performance improvements when incorporated into various SSL methods.The universal-add-on evaluation uses CIFAR10 with four labeled instances per class.

6. Summary

The paper identifies biased pseudo-labeling as a previously unknown issue and proposes DebiasPL to alleviate its influence without prior knowledge of the true data distribution. The method improves semi-supervised and transductive zero-shot learning and is implemented through adaptive pseudo-labeling and loss updates.

  • Summary: DebiasPL dynamically alleviates biased pseudo-label influence without using prior knowledge of the true data distribution.The approach is presented as a universal add-on for semi-supervised and transductive zero-shot learning.
  • Summary: The training procedure initializes a momentum estimate of class probabilities and updates it from unlabeled predictions.The estimate is initialized uniformly and updated using a momentum coefficient.
  • Summary: Debiased pseudo-labels are obtained by adjusting weak-view logits with the logarithm of the estimated class-probability vector.The resulting maximum probability determines the pseudo-label and its confidence-based mask.
  • Summary: The objective combines supervised cross-entropy with a masked unlabeled loss using adaptive marginal adjustment.The unlabeled logits are adjusted before calculating the pseudo-label loss.
  • Summary: The optimization step updates model parameters and the exponential-moving-average model.This completes the semi-supervised training loop after supervised and unlabeled losses are combined.

7. Appendix

The appendix details DebiasPL’s benchmark implementations and ablations, showing gains from its components and sensitivity to pseudo-label confidence thresholds. It also documents the precision–imbalance trade-off underlying threshold selection.

  • Ablation studies: The adaptive debiasing module and marginal loss significantly improve performance on both CIFAR10 and CIFAR10-LT SSL benchmarks.The ablation is reported over five folds, with four labeled samples per class for CIFAR10 and 30% labeled data for CIFAR10-LT.
  • Ablation studies: The marginal loss yields smaller gains on balanced data than on unbalanced data, where pseudo-label imbalance is more consequential.This comparison isolates the differing contribution of the marginal-loss component across CIFAR10 and CIFAR10-LT.
  • Ablation studies: More than 7% improvement is obtained on CIFAR10-LT with the optimal debias factor λ=0.5, while overly strong debiasing makes fitting difficult.When λ=0, the model is identical to FixMatch; excessively small λ values barely eliminate bias.
  • Threshold analysis: Increasing the CLIP confidence threshold raises pseudo-label imbalance; at τclip=0.95, more than 60 categories receive zero predictions.The threshold is nevertheless optimal for ImageNet zero-shot learning because it supplies higher-precision labeled data.
  • Threshold analysis: DebiasPL alleviates the higher imbalance caused by large CLIP thresholds and produces more than 10% performance gains.The appendix frames threshold selection as a trade-off between lower imbalance at smaller thresholds and lower precision with more outliers.
Loading 2201.01490v2…