PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Dynamics of Logits Debiasing for Long-Tailed Semi-Supervised Learning

Yue Cheng, Jiajun Zhang, Xiaohui Gao, Weiwei Xing, Zhanxing Zhu

arXiv:2608.30699v1cs.LGcs.AI

TL;DR

Long-tailed class imbalance biases SSL pseudo-labels and leaves the mechanisms of LTSSL debiasing insufficiently understood. The paper decomposes learning dynamics, uses baseline-image logits to characterize accumulated bias, and proposes DyTrim; experiments report consistent gains across benchmarks, architectures, and SSL methods.

  • Problem

    Long-tailed SSL suffers from majority-class pseudo-label bias, while the mechanisms by which LTSSL methods implicitly debias logits remain poorly understood.

  • Method

    The paper derives step-wise logit-update decompositions, interprets task-irrelevant baseline-image logits as accumulated-bias indicators, and introduces DyTrim dynamic pruning.

  • Results

    DyTrim consistently improves LTSSL performance across standard benchmarks, SSL methods, and backbone architectures.

  • Takeaways & Limitations

    The analysis unifies logit adjustment, reweighting, and resampling as gradient-dynamics interventions and supports reallocating gradient budgets through dynamic pruning.

  • Takeaways & Limitations

    The per-step decomposition does not by itself capture the accumulated effects governed by iterative interactions between labeled and unlabeled updates.

Abstract

from arXiv · show

Long-tailed distributions are prevalent in real-world semi-supervised learning (SSL), where pseudo-labels tend to favor majority classes, leading to degraded generalization. While many long-tailed semi-supervised learning (LTSSL) methods have been proposed, the mechanisms by which they implicitly debias logits remain poorly understood. In this work, we revisit LTSSL through the lens of learning dynamics and provide a theoretical characterization of logits debiasing. Specifically, we derive a step-wise decomposition of the logits updates, showing that predictions are dominated by class-imbalance bias that reliably reflects label priors. To expose this effect, we use the logits of a task-irrelevant baseline image as an indicator of accumulated bias and prove that they converge to the class prior. This provides a unified view where LTSSL remedies such as logit adjustment, reweighting, and resampling correspond to reshaping gradient dynamics. Based on this insight, we propose DyTrim, a principle-based dynamic pruning framework that reallocates gradient budget through class-aware pruning on labeled data and confidence-based soft pruning on unlabeled data. We provide theoretical guarantees that DyTrim reduces class bias and improves generalization. Extensive experiments on standard LTSSL benchmarks show consistent gains across architectures and methods. Code available at: https://jiajun0425.github.io/DyTrim

1 INTRODUCTION

The paper studies why long-tailed class imbalance biases pseudo-labels toward majority classes and develops a learning-dynamics account of LTSSL debiasing. It proposes DyTrim, a pruning framework with theoretical and empirical support.

  • Long-tailed data commonly biases SSL pseudo-labels toward majority classes, challenging generalization on real-world datasets.
  • Existing LTSSL methods address imbalance through distribution alignment, data rebalancing, logit adjustment, and foundation model-based approaches.
  • The paper decomposes logit updates step by step and shows class imbalance dominates predictions, limiting the use of inter-sample similarity.
  • The analysis interprets baseline-image logits as an indicator of accumulated network bias and unifies logit adjustment, reweighting, and resampling as gradient-dynamics interventions.
  • DyTrim reallocates gradient budget through class-wise pruning for labeled data and confidence-based pruning for unlabeled data, with theoretical guarantees and benchmark gains.

2 PRELIMINARIES

The paper formalizes imbalanced labeled and unlabeled datasets, describes FixMatch’s pseudo-labeling loss, and introduces per-step influence analysis for SSL predictions.

  • Labeled and unlabeled class counts define imbalance ratios γl and γu, while minibatches contain B labeled and μB unlabeled samples.
  • FixMatch predicts weakly augmented unlabeled examples, retains hard pseudo-labels above confidence threshold τ, and trains with supervised and consistency losses.
  • Per-step learning dynamics use predicted probabilities, an empirical neural tangent kernel, and the loss gradient to characterize how updates affect an observation.
  • In MNIST, accurate pseudo-labels reinforce correct predictions, whereas inaccurate pseudo-labels can reduce correct-class probability; class imbalance masks this effect.

3 LEARNING DYNAMICS OF LONG-TAILED SEMI-SUPERVISED DEBIASING

The paper explains LTSSL debiasing through decomposed learning dynamics and uses baseline-image logits to measure accumulated class bias. This analysis motivates theoretical and empirical evaluation of debiasing behavior.

  • 3.1 LEARNING DYNAMICS OF SEMI-SUPERVISED LEARNING: FixMatch updates decompose into supervised influence from labeled samples and consistency influence from unlabeled samples and pseudo-labels.
  • 3.1 LEARNING DYNAMICS OF SEMI-SUPERVISED LEARNING: The per-step decomposition characterizes individual updates, but accumulated effects depend on iterative interactions between labeled and unlabeled updates.
  • 3.2 LEARNING DYNAMICS ANALYSIS OF ACCUMULATED BIAS UNDER CLASS IMBALANCE: A task-irrelevant baseline image provides an attribution indicator linking class-imbalance bias with learning dynamics.
  • 3.2 LEARNING DYNAMICS ANALYSIS OF ACCUMULATED BIAS UNDER CLASS IMBALANCE: For solid-color inputs under affine normalization, baseline logits are independent of pixel intensity and are controlled by normalization-layer affine parameters.
  • 3.2 LEARNING DYNAMICS ANALYSIS OF ACCUMULATED BIAS UNDER CLASS IMBALANCE: Baseline logits encode the class-imbalance ratio and align with the training class prior, while removing bias terms eliminates this alignment.
  • 3.2 LEARNING DYNAMICS ANALYSIS OF ACCUMULATED BIAS UNDER CLASS IMBALANCE: As majority-class samples accumulate, baseline-image predictions are progressively squeezed toward a biased long-tailed distribution.
  • 3.2 LEARNING DYNAMICS ANALYSIS OF ACCUMULATED BIAS UNDER CLASS IMBALANCE: The baseline-image dynamics are mainly governed by output sensitivity and gradient signals because its coupling kernel is typically small away from the data manifold.

4 DYNAMICS ANALYSIS OF LOGITS DEBIASING IN SEMI-SUPERVISED

The paper unifies logits adjustment, reweighting, resampling, and dynamic pruning as changes to gradient dynamics for mitigating long-tailed bias. DyTrim instead gates sample participation, using class-aware labeled pruning and confidence-based unlabeled pruning to reallocate update budget.

  • Unified dynamics: The framework analyzes logits debiasing methods through a unified per-step influence decomposition that tracks how they reshape gradient flow.It compares logits adjustment, reweighting, resampling, and pruning within the same learning-dynamics perspective.
  • Existing remedies: Logits adjustment modifies gradient terms through class-prior shifts, allowing training to compensate for class imbalance.The adjusted logits are described as subtracting class-prior estimates, which changes the gradient flow for unlabeled samples.
  • Existing remedies: Reweighting scales both similarity kernels and gradient terms by class-dependent weights, giving higher-weight classes stronger influence on updates.The analysis assumes labeled and unlabeled class-weight distributions remain proportionally consistent and fixed during training.
  • Dynamic pruning: Dynamic pruning directly changes which samples participate in each update rather than modifying losses or sampling frequencies.Its gating can zero out low-utility kernel–gradient interactions while preserving the underlying kernel.
  • DyTrim: DyTrim uses baseline-guided class-aware pruning for labeled data and label-insensitive confidence-based soft pruning for unlabeled data.The two mechanisms address long-tailed labeled data and unlabeled data with unknown distributions or imbalance ratios.

5 EXPERIMENT

Experiments evaluate DyTrim across long-tailed datasets, imbalance settings, SSL methods, and backbones. DyTrim consistently outperforms competing approaches, with reported gains over CDMAD and FixMatch across several benchmarks.

  • Experimental scope: Experiments cover CIFAR10-LT, CIFAR100-LT, STL10-LT, and ImageNet-127, with detailed settings deferred to the appendix.The evaluation includes multiple datasets representing different scales and imbalance conditions.
  • Standard benchmarks: DyTrim achieves the best performance across CIFAR-10-LT imbalance ratios and improves average bACC by 1.2% and GM by 1.4% over CDMAD.When integrated with FlexMatch and FreeMatch, it further boosts average bACC/GM by 2–3% without additional computational overhead.
  • Standard benchmarks: DyTrim consistently outperforms competing approaches on CIFAR-100-LT under stronger imbalance and more classes.The cited comparison concerns the setting where the unlabeled imbalance ratio is known and matched to the labeled ratio.
  • Distribution mismatch: DyTrim remains most effective when labeled and unlabeled distributions are mismatched, outperforming CDMAD on CIFAR-10-LT and STL-10-LT.It also outperforms prior techniques such as CDMAD on the large-scale ImageNet-LT benchmark.
  • Backbone evaluation: With ViT backbones, DyTrim improves CIFAR-10-LT bACC by 0.6% over CDMAD and nearly 4% over FixMatch at γl = γu = 100.Under inconsistent imbalance, it surpasses CDMAD by almost 2%; on CIFAR-100-LT it matches or slightly improves upon CDMAD while outperforming FixMatch.
  • Extensibility: DyTrim boosts performance across SSL frameworks, datasets, and imbalance ratios from γ = 1 ∼150 without architecture-specific tuning.Reported gains reach +1.4% for CDMAD on CIFAR10-LT and +2.7% on STL10-LT at γl=20.

6 CONCLUSION

The paper characterizes LTSSL class bias through learning dynamics and introduces DyTrim to mitigate imbalance by reallocating gradient budgets through pruning.

  • A step-wise logit-update decomposition shows how class imbalance dominates predictions and limits the model’s use of inter-sample similarity.
  • The framework connects logit adjustment, reweighting, and resampling to changes in gradient dynamics.
  • DyTrim uses dynamic pruning to reallocate gradient budgets for class-imbalance mitigation.
  • Experiments across multiple benchmarks and SSL methods show that DyTrim consistently improves performance.
  • Existing dynamic pruning methods reduce training iterations while maintaining performance, but had not explored pruning as an LTSSL debiasing mechanism.

B.1 MORE ABOUT TRAINING LOSSES OF FIXMATCH

This section describes the training losses and learning-dynamics analysis for SSL, then develops a baseline-image construction whose normalized output is controlled by affine bias parameters.

  • FixMatch: FixMatch combines supervised loss on labeled data with consistency loss on confident pseudo-labels from strongly augmented unlabeled data.
  • FlexMatch: FlexMatch replaces FixMatch’s fixed threshold with class-dependent thresholds that lower barriers for hard-to-learn classes and raise them for easy classes.
  • FreeMatch: FreeMatch uses EMA-based global confidence thresholds, class-specific statistics, and fairness regularization to encourage diverse predictions.
  • Learning dynamics: A one-step Taylor expansion and SGD chain-rule calculation decompose supervised and consistency learning dynamics through neural tangent-kernel evaluations.
  • Baseline image: For a solid-color baseline image, affine normalization makes the logits independent of the color value and dependent only on the normalization-layer bias term.

C.3 PROOF OF THEOREM 1

The proof links baseline-image predictions to class priors and interprets LTSSL debiasing and DyTrim pruning as modifications of per-step gradient influence.

  • Baseline prediction: Theorem 1 states that the baseline prediction at the normalized-zero feature state captures the conditional class distribution induced by the long-tailed training distribution.
  • Baseline prediction: When the baseline-image feature variance vanishes, the model output is determined solely by the bias term b⋆.
  • Learning dynamics: The one-step dynamics for baseline predictions are decomposed using the empirical neural tangent kernel and the loss-gradient residual.
  • Resampling: Resampling changes the expectation measure and reweights update magnitudes, amplifying tail-class contributions at each training step.
  • DyTrim: DyTrim’s class-aware labeled pruning implicitly reweights classes, while confidence-based unlabeled pruning and gradient rescaling preserve approximately the same gradient expectation.
  • Baseline-image ablation: On CIFAR-10-LT, solid-color baseline images outperform noise or mean-based baselines, with white and black images delivering the strongest results.

D.2 DETAIL OF THE BIAS TERM AND RUNNING STATISTICS

This section examines normalization bias and running statistics, then details DyTrim’s score-based pruning and evaluates it with several backbones.

  • Bias term: With the BN bias frozen at zero, the baseline-image output becomes zero during training and loses the benefit associated with the learned bias.
  • Running statistics: Without running estimates, batch statistics are used at evaluation, causing the baseline-image output to approach β and removing the benefit of g∗θ.
  • Extension: The analysis is extended from the normalization setting to a non-linear neural network through a corollary.
  • Score updates: DyTrim updates retained-sample scores with current-epoch losses while retaining the scores of pruned labeled samples for dynamic adaptation.
  • Unlabeled pruning: For retained unlabeled samples, DyTrim scales gradients by 1/(1 −r), while pruning decisions use consistency losses and a confidence threshold.

G.2 IMPLEMENTATION DETAILS

Experiments use fixed training settings, dataset-specific class sizes, and balanced metrics, comparing DyTrim with vanilla, SSL, and CISSL baselines.

  • Training configuration: Training runs for 250,000 iterations using SGD with η = 0.0015 and batch size 32.The schedule is 500 epochs with 500 steps per epoch.
  • Training configuration: DyTrim uses an unlabeled pruning ratio of 0.7 and sets δ = 0.875, aligned with InfoBatch.
  • Unlabeled-data pruning: Unlabeled selection retains low-confidence and poorly learned samples for pruning, randomly selects part of well-learned samples, and rescales selected weights.
  • Dataset settings: CIFAR-10-LT and CIFAR-100-LT have largest labeled classes of 1,500 and 150 samples, while their largest unlabeled classes contain 3,000 and 300 samples.
  • Evaluation: Evaluation uses bACC and GM for CIFAR-10-LT and STL-10-LT, but only bACC for CIFAR-100-LT and ImageNet-127.
  • Baselines: Comparisons include vanilla cross-entropy, CIL, SSL, and CISSL algorithms combined with FixMatch.

H.2 ADDITIONAL RESULTS ON CIFAR-10-LT

Additional experiments show DyTrim consistently improves long-tailed semi-supervised classification, reduces class bias, and benefits both labeled and unlabeled pruning across datasets and settings.

  • CIFAR-10-LT results: More than 1% improvement in both bACC and GM over CDMAD occurs with FixMatch under reverse-direction imbalance.
  • CIFAR-10-LT results: DyTrim improves FlexMatch by approximately 1.1–1.3% and FreeMatch by around 0.9–1.5% in the same challenging setting.
  • Debiasing analysis: DyTrim reduces classifier bias induced by class imbalance, as shown by baseline-image logit dynamics.
  • ImageNet-127: On Small-ImageNet-127, DyTrim exceeds CDMAD by 3.0% bACC at 32 × 32 and 1.2% at 64 × 64.
  • ImageNet-LT: On ImageNet-LT, DyTrim reaches 37.2%, compared with CDMAD’s 35.4% bACC and FixMatch’s 20.0%.
  • Ablation studies: Ablations show that combining labeled and unlabeled pruning with rescaling performs best, while removing rescaling lowers bACC by 0.8–2.1 points.
  • Hyperparameter analysis: DyTrim’s best pruning-ratio performance occurs at r = 0.3, while r ≥ 0.1 performs better across both architectures.
  • Pruning behavior: Pruning flattens class distributions by reducing head-class proportions and retaining or slightly increasing tail-class proportions.

I.2 DETAILS OF THE CHANGE OF LOGITS’S PROBABILITY DISTRIBUTION

Visualization experiments compare class predictions and representations across FixMatch, InfoBatch, CDMAD, and DyTrim under consistent, inconsistent, and balanced imbalance settings.

  • Confusion matrices: Figures 15–17 compare test-set confusion matrices across four methods under γl = 100, γu = 100; γl = 100, γu = 1; and γl = γu = 1.
  • Confusion matrices: FixMatch+DyTrim produces more balanced class predictions than FixMatch, InfoBatch, and CDMAD in the visualized settings.
  • Balanced setting: Under balanced data, DyTrim performs better on the pruned dataset than methods such as CDMAD trained on the full dataset.
  • Representation visualization: DyTrim representations show clearer class boundaries than the compared methods in the reported visualizations.
  • Representation visualization: Figures 18–19 compare t-SNE representations under γl = 100 with γu = 100 or γu = 1, using colors to indicate CIFAR-10 classes.

J LIMITATION

The method’s main scope boundary is its reliance on a task-irrelevant baseline image and limited coverage of certain architectures and semi-supervised methods.

  • Scope limitations: Using the baseline image as a training sample may reduce its ability to reflect accumulated classifier bias.
  • Scope limitations: DyTrim does not cover architectures with auxiliary classification heads or semi-supervised methods based on mixup-style interpolations.
Loading 2608.30699v1…