Source-linked AI summary
Disentangling Label Distribution for Long-tailed Visual Recognition
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, Buru Chang
TL;DR
Long-tailed visual recognition is formulated as a label-shift problem because real-world targets may not be uniform, while model predictions entangle source label frequencies with class evidence. The paper introduces PC Softmax and LADE to disentangle source label distributions, with both outperforming state-of-the-art methods on benchmark datasets and LADE adapting to various shifted targets.
Problem
Long-tailed visual recognition commonly evaluates on uniform targets despite potentially shifted real-world targets, while model predictions couple source label distributions with conditional predictions.
Method
PC Softmax post-processes cross-entropy and Softmax predictions to match the target label distribution, while LADE uses the Donsker-Varadhan representation to disentangle source labels during training.
Results
PC Softmax and LADE outperform state-of-the-art methods on real-world benchmark datasets, and LADE achieves state-of-the-art performance across various shifted target label distributions.
Takeaways & Limitations
Disentangling the source label distribution supports long-tailed recognition under target distributions that differ from the source, including non-uniform shifted targets.
Takeaways & Limitations
The formulation assumes label shift: source and target class-conditional feature distributions are equal, while their label distributions differ.
Abstract
from arXiv · showhide
The current evaluation protocol of long-tailed visual recognition trains the classification model on the long-tailed source label distribution and evaluates its performance on the uniform target label distribution. Such protocol has questionable practicality since the target may also be long-tailed. Therefore, we formulate long-tailed visual recognition as a label shift problem where the target and source label distributions are different. One of the significant hurdles in dealing with the label shift problem is the entanglement between the source label distribution and the model prediction. In this paper, we focus on disentangling the source label distribution from the model prediction. We first introduce a simple but overlooked baseline method that matches the target label distribution by post-processing the model prediction trained by the cross-entropy loss and the Softmax function. Although this method surpasses state-of-the-art methods on benchmark datasets, it can be further improved by directly disentangling the source label distribution from the model prediction in the training phase. Thus, we propose a novel method, LAbel distribution DisEntangling (LADE) loss based on the optimal bound of Donsker-Varadhan representation. LADE achieves state-of-the-art performance on benchmark datasets such as CIFAR-100-LT, Places-LT, ImageNet-LT, and iNaturalist 2018. Moreover, LADE outperforms existing methods on various shifted target label distributions, showing the general adaptability of our proposed method.
1. Introduction
Long-tailed recognition is reframed as label shift because real-world target distributions may differ from the long-tailed source distribution. The paper proposes disentangling source-frequency effects from predictions through PC Softmax and LADE.
- Real-world data are often long-tailed, with head classes dominating samples and tail classes having few examples, degrading recognition performance.
- The standard protocol trains on a long-tailed source distribution but evaluates on a uniform target distribution, although target distributions may be arbitrary.
- Cross-entropy predictions become entangled with the source label distribution, preventing direct use when the target distribution shifts.
- PC Softmax post-processes predictions to disentangle source frequencies and incorporate the target label distribution, outperforming state-of-the-art methods.
- LADE uses the Donsker-Varadhan representation to disentangle the source distribution during training and adapt to arbitrary target distributions.
- LADE achieves state-of-the-art performance across CIFAR-100-LT, Places-LT, ImageNet-LT, and iNaturalist 2018, including varied target distributions.
2. Related work
Related work addresses long-tailed recognition through resampling, class weighting, representation strategies, and label-shift estimation. The paper connects these areas through explicit source–target label-distribution shifts and DV-based disentanglement.
- Existing long-tailed methods mainly rebalance optimization by changing data sampling or class-wise loss weights.
- These strategies can under-represent head classes or encourage memorization of tail classes, motivating representation-preserving and tail-enhancement methods.
- Other approaches use effective sample numbers, larger tail-class margins, momentum-confounding adjustment, knowledge transfer, augmentation, or domain adaptation.
- Label-shift studies assume ps(y) ≠ pt(y) while ps(x|y) = pt(x|y), using black-box predictors, expectation-maximization, domain adaptation, or importance weighting.
- The Donsker-Varadhan representation is a variational form of KL divergence whose optimal bound gives a log-likelihood ratio, though direct use can be unstable.
3. Method
The method addresses label shift by disentangling the source label distribution from model predictions, first through post-compensation and then through training-time regularization with LADE.
- Problem formulation: Long-tailed recognition commonly trains on a long-tailed source distribution but evaluates on a uniform target, although target distributions may be arbitrary.
- PC Softmax: PC Softmax modifies model logits during inference to remove source-distribution entanglement and incorporate an arbitrary target label distribution.For Softmax regression, this adjustment estimates the target distribution while preserving categorical probabilities.
- PC Softmax: PC Softmax surpasses previous state-of-the-art long-tailed visual recognition methods, establishing disentanglement as a strong baseline.
- LADER and LADE: LADE uses the regularized Donsker-Varadhan representation to train logits that directly disentangle the source label distribution from model predictions.The objective models a likelihood ratio using logits parameterized by a deep neural network, with Monte Carlo approximation over minibatches.
- LADER and LADE: LADER regularizes logits toward log(p_u(x|y)/p_u(x)), and LADE combines this regularizer with cross-entropy training.The regularization strength is controlled by α, with class-specific weights α_c = p_s(y = c) applied to L_LADER^c.
- Target-distribution adaptation: Unlike Balanced Softmax, which covers only uniform target distributions, LADE is designed to handle arbitrary target distributions without retraining.Balanced Softmax is equivalent to LADE when α = 0, despite the methods’ different derivations.
4. Experiments
The experiments evaluate PC Softmax and LADE on standard long-tailed benchmarks, shifted target label distributions, and calibration. LADE consistently advances performance across these settings and regularizes logits as intended.
- Experimental setup: The evaluation compares PC Softmax and LADE with baseline, two-stage, and other state-of-the-art methods using top-1 accuracy.ImageNet-LT and Places-LT results also report Many, Medium, and Few class groups.
- Balanced test label distribution: LADE achieves better performance than previous methods on CIFAR-100-LT, Places-LT, ImageNet-LT, and iNaturalist 2018 under uniform target label distributions.PC Softmax is comparable to or better than previous state-of-the-art results on these benchmarks.
- Balanced test label distribution: 38.8% top-1 overall accuracy is LADE’s Places-LT state-of-the-art result without two-stage training.PC Softmax also surpasses the previous state-of-the-art, despite differing from Softmax only during inference.
- Balanced test label distribution: 53.0% top-1 overall accuracy is LADE’s ImageNet-LT result with 180 epochs, exceeding previous state-of-the-art methods; PC Softmax reaches 52.8%.LADE also achieves the best result when methods are trained for 90 epochs, although it is under-fitting at that duration.
- Balanced test label distribution: 70.0% top-1 accuracy is LADE’s best result on iNaturalist 2018, and it surpasses PC Softmax by +0.7%.The results indicate that LADE scales well on large datasets, whereas PC Softmax performs better on smaller datasets.
- Shifted test label distribution: Across shifted ImageNet-LT target distributions, PC Softmax provides consistent gains and LADE outperforms all other methods in every imbalance setting.The performance gap between LADE and PC Softmax widens as the test distribution becomes more imbalanced.
- Further analysis: As α increases, LADE-regularized logits converge toward log C = log 100, while experiments also show improved confidence calibration.The logit behavior matches the stated theoretical value under the visualization analysis.
5. Conclusion
The paper argues that disentangling the source label distribution from model predictions benefits long-tailed visual recognition. It introduces PC Softmax and LADE, which outperform existing methods on benchmark and shifted target distributions, while LADE also improves confidence calibration.
- PC Softmax post-processes cross-entropy and Softmax predictions to match an arbitrary target label distribution.It disentangles the source label distribution during inference.
- LADE directly disentangles the source label distribution during training using the optimal bound of Donsker-Varadhan representation.
- PC Softmax and LADE outperform state-of-the-art long-tailed recognition methods on real-world benchmark datasets.
- LADE achieves state-of-the-art performance across various shifted target label distributions.
- LADE is effective for confidence calibration and is proposed for future extension to long-tailed object detection and segmentation.
Appendix
The appendix provides derivations, experimental settings, ablations, shifted-distribution evaluations, calibration procedures, and reliability-diagram results for LADE and PC Softmax.
- Derivation: The appendix derives PC Softmax from source and target conditional probabilities and logits, then reconstructs the post-compensated Softmax output.
- Ablation study: Removing either the DV-representation regularizer or LADER causes performance degradation, with severe drops on ImageNet-LT and iNaturalist 2018 when the former is removed.
- Shifted target distributions: LADE outperforms all other methods on shifted CIFAR-100-LT test distributions, consistent with its ImageNet-LT results across dataset scales.
- Confidence calibration: Calibration is evaluated using Expected Calibration Error, Classwise Expected Calibration Error, and Negative Log Likelihood, with LADE showing better overall results than baselines on CIFAR-100-LT and ImageNet-LT.The evaluation uses four metrics overall, including the listed measures and the calibration protocol defined by test-sample bins.
- Confidence calibration: Reliability diagrams report ResNet-32 calibration on CIFAR-100-LT with imbalance ratio 50.