Source-linked AI summary
Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, Boqing Gong
TL;DR
Long-tailed training data mismatches the expectation of balanced performance because tail-class examples may not represent inference-time variation. The paper reframes class balancing as domain adaptation, adds meta-learned conditional-distribution weights, and reports advantages across six datasets and three losses.
Problem
Long-tailed training sets and more balanced inference expectations differ, while existing class-balanced methods assume shared class-conditioned distributions that may fail for tail classes.
Method
The method augments class-balanced learning with two-component example weights, estimating conditional-distribution differences through a meta-learning re-weighting framework.
Results
The two-component weighting is advantageous over class-balanced methods across six benchmark datasets and three losses.
Takeaways & Limitations
Class-balanced learning can be extended by explicitly modeling source–target differences in class-conditioned distributions rather than assuming they are identical.
Takeaways & Limitations
Future work is needed to explore other techniques for estimating conditional weights and additional domain-adaptation strategies.
Abstract
from arXiv · showhide
Object frequency in the real world often follows a power law, leading to a mismatch between datasets with long-tailed class distributions seen by a machine learning model and our expectation of the model to perform well on all classes. We analyze this mismatch from a domain adaptation point of view. First of all, we connect existing class-balanced methods for long-tailed classification to target shift, a well-studied scenario in domain adaptation. The connection reveals that these methods implicitly assume that the training data and test data share the same class-conditioned distribution, which does not hold in general and especially for the tail classes. While a head class could contain abundant and diverse training examples that well represent the expected data at inference time, the tail classes are often short of representative training data. To this end, we propose to augment the classic class-balanced learning by explicitly estimating the differences between the class-conditioned distributions with a meta-learning approach. We validate our approach with six benchmark datasets and three loss functions.
1. Introduction
Long-tailed real-world data creates a mismatch between training distributions and the expectation of balanced performance across classes. The paper reframes class balancing as domain adaptation and augments it by modeling class-conditioned differences.
- Real-world object frequencies often follow long-tailed power-law distributions, with a few classes dominating datasets.
- Long-tailed training data can bias conventional classifiers toward head classes and underrepresent tail-class performance.The mismatch arises because test-time expectations differ from the long-tailed training distribution.
- Class-balanced methods correspond to target shift by changing class proportions while assuming identical class-conditioned distributions across training and test data.Formally, they assume Ps(y) ≠ Pt(y) and Ps(x|y) = Pt(x|y).
- The shared conditional-distribution assumption is especially implausible for tail classes because limited examples may not represent inference-time variation.Head classes may have abundant and diverse examples, whereas tail classes such as King Eider may not.
- The proposed method adds conditional-distribution weighting to classic class-wise weighting and estimates it with meta-learning.The meta-learning framework is initialized using prior knowledge of the two-component weights and expands the search space by removing two constraints.
- Experiments across artificially and naturally long-tailed datasets and three losses show that two-component weighting outperforms class-balanced methods.The evaluated losses are cross-entropy, focal loss, and label-distribution-aware margin loss.
2. Related work
Related work covers class-balanced, metric-learning, hard-example, and domain-adaptation approaches to long-tailed recognition. The paper distinguishes its loss-agnostic weighting strategy from these lines of work.
- Metric learning and hinge-loss methods commonly combine long-tail-aware sampling or grouping with large-margin losses.The paper describes its own approach as loss-agnostic and evaluates it with different loss functions.
- The paper positions its method as a loss-agnostic alternative within long-tailed recognition research.
- Hard-example mining can shift attention toward tail classes, but may be sensitive to outliers or let a minority of examples dominate training.
- Domain-adaptation methods address mismatches between training and test distributions to improve generalization to inference-time data.Related approaches include methods for handling imbalance through pseudo-label learning and generation.
3. Class balancing as domain adaptation
The paper formulates long-tailed recognition as minimizing error on a more balanced target domain despite a long-tailed source domain. Existing class balancing corrects class proportions but assumes shared class-conditioned distributions, which the paper argues fails especially for tail classes.
- The source domain has a heavy-tailed class marginal, while the desired inference-time target domain has a more balanced class distribution.The target class distribution may be uniform, and the training objective uses a differentiable surrogate such as cross-entropy.
- Importance sampling connects target-domain error to the long-tailed source domain through class-wise and conditional-distribution factors.The class-wise factor is wy = Pt(y)/Ps(y), while the conditional discrepancy is ˜ϵx,y = Pt(x|y)/Ps(x|y) − 1.
- Existing class-balanced methods estimate class-wise weights while setting the conditional discrepancy to zero.This is equivalent to assuming Ps(x|y) = Pt(x|y) for every class, the target-shift assumption.
- The paper argues that shared conditional distributions do not generally hold, especially when tail classes lack representative training examples.Dog can have abundant representative data, whereas King Eider may not.
- The proposed improvement explicitly models the difference between source and target conditional distributions on top of class-balanced methods.
4. Modeling the conditional differences
The method models each example’s training weight with class-wise and conditional components, then meta-learns the conditional component using a balanced development set. It initializes and searches these weights to improve stability and preserve a sufficiently large optimization space.
- Two-component weighting: Each example receives a class-wise weight plus a conditional weight to account for differences between source and target class-conditioned distributions.The approach allows Ps(x|y) ≠ Pt(x|y), rather than assuming identical conditional distributions.
- Class-wise weights: Class-wise weights use the effective-number formulation wy ≈ (1 − β)/(1 − β^ny), with β recommended as (n − 1)/n.Here ny is the number of training examples in class y.
- Meta-learning conditional weights: Conditional weights are estimated by holding out a balanced development set and selecting weights that produce the best-performing recognition model on it.The remaining training data is used to learn the model whose development-set performance guides the conditional-weight search.
- Greedy optimization: The optimization alternates one-step model updates using current weights with development-set updates to the conditional weights.This greedy procedure avoids brute-force search over weight sets and repeatedly updates θ and ϵ.
- Differences from L2RW: Unlike L2RW, the method retains principled two-component weights, avoids clipping and mini-batch normalization, and searches each conditional weight over the full real space.The authors argue that these L2RW constraints can exclude optimal weights; observed total weights did not become negative in their experiments.
- Overall algorithm: Algorithm 1 implements the procedure stochastically: cross-entropy pretraining is followed by meta-learning with a balanced development set while continuing model updates.The second stage initializes conditional weights at zero and updates them using development-set loss.
5. Experiments
Experiments evaluate the approach across six long-tailed datasets, three losses, comparisons with competing methods, and ablations of its weighting design. Results show consistent gains over class-balanced methods, improved tail-class accuracy without sacrificing head-class performance, and benefits from the two-component weights.
- Datasets: Experiments cover six datasets spanning artificially and naturally long-tailed recognition, with imbalance factor defined as the largest divided by smallest class size.The datasets include CIFAR-LT, ImageNet-LT, Places-LT, and iNaturalist 2017/2018.
- Comparisons: The method is evaluated with cross-entropy, focal loss, and LDAM, and outperforms competing methods in each loss-based comparison on CIFAR-LT.Its advantage generally becomes more significant as the imbalance factor increases, while balanced data does not harm vanilla-loss performance relative to L2RW.
- Class-wise behavior: On CIFAR-LT-10 with imbalance factor 200, the method maintains about the cross-entropy model’s head-class accuracy while significantly improving the last five tail classes.L2RW improves tail classes but sacrifices head-class accuracy in the same comparison.
- Class-wise behavior: The learned conditional weights are more prominent for tail classes than head classes in most training epochs, supporting greater attention to scarce tail examples.This pattern is shown for imbalance factors 100 and 10 on CIFAR-LT-10.
- Ablations: Ablations show that pre-training, two-component weights, and removing L2RW’s clipping and normalization progressively produce the strongest results among tested variations.The two-component weights benefit both L2RW and the proposed approach; removing conditional weights worsens performance relative to the original algorithm.
- Large-scale datasets: On Places-LT, the method reaches 62.90% top-1 error and 39.86/29.87% top-3/5 error, on par with 64.1% by OLTR and 63.3% by cRT.The weighting can also be applied to OLTR and cRT.
6. Conclusion
The paper reframes long-tailed recognition as a domain-adaptation mismatch and models differences between class-conditioned source and target distributions. Experiments across six datasets support the approach, while future work considers alternative conditional-weight estimators and other adaptation techniques.
- The domain-adaptation perspective decomposes long-tailed mismatch into class-wise differences and discrepancies between class-conditioned distributions.
- Existing class-balanced methods implicitly assume that training and test sets share the same class-conditioned distribution.
- The proposed method explicitly models the ratio between source and target class-conditioned distributions to relax that assumption.
- Experiments on six datasets verify the effectiveness of the proposed approach.
- Future work will explore other conditional-weight estimators, domain-invariant features, and data-sampling strategies.
Appendices
The appendices document additional experiments, including repeated CIFAR-LT-10 runs and detailed comparisons on large-scale long-tailed datasets.
- The appendices include multiple experimental runs on CIFAR-LT-10 under different imbalance factors.
- They also provide detailed comparisons of methods on large-scale long-tailed datasets.
A. Multiple runs on CIFAR-LT-10
Repeated CIFAR-LT-10 experiments show that the approach produces mean error rates consistent with the main-paper results across imbalance factors 200, 100, and 50.
- Five runs with different random seeds report mean top-1 errors and standard deviations for imbalance factors 200, 100, and 50.
- The mean error rates are consistent with the results reported in Table 2 of the main paper.
B. Detailed comparison of various methods on large-scale long-tailed datasets
The appendix compares top-1 errors and experimental setups across ImageNet-LT, Places-LT, and iNaturalist 2018. It cautions that differing setups complicate fair comparisons and recommends iNaturalist 2017 over 2018 because of the latter’s extremely small validation set.
- B. Detailed comparison of various methods on large-scale long-tailed datasets: The appendix reports top-1 errors for various methods on ImageNet-LT, Places-LT, and iNaturalist 2018.
- B. Detailed comparison of various methods on large-scale long-tailed datasets: Experimental setups are reported alongside errors because differences in initialization, minibatch sampling, losses, and trainable layers hinder fair comparison.
- B. Detailed comparison of various methods on large-scale long-tailed datasets: The authors advocate iNaturalist 2017 rather than 2018 because iNaturalist 2018 has an extremely small validation set.
- B. Detailed comparison of various methods on large-scale long-tailed datasets: The comparison includes tables for ImageNet-LT, Places-LT, and iNaturalist 2018.