Source-linked AI summary
Decoupling Representation and Classifier for Long-Tailed Recognition
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, Yannis Kalantidis
TL;DR
Long-tailed recognition raises whether class imbalance harms representation learning or mainly distorts classifier decisions. The paper separates representation learning from classification and systematically compares balancing strategies. It finds that instance-balanced sampling can produce generalizable representations, while classifier rebalancing achieves state-of-the-art benchmark performance.
Problem
Long-tailed training causes head classes to dominate, while jointly learning representations and classifiers leaves unclear whether performance gains come from representations or classifier boundaries.
Method
The paper decouples representation learning from classification and evaluates alternative sampling strategies, classifier rebalancing, and weight normalization.
Results
The study finds that instance-balanced sampling learns generalizable representations, while decoupled classifier adjustment achieves state-of-the-art performance across three long-tailed benchmarks.
Takeaways & Limitations
Strong long-tailed recognition can be obtained by keeping natural-sampling representations and properly rebalancing the classifier, without carefully designed losses or memory units.
Takeaways & Limitations
The paper does not examine all loss-reweighting approaches and reports that some recent methods are difficult to train, reproduce, and tune extensively for specific datasets.
Abstract
from arXiv · showhide
The long-tail distribution of the visual world poses great challenges for deep learning based classification models on how to handle the class imbalance problem. Existing solutions usually involve class-balancing strategies, e.g., by loss re-weighting, data re-sampling, or transfer learning from head- to tail-classes, but most of them adhere to the scheme of jointly learning representations and classifiers. In this work, we decouple the learning procedure into representation learning and classification, and systematically explore how different balancing strategies affect them for long-tailed recognition. The findings are surprising: (1) data imbalance might not be an issue in learning high-quality representations; (2) with representations learned with the simplest instance-balanced (natural) sampling, it is also possible to achieve strong long-tailed recognition ability by adjusting only the classifier. We conduct extensive experiments and set new state-of-the-art performance on common long-tailed benchmarks like ImageNet-LT, Places-LT and iNaturalist, showing that it is possible to outperform carefully designed losses, sampling strategies, even complex modules with memory, by using a straightforward approach that decouples representation and classification. Our code is available at https://github.com/facebookresearch/classifier-balancing.
1 INTRODUCTION
Long-tailed recognition suffers because head classes dominate training and tail-class accuracy drops. The paper decouples representation learning from classification to determine whether gains come from better representations or rebalanced classifier boundaries.
- Long-tailed data contain highly varying class frequencies, causing standard approaches to perform poorly and tail classes to receive weaker recognition.
- Jointly learning representations and classifiers obscures whether long-tailed gains arise from representation quality or shifted classifier decision boundaries.
- The paper evaluates sampling strategies and classifier-training strategies separately under joint and decoupled learning schemes.
- Instance-balanced sampling learns the best and most generalizable representations in the authors’ long-tailed experiments.
- Classifier retraining with class-balanced sampling or temperature-controlled weight normalization can readjust decision boundaries without additional training in the normalization case.
- Decoupled learning with standard networks achieves higher accuracy than established methods across ImageNet-LT, Places-LT, and iNaturalist.
2 RELATED WORK
Prior long-tailed recognition work mainly rebalances data, reweights losses, or transfers knowledge from head to tail classes. These approaches address imbalance through different training distributions, objectives, or feature-transfer mechanisms.
- Data-distribution methods rebalance training through oversampling minority classes, undersampling majority classes, or class-balanced sampling.
- Class-balanced loss methods assign different weights at class or sample level to improve generalization for tail classes.
- Transfer-learning methods move features such as intra-class variance or semantic deep features from head classes to under-represented tail classes.
- Long-tail recognition differs from low-shot recognition because training accesses both head and tail classes with a continuous class-frequency decline.
3 LEARNING REPRESENTATIONS FOR LONG-TAILED RECOGNITION
The paper studies how sampling affects representation learning separately from classifier learning in long-tailed recognition. It defines representation and classifier notation, compares sampling schemes, and notes practical difficulty in reproducing sophisticated loss-reweighting methods.
- The study disentangles representation learning from classifier learning to identify what matters for long-tailed recognition.
- Notation: Representations are produced by a deep CNN f(x; θ), while predictions use a classifier g, commonly the linear function g(z) = W^Tz + b.
- Sampling strategies: Sampling strategies assign class-dependent probabilities intended to rebalance the data distribution, with q values defining different strategies.
- Sampling strategies: Instance-balanced sampling selects each training example equally often in expectation, making class-j sampling proportional to its training cardinality n_j.
- Sampling strategies: Class-balanced sampling gives every class equal selection probability to address under-fitting of few-shot classes.
- Sampling strategies: Progressively-balanced sampling starts with instance-balanced sampling and later uses class-balanced sampling, requiring a switching-epoch hyper-parameter.
- Loss re-weighting strategies: The authors restrict their loss-reweighting study because recent high-performing methods can be difficult to train, reproduce, and tune across datasets.
4 CLASSIFICATION FOR LONG-TAILED RECOGNITION
The paper decouples representation learning from classification to study how classifier training and weight adjustment can rectify long-tailed decision boundaries. It presents classifier retraining, nearest class mean classification, τ-normalization, and learnable weight scaling as alternatives to joint training.
- Decoupled classification: Decoupling fixes the representations while learning classifiers that rectify decision boundaries between head and tail classes.The section frames long-tailed recognition as separate representation-learning and classification stages rather than one jointly trained model.
- Classifier Re-training (cRT): Classifier retraining reinitializes and optimizes W and b for a few epochs with class-balanced sampling while keeping representations fixed.
- Nearest Class Mean classifier (NCM): Nearest Class Mean classification assigns examples to the closest class-specific mean representation using cosine similarity or Euclidean distance on L2-normalized means.Cosine similarity inherently normalizes the classifier and alleviates weight imbalance.
- τ-normalized classifier: Joint instance-balanced training produces classifier weight norms correlated with class cardinality, whereas class-balanced fine-tuning makes the norms more similar.
- τ-normalized classifier: τ-normalization rescales classifier weights by their L2 norms using a temperature hyper-parameter, leaves directions unchanged, and discards the bias term.τ = 1 gives standard L2-normalization, τ = 0 applies no scaling, and the method uses τ ∈ (0, 1) empirically.
- Learnable weight scaling (LWS): Learnable Weight Scaling learns the per-class scaling factors with class-balanced sampling while keeping representations and classifier weights fixed.
5 EXPERIMENTS
Experiments across long-tailed benchmarks show that decoupling representation learning from classifier training substantially improves recognition, especially for medium- and few-shot classes. Instance-balanced sampling provides strong representations, while classifier re-balancing yields competitive or state-of-the-art performance without complex additions.
- Experimental setup: Experiments use Places-LT, ImageNet-LT, and iNaturalist 2018, evaluating top-1 accuracy overall and across many-, medium-, and few-shot class splits.ImageNet-LT and Places-LT are artificially long-tailed, while iNaturalist 2018 is naturally long-tailed.
- Sampling strategies and decoupled learning: Decoupled methods outperform joint training in most cases, with cRT and τ-normalization exceeding the jointly trained baseline by 5% and the best progressively-balanced setup by 2% overall.The gains reach 5% for medium-shot and 11% for few-shot classes, while joint instance-balanced training remains strongest for many-shot classes.
- Sampling strategies and decoupled learning: 49.5% accuracy is achieved by retraining only the classifier, compared with 46.3% and 48.8% when fine-tuning the whole network.Keeping the representation frozen performs best, with the trend becoming more evident for medium- and few-shot classes.
- Sampling strategies and decoupled learning: Instance-balanced sampling gives the best decoupled performance across overall and non-many-shot evaluations, suggesting that imbalance may not prevent learning high-quality representations.Among classifier-balancing methods, cRT and τ-normalization are generally stronger than NCM, except that NCM is comparable for few-shot classes.
- Sampling strategies and decoupled learning: Classifier weight norms increase with the number of training instances, while class-balanced fine-tuning and τ-normalization make classifier norms more similar.Increasing τ sharply decreases many-shot accuracy while sharply increasing few-shot accuracy.
- Comparison with the state of the art on long-tailed datasets: 69.3% accuracy on iNaturalist 2018 with ResNet-50 and 72.5% with ResNet-152 establish new state-of-the-art results after longer representation training and classifier normalization.On ImageNet-LT, properly re-balanced classifiers outperform the state of the art across backbone architectures; on Places-LT, all three decoupled methods outperform prior approaches, with τ-normalization best.
6 CONCLUSIONS
The paper compares joint and decoupled learning for long-tailed recognition and finds that instance-balanced representations can support strong performance after classifier re-balancing. It reports state-of-the-art results without carefully designed losses or memory units.
- Instance-balanced sampling can produce generalizable representations that achieve state-of-the-art long-tailed recognition after classifier re-balancing.This finding contrasts with the stronger dependence on sampling strategies observed under joint representation-classifier training.
- The study compares joint representation-classifier learning with straightforward decoupled methods across three long-tailed recognition benchmarks.The benchmarks are ImageNet-LT, Places-LT, and iNaturalist.
- The paper argues that strong performance can be obtained without carefully designed losses or memory units.
B.1 SAMPLING STRATEGIES
Joint learning benefits from improved sampling strategies, while classifier normalization balances decision boundaries and gives few-shot classes more feature-space coverage.
- Better sampling strategies improve accuracy when representations and classifiers are trained jointly.The explored strategies include instance-balanced, class-balanced, square-root, and progressively-balanced sampling.
- Progressively-balanced sampling transitions from instance-balanced to class-balanced sampling as training epochs progress.
- Balancing classifier norms produces more balanced decision boundaries and allows few-shot classes to occupy more feature-space area.
- Table 5 reports ImageNet-LT accuracy for joint learning under different sampling strategies.
B.4 VARYING THE BACKBONE ARCHITECTURE SIZE
The experiments examine how backbone capacity and training duration affect long-tailed recognition across ImageNet-LT and iNaturalist 2018.
- Different backbone architectures and capacities are compared for long-tailed recognition on ImageNet-LT.The comparison includes OLTR variants, cRT, cRT with memory, and τ-normalized classifiers.
- On iNaturalist 2018, larger models and longer training produce much higher accuracy than 90-epoch training.
- Table 6 provides comparative analysis of different classifier-learning approaches for long-tail recognition.
B.5 ON THE EXPLORATION OF DETERMINING τ
The paper examines alternatives to validation-based selection of τ, including training-set selection and learning τ directly with balanced sampling.
- Selecting τ requires a validation set, which may be disadvantageous in some practical scenarios.
- Training-set selection produces τ values and test performance very close to those obtained with validation-set selection.The training-set procedure averages class-specific top-1 accuracies to choose τ.
- τ can be learned as a parameter on the training set with balanced sampling while the backbone and classifier remain fixed.This approach reduces the manual effort of searching for the best τ values.
- Table 9 compares τ determined using training and validation sets across all three datasets.
B.6 COMPARING MLP CLASSFIIER WITH LINEAR CLASSIFIER
The section compares MLP classifiers with linear and cosine-similarity alternatives on fixed representations. Deeper MLP classifiers perform worse on both ImageNet-LT and iNaturalist, while cosine-similarity variants perform comparably.
- MLPs use 2 or 3 layers, 2048 or 512 hidden neurons, ReLU, and balanced sampling for 10 epochs on fixed representations.Training uses batch size 512 and a cosine learning-rate schedule that decreases the rate to zero.
- On ImageNet-LT, deeper MLP classifiers reduce performance, suggesting the backbone already learns discriminative representations.The experiments use a ResNeXt50 backbone.
- On iNaturalist, deeper classifiers cause an even more severe performance drop when using representations from a 200-epoch ResNet50.Only a 2048-dimensional hidden layer is considered because iNaturalist contains many classes.
- Cosine-similarity classifiers with and without the final ReLU achieve comparable results.The variants are denoted “cos” and “cos(noRelu)” and are compared with the linear classifier.