Source-linked AI summary
ResLT: Residual Learning for Long-tailed Recognition
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, Jiaya Jia
TL;DR
Long-tailed recognition suffers because head classes dominate training, while re-sampling and re-weighting can overfit tail classes or make optimization difficult. ResLT addresses this through parameter-space re-balancing with residual fusion, and reports effectiveness across several long-tailed benchmarks, including consistent superiority over two-stage methods on four datasets. The paper also notes that further theoretical analysis remains future work.
Problem
Long-tailed data causes head classes to dominate training, while existing re-sampling and re-weighting approaches can overfit tail classes or create optimization difficulties.
Method
ResLT re-balances in parameter space using an all-class main branch and residual branches specialized for medium+tail and tail classes.
Results
ResLT consistently surpasses two-stage methods on long-tailed CIFAR-10, CIFAR-100, ImageNet, and iNaturalist 2018, and is validated on Places as well.
Takeaways & Limitations
Residual fusion provides an effective parameter-space perspective for re-balancing head and tail classes across representative long-tailed benchmarks.
Takeaways & Limitations
The paper leaves more theoretical analysis of ResLT and its trade-off between head- and tail-class accuracy for future work.
Abstract
from arXiv · showhide
Deep learning algorithms face great challenges with long-tailed data distribution which, however, is quite a common case in real-world scenarios. Previous methods tackle the problem from either the aspect of input space (re-sampling classes with different frequencies) or loss space (re-weighting classes with different weights), suffering from heavy over-fitting to tail classes or hard optimization during training. To alleviate these issues, we propose a more fundamental perspective for long-tailed recognition, i.e., from the aspect of parameter space, and aims to preserve specific capacity for classes with low frequencies. From this perspective, the trivial solution utilizes different branches for the head, medium, and tail classes respectively, and then sums their outputs as the final results is not feasible. Instead, we design the effective residual fusion mechanism -- with one main branch optimized to recognize images from all classes, another two residual branches are gradually fused and optimized to enhance images from medium+tail classes and tail classes respectively. Then the branches are aggregated into final results by additive shortcuts. We test our method on several benchmarks, i.e., long-tailed version of CIFAR-10, CIFAR-100, Places, ImageNet, and iNaturalist 2018. Experimental results manifest the effectiveness of our method. Our code is available at https://github.com/jiequancui/ResLT.
1 INTRODUCTION
Long-tailed recognition is difficult because head classes dominate training, while common re-sampling and re-weighting methods can overfit tail classes, impair generalization, or complicate optimization. ResLT instead re-balances in parameter space through residual fusion, preserving specialized capacity and improving recognition across representative benchmarks.
- Long-tailed data makes head classes dominate training, producing much higher head-class accuracy than tail-class accuracy.
- Re-sampling can overfit tail classes or impair generalization by discarding many head-class images, while re-weighting can cause optimization difficulties.
- Residual fusion uses an all-class main branch plus residual branches for medium+tail and tail classes, whose outputs are combined through additive shortcuts.
- ResLT directly re-balances model parameters, preserving specialized capacity for head, medium, and tail classes through three sub-branches.
- ResLT was validated on long-tailed CIFAR-10, CIFAR-100, ImageNet, Places, and iNaturalist 2018, consistently surpassing two-stage methods on four listed benchmarks.
2 RELATED WORK
Related work addresses long-tailed recognition through re-sampling, re-weighting, two-stage optimization, and knowledge transfer. These approaches trade off tail emphasis, data use, optimization, representation quality, or end-to-end training.
- Re-sampling Strategy: Re-sampling either over-samples tail images, risking over-fitting, or under-samples head images, reducing data use and generalization.
- Re-weighting Strategy: Re-weighting assigns class or sample weights, but large-scale training can become difficult to optimize.
- Two-stage Methods: Two-stage methods first learn with vanilla or uniform sampling and later rebalance or fine-tune the classifier, improving long-tailed recognition at the cost of end-to-end training.
- Meta Learning and Transfer Learning: Meta and transfer learning methods transfer information from head classes to tail classes using meta regression, memory banks, or dynamic embeddings.
3 METHOD
ResLT re-balances long-tailed recognition in parameter space by preserving specialized capacity and combining branches through residual fusion. Its nested branch assignments and additive shortcuts address the weaknesses of naive branch fusion, while ablations identify both mechanisms as essential.
- Re-balancing in Parameter Space: Parameter-space re-balancing preserves specialized capacity for head, medium, and tail classes instead of modifying sampling or class weights.The proposed perspective targets model parameters directly to avoid over-fitting and optimization difficulties associated with prior approaches.
- Residual Fusion Mechanism: Naive fusion methods with separate head, medium, and tail branches perform unsatisfactorily and can underperform the vanilla baseline.This motivates replacing independent branch outputs with the residual fusion mechanism.
- Residual Fusion Mechanism: ResLT uses a main branch for all classes and residual branches for medium+tail and tail classes, whose outputs are added for the final prediction.The branches are implemented with shared and specialized parameters, and nested class assignments preserve class-specific capacity.
- Parameter Specialization: Parameter specialization remains important even though the specialized sub-branches use only a 1x1 grouped convolution with three groups.Ablations compare models with and without this mechanism on CIFAR-10-LT and CIFAR-100-LT.
- Residual Fusion Mechanism: Residual fusion combines nested class assignments with additive shortcuts; removing either design element lowers performance relative to the full module.Variants without the residual relationship or additive shortcuts are inferior to ResLT, while a weak variant remains below ResLT.
- Relation to Re-weighting: ResLT is designed specifically for long-tailed data, adaptively enhancing tail predictions through learned residual branches rather than fixed class-weight scalars.The method explicitly preserves class-specific capacity and differs from naive re-weighting despite calculating tail losses through multiple branches.
4.1 Datasets and Experimental Setting
Experiments use long-tailed versions of CIFAR-10, CIFAR-100, ImageNet, Places, and iNaturalist 2018, with evaluation stratified by class frequency. Settings include established backbones and reported top-1 accuracy comparisons.
- Datasets: The benchmarks comprise long-tailed CIFAR-10, CIFAR-100, ImageNet, Places, and iNaturalist 2018 datasets.CIFAR datasets use imbalance factors 0.01, 0.02, and 0.1; ImageNet-LT and Places-LT follow long-tailed sampling.
- Datasets: ImageNet-LT contains 115.8K images across 1,000 categories, while Places-LT contains 184.5K images across 365 categories.Class cardinalities range from 5 to 1,280 for ImageNet-LT and from 5 to 4,980 for Places-LT.
- Datasets: iNaturalist 2018 contains 437.5K images from 8,142 categories and combines extreme imbalance with fine-grained classification.
- Evaluation protocol: ImageNet-LT, Places-LT, and iNaturalist 2018 are evaluated using Many-shot, Medium-shot, and Few-shot class splits.The splits contain more than 100, 20–100, and fewer than 20 images, respectively.
- Training settings: CIFAR experiments use ResNet-32, while larger datasets use ResNet-152 for Places-LT, multiple backbones for ImageNet-LT, and ResNet-50 or ResNet-152 for iNaturalist 2018.CIFAR models are trained for 200 epochs with SGD and momentum 0.9.
4.2 Ablation Study
Ablations indicate that ResLT’s gains are not explained by merely adding parameters, and that its residual branches specialize for different frequency groups while improving medium- and tail-class performance.
- Parameter ablation: ResLT significantly surpasses grouped-convolution and three-separate-convolution baselines, indicating its gains do not come from a small number of extra parameters.
- Baseline comparison: ResLT significantly outperforms BBN, another method with extra parameters.
- Model size: Across increasing model sizes, ResLT consistently surpasses BBN on CIFAR-LT and LWS on ImageNet-LT.The comparison uses ResNet-32 variants for CIFAR-LT and ResNet-10, ResNeXt-50, and ResNeXt-101 for ImageNet-LT.
- Branch specialization: The main branch and two residual branches are associated with head, medium, and tail classes, respectively, reserving specialized parameter capacity.
- Branch specialization: Adding the residual branches significantly enhances medium- and tail-class performance over the single main branch.
4.2.4 Accuracy on Many-shot, Medium-shot, and Few-shot
ResLT improves the balance among Many-shot, Medium-shot, and Few-shot classes on long-tailed benchmarks. Its stronger Medium-shot and Few-shot performance is accompanied by only a small degradation on Many-shot classes.
- Accuracy by class frequency: ResLT usually achieves higher Medium-shot and Few-shot performance than LWS and cRT on ImageNet-LT.
- Accuracy by class frequency: Overall improvements arise from strong Medium-shot and Few-shot performance with a small degradation on Many-shot classes.
- Evaluation: ImageNet-LT class-frequency results are summarized in Table 6 using top-1 accuracy across various ResNet backbones.
- Branch analysis: The analysis reports branch performance for Many-shot, Medium-shot, and Few-shot classes on CIFAR100-LT and CIFAR10-LT with imbalance ratio 0.01.
- Accuracy by class frequency: ResLT provides a better trade-off among Many-shot, Medium-shot, and Few-shot classes.
4.2.5 How does the hyper-parameter α affect our method?
The residual mechanism remains effective across tested values of α, with dataset-specific settings used in the experiments. Increasing the number of groups beyond three provides no obvious improvement.
- Hyper-parameter α: α is set to 0.995 for CIFAR-10-LT and CIFAR-100-LT, 0.99 for ImageNet-LT, and 0.90 for iNaturalist 2018.
- Hyper-parameter α: Even α = 1, using only Lbranch during training, performs much better than the cited baselines.
- Evaluation: The reported ImageNet-LT and iNaturalist 2018 comparisons include training-epoch and inference-speed measurements.
- Number of groups: When the number of groups exceeds 3, there are no obvious improvements on CIFAR-10-LT and CIFAR-100-LT with imbalance factor 0.01.
4.2.7 Feature visualizations
ResLT representations on CIFAR-10-LT and CIFAR-100-LT are more compact and better separated than those learned with vanilla cross-entropy, supporting improved performance. The paper also frames this result as a better head–tail accuracy trade-off, while leaving deeper theoretical analysis for future work.
- Discussion: ResLT can learn a better trade-off between head-class and tail-class accuracy through its residual learning mechanism.The authors relate this trade-off to prior observations about long-tailed classification.
- Limitations: More theoretical analysis of ResLT is left for future work.
4.3 Comparisons with Previous Methods
ResLT is evaluated against one-stage and two-stage long-tailed recognition methods across CIFAR, ImageNet, Places, and iNaturalist benchmarks. It consistently outperforms strong comparison methods, including under matched training and inference settings.
- Comparison setup: The comparisons cover one-stage methods such as mixup, LDAM, BBN, and Causal Norm, plus two-stage methods including Decouple and RIDE.Two-stage comparisons allow different sampling or loss strategies across training stages.
- ImageNet-LT: 1.4% higher than Decouple and 1.0% higher than Causal Norm, ResLT improves ImageNet-LT performance with ResNeXt-50.ResLT also surpasses LWS by 0.8%–1.4% across ResNet-10, ResNeXt-50, and ResNeXt-101.
- ImageNet-LT: 57.6% top-1 accuracy with 3 experts, ResLT outperforms RIDE by 1.6% on ImageNet-LT under the matched RIDEResNeXt-50 comparison.Inference costs are comparable: 17.2 ms for ResLT versus 17.0 ms for RIDE with batches of 64 images.
- Places-LT: 39.8% Top-1 accuracy, ResLT exceeds the previous Decoupling method by 1.9% on Places-LT.The experiment uses a ResNet-152 backbone pre-trained on the full ImageNet dataset.
- iNaturalist 2018: 70.2% top-1 accuracy, ResLT surpasses Kang et al. by 0.7% on iNaturalist 2018 in an end-to-end setting.With 3 experts, ResLT reaches 72.9% top-1 accuracy and surpasses RIDE by 1.2%.
- CIFAR-LT: 2.78%, 2.96%, and 1.76% higher than BBN, ResLT improves CIFAR-100-LT across imbalance factors 0.01, 0.02, and 0.1.It also surpasses Causal Norm by 1.24% and 1.19% at imbalance ratios 0.01 and 0.1.
5 CONCLUSION
The paper presents residual fusion as a parameter-space perspective for re-balancing long-tailed recognition. Experiments across representative challenging benchmarks support the method's effectiveness, while further theoretical analysis remains future work.
- Conclusion: ResLT introduces residual fusion to re-balance head and tail classes directly in parameter space.The conclusion presents this as a novel perspective for understanding and addressing long-tailed recognition.
- Conclusion: Experiments on various representative and challenging benchmarks manifest the effectiveness of the proposed method.
- Conclusion: Ablation studies identify parameter specialization and residual learning as key components, while more theoretical analysis is deferred to future work.