Source-linked AI summary
ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot
Jiarui Cai, Yizhou Wang, Jenq-Neng Hwang
TL;DR
Long-tailed recognition methods face a seesaw between majority and minority accuracy, while multi-stage solutions depend on pre-training and are difficult to transfer to tasks such as detection and segmentation. ACE uses complementary experts with adaptive optimization in a single end-to-end stage. Across standard long-tailed benchmarks, it improves one-stage performance by 3-10% and improves majority and minority groups simultaneously.
Problem
Long-tailed recognition must handle scarce tail categories without sacrificing majority accuracy, but existing solutions rely on trade-offs or multi-stage training that depends on pre-trained representations and is difficult to integrate into detection and segmentation.
Method
ACE trains parallel experts with a shared backbone on diverse overlapping subsets, suppresses interference from unseen categories, and adapts each expert’s learning pace end-to-end.
Results
3-10% accuracy gain establishes ACE as the new one-stage state of the art, with improvements reported across many-shot, medium-shot, and few-shot groups.
Takeaways & Limitations
ACE is reported to break the seesaw trade-off by improving majority and minority performance simultaneously in one stage, with performance comparable to multi-stage methods.
Takeaways & Limitations
The distribution-aware planner assumes the total number of categories is a multiple of the number of experts for simplicity and without loss of generality.
Abstract
from arXiv · showhide
One-stage long-tailed recognition methods improve the overall performance in a "seesaw" manner, i.e., either sacrifice the head's accuracy for better tail classification or elevate the head's accuracy even higher but ignore the tail. Existing algorithms bypass such trade-off by a multi-stage training process: pre-training on imbalanced set and fine-tuning on balanced set. Though achieving promising performance, not only are they sensitive to the generalizability of the pre-trained model, but also not easily integrated into other computer vision tasks like detection and segmentation, where pre-training of classifiers solely is not applicable. In this paper, we propose a one-stage long-tailed recognition scheme, ally complementary experts (ACE), where the expert is the most knowledgeable specialist in a sub-set that dominates its training, and is complementary to other experts in the less-seen categories without being disturbed by what it has never seen. We design a distribution-adaptive optimizer to adjust the learning pace of each expert to avoid over-fitting. Without special bells and whistles, the vanilla ACE outperforms the current one-stage SOTA method by 3-10% on CIFAR10-LT, CIFAR100-LT, ImageNet-LT and iNaturalist datasets. It is also shown to be the first one to break the "seesaw" trade-off by improving the accuracy of the majority and minority categories simultaneously in only one stage. Code and trained models are at https://github.com/jrcai/ACE.
1. Introduction
Long-tailed recognition suffers because models favor sample-rich classes, while methods that improve tail performance often sacrifice head accuracy. ACE addresses this trade-off with complementary experts trained end-to-end in one stage and reports simultaneous gains across majority and minority groups.
- Motivation: Long-tailed datasets cause models dominated by sample-rich classes to lose sight of scarce tail categories.This limits practical use in applications including biological identification, disease classification, spam detection, detection, and instance segmentation.
- Limitations of Existing Methods: One-stage re-balancing methods improve minority accuracy but exhibit a seesaw trade-off by sacrificing majority-class accuracy.Existing multi-stage approaches address the imbalance through backbone pre-training followed by classifier adjustment or expert construction, but remain sensitive to pre-trained representations and re-balancing.
- ACE: ACE draws on complementary specialist expertise, with experts specializing in dominant subsets while incorporating information from experts exposed to more data.This design is motivated by specialist consultation for difficult and rare cases.
- ACE: ACE uses parallel experts with a shared backbone, diverse overlapping imbalanced subsets, distribution-adaptive updates, and split-based output aggregation.The method is trained end-to-end without pre-training or staged training.
- Results: 3-10% accuracy gain makes ACE the new one-stage state of the art across CIFAR10-LT, CIFAR100-LT, ImageNet-LT, and iNaturalist2018.It is also reported as the first method to improve all three frequency groups: many-shot, medium-shot, and few-shot.
2. Related Works
Long-tailed recognition methods address imbalance through data re-balancing, staged transfer, or multi-expert architectures. Re-balancing and augmentation methods face conflicting head-tail effects, while multi-expert methods separately process relatively balanced subgroups but retain trade-offs or additional training mechanisms.
- Method Categories: Long-tailed recognition methods are grouped into data-distribution readjustment, two-stage transfer learning, and multi-expert or multi-branch frameworks.These categories organize the main strategies used to address class imbalance.
- Re-balancing: Re-balancing uses under-sampling, over-sampling, or frequency-based loss re-weighting to alter the training distribution.Naive class-balanced sampling can overfit scarce classes, motivating square-root or progressively adjusted sampling.
- Augmentation: Strong augmentation increases training-set diversity, but long-tailed methods commonly sacrifice either head or tail performance because representation and classifier learning favor different sampling biases.Mixup, ReMix, and tail-class synthesis are representative augmentation approaches.
- Two-Stage Learning: Two-stage methods first learn representations on imbalanced data, then re-balance classifiers through sampling, re-weighting, parameter scaling, or memory-based transfer.Examples include DRS, DRW, τ-norm, LWS, OLTR, and IEM.
- Multi-Expert Networks: Multi-expert networks process relatively balanced subgroups separately, using branches or diverse classifiers to address long-tailed recognition.BBN uses normal and reversed sampling, while LFME and RIDE learn diverse classifiers in parallel with additional knowledge-transfer mechanisms.
3. Proposed Methodology
ACE combines a shared backbone with complementary experts trained on diverse, overlapping category splits. Each expert specializes in target categories, suppresses interfering categories, and contributes through distribution-aware optimization and group-average aggregation.
- Architecture: ACE branches multiple experts from a shared backbone and assigns each diverse, overlapping target and interfering category splits.Different target-category dominances encourage specialization, while overlap lets experts support one another on sample-few categories.
- Distribution-aware Planner: The distribution-aware planner exposes smaller classes to more experts while allowing medium-shot and few-shot classes to dominate some experts.Each expert trains on the mini-batch subset whose labels belong to its assigned target categories.
- Prediction Aggregation: Outputs are normalized, averaged among experts trained with each class, and passed through SoftMax to produce classification confidence.Group averaging combines the strengths of the specialists while preserving class-specific expert contributions.
- Objective Functions: Each expert uses classification loss on its target categories and complementary loss to suppress outputs for interfering categories.The complement loss acts as soft regularization, minimizing non-target logits so experts operate complementarily rather than competitively.
- Distribution-adaptive Optimizer: The optimizer assigns smaller learning weights to experts trained with less data to reduce over-fitting and support simultaneous convergence.Only the first expert updates the shared backbone because overlapping data could otherwise duplicate errors and harm representation learning.
4. Experiments
ACE is evaluated across long-tailed datasets and compared with one-stage, multi-stage, and architectural variants. It achieves broad gains across frequency groups, while complementary loss, aggregation, and distribution-aware optimization address expert competition and overfitting.
- Experimental settings: ACE is evaluated on CIFAR10-LT, CIFAR100-LT, ImageNet-LT, and iNaturalist2018 using multiple backbone and imbalance settings.The experiments include ResNet-32, ResNet-10, ResNet-50, and ResNeXt-50 configurations.
- Performance: ACE is the best one-stage method on CIFAR-LT and improves all many-, medium-, and few-shot groups in a single stage.On CIFAR100-LT-100, it surpasses the previous one-stage SOTA BBN by 9.0%.
- Performance: 6.4% and 7.3% are ACE’s reported gains over BBN on ImageNet-LT with ResNet-50 and ResNeXt-50, respectively, while the gain is 3.9% on iNaturalist2018.These comparisons use multiple backbone models and datasets.
- How ACE works: Complementary loss makes experts contribute jointly on shared splits and suppresses interfering classes, unlike variants where experts compete or a full-set expert dominates.The complementary-loss design gives experts similar scales on shared data while allowing minority-split experts to provide support.
- How ACE works: The distribution-aware optimizer controls expert learning speed according to data assignments, reducing overfitting that otherwise improves majority categories while hurting the tails.Uniform optimization favors majority improvements, whereas square-root scaling performs better on medium- and few-shot splits.
- How ACE works: Scaling logits before group averaging preserves head-class accuracy, whereas alternative aggregation schemes can amplify overconfidence and expert competition.ACE combines scaling with learning-speed adjustment and complementary loss to improve all frequency groups.
5. Conclusion
ACE addresses long-tailed recognition by jointly optimizing complementary experts, data/objective constraints, and distribution-adaptive learning within one network. It achieves strong one-stage performance while improving dominating and minority groups across all three frequency splits.
- ACE uses a multi-expert network to optimize representation learning and classifier learning in a uniform network.
- Complementary constraints suppress non-targeted groups and promote both dominating and minority groups.
- A distribution-adaptive optimization scheme adjusts each expert’s learning pace to avoid over-fitting.
- 3∼10% accuracy gain makes ACE the new SOTA among one-stage long-tailed recognition methods.
- ACE is the first method reported to improve performance on all three frequency splits, with potential extension to detection and segmentation.