Long-tailed Recognition by Routing Diverse Distribution-Aware Experts

Xudong Wang; Long Lian; Zhongqi Miao; Ziwei Liu; Stella X. Yu

Source-linked AI summary

Long-tailed Recognition by Routing Diverse Distribution-Aware Experts

Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, Stella X. Yu

arXiv:2010.01809v4Oct 5, 2020cs.CV

TL;DR

Long-tailed recognition must handle severe class imbalance and many small-data tail problems, while existing methods leave a large head-tail bias gap and increase variance. RIDE uses diverse distribution-aware experts with dynamic routing to address these issues, achieving 5%∼7% higher accuracies than current SOTA methods across three benchmarks.

Problem
Long-tailed recognition lacks a principled way to analyze the apparent head-tail performance trade-off while addressing imbalance and tail-specific small-data learning.
Method
RIDE trains partially shared, distribution-aware diverse experts and dynamically routes instances to additional experts when necessary.
Results
5%∼7% higher accuracies than current SOTA methods are reported on CIFAR100-LT, ImageNet-LT, and iNaturalist.
Takeaways & Limitations
RIDE reduces model bias and variance throughout while maintaining computational costs comparable to a single expert and working across backbones and training schemes.
Takeaways & Limitations
The bias-variance analysis treats the training data D as a random variable representing fluctuating realizations from an underlying distribution.

Abstract

from arXiv · show

Natural data are often long-tail distributed over semantic classes. Existing recognition methods tackle this imbalanced classification by placing more emphasis on the tail data, through class re-balancing/re-weighting or ensembling over different data groups, resulting in increased tail accuracies but reduced head accuracies. We take a dynamic view of the training data and provide a principled model bias and variance analysis as the training data fluctuates: Existing long-tail classifiers invariably increase the model variance and the head-tail model bias gap remains large, due to more and larger confusion with hard negatives for the tail. We propose a new long-tailed classifier called RoutIng Diverse Experts (RIDE). It reduces the model variance with multiple experts, reduces the model bias with a distribution-aware diversity loss, reduces the computational cost with a dynamic expert routing module. RIDE outperforms the state-of-the-art by 5% to 7% on CIFAR100-LT, ImageNet-LT and iNaturalist 2018 benchmarks. It is also a universal framework that is applicable to various backbone networks, long-tailed algorithms, and training mechanisms for consistent performance gains. Our code is available at: https://github.com/frank-xwang/RIDE-LongTailRecognition.

1 INTRODUCTION

Long-tailed recognition must address both extreme class imbalance and many small-data learning problems. The paper analyzes fluctuating training data through model bias and variance, then proposes RIDE to reduce both while preserving accuracy across class-frequency splits.

Long-tailed recognition combines extreme class imbalance with numerous small-data learning problems in tail classes.
Existing methods rebalance classes or ensemble models trained on different data distributions, improving tail accuracy but often reducing head accuracy.
A dynamic training-set view treats the dataset as fluctuating samples from an underlying distribution rather than a fixed set.
Existing long-tail methods reduce tail bias but retain a large head-tail bias gap and increase model variance across class splits.For CE, head and tail bias are 0.3 and 0.9; existing methods leave a 0.3 versus 0.8 gap, while increasing variance overall.
Hard-negative confusion increasingly affects tail classes, contributing to the persistent head-tail bias gap.The analysis measures the largest softmax score among non-ground-truth classes as a confusion indicator.
RIDE uses multiple experts, distribution-aware diversity, and dynamic routing to reduce variance, tail bias, and computational cost.Its routing module invokes additional experts only when needed, while shared reduced-complexity expert architectures can lower cost below a same-backbone baseline.
RIDE achieves 5%∼7% higher accuracies than current SOTA methods on CIFAR100-LT, ImageNet-LT, and iNaturalist.The framework also applies to different backbones and existing long-tailed algorithms such as focal loss, LDAM, and τ-norm.

2 RELATED WORKS

Prior long-tailed recognition work addresses imbalance through few-shot learning, rebalancing, knowledge transfer, and multi-expert grouping. RIDE differs by jointly optimizing partially shared experts and dynamically assigning them per instance while reducing complexity.

Few-shot learning: Few-shot recognition commonly uses meta-learning and data augmentation or generation to generalize from small training sets.
Re-balancing/re-weighting: Rebalancing methods alter sampling, generate additional tail examples, or weight tail classes more heavily in the loss.
Knowledge transfer: Knowledge-transfer methods move features from head to tail classes but often lack effective transfer control, resulting in head-performance loss.
Ensembling and grouping: Grouping and ensembling methods train experts on class-size groups or branches focused on head and tail data, yet still lose head performance or overall generalization.
RIDE shares earlier layers, reduces later channels, jointly optimizes experts, and routes additional experts only when needed.This design reaches higher accuracies with smaller model complexity and computational cost.

3 RIDE: ROUTING DIVERSE DISTRIBUTION-AWARE EXPERTS

RIDE combines jointly optimized, distribution-aware experts with dynamic routing to improve long-tailed classification while controlling variance, bias, and computation. Shared backbones, reduced expert channels, individual losses, diversity regularization, and optional self-distillation define the framework.

Model architecture and training: RIDE uses a shared backbone with independent, channel-reduced expert layers, jointly optimized before dynamic deployment.The shared early layers encode generic features, while reduced later channels help limit overfitting on small tail classes.
Model architecture and training: Individual expert losses discourage correlated experts by requiring each expert to classify well independently.The paper reports that this design contributes a large portion of the performance gain in most experiments.
Model architecture and training: A distribution-aware diversity loss encourages complementary decisions by maximizing divergence between experts’ class probabilities.Class-dependent temperatures are lower for classes with fewer instances, increasing probability sensitivity for tail classes while preserving head-class complexity.
Dynamic routing: At inference, a learned router sequentially activates additional experts only when needed, and averages active-expert logits for the final softmax decision.The router uses image features and current mean logits, with a threshold of 0.5 determining whether to continue routing.
Dynamic routing: RIDE reduces test-time cost while retaining multiple-expert predictions, and optional self-distillation from six experts improves most experiments by 0.4%–0.8%.The framework can use an arbitrary number of experts to trade classification accuracy against computational cost.

4 EXPERIMENTS

Experiments across long-tailed benchmarks show that RIDE consistently improves accuracy across training methods, backbones, and class-frequency splits while controlling computational cost. Ablations and routing analyses indicate that its gains come from multiple experts, diversity-aware training, and selective expert deployment.

CIFAR100-LT: RIDE surpasses SOTA methods by more than 5.3% on CIFAR100-LT while using about 10% less average computational cost than BBN-style two-expert baselines.The comparison uses CIFAR100-LT results and reports fewer GFlops than prior multi-expert methods.
Universal framework: RIDE delivers consistent accuracy gains with end-to-end and two-stage training mechanisms, including focal loss, CE, LDAM, cRT, τ-norm, and cosine classifiers.With a simple cosine classifier, RIDE reaches on-par performance with current SOTA methods.
iNaturalist 2018: RIDE exceeds current SOTA by 6.3% on iNaturalist 2018 and obtains very similar many-, medium-, and few-shot results.Unlike BBN, which decreases many-shot performance by about 23%, RIDE increases few-shot accuracy without reducing many-shot accuracy.
Ablation studies: On CIFAR100-LT, architectural changes contribute 2.7%–4.3%, diversity loss adds about 0.9%, and all components together deliver 7.1% over baseline LDAM.The router greatly reduces computational cost, while knowledge distillation provides another 0.6% when used.
Expert analysis: Adding more experts yields a 16% relative gain for few-shot classes versus 3.8% for many-shot classes on CIFAR100-LT.The few-shot split benefits more from additional experts than the many-shot split.
Expert analysis: The router assigns additional experts mainly to few-shot instances, while most many-shot instances require only the first expert.Low-confidence tail instances often seek a second or third opinion, reducing the cost of always evaluating every expert.
Backbone analysis: With Swin-T and Swin-S on ImageNet-LT, adding RIDE still produces around 3%–6% improvement after distribution-aware re-weighting and margin regularization.The method improves all three class-frequency subsets, although the largest gains are often in few-shot classes.

5 SUMMARY

The paper frames long-tailed recognition through model bias and variance and proposes RIDE to reduce both across class-frequency groups. RIDE uses partially shared, distribution-aware experts and routes additional experts only when needed, achieving consistent gains across backbones and training schemes.

5 SUMMARY: RIDE addresses the persistent head-tail bias gap and increased variance observed in existing long-tail classifiers.The paper reports that existing methods increase model variance across classes while insufficiently reducing the head-tail bias gap.
5 SUMMARY: RIDE trains partially shared diverse distribution-aware experts and dynamically routes instances to additional experts when necessary.The routing design keeps computational costs comparable to a single expert.
5 SUMMARY: RIDE works with various backbones and training schemes while producing consistent gains over state-of-the-art methods.The summary characterizes the framework as broadly applicable across long-tailed recognition settings.

A.1 DATASETS AND IMPLEMENTATIONS

RIDE is evaluated on three major long-tailed recognition benchmarks using multiple dataset settings and backbone networks. The experiments cover CIFAR100-LT, ImageNet-LT, and iNaturalist-2018.

Benchmarks: Experiments cover three major long-tailed recognition benchmarks: CIFAR100-LT, ImageNet-LT, and iNaturalist-2018.The study also evaluates different backbone networks to assess RIDE's effectiveness and universality.
CIFAR100-LT: CIFAR100-LT uses an exponential decay in class sample sizes with an imbalance factor of 100.The imbalance factor is the ratio between the most frequent and least frequent classes.
ImageNet-LT: ImageNet-LT contains 115.8k images across 1,000 categories, with class sizes ranging from 1,280 to 5 images.The dataset follows a Pareto distribution with power value α = 6, and experiments use ResNet-10, ResNet-50, and ResNeXt-50.
iNaturalist-2018: iNaturalist-2018 contains 437,513 training images from 8,142 classes and a balanced test set of 24,426 images.Experiments use ResNet-50 and a batch size of 512.

A.2 ADDITIONAL EXPERIMENTS

Additional experiments examine distillation, expert count, backbone choice, ensemble efficiency, feature embeddings, and performance on balanced data. Across these settings, RIDE maintains strong accuracy and computational efficiency, while its gains are especially pronounced for long-tailed data.

Distillation: Self-distillation is optional but recommended, providing 0.4%∼0.8% further improvement in most experiments.Distillation transfers knowledge from a more powerful model with more experts to a model with fewer experts.
ImageNet-LT experiments: With 2 experts on ImageNet-LT ResNet-50, RIDE gains about 7% accuracy while using about 10% less computational cost than the baseline.RIDE improves many-shot, medium-shot, and few-shot performance simultaneously.
ImageNet-LT experiments: From 3 to 4 experts, RIDE preserves many-shot accuracy while increasing few-shot accuracy on ImageNet-LT.This result indicates that adding experts can improve tail performance without reducing head performance in this setting.
Backbone comparisons: RIDE achieves lower computational cost and higher performance than competing methods with ResNet-10 and ResNeXt-50 backbones.These comparisons are reported for ImageNet-LT experiments.
Ensemble comparison: RIDE with 4 experts costs less computationally than an ensemble of 2 LDAM models while achieving higher accuracy than LDAM ensembles.The comparison spans RIDE models with 2–4 experts and LDAM ensembles with 1–7 models.
Embedding visualization: RIDE produces more compact and better-separated head and tail embeddings than LDAM on CIFAR100-LT.The improved embedding structure reduces the classifier's difficulty in distinguishing tail categories.
Balanced-data experiment: On balanced CIFAR100, RIDE improves accuracy by about 1%, substantially less than its gains on CIFAR100-LT.The experiments therefore report greater benefits from RIDE on long-tailed than balanced datasets.
Evaluation metrics: The ImageNet-LT comparisons report Top-1 accuracy together with many-shot, medium-shot, and few-shot performance across ResNet-10, ResNet-50, and ResNeXt-50.The results are presented in Tables 7–9.