PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition

Yifan Zhang, Bryan Hooi, Lanqing Hong, Jiashi Feng

arXiv:2107.09249v4cs.CV

TL;DR

Long-tailed recognition usually assumes a uniform test distribution, despite practical test distributions being unknown and potentially long-tailed or inversely long-tailed. SADE learns experts with diverse distribution skills and aggregates them self-supervised at test time; it achieves state-of-the-art performance on vanilla and test-agnostic recognition.

  • Problem

    Test-agnostic long-tailed recognition addresses long-tailed training data when the test class distribution is unknown and not necessarily uniform.

  • Method

    SADE learns skill-diverse experts from one stationary long-tailed dataset and aggregates them at test time using prediction stability from unlabeled perturbed views.

  • Results

    SADE achieves new state-of-the-art performance on both vanilla and test-agnostic long-tailed recognition, including 58.8% accuracy on ImageNet-LT.

  • Takeaways & Limitations

    Self-supervised expert aggregation provides a practical approach for handling unknown test class distributions without knowing their true distribution in advance.

  • Takeaways & Limitations

    The formulation assumes the training classes are ordered by decreasing cardinality, with a much larger head-class count than tail-class count.

Abstract

from arXiv · show

Existing long-tailed recognition methods, aiming to train class-balanced models from long-tailed data, generally assume the models would be evaluated on the uniform test class distribution. However, practical test class distributions often violate this assumption (e.g., being either long-tailed or even inversely long-tailed), which may lead existing methods to fail in real applications. In this paper, we study a more practical yet challenging task, called test-agnostic long-tailed recognition, where the training class distribution is long-tailed while the test class distribution is agnostic and not necessarily uniform. In addition to the issue of class imbalance, this task poses another challenge: the class distribution shift between the training and test data is unknown. To tackle this task, we propose a novel approach, called Self-supervised Aggregation of Diverse Experts, which consists of two strategies: (i) a new skill-diverse expert learning strategy that trains multiple experts from a single and stationary long-tailed dataset to separately handle different class distributions; (ii) a novel test-time expert aggregation strategy that leverages self-supervision to aggregate the learned multiple experts for handling unknown test class distributions. We theoretically show that our self-supervised strategy has a provable ability to simulate test-agnostic class distributions. Promising empirical results demonstrate the effectiveness of our method on both vanilla and test-agnostic long-tailed recognition. Code is available at \url{https://github.com/Vanint/SADE-AgnosticLT}.

1 Introduction

Existing long-tailed recognition methods typically target uniform test distributions, although practical test distributions may vary arbitrarily. SADE learns experts with diverse class-distribution skills and aggregates them self-supervised at test time to handle unknown distributions.

  • Real-world recognition datasets are often long-tailed, causing models trained on imbalanced data to favor head classes and perform poorly on tail classes.
  • Most existing methods assume uniform test classes, but practical test data may be uniform, long-tailed, or inversely long-tailed.
  • SADE learns multiple experts from one long-tailed dataset, with each expert specializing in a different class distribution.
  • Different training losses produce models skilled for class distributions with different skewness, providing a basis for learning diverse experts.
  • SADE aggregates experts using prediction stability on unlabeled perturbed test views, and this strategy theoretically simulates unknown test class distributions.
  • 58.8% accuracy on ImageNet-LT gives SADE more than a 2% gain over RIDE and ACE, while SADE also outperforms test-distribution-aware LADE.

2 Related Work

Related work covers long-tailed recognition methods for rebalancing class performance and test-time training methods for distribution shifts. SADE applies test-time adaptation to unknown class-distribution shifts in long-tailed recognition.

  • Long-tailed recognition methods include class re-balancing, logit adjustment, and ensemble learning.
  • Class re-balancing uses re-sampling or cost-sensitive learning, while logit adjustment modifies inference logits using training-label frequencies.
  • Test-time training is a transductive paradigm for handling distribution shifts between training and test data.
  • This study uses test-time training to address class-distribution shifts in test-agnostic long-tailed recognition.

3 Problem Formulation

The task starts with long-tailed training data but allows uniform, long-tailed, or inversely long-tailed test distributions. Existing methods may fail when the actual test distribution differs from the assumed one.

  • The training set is long-tailed, with head classes containing many samples and tail classes containing few samples.
  • The imbalance ratio is defined as max(n_k)/min(n_k) = n_1/n_C, with classes ordered by decreasing sample count.
  • Existing methods commonly assume a uniform test distribution p_t(y) = 1/C while training from a long-tailed distribution p_s(y).
  • Actual test data may be long-tailed, p_t(y) = p_s(y), or inversely long-tailed, p_t(y) = inv(p_s(y)).
  • Table 1 evaluates existing methods on uniform, forward long-tailed, and backward long-tailed test distributions with imbalance ratios of 10 and 50.

4 Method

SADE learns three experts with different class-distribution preferences, then aggregates them at test time using prediction stability from unlabeled augmented views. Theoretical analysis links this stability objective to matching the unknown test distribution while reducing prediction entropy.

  • Skill-diverse Expert Learning: Experts share backbone fθ but have independent expert networks, enabling the three-expert model to learn differentiated distribution preferences.SADE’s architecture combines one shared feature extractor with independent E1, E2, and E3 networks.
  • Skill-diverse Expert Learning: SADE trains forward, uniform, and backward experts to handle long-tailed, uniform, and inversely long-tailed class distributions.The three experts use different expertise-guided losses and target different distribution skews.
  • Skill-diverse Expert Learning: The forward expert uses softmax loss, the uniform expert uses balanced softmax, and the backward expert uses a new inverse softmax loss.The inverse loss reverses the training-prior adjustment so E3 simulates an inversely long-tailed distribution.
  • Test-time Self-supervised Aggregation: Prediction stability identifies stronger experts because their predictions remain more consistent across augmented views of samples from favorable classes.The empirical observation uses cosine similarity between predictions for two views, and Table 2 reports this relationship for CIFAR100-LT.
  • Test-time Self-supervised Aggregation: At test time, SADE freezes expert parameters and learns normalized aggregation weights by maximizing cosine similarity between predictions from two augmented views of unlabeled samples.The final prediction is σ(w1·v1 + w2·v2 + w3·v3), with w1 + w2 + w3 = 1; the procedure can operate online on streaming test data.
  • Theoretical Analysis: Theorem 1 states that maximizing prediction stability increases mutual information with the test class distribution and decreases prediction entropy.This provides a theoretical basis for learning aggregation weights that simulate the unknown test distribution and improve output confidence.

5 Experiments

Experiments evaluate SADE on vanilla and test-agnostic long-tailed recognition across several benchmark datasets and unknown test distributions. Results show that skill-diverse experts and self-supervised aggregation improve performance across these settings.

  • Experimental Setups: SADE is evaluated on ImageNet-LT, CIFAR100-LT, Places-LT, and iNaturalist 2018 using uniform, forward long-tailed, and backward long-tailed test distributions.The evaluation uses micro accuracy for multiple test sets with different class distributions.
  • Vanilla Long-tailed Recognition: SADE performs best on vanilla long-tailed recognition across the evaluated datasets, with more than 2% accuracy gain on ImageNet-LT over RIDE and ACE.Compared methods include two-stage, logit-adjusted, ensemble, classifier-design, and representation-learning approaches.
  • Skill-diverse Expert Learning: The three learned experts specialize in many-shot classes, uniform distributions, and few-shot classes, respectively, producing a non-trivial ensemble gain over RIDE.RIDE independently trains experts with cross-entropy and uses KL-Divergence, but prediction-divergence maximization does not produce visibly diverse experts.
  • Test-time Self-supervised Aggregation: Self-supervised aggregation learns higher weights for the forward expert on forward long-tailed distributions and for the backward expert on backward long-tailed distributions.The learned weights adapt to unknown test class distributions and improve ensemble performance.
  • Test-agnostic Recognition: SADE remains effective on partial test distributions and outperforms directly applied test-time training methods under class-distribution shifts, especially inverse long-tailed shifts.Partial distributions contain only many-shot, medium-shot, or few-shot classes; compared test-time methods include pseudo-labeling, BBSE estimation, and Tent.

6 Conclusion

The paper studies test-agnostic long-tailed recognition, where test class distributions are unknown and need not be uniform. It introduces SADE and reports state-of-the-art performance on both vanilla and test-agnostic recognition.

  • Conclusion: Test-agnostic long-tailed recognition has long-tailed training data but an unknown, potentially non-uniform test class distribution.The task addresses both class imbalance and unknown distribution shift between training and test data.
  • Conclusion: SADE combines skill-diverse expert learning with test-time self-supervised aggregation.The paper theoretically analyzes the method and empirically evaluates both strategies.
  • Conclusion: SADE achieves new state-of-the-art performance on vanilla and test-agnostic long-tailed recognition.

Checklist

The checklist reports that the paper addresses contributions, limitations, assumptions, reproducibility, datasets, and supplementary materials. It also states that the work is fundamental research without particular negative societal impacts.

  • Contributions and Scope: The authors report that they described the work’s contributions, scope, and potential limitations.Potential limitations are discussed in Appendix H.
  • Ethics: The paper reports no particular negative societal impacts because it is fundamental research.
  • Theoretical Results: The paper reports stating assumptions for theoretical results and providing complete proofs in Appendix A.
  • Reproducibility: The authors report providing code, data, training details, and experimental settings needed to reproduce the main results.The materials are supplied in the supplementary material or through a URL.
  • Assets and Data: The experiments use open-source benchmark datasets, and the authors report citing existing assets and addressing their licenses where applicable.The datasets are publicly available and identified as open-source benchmark datasets.
  • Supplementary Materials: Supplementary materials contain proofs, pseudocode, experimental details, additional results, ablations, model-complexity discussion, and limitations.

A Proofs for Theorem 1

The appendix proves a theorem connecting the self-supervised aggregation objective to conditional prediction entropy under stated distributional assumptions. It also documents the algorithms, evaluation distributions, and dataset settings used in the analysis.

  • Theorem 1 Proof: The proof defines prediction and label random variables, the number of classes, test samples by class, and class hard means.
  • Theorem 1 Proof: The self-supervised aggregation objective maximizes prediction stability across augmented views of unlabeled test samples.The first view can be simplified to the original data, and sufficiently strong augmentations are assumed to simulate representative same-class views.
  • Theorem 1 Proof: The analysis relates the objective to prediction concentration around class hard means and interprets it through conditional cross-entropy and conditional entropy.The proof uses a Gaussian conditional distribution centered around the class hard mean and derives an upper bound on conditional entropy.
  • Theorem 1 Proof: Under the stated Gaussian condition, the bound is tight, so minimizing the objective is equivalent to minimizing conditional entropy.The proof concludes Theorem 1 after combining the entropy and mutual-information relations.
  • Algorithms: SADE’s training algorithm uses three experts optimized with cross-entropy, a balancing loss, and an inverse softmax loss.The resulting model contains a shared backbone and three expert networks.
  • Algorithms: At test time, expert weights are initialized uniformly, updated using augmented unlabeled test views, and retained above a 0.05 stopping threshold.For vanilla recognition, predictions are obtained by averaging expert logits before softmax; test-agnostic recognition uses adaptive aggregation.
  • Experimental Settings: Experiments use four long-tailed benchmark datasets and construct uniform, forward, and backward test distributions with specified imbalance-ratio ranges.Forward and backward distributions reverse the class-order skew, with iNaturalist 2018 using a smaller ratio range because each class has only three test samples.

C.3 More Implementation Details of Our Method

The method uses shared-backbone multi-expert architectures and evaluates performance across uniform, long-tailed, and unknown test distributions. Its experts are trained for distinct distributional skills, with empirical results reported across datasets, architectures, and evaluation settings.

  • Implementation: Experiments use ResNeXt-50, ResNet-32, ResNet-152, and ResNet-50 backbones for ImageNet-LT, CIFAR100-LT, Places-LT, and iNaturalist 2018, respectively.The implementation uses cosine classifiers and reports multi-GPU training configurations in the supplementary statistics.
  • Architecture: The three-expert ResNet shares its first two stages, gives later stages and classifiers to individual experts, and averages their logits before softmax.Reducing each expert’s filters by one quarter lowers computational complexity relative to fully independent experts.
  • Test distributions: SADE consistently outperforms baselines across benchmark test distributions, especially backward long-tailed distributions with relatively large imbalance ratios.The comparison includes uniform, forward, and backward distributions, while supplementary tables cover additional dataset settings.
  • Expert learning: SADE trains forward, uniform, and backward experts with cross-entropy, balanced softmax, and inverse softmax objectives, respectively.These objectives target the training, uniform, and inversely long-tailed class distributions.

D.4 More Results on Test-time Self-supervised Aggregation

The self-supervised aggregation strategy learns expert weights from unlabeled test data without knowing the true test distribution. It adapts weights to forward, uniform, and backward shifts, with larger gains under stronger imbalance but smaller improvements on iNaturalist 2018.

  • Weight adaptation: SADE learns suitable aggregation weights for test-agnostic class distributions without relying on the true test class distribution.This behavior is reported in Table 19 across ImageNet-LT, CIFAR100-LT, Places-LT, and iNaturalist 2018.
  • Weight adaptation: On uniform ImageNet-LT test data, the learned expert weights are 0.33, 0.33, and 0.34 for the three experts.Forward distributions receive higher weight for the forward expert, while backward distributions receive higher weight for the backward expert.
  • Interpretation: The learned weights empirically support the theoretical claim that self-supervised aggregation can simulate unknown test class distributions.The authors connect adaptive weighting to handling unknown class-distribution shifts at test time.
  • Performance: 7.7% and 9.2% gains on CIFAR100-LT Forward-LT-50 and Backward-LT-50 show larger improvements for imbalanced test data.The results are attributed to appropriate aggregation and improved performance on dominant test classes.
  • Scope boundary: On iNaturalist 2018, relatively low test imbalance produces relatively balanced simulated distributions and less significant performance improvement.The authors project stronger gains if future applications contain more highly imbalanced iNaturalist test samples.

E Ablation Studies on Skill-diverse Expert Learning

The ablations examine expert count, inverse-softmax weighting, and test-time optimization hyperparameters. They support three experts, show robustness to training duration and batch size, and indicate that learning rate affects convergence and weight sharpness.

  • Number of experts: SADE uses forward and backward experts to span possible test distributions and a uniform expert to retain accuracy on uniform tests.The framework can be extended beyond three experts by adjusting the diversity hyperparameter for added experts.
  • Inverse-softmax loss: Increasing λ makes the backward expert simulate a more inversely long-tailed distribution, improving few-shot-class ensemble performance.λ values of 2 or 3 provide a better head–tail trade-off in the reported ImageNet-LT ablation.
  • Epoch number: More than 5 test-time training epochs yield converged expert weights on ImageNet-LT, supporting robustness to longer optimization.The reported epoch ablation evaluates training durations from 1 to 100 epochs.
  • Batch size: Learned expert weights remain nearly unchanged across batch sizes from 64 to 256, indicating batch-size insensitivity.The corresponding performance is reported separately for various test class distributions.
  • Learning rate: A learning rate of 0.001 prevents convergence within 5 epochs, whereas increasing it produces sharper weights that better fit unknown test distributions.The ablation varies learning rates from 0.001 to 0.5 and reports corresponding performance changes.

F.4 Results of Prediction Confidence

The appendix evaluates test-time self-supervised aggregation through prediction confidence, runtime, streaming-data performance, and model complexity. The strategy improves confidence and performs well online, while achieving stronger performance than RIDE at the same parameter and MAC costs, though multi-expert models remain more complex than the backbone.

  • Prediction Confidence: 0.015 confidence improvement on Forward-LT-50 demonstrates higher prediction confidence over more than 10,000 test samples.The improvement is measured by the hard mean of the highest prediction probability; larger class-imbalance ratios produce more apparent gains.
  • Runtime Cost: 0.009 seconds per sample is the average additional test-time cost of self-supervised aggregation on V100 GPUs.The reported runtime evaluation includes Forward-LT-50 and Forward-LT-25 test distributions.
  • Streaming Test Data: The test-time strategy performs well on streaming ImageNet-LT data, although the supplied passages provide no streaming accuracy value.The strategy can operate online without access to all test data in advance.
  • Model Complexity: SADE and RIDE use the same multi-expert parameter and MAC budgets, but SADE performs much better on test-agnostic long-tailed recognition.Both ensemble methods use about 1.5x the backbone parameters and 1.4x its MACs when the efficient expert-assignment trick is not used.
  • Model Complexity: The multi-expert scheme increases model parameters and computational costs relative to the original backbone.The paper treats this increment as acceptable for the test-agnostic task and leaves further acceleration to future work.

H Potential Limitations

The paper evaluates test-agnostic long-tailed recognition only for long-tailed classification, while extensions to detection, segmentation, more efficient models, and more complex test distributions remain future work.

  • Scope and Future Work: The study focuses on long-tailed classification, leaving extension to object detection and instance segmentation for future work.The authors also identify further acceleration and evaluation on more complex test class distributions as future directions.
Loading 2107.09249v4…