PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Long-Tailed Learning: A Survey

Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, Jiashi Feng

arXiv:2110.04596v2cs.CV

TL;DR

Long-tailed class imbalance biases deep recognition models toward dominant classes and limits performance on tail classes. The paper surveys pre-mid-2021 methods using a taxonomy of three categories, empirically evaluates state-of-the-art approaches with relative accuracy, and discusses applications and future research. Its analysis reports positive progress in addressing imbalance, while finding that most methods improve tail performance at the cost of lower head-class performance.

  • Problem

    Long-tailed class imbalance makes deep recognition models biased toward dominant classes and poor on tail classes, while the field lacks a systematic review of its rapid progress.

  • Method

    The survey reviews methods under class re-balancing, information augmentation, and module improvement, then evaluates state-of-the-art approaches using relative accuracy.

  • Results

    Existing research makes positive progress in addressing class imbalance; ensemble-based methods are current state-of-the-art, while most methods trade lower head-class performance for improved tail-class performance.

  • Takeaways & Limitations

    The survey provides a consolidated understanding of deep long-tailed learning and identifies applications, method directions, and task settings for future research.

  • Takeaways & Limitations

    Existing methods tend to fail in fine-grained real-world applications because they focus on class imbalance without addressing the required finer discrimination abilities.

Abstract

from arXiv · show

Deep long-tailed learning, one of the most challenging problems in visual recognition, aims to train well-performing deep models from a large number of images that follow a long-tailed class distribution. In the last decade, deep learning has emerged as a powerful recognition model for learning high-quality image representations and has led to remarkable breakthroughs in generic visual recognition. However, long-tailed class imbalance, a common problem in practical visual recognition tasks, often limits the practicality of deep network based recognition models in real-world applications, since they can be easily biased towards dominant classes and perform poorly on tail classes. To address this problem, a large number of studies have been conducted in recent years, making promising progress in the field of deep long-tailed learning. Considering the rapid evolution of this field, this paper aims to provide a comprehensive survey on recent advances in deep long-tailed learning. To be specific, we group existing deep long-tailed learning studies into three main categories (i.e., class re-balancing, information augmentation and module improvement), and review these methods following this taxonomy in detail. Afterward, we empirically analyze several state-of-the-art methods by evaluating to what extent they address the issue of class imbalance via a newly proposed evaluation metric, i.e., relative accuracy. We conclude the survey by highlighting important applications of deep long-tailed learning and identifying several promising directions for future research.

1 INTRODUCTION

Long-tailed class distributions make deep recognition models biased toward dominant classes and weak on tail classes. This survey organizes recent methods, evaluates their handling of imbalance, and identifies applications and future directions.

  • Long-tailed datasets contain many samples for a small portion of classes and few samples for the remainder, making deep recognition training challenging.
  • Deep models trained with empirical risk minimization can favor head classes and perform poorly on tail classes with limited data.
  • The survey addresses the lack of a systematic review by comprehensively examining long-tailed learning studies conducted before mid-2021.
  • Existing methods are grouped into class re-balancing, information augmentation, and module improvement, with nine more specific sub-categories.
  • The survey evaluates state-of-the-art methods using relative accuracy to assess how they handle class imbalance.
  • It also discusses real-world applications and identifies potential directions and task settings for future research.

2 PROBLEM OVERVIEW

Deep long-tailed learning trains recognition models when class frequencies are highly uneven, creating head-class bias and limited tail-class performance. The survey defines the problem, reviews datasets, metrics, applications, and related tasks.

  • Problem Definition: Long-tailed learning involves a small fraction of classes with many samples and many classes with few samples.
  • Problem Definition: The task is difficult because class imbalance biases models toward head classes, while scarce tail samples hinder tail-class classification.
  • Datasets and Applications: Benchmark coverage spans image classification, detection, segmentation, multi-label classification, video recognition, and visual relation learning.
  • Evaluation: Long-tailed learning is evaluated across all classes and head, middle, and tail subsets using metrics that treat classes equally.
  • Applications: Real-world applications remain challenging because fine-grained class differences, domain shifts, or relational reasoning accompany class imbalance.
  • Related Tasks: Long-tailed learning is related to class-imbalanced, few-shot, and out-of-domain generalization tasks but differs in distributional structure and typical scale.

3 CLASSIC METHODS

The survey organizes deep long-tailed learning methods by technical contribution into class re-balancing, information augmentation, and module improvement.

  • The taxonomy contains three main categories: class re-balancing, information augmentation, and module improvement.
  • Class Re-balancing: Class re-balancing includes re-sampling, class-sensitive learning, and logit adjustment.
  • Information Augmentation: Information augmentation comprises transfer learning and data augmentation.
  • Module Improvement: Module improvement covers representation learning, classifier design, decoupled training, and ensemble learning.

3.1 Class Re-balancing

Class re-balancing addresses long-tailed learning by modifying sampling, losses, or logits to reduce imbalance effects. It is practical and competitive, but often trades head-class performance for tail-class gains.

  • Class re-balancing targets the negative influence of unequal training sample numbers through re-sampling, class-sensitive learning, and logit adjustment.
  • Re-sampling: Random sampling overrepresents head classes in mini-batches, biasing models toward head classes and weakening tail-class performance.
  • Re-sampling: Random over-sampling can overfit duplicated tail data, whereas random undersampling can discard valuable majority-class information.
  • Re-sampling: Decoupling found square-root and progressively-balanced sampling better for standard training, but both require class-frequency information in advance.
  • Re-sampling: Adaptive and meta-learning approaches use training statistics or balanced validation performance to adjust class sampling rates.
  • Re-sampling: Bi-level class-balanced sampling addresses tasks with simultaneous image-level and instance-level imbalance.
  • Class-sensitive Learning: Class-sensitive learning modifies losses to counter uneven gradients, including frequency-based weights, logit adjustments, hardness-based weights, and gradient suppression controls.
  • Discussion: Class re-balancing is relatively simple, can achieve comparable or better performance, and includes methods with theoretical motivation or guarantees.

3.2 Information Augmentation

Information augmentation introduces additional knowledge through transfer learning and data augmentation to address long-tailed class imbalance. The survey describes complementary schemes spanning pre-training, distillation, model transfer, self-training, and sample- or feature-level augmentation.

  • Overview: Information augmentation introduces additional knowledge through transfer learning and data augmentation for long-tailed learning.The two method types are transfer learning and data augmentation.
  • Transfer Learning: Transfer learning uses source-domain knowledge through model pre-training, knowledge distillation, head-to-tail model transfer, or self-training.These are identified as the four main transfer schemes in long-tailed learning.
  • Transfer Learning: Model pre-training can first learn representations from long-tailed data or self-supervised objectives before standard or class-balanced training.DSTL fine-tunes on a more class-balanced subset, while SSP precedes standard long-tailed training with self-supervised learning.
  • Transfer Learning: Self-training uses labeled data to generate pseudo-labels for unlabeled data, but long-tailed labeled and unlabeled distributions can bias pseudo-labels toward head classes.DARS aligns pseudo-label frequencies with labeled-data frequencies, while CReST selects more tail-class samples for online pseudo-labeling.
  • Data Augmentation: Data augmentation addresses imbalance at sample or feature levels through head-to-tail transfer and non-transfer methods.Examples include M2m translation, feature-variance transfer, feature-space generation, mixup, and class-wise feature sampling.
  • Summary: Information augmentation is complementary to class re-balancing and module improvement, but class-agnostic augmentation can favor head classes and remains an open challenge.The survey states that carefully designed methods can improve tail-class performance without sacrificing head-class performance.

3.3 Module Improvement

Module improvement addresses long-tailed learning through representation learning, classifier design, decoupled training, and ensemble learning. The surveyed methods modify features, classifiers, training stages, or entire architectures, while combination and practicality depend on the design.

  • Overview: Module improvement comprises representation learning, classifier design, decoupled training, and ensemble learning.These categories target the feature extractor, classifier, both through separate stages, or the whole architecture.
  • Representation Learning: Representation learning uses metric learning, prototype learning, and sequential training to improve features for long-tailed data.Examples include distance-based losses, class-specific prototypes, hierarchical clustering, and differentiated head- and tail-class training.
  • Representation Learning: Sequential training treats head and tail subsets differently, first learning noise-resistant head-class features and then enhancing tail-class inter-class discrimination.Unequal-training uses hard identity mining and a center-dispersed loss in its tail-class stage.
  • Classifier Design: Long-tailed imbalance can produce larger classifier weight norms for head classes, biasing linear classifiers toward dominant classes.This motivates cosine, τ-normalized, nearest-class-mean, hierarchical, causal, and knowledge-transfer classifier designs.
  • Classifier Design: Causal classifiers estimate long-tailed bias with multi-head weights and features, then subtract the bad causal effect from prediction logits during inference.The method records bias using exponential moving average features and uses α to control direct and indirect effects.
  • Decoupled Training: Decoupled classifier learning can yield significant performance gains with limited computation cost, but accumulated training stages reduce practicality for integration.The survey presents this trade-off as a critique of decoupled training.

4 EMPIRICAL STUDIES

The survey evaluates long-tailed methods using accuracy, upper reference accuracy, and relative accuracy, finding overall progress but persistent trade-offs across class subsets. Ensemble-based methods, particularly SADE, achieve the strongest overall results, while relative accuracy separates imbalance handling from gains due to other factors.

  • Evaluation metrics: Relative accuracy complements accuracy by reducing the influence of factors beyond class imbalance, such as data augmentation and network architecture.MiSLAS has higher accuracy but lower relative accuracy than Balanced Softmax under 90 epochs because its upper reference accuracy is also higher.
  • Overall trends: 93.0 is the reported relative accuracy of SADE, indicating positive progress while leaving room for improvement.
  • Training duration: Most methods perform better after 200 rather than 90 training epochs, although De-confound and Decouple-CB-CRT perform better with shorter training.The authors speculate that additional epochs can increase training difficulty and interact with learning-rate decay for these exceptions.
  • Method types: Class re-balancing, information augmentation, and module improvement all produce gains, with ensemble-based methods leading in both accuracy and relative accuracy.LADE, Balanced Softmax, and LDAM are highlighted among class-rebalancing methods; SSP and RSG help with information augmentation, while SADE and RIDE lead module improvement.
  • Class subsets: Most methods improve tail- and middle-class performance at the cost of lower head-class performance, making balanced improvement across all classes a central challenge.Information augmentation and ensemble learning are identified as potential ways to improve long-tailed performance without sacrificing head-class accuracy.
  • Overall trends: SADE achieves the best overall accuracy and relative accuracy, although RIDE is stronger on head classes and BBN on tail classes.The survey attributes SADE’s higher average performance to a better trade-off among classes and identifies ensemble learning with class re-balancing as current best practice.

5 FUTURE DIRECTIONS

Future research should extend deep long-tailed learning beyond standard image classification settings to address changing test distributions, open classes, decentralized and continual data, domain shifts, noise, and continuous labels.

  • Test-agnostic long-tailed learning: Test-agnostic long-tailed learning relaxes the balanced-test-distribution assumption by allowing arbitrarily skewed test class distributions.Existing methods may fail when practical test distributions are long-tailed or inversely long-tailed.
  • Open-set long-tailed learning: Open-set long-tailed learning combines long-tailed training data with balanced evaluation over head, tail, and open classes.Its challenges include sharing visual knowledge between head and tail classes and reducing confusion between tail and open classes.
  • Federated long-tailed learning: Federated long-tailed learning addresses decentralized training when long-tailed samples are distributed across mobile devices or the Internet of Things.It must handle both long-tail imbalance and unknown class-distribution shifts among local datasets.
  • Class-incremental long-tailed learning: Class-incremental long-tailed learning studies sequentially arriving classes without future class or frequency information while preserving previous knowledge.The setting combines evolving imbalance with catastrophic forgetting.
  • Other directions: Multi-domain, robust, and regression settings extend the problem to domain shifts, noisy data, and continuous label spaces.These settings require addressing imbalance alongside distribution variation, image or label noise, or the absence of discrete class boundaries.
  • Long-tailed video learning: Long-tailed video learning adds temporal structure to the imbalance problem and is enabled for further study by the VideoLT dataset.The survey identifies it as more difficult than image-level long-tailed learning because of the additional temporal dimension.

6 CONCLUSION

The survey reviews deep long-tailed learning methods through a three-part taxonomy, evaluates their handling of imbalance with relative accuracy, and discusses applications and future directions.

  • The survey reviews methods proposed before mid-2021 using the taxonomy of class re-balancing, information augmentation, and module improvement.
  • It empirically analyzes state-of-the-art methods according to how effectively they address class imbalance using relative accuracy.
  • The survey discusses application scenarios and identifies potential directions for method innovation and future task settings.
Loading 2110.04596v2…