PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Long-Tailed Visual Recognition

Lu Yang, He Jiang, Qing Song, Jun Guo

arXiv:2205.13775v1cs.CV

TL;DR

Long-tailed data causes head classes to dominate model performance while tail classes remain under-learned. This survey organizes existing approaches, evaluates imbalance metrics and datasets, and finds the phenomenon widespread and insufficiently studied.

  • Problem

    Tail classes are often severely under-represented, making their decision boundaries and effective feature extractors difficult to learn while head-class gradients dominate.

  • Method

    The survey categorizes long-tailed visual recognition studies into ten representation-learning categories, compares four imbalance metrics, and analyzes benchmarks across 20 widely used visual datasets.

  • Results

    The survey finds long-tailed distributions widespread across visual datasets and reports that no single method greatly solves the problem, with combined strategies often used in practice.

  • Takeaways & Limitations

    The survey supports using the Gini coefficient to evaluate dataset long-tailedness and highlights the need for continued research across methods and application areas.

  • Takeaways & Limitations

    Long-tailed instance-level recognition, especially instance segmentation using mask information, remains less studied than image-level recognition.

Abstract

from arXiv · show

The heavy reliance on data is one of the major reasons that currently limit the development of deep learning. Data quality directly dominates the effect of deep learning models, and the long-tailed distribution is one of the factors affecting data quality. The long-tailed phenomenon is prevalent due to the prevalence of power law in nature. In this case, the performance of deep learning models is often dominated by the head classes while the learning of the tail classes is severely underdeveloped. In order to learn adequately for all classes, many researchers have studied and preliminarily addressed the long-tailed problem. In this survey, we focus on the problems caused by long-tailed data distribution, sort out the representative long-tailed visual recognition datasets and summarize some mainstream long-tailed studies. Specifically, we summarize these studies into ten categories from the perspective of representation learning, and outline the highlights and limitations of each category. Besides, we have studied four quantitative metrics for evaluating the imbalance, and suggest using the Gini coefficient to evaluate the long-tailedness of a dataset. Based on the Gini coefficient, we quantitatively study 20 widely-used and large-scale visual datasets proposed in the last decade, and find that the long-tailed phenomenon is widespread and has not been fully studied. Finally, we provide several future directions for the development of long-tailed learning to provide more ideas for readers.

1 Introduction

Long-tailed data challenges deep learning because head classes dominate naturally distributed datasets while tail classes receive little data. This survey organizes existing methods, evaluates long-tailedness, and identifies widespread but insufficiently studied imbalance.

  • Motivation: Long-tailed datasets contain a few data-rich head classes and many data-poor tail classes, making equal attention to tail classes important.The phenomenon is associated with power-law behavior in nature and appears across domains.
  • Motivation: Deep learning depends heavily on large, well-annotated datasets, but artificially balanced benchmarks do not reflect natural data distributions.Collecting enough tail examples to preserve balance is difficult, motivating long-tailed learning.
  • Survey scope: The survey comprehensively reviews mainstream long-tailed methods, grouping them into ten representation-learning categories and outlining each category’s highlights and limitations.It also provides background, datasets, benchmarks, and method summaries across the paper’s sections.
  • Survey scope: The authors compare four imbalance metrics and recommend the Gini coefficient for evaluating dataset long-tailedness.The recommendation follows a comparative analysis of the metrics’ characteristics.
  • Empirical scope: A structured study of 20 widely used large-scale visual datasets finds long-tailed distributions widespread and insufficiently studied in some fields.The survey frames this analysis as extending beyond the existing research scope.
  • Future directions: The survey concludes by identifying open problems and opportunities for future long-tailed visual-recognition research.These directions are intended to facilitate further research.

2 Overview

Long-tailed visual recognition concerns learning when class frequencies follow a highly skewed distribution. The survey examines its learning challenges, relationships to imbalance and few-shot learning, and effects across recognition, detection, and segmentation.

  • Problem definition: Long-tailed distributions have a prominent head of frequent classes and an extended tail of classes with few samples.The phenomenon is described as a colloquial expression of power-law and Pareto characteristics.
  • Problem definition: Insufficient tail data makes decision boundaries difficult to determine, while head-class gradients can overwhelm tail-class learning.These effects hinder effective feature extraction and classifier performance.
  • Problem definition: Long-tailed training sets usually differ from more balanced test sets, creating a source-target distribution gap that harms tail-class performance under conventional methods.The passage names cross-entropy and simple fine-tuning as examples of conventional methods.
  • Task challenges: Object detection adds competition between head and tail box categories, causing tail samples to be lost or classified as background during box sampling.Long-tailed distributions can also contribute to missed detections during NMS.
  • Task challenges: Instance segmentation inherits detection limitations, while sparse tail samples make tail objects difficult to distinguish from background and produce inaccurate masks.The survey specifically discusses two-stage methods such as Mask R-CNN.
  • Related fields: Long-tailed recognition overlaps with imbalance learning and few-shot learning but combines large class counts, imbalanced data, and few-shot tail classes.The survey compares these areas using training data, test imbalance, tail sample counts, class numbers, and evaluation range.

3 Long-tailed Datasets and Metrics

This section surveys representative long-tailed visual datasets and compares four metrics for quantifying distribution imbalance. It recommends the Gini coefficient because it is bounded, less sensitive to extremes and absolute sample counts, and supports dataset comparisons.

  • Representative Datasets: Representative datasets span classification, detection, segmentation, and face recognition, including CIFAR-10/100-LT, ImageNet-LT, Places-LT, iNaturalist, LVIS, and MS1M-LT.MS1M-LT contains 887,530 images across 74,532 identities; iNaturalist 2017 and 2018 contain 5,089 and 8,142 classes, respectively.
  • Quantitative Metrics: Accurate measurement of long-tailedness is presented as a prerequisite for addressing long-tailed visual recognition.The review compares imbalance factor, standard deviation, mean/median, and Gini coefficient.
  • Quantitative Metrics: The imbalance factor β uses the largest-to-smallest class size ratio but is easily affected by extreme classes and cannot reflect overall dataset characteristics.β = max {n1, n2, ..., nk} /min {n1, n2, ..., nk}.
  • Quantitative Metrics: Standard deviation is affected by absolute sample counts and can misidentify long-tailedness, with balanced COCO at σ=29,360 and long-tailed MS1M-LT at σ=18.These values illustrate why dispersion alone does not objectively express long-tailedness.
  • Quantitative Metrics: The mean-to-median ratio γ distinguishes uniform from imbalanced distributions but is affected by individual cases and has an open-ended range.Values near 1 indicate uniformity, whereas values significantly above 1 indicate imbalance.
  • Gini Coefficient: The review recommends Gini coefficient δ because it is bounded in (0,1), less affected by extreme samples and absolute data volume, and makes datasets more comparable.CIFAR has δ=0, ImageNet-1K has δ=0.013, while long-tailed datasets generally exceed 0.5 and can reach 0.8.
  • Long-tailedness Analysis: COCO has δ=0.564, whereas LVIS v0.5 and v1.0 have δ=0.825 and δ=0.820, motivating task-specific standards for long-tailedness.The review notes that instance-level annotation makes manual distribution control more difficult than image-level annotation.
  • Long-tailedness Analysis: Using the Gini coefficient, the survey analyzes 20 mainstream large-scale visual datasets to characterize research status and future directions.The resulting quantitative analysis is intended to guide the selection of long-tailed learning solutions.

4 Long-tailed Visual Recognition

This section reviews deep-learning methods for long-tailed visual recognition from 2016 onward. The studies are organized into ten representation-learning categories, with each category’s highlights and limitations outlined.

  • Research on long-tailed visual recognition has grown increasingly abundant since 2016.
  • The survey mainly reviews each study’s core method despite many works combining multiple techniques.
  • The reviewed methods are finely summarized into ten categories from the perspective of representation learning.
  • Color scatter diagrams illustrate category principles, class identities, instance counts, unlabeled data, and repeated sampling.

4.1 Data Processing

Data processing methods address long-tailed imbalance through over-sampling, under-sampling, and data augmentation. These approaches rebalance class exposure or enrich tail-class data, but sampling can overfit tails or discard useful head-class information.

  • Data processing uses over-sampling, under-sampling, and data augmentation to make classes more balanced from the data perspective.
  • Over-sampling: Over-sampling increases tail-class exposure through class-aware, replication-factor, hybrid, and instance-level sampling strategies.
  • Under-sampling: Under-sampling reduces head-class exposure through random deletion, ensembles, sequential removal, and data-cleaning procedures.
  • Limitations: Over-sampling may overfit noisy tail classes, whereas under-sampling may underlearn head classes and miss valuable data.
  • Data augmentation: Data augmentation compensates for limited tail-class samples by generating or synthesizing new data from similar samples, other sources, or feature representations.

4.2 Cost Sensitive Weighting

Cost-sensitive weighting addresses imbalance by increasing the influence of tail classes or hard samples through class-level and instance-level weights. The survey describes adaptive weighting methods while noting difficulties in choosing stable weights and optimizing large-scale models.

  • Cost-sensitive weighting assigns different weights to classes or samples to increase the influence of tail examples.
  • Class-level re-weighting: Class-level methods include inverse-frequency weighting, smoothed weights, effective-number weighting, difficulty-balanced losses, and gradient-based losses.
  • Class-level re-weighting: Equalization Loss ignores frequent-class gradients on rare classes while preserving background gradients, and later methods adaptively rebalance gradient contributions.
  • Instance-level re-weighting: Instance-level re-weighting targets hard samples because extreme imbalance can cause tail classes to receive fewer learning iterations.
  • Limitations: Simple frequency-based weights can be difficult to set, leave head classes dominant, hinder optimization, and require dataset-sensitive hyper-parameters.

4.3 Decoupling Methods

Decoupling methods separate representation learning from classifier learning because rebalancing can damage learned representations. They commonly learn features with ordinary sampling and then rebalance or calibrate the classifier, but this sacrifices end-to-end training and retains second-stage limitations.

  • Decoupling separates representation learning from classifier learning to avoid harming representations through rebalancing.
  • The first stage learns representations with standard sampling, while the second freezes the representation learner and trains a balanced classifier.
  • Methods such as BAGS group classes for balanced softmax, while BBN combines conventional and rebalancing branches through an adaptive trade-off.
  • SimCal, DisAlign, MiSLAS, and LADC calibrate classifier bias, class priors, confidence, stage shifts, or tail distributions.
  • Decoupling has become a mainstream direction and can combine with data processing or cost-sensitive weighting for better learning effects.
  • Limitations: Two-stage learning departs from end-to-end training, and its second-stage resampling or re-weighting retains earlier limitations.

4.4 Other Long-tailed Visual Recognition Methods

Other long-tailed methods draw on metric, transfer, meta-learning, and related strategies to improve representation and classification across imbalanced classes. These approaches target class boundaries, knowledge transfer, adaptive training, and feature learning, while retaining distinct limitations.

  • Metric Learning: Metric learning separates classes by reducing distances among similar samples and increasing distances among dissimilar samples.It can clarify decision boundaries for tail classes, but requires suitable sample combinations and loss functions.
  • Metric Learning: Margin-based methods regularize minority classes more strongly, giving them larger class-dependent margins based on label frequencies.LDAM uses class sample counts to design a label-distribution-aware loss.
  • Other Methods: The survey also covers contrastive learning, classifier calibration, and hybrid feature-classifier training as additional long-tailed strategies.PaCo introduces learnable class-wise centers to counter high-frequency-class bias in supervised contrastive learning.
  • Transfer Learning: Transfer learning uses richer head-class training resources to guide under-represented tail classes without damaging head-class performance.Some approaches transfer model parameters or extend tail feature distributions, but complex module designs can hinder knowledge transfer.
  • Meta Learning: Meta-learning adapts training through balanced meta-data, meta-models, adaptive parameters, or sample weighting.L2RW uses a small unbiased validation set to update training-loss weights online.

4.4.5 Knowledge Distilling

Knowledge distillation, grouping, semi-supervised learning, causal inference, adversarial training, and robust optimization extend long-tailed recognition beyond standard loss and representation methods. Each approach offers a mechanism for balancing or enriching tail-class learning but introduces practical constraints.

  • Knowledge Distilling: Knowledge distillation uses teacher predictions to balance head- and tail-class learning in a lower-complexity student model.LFME trains experts on balanced subsets, DiVE adds virtual examples, and BKD combines instance-balanced classification with class-balanced distillation.
  • Knowledge Distilling: Distillation methods depend on reliable expert models and require appropriate strategies for transferring knowledge or adding virtual data.The survey identifies expert selection and distillation design as continuing challenges.
  • Grouping: Grouping divides data by instance counts or semantic, visual, and geometric relations so models can learn separately for each group.Instance-based grouping may block inter-group knowledge exchange, while semantic grouping requires additional knowledge.
  • Semi-supervised Learning: Semi-supervised learning expands tail classes with pseudo-labels or weakly labeled external data.It compensates for insufficient tail representation learning but requires additional training and accessible unlabeled data.
  • Other Methods: Causal inference, adversarial training, and distributionally robust optimization address confounding, robustness, and representation quality in long-tailed settings.RoBal targets adversarial boundary errors, while DRO-LT encourages high-quality representations for both head and tail classes.

4.5 Summary

The survey finds that long-tailed methods have distinct emphases but no single method solves the problem broadly. Practical systems therefore combine data processing, reweighting, representation enhancement, and decoupling strategies.

  • Summary: No method can greatly solve the long-tailed problem across settings.The surveyed approaches differ in their focus on data processing, loss functions, architectures, and training methods.
  • Summary: Practical applications often combine semi-supervised expansion, tail-class oversampling, class-level reweighting, and decoupling.These combinations respectively expand tail data, increase sampling frequency, balance training gradients, and enhance representation ability.
  • Summary: Long-tailed recognition remains an active research area with substantial scope for future development.The survey predicts continued growth in methods for solving the problem.

5 Performance Comparison

The survey compares representative long-tailed methods across classification, detection, and instance-segmentation benchmarks. Results show strong progress on some datasets, saturation on CIFAR-LT, smaller gains on Places-LT, and continuing gaps in instance-level recognition.

  • CIFAR-LT: 2021 MARC reaches 85.3% versus 84.9% for 2020 BALMS at β = 100 on CIFAR-10-LT, a 0.4-point gain.MARC does not improve the other three reported metrics, indicating CIFAR-LT saturation.
  • ImageNet-LT and Places-LT: 58.2% overall top-1 accuracy makes 2021 PaCo 6.1 points higher than 2020 MBJ on ImageNet-LT.The comparison remains notable despite PaCo using RandAugment and longer training.
  • ImageNet-LT and Places-LT: 41.2% overall top-1 accuracy makes 2021 PaCo 2.5 points higher than 2020 BALMS on Places-LT.The survey attributes the relatively small improvement to the need for scene-centric-specific solutions.
  • iNaturalist: 73.2% overall top-1 accuracy makes 2021 PaCo 2.8 points higher than the best 2020 iNaturalist 2018 method.The survey considers iNaturalist 2018 unsaturated and important for domain-specific long-tailed research.
  • LVIS: EOD, Seesaw Loss, LDA, and EQL v2 lead LVIS v1.0 with small performance gaps, mainly using class-level reweighting and tail-class resampling.The survey identifies class-level reweighting as the mainstream solution for long-tailed detection and instance segmentation.
  • LVIS: Most detection studies extend image-level methods, leaving instance-level long-tailed recognition comparatively underexplored.Long-tailed instance segmentation especially makes limited use of mask information.

6 Analysis of Long-tailed Phenomenon

The survey finds long-tailed distributions widespread across visual datasets, with increasing imbalance and substantially poorer performance on body and tail classes, especially in detection and segmentation.

  • The survey analyzes 20 widely used large-scale datasets across classification, detection, segmentation, re-identification, face, parsing, and video recognition.
  • Long-tailed datasets have increased since 2016 as researchers have reduced artificial control over dataset balance.
  • Object detection and segmentation datasets show more severe long-tailed distributions than classification datasets.
  • The Gini coefficient generally increases in newer datasets, indicating increasingly severe long-tailedness.
  • 6.2 Analysis of Performance: On Object365, head-class APbox exceeds body-class performance by 1.43×–1.48× and tail-class performance by up to 8.40× across Faster R-CNN, RetinaNet, and FCOS.
  • 6.2 Analysis of Performance: On ADE20K, head-class mIoU exceeds body classes by 1.47×–1.62× and tail classes by 1.87×–2.39× across three segmentation models.
  • 6.2 Analysis of Performance: Performance declines as class instance counts decrease, while long-tailed recognition remains insufficiently studied beyond benchmark datasets.

7 Future Directions

The survey identifies future directions that extend long-tailed learning beyond current methods and benchmarks, including large models, self-supervision, vision-language learning, adversarial settings, and additional tasks.

  • Large model with Large-scale Data: Large models trained on large-scale data may address long-tailed recognition without explicitly modeling label frequency.
  • Long-tailed Adversarial Learning: Long-tailed adversarial learning is proposed because both attacks and defenses in adversarial learning increasingly exhibit long-tailed distributions.
  • Self-supervised Long-tailed Learning: Self-supervised learning may improve representation learning on long-tailed datasets by reducing reliance on labels.
  • Vision-Language Long-tailed Learning: Vision-language models may improve representation ability for few-sample classes by learning from multiple annotation modes.
  • More Task Settings: More long-tailed visual recognition tasks remain to be explored because many fields naturally exhibit long-tailed data distributions.

8 Conclusions

The survey consolidates long-tailed visual recognition datasets, methods, phenomena, and future directions, while identifying the phenomenon as widespread and research as incomplete.

  • The survey organizes long-tailed studies into ten representation-learning categories and analyzes their highlights and limitations.
  • It compiles generalized long-tailed datasets and benchmarks results on 8 datasets.
  • The survey finds long-tailed distributions widespread and identifies many unresolved problems and future research areas.
Loading 2205.13775v1…