PaperlayerView exact revision on arXiv

Source-linked AI summary

Equalization Loss for Long-Tailed Object Recognition

Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, Junjie Yan

arXiv:2003.05176v2cs.CV

TL;DR

Long-tailed datasets leave rare categories vulnerable to discouraging gradients from other categories and background samples. The paper introduces equalization loss, which reduces negative-sample influence for rare classes, and reports significant improvements across long-tailed detection and classification settings. Its conclusion is that the method is simple but effective across frameworks and architectures.

  • Problem

    Rare categories in long-tailed datasets receive frequent discouraging gradients because samples from other categories and background act as negatives, hurting recognition.

  • Method

    Equalization loss introduces class-specific weights and an ignoring strategy that mainly reduces negative-sample influence for rare categories.

  • Results

    The method brings significant improvement across long-tailed object detection and image classification datasets, frameworks, and network architectures.

  • Takeaways & Limitations

    Equalization loss brings categories with different frequencies to a more equal status during parameter updating and improves rare-object discrimination without harming frequent-category accuracy.

Abstract

from arXiv · show

Object recognition techniques using convolutional neural networks (CNN) have achieved great success. However, state-of-the-art object detection methods still perform poorly on large vocabulary and long-tailed datasets, e.g. LVIS. In this work, we analyze this problem from a novel perspective: each positive sample of one category can be seen as a negative sample for other categories, making the tail categories receive more discouraging gradients. Based on it, we propose a simple but effective loss, named equalization loss, to tackle the problem of long-tailed rare categories by simply ignoring those gradients for rare categories. The equalization loss protects the learning of rare categories from being at a disadvantage during the network parameter updating. Thus the model is capable of learning better discriminative features for objects of rare classes. Without any bells and whistles, our method achieves AP gains of 4.1% and 4.8% for the rare and common categories on the challenging LVIS benchmark, compared to the Mask R-CNN baseline. With the utilization of the effective equalization loss, we finally won the 1st place in the LVIS Challenge 2019. Code has been made available at: https: //github.com/tztztztztz/eql.detectron2

1. Introduction

Long-tailed recognition is difficult because rare categories face stronger inter-class suppression during training. The paper proposes equalization loss to reduce these discouraging gradients and reports improvements across datasets and tasks.

  • Problem: Long-tailed datasets make rare-category detection difficult because other-category and background samples act as negatives for each rare category.Rare categories can therefore be overwhelmed by majority categories and predicted as negatives.
  • Problem: Previous methods mainly address batch-sampling imbalance or foreground-background imbalance, leaving severe imbalance among foreground categories unresolved.The paper identifies inter-class imbalance as a distinct challenge beyond positive-negative sample imbalance.
  • Motivation: For frequent categories, positive gradients dominate on average, whereas negative gradients dominate for rare categories during learning.The analysis uses average gradient norms from the last classifier layer, with categories ordered by instance counts.
  • Method: Equalization loss introduces class- and sample-specific weights that reduce negative-sample influence for rare categories.The method uses an ignoring strategy to alleviate overwhelmed discouraging gradients during parameter updates.
  • Results: EQL significantly improves rare-category performance without harming frequent-category accuracy, bringing categories with different frequencies to a more equal training status.The method is evaluated on object detection, instance segmentation, image classification, Open Images, and LVIS.

2. Related Works

Prior long-tailed recognition methods balance data, losses, or feature representations through distinct strategies. The paper positions its approach alongside these families while targeting foreground-category imbalance.

  • Re-sampling Methods: Re-sampling methods oversample minority classes or undersample frequent classes to balance training data.Oversampling can risk overfitting, while under-sampling is infeasible for extreme long-tailed distributions.
  • Re-sampling Methods: Class-aware sampling first samples a category and then uniformly samples an image containing that category.This is presented as a typical class-balanced oversampling technique.
  • Re-weighting Methods: Re-weighting methods assign weights using class frequency or sample difficulty to make training cost-sensitive.Examples include inverse-frequency weighting, inverse-square-root weighting, and weighting hard versus easy samples.
  • Feature Manipulation: Feature-manipulation methods improve tail representations by enlarging inter-class distances, reducing intra-class variation, or transferring feature variance or semantics.These approaches operate directly on feature representations through specialized modules or transformations.

3. Equalization Loss

Equalization loss addresses long-tailed category imbalance by suppressing discouraging gradients that disproportionately affect rare categories, while retaining background gradients to limit false positives.

  • Motivation: For a foreground sample of category c, every other category receives a discouraging gradient, causing rare categories to accumulate discouraging gradients more frequently than encouraging ones.This can lead even positive samples of rare categories to receive relatively low predicted probabilities.
  • Equalization Loss: Equalization loss introduces a weight term into sigmoid cross-entropy to ignore frequent-category negative gradients for rare categories.Its goal is to reduce the suppression of rare-category learning caused by scarce annotations and long-tailed frequencies.
  • Equalization Loss: The weighting scheme uses proposal foreground status, category frequency, and a threshold function to distinguish tail categories from other categories.E(r) identifies foreground proposals, f_j measures category frequency, and T_λ(x) returns 1 when x < λ and 0 otherwise.
  • Design Choices: Equalization loss ignores negative gradients for rare categories below a frequency threshold but preserves gradients from background samples.Ignoring all negative samples would leave rare categories without negative training examples and produce many false positives.
  • Extension to Image Classification: A Softmax Equalization Loss extends the same idea to softmax classification by introducing a random variable that maintains negative-sample gradients with probability γ.Image classification lacks the background category used in object detection, motivating the additional random gradient-maintenance mechanism.

4. Experiments on LVIS

Experiments on LVIS evaluate equalization loss across models, thresholds, threshold functions, excluding-function variants, and competing imbalance methods. The method consistently improves long-tailed categories, while preserving frequent-category performance better than alternatives.

  • 4.1. LVIS Dataset: LVIS contains 1230 categories divided into rare, common, and frequent groups, with evaluation using mask AP across IoU thresholds from 0.5 to 0.95.The experiments train on 57k images, evaluate on a 5k-image validation set, and also report results on 20k test images.
  • 4.2. Main Results: Equalization loss consistently improves performance across Mask R-CNN and Cascade Mask R-CNN models, with gains mainly from rare and common categories.The improvement across frameworks and backbones indicates effectiveness on long-tailed category groups.
  • 4.3. Effectiveness of Equalization Loss: AP improves as the threshold includes more tail categories, but performance degenerates when frequent categories are included.Setting λ to λr significantly improves APr with marginal influence on APc and APf; λc improves APr and APc while APf declines slightly.
  • 4.4. Ablation Studies: The proposed threshold function is compared with exponential and Gompertz decay functions using their best hyper-parameters.Figure 3 illustrates the alternative threshold designs, while Table 3 reports their performance comparison.
  • 4.4. Ablation Studies: 0.6 points AP gain is achieved with E(r) compared with EQL without E(r), while removing E(r) increases false positives and sharply reduces APf.Ignoring background gradients leaves insufficient background supervision; LVIS evaluation can ignore some false positives for rare categories, affecting the observed APr result.
  • 4.5. Comparison with Other Methods: Equalization loss improves APr and APc significantly with slight effect on APf, surpassing re-sampling and re-weighting approaches.Re-sampling improves rare and common categories at the expense of frequent categories, whereas re-weighting provides consistent but trivial overall gains.
  • 4.6. LVIS Challenge 2019: 28.9 mask AP is achieved on the LVIS v0.5 test set, outperforming the ResNeXt-101 Mask R-CNN baseline by 8.4%.The result combines equalization loss with larger backbones, deformable convolution, synchronized batch normalization, and extra data.

5. Experiments on Open Images Detection

Experiments on Open Images evaluate the method on OID19, a 500-category detection setting with substantial long-tailed frequency variation. Equalization loss improves performance most strongly for categories with the fewest samples.

  • OID19 training uses 12.2M bounding boxes across 500 categories and 1.7M images, with approximately 10k validation images.
  • 2.6 AP points over sigmoid cross-entropy and 10.88 AP points over class-aware sampling are achieved for the 100 categories with the fewest samples.
  • Equalization loss improves more on categories with fewer samples than on categories with more samples.

6. Experiments on Image Classification

Image-classification experiments on CIFAR-100-LT and ImageNet-LT test whether equalization loss transfers beyond object detection. The method improves strong baselines and surpasses prior approaches, particularly for classes with fewer training samples.

  • CIFAR-100-LT: CIFAR-100-LT contains 9,502 training images, with 500 images for the most frequent category and 2 for the rarest.
  • ImageNet-LT: ImageNet-LT contains 1,000 categories with 1,280 to 5 images per category, 116k training images, and 50k test images.
  • CIFAR-100-LT: 2% improvement over a strong CIFAR-100-LT baseline comes from classes with fewer training samples.
  • ImageNet-LT: A wide range of γ values consistently improves over the softmax-loss baseline on ImageNet-LT.
  • Cross-dataset results: Equalization loss surpasses prior state-of-the-art approaches on CIFAR-100-LT and ImageNet-LT.

7. Conclusion

The paper analyzes inter-class competition in long-tailed datasets and proposes equalization loss to reduce discouraging gradients affecting tail categories. It reports improvements across object-detection and image-classification settings.

  • Equalization loss alleviates overwhelmed discouraging gradients on tail categories.
  • The method brings significant improvements across frameworks and network architectures on long-tailed object-detection and image-classification datasets.

Appendix A. Details of LVIS Challenge 2019

The LVIS Challenge solution combines equalization loss with additional data, architectural enhancements, sampling, and multi-scale evaluation. These additions raise Mask AP on both validation and test sets.

  • Additional training data includes COCO and approximately 20k Open Images V5 images sharing 110 categories with LVIS.
  • The challenge baseline combines equalization loss with ResNeXt-101-64x4d, deformable convolution, synchronized batch normalization, repeat-factor sampling, multi-scale training, and COCO data.
  • 30.1% AP is achieved on the LVIS v0.5 validation set by the enhanced challenge baseline.
  • 36.4 Mask AP on validation and 28.9 Mask AP on the test set are achieved with the combined enhancements.
  • Table 12 states that equalization loss plays an important role in achieving the highest AP on rare and common categories.
Loading 2003.05176v2…