PaperlayerView exact revision on arXiv

Source-linked AI summary

Equalization Loss v2: A New Gradient Balance Approach for Long-tailed Object Detection

Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, Quanquan Li

arXiv:2012.08548v2cs.CVcs.LG

TL;DR

Long-tailed object detection is hindered by imbalanced positive and negative gradients, while decoupled methods require extra fine-tuning and end-to-end EQL remains weaker. EQL v2 introduces gradient-guided reweighting for independent category subtasks and surpasses prior methods on LVIS while generalizing to OpenImages.

  • Problem

    Long-tailed object detection must overcome positive gradients being overwhelmed by negative gradients, while existing decoupled training requires extra fine-tuning and EQL remains less accurate.

  • Method

    EQL v2 independently models each category and dynamically up-weights positive gradients and down-weights negative gradients according to accumulated gradient ratios.

  • Results

    EQL v2 surpasses end-to-end and decoupled methods on LVIS, with about 6 points overall AP over baseline and 9 points AP over baseline on OpenImages.

  • Takeaways & Limitations

    Balanced positive-to-negative gradients provide an effective end-to-end approach for long-tailed object detection across multiple frameworks and datasets.

Abstract

from arXiv · show

Recently proposed decoupled training methods emerge as a dominant paradigm for long-tailed object detection. But they require an extra fine-tuning stage, and the disjointed optimization of representation and classifier might lead to suboptimal results. However, end-to-end training methods, like equalization loss (EQL), still perform worse than decoupled training methods. In this paper, we reveal the main issue in long-tailed object detection is the imbalanced gradients between positives and negatives, and find that EQL does not solve it well. To address the problem of imbalanced gradients, we introduce a new version of equalization loss, called equalization loss v2 (EQL v2), a novel gradient guided reweighing mechanism that re-balances the training process for each category independently and equally. Extensive experiments are performed on the challenging LVIS benchmark. EQL v2 outperforms origin EQL by about 4 points overall AP with 14-18 points improvements on the rare categories. More importantly, it also surpasses decoupled training methods. Without further tuning for the Open Images dataset, EQL v2 improves EQL by 7.3 points AP, showing strong generalization ability. Codes have been released at https://github.com/tztztztztz/eqlv2

1. Introduction

Long-tailed object detection suffers from category imbalance and gradient suppression of tail classes. EQL v2 addresses this by independently rebalancing positive and negative gradients, achieving strong results across LVIS and OpenImages.

  • Motivation: Long-tailed datasets follow a Zipfian distribution, creating extreme category imbalance in natural images.Tail classes have insufficient annotations, while head classes dominate training.
  • Motivation: Decoupled training improves accuracy through classifier fine-tuning but requires an extra stage and may learn representations suboptimally.Its classifier is retrained with the representation frozen.
  • Motivation: EQL protects tail categories by blocking some negative gradients, but its end-to-end accuracy remains below decoupled methods.Gradient analysis shows ratios near 1 for head categories but near 0 for tail categories.
  • EQL v2: EQL v2 models each category as an independent task and dynamically reweights positive and negative gradients using accumulated gradient ratios.The same simple reweighting rule is applied equally across category-specific subtasks.
  • Results: About 6 points overall AP and 17–20 points tail-category AP are gained on LVIS versus baseline models, while OpenImages gains 9 points AP with unchanged hyper-parameters.EQL v2 also outperforms EQL by about 4 points AP and surpasses existing end-to-end and decoupled methods on LVIS.

2. Related Work

Related work addresses long-tailed recognition through detector design, resampling, loss reweighting, incremental learning, and decoupled classification. The paper positions EQL v2 as a gradient-balancing alternative to these approaches.

  • Long-tailed detection: Standard object detectors deteriorate severely when category frequencies become long-tailed rather than balanced.Long-tailed detection must locate objects across scales and image locations, making it harder than classification.
  • Existing approaches: Resampling can overfit tail classes and underfit head classes, while loss reweighting is sensitive to dataset-specific hyper-parameters.These limitations motivate alternatives beyond direct data or loss adjustment.
  • Reweighting: EQL blocks suppressing gradients for tail categories but uses category frequencies and a threshold to split head and tail classes.Unlike EQL, the paper treats background and foreground uniformly.
  • Decoupled training: SimCal and BAGS follow decoupled-training principles by using extra classifier branches or separate group-wise softmax classification.These methods address head-class domination through classifier fine-tuning or category grouping.

3. Equalization Loss v2

EQL v2 models detection as independent category tasks and uses accumulated positive-to-negative gradient ratios to dynamically rebalance each task. This gradient-guided strategy enables end-to-end training without the fine-tuning stage used by the compared decoupled methods.

  • EQL v2 treats detection as a set of independent sub-tasks, with one category classifier responsible for each category.The method applies its rebalancing separately to each category task.
  • Positive-to-negative sample ratios can differ substantially across category classifiers, especially for rare categories.The dataset-level ratio is determined by category instance counts relative to the total number of instances.
  • Gradient statistics are used instead of sample counts because many easy negative samples may contribute less than a few hard positive samples.The method therefore measures training balance through positive and negative gradients with respect to each classifier output.
  • EQL v2 dynamically up-weights positive gradients and down-weights negative gradients for each classifier according to its accumulated gradient ratio.The accumulated ratio g_j^(t) determines the current positive and negative reweighting factors.
  • The reweighted positive and negative gradients are applied to the current batch, and the accumulated ratio is updated for the next iteration.This makes the balancing mechanism iterative rather than fixed from dataset statistics alone.
  • EQL v2 is evaluated as an end-to-end method without a fine-tuning stage, unlike the compared decoupled training methods.The comparison uses LVIS v1.0 validation with ResNet-50-FPN Mask R-CNN under a 1x schedule.

4. Experiments

Experiments evaluate EQL v2 on LVIS using standard and extended training settings, including a category-agnostic task and gradient reweighing components.

  • Dataset and Evaluation Metric: LVIS v1.0 contains 1203 categories, about 100k training images, and 1.3M instances, with AP, APr, APc, and APf reported.Results are reported on a 20k-image validation set; rare, common, and frequent categories are defined by image counts.
  • Implementation Details: Models use SGD with momentum 0.9, weight decay 0.0001, ImageNet initialization, a 16-image batch across 16 GPUs, and learning rate 0.02.Training uses scale jitter and horizontal flipping without test-time augmentation.
  • Implementation Details: EQL v2 adds a category-agnostic objectness branch whose positive samples are all other tasks’ positive samples during training.The gradient-guided reweighing mechanism is not applied to this category-agnostic task.
  • Ablation Studies: The baseline Mask R-CNN uses ResNet-50 and FPN, while the longer 3x schedule trains larger backbones for 36 epochs with two learning-rate reductions.The learning rate is divided by 10 at epochs 28 and 34 under the 3x schedule.
  • Ablation Studies: 16.1% baseline AP rises to 23.7% AP with EQL v2’s components, while rare-category gains from negative-gradient down-weighting and positive-gradient up-weighting reach 5.4% and 7.6%.The ablation attributes rare-category improvement to balancing positive and negative gradients without re-sampling.

4.4. Main Results

EQL v2 performs strongly against decoupled and end-to-end long-tailed detection methods across LVIS settings, while retaining joint end-to-end training without extra fine-tuning.

  • Comparison with Decoupled Training methods: 23.7% overall AP is achieved by EQL v2, with APr increasing by 14.9% and APc by 10.8% over the compared decoupled-training results.EQL v2 surpasses cRT, LWS, and BAGS while jointly learning the representation and classifier.
  • Comparison with End-to-End Training methods: 7.6% and 5.1% are the margins by which EQL v2 outperforms naive softmax and EQL, respectively, under random sampling.EQL v2 achieves 23.7% AP versus 22.2% for RFS and does not increase training time through re-sampling.
  • Larger Model & Longer Training: 14.9% rare-category AP improves to 19.1% with EQL v2 under a larger model and longer training schedule.The corresponding Mask R-CNN longer schedule does not substantially improve rare categories.
  • Comparison with State-of-the-Art Methods: 4.3% is EQL v2’s advantage over EQL with ResNet-50-FPN on LVIS v0.5, while larger-framework comparisons report advantages of 1.8% and 0.8% AP.The latter comparisons are against De-confound-TDE and BAGS, respectively.
  • Comparison with State-of-the-Art Methods: 6.3% higher rare-category performance than De-confound-TDE is reported for EQL v2 on LVIS v1.0.EQL v2 also achieves higher overall AP across the evaluated backbones and frameworks.

4.6. Model Analysis

EQL v2 balances positive and negative gradients throughout training and produces more balanced classifier weights, while preserving representations that remain effective under classifier retraining.

  • Gradient balance: EQL v2 preserves a more balanced positive-to-negative gradient ratio throughout training, unlike the baseline where negative gradients overwhelm positives for tail classes.Training longer does not substantially correct the baseline imbalance.
  • Classifier balance: EQL v2 produces a more balanced fc-cls weight norm than baseline and repeat factor sampling models.The comparison uses the L2 weight norm of the final classifier layer.
  • Representation quality: Models initialized with EQL v2 consistently achieve higher AP than standard-training models when evaluated with decoupled classifier retraining.The reported comparisons include cRT, LWS, and BAGS retraining settings.
  • Representation quality: BAGS retraining increases EQL v2 AP by 0.3%, whereas cRT and LWS reduce it by 1.3% and 0.6%, respectively.EQL v2 reaches 23.7% AP before these retraining procedures.

4.7. Influence of hyper-parameters

EQL v2 remains effective across mapping-function shapes and classifier choices, with gradient balancing applicable to both sigmoid and 2C-softmax formulations.

  • Mapping function: Detection AP is not sensitive to the mapping function’s γ and µ shapes, increasing stably across a wide range of both hyper-parameters.The corresponding mapping functions are visualized in Figure 4.
  • Experimental settings: The mapping-function analysis varies γ and µ with α fixed at 4, while the α analysis fixes γ at 12 and µ at 0.8.These settings correspond to the reported hyper-parameter studies.
  • Classifier choice: EQL v2’s gradient-balancing idea extends beyond sigmoid classifiers to 2C-softmax classifiers.The 2C-softmax formulation uses two output logits per category task and introduces an others category during inference.
  • Classifier choice: With only objectiveness-task addition and negative-gradient down-weighting, 2C-softmax achieves higher accuracy than C-sigmoid.After positive-gradient up-weighting, the two designs reach comparable results.

4.9. Experiments on Open Images Detection

On OpenImages, EQL v2 improves substantially over baseline and EQL, especially for tail categories, while reusing LVIS hyper-parameters and equalizing accuracy across category groups.

  • Overall results: EQL v2 reaches 52.6 AP on OpenImages, exceeding the baseline by 9.5 AP and EQL by 7.3 AP.The evaluation uses the Open Images Challenge 2019 validation set.
  • Tail performance: EQL v2 improves the tail group AP by 22.3 points, compared with EQL’s 6.4 AP improvement.Categories are grouped into five groups by instance count, with AP1 representing the least-annotated group.
  • Category balance: EQL v2 achieves similar accuracy across category groups, unlike baseline and EQL models, which retain a large head-to-tail performance gap.The comparison also reports considerable gains over EQL with a larger ResNet-101 backbone.
  • Generalization: EQL v2 keeps its LVIS hyper-parameters on OpenImages, whereas EQL’s hyper-parameters are tuned for the dataset.Further tuned EQL v2 settings are also reported separately.

5. Conclusion

The paper identifies balanced positive and negative gradients as central to long-tailed detection and proposes EQL v2 to dynamically enforce that balance end-to-end. EQL v2 improves tail categories across frameworks and surpasses existing methods on LVIS, including decoupled approaches.

  • Balanced positive and negative gradients are presented as the key to improving long-tailed object detection.
  • EQL v2 dynamically balances the positive-to-negative gradient ratio during training.
  • EQL v2 delivers large improvements for tail categories across various frameworks and beats existing methods on LVIS, including decoupled training methods.

Appendix A. Mapping Function Types

The appendix compares several mapping functions and reports that the proposed sigmoid-like function achieves the highest AP.

  • The proposed sigmoid-like mapping function achieves the highest AP among the compared variants.
  • The mapping-function ablation evaluates performance using AP.
  • Table 11 compares different mapping function types.

Appendix B. More Ablations of Hyper-Params

The appendix studies µ and γ, finding that µ should be relatively high, while γ is robust when µ remains in a reasonable range.

  • Lowering µ significantly degrades accuracy.
  • Higher µ values, such as 0.8 or 0.9, are preferred.
  • γ is more robust when µ lies within a reasonable range.
  • Table 12 presents additional ablations of γ and µ.
Loading 2012.08548v2…