PaperlayerView exact revision on arXiv

Source-linked AI summary

Seesaw Loss for Long-Tailed Instance Segmentation

Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, Dahua Lin

arXiv:2008.10032v4cs.CV

TL;DR

Long-tailed instance-segmentation datasets cause head-class negatives to overwhelm tail-class gradients, biasing classifiers against rare categories. The paper proposes Seesaw Loss, which dynamically combines mitigation and compensation factors to rebalance these gradients. It reports consistent gains across frameworks and samplers, including 6.0% and 2.1% AP improvements for Mask R-CNN on LVIS.

  • Problem

    Long-tailed category distributions produce overwhelming negative gradients for tail classes, causing their objects to be misclassified as backgrounds or head categories.

  • Method

    Seesaw Loss dynamically re-balances positive and negative gradients per category using sample-ratio-based mitigation and false-positive compensation factors.

  • Results

    Seesaw Loss consistently improves instance-segmentation performance across frameworks and samplers, including 6.0% AP and 2.1% AP gains for Mask R-CNN on LVIS.

  • Takeaways & Limitations

    Seesaw Loss provides a simpler end-to-end solution for long-tailed instance segmentation without relying on complex training pipelines.

  • Takeaways & Limitations

    Online accumulation is designed to remain robust when the full training-set distribution is unavailable or data sampling affects category frequencies.

Abstract

from arXiv · show

Instance segmentation has witnessed a remarkable progress on class-balanced benchmarks. However, they fail to perform as accurately in real-world scenarios, where the category distribution of objects naturally comes with a long tail. Instances of head classes dominate a long-tailed dataset and they serve as negative samples of tail categories. The overwhelming gradients of negative samples on tail classes lead to a biased learning process for classifiers. Consequently, objects of tail categories are more likely to be misclassified as backgrounds or head categories. To tackle this problem, we propose Seesaw Loss to dynamically re-balance gradients of positive and negative samples for each category, with two complementary factors, i.e., mitigation factor and compensation factor. The mitigation factor reduces punishments to tail categories w.r.t. the ratio of cumulative training instances between different categories. Meanwhile, the compensation factor increases the penalty of misclassified instances to avoid false positives of tail categories. We conduct extensive experiments on Seesaw Loss with mainstream frameworks and different data sampling strategies. With a simple end-to-end training pipeline, Seesaw Loss obtains significant gains over Cross-Entropy Loss, and achieves state-of-the-art performance on LVIS dataset without bells and whistles. Code is available at https://github.com/open-mmlab/mmdetection.

1. Introduction

Long-tailed category distributions bias instance-segmentation classifiers toward head classes and backgrounds. Seesaw Loss addresses this by dynamically re-balancing category gradients, improving tail and overall performance across settings.

  • Motivation: Long-tailed datasets contain few head classes with abundant instances and many tail classes with relatively few instances.This imbalance contrasts with the relatively balanced distributions of common benchmarks.
  • Motivation: Head-class instances generate overwhelming negative samples for tail classes, heavily imbalancing their positive and negative gradients.The resulting biased classifier learning suppresses tail-class probabilities and accuracy.
  • Approach: Seesaw Loss dynamically re-balances each category’s positive and negative gradients using mitigation and compensation factors.Mitigation scales penalties according to cumulative category sample ratios, while compensation increases penalties for false positives.
  • Properties: Seesaw Loss is dynamic, self-calibrated, and distribution-agnostic rather than relying on static groups, constant weights, or precomputed dataset distributions.Its factors jointly reduce excessive tail-class punishment while limiting false positives.
  • Results: 6.0% AP and 2.1% AP improvements are reported for Mask R-CNN with random and repeat-factor sampling, respectively, on LVIS.Cascade Mask R-CNN gains 6.4% AP and 2.3% AP under the same samplers, while ImageNet-LT accuracy improves by 6%.

2. Related Work

Related work spans object detection and instance-segmentation architectures, alongside long-tailed learning methods that alter training procedures or classifiers. These approaches motivate Seesaw Loss’s focus on simpler gradient re-balancing.

  • Object Detection: Object detection research includes two-stage, single-stage, anchor-free, and cascade pipelines with progressively refined bounding-box predictions.Two-stage methods generate proposals before classification and refinement, while single-stage methods predict boxes directly.
  • Instance Segmentation: Mask R-CNN adds a mask-prediction branch to Faster R-CNN, connecting object detection and instance segmentation.Cascading frameworks also adopt this detection-and-segmentation architecture.
  • Long-Tailed Recognition: Long-tailed approaches use multi-stage training, class-balanced fine-tuning, group-specific classifiers, or modified classifier weights.A decoupling pipeline first learns representations with natural sampling and then fine-tunes classifiers with class-balanced sampling.

3. Methodology

Seesaw Loss re-balances category-specific gradients with mitigation and compensation factors, then extends the design with normalized activations and a separate objectness branch for detection.

  • Seesaw Loss: Cross-Entropy Loss becomes biased because abundant head-class instances generate overwhelming negative gradients for tail classes.Tail classifiers consequently receive few positive signals and their predicted probabilities are suppressed.
  • Seesaw Loss: Seesaw Loss introduces a tunable balancing factor S_ij that adjusts penalties on class j from positive samples of class i.The factor is defined as the product of mitigation and compensation factors.
  • Mitigation Factor: The mitigation factor reduces penalties on relatively rare classes using cumulative category counts gathered online during training.Online accumulation adapts to unavailable dataset distributions and remains robust to sampling methods.
  • Compensation Factor: The compensation factor increases penalties for misclassified negative classes when their predicted probability exceeds that of the ground-truth class.It targets individual false positives rather than uniformly modifying an entire category.
  • Normalized Linear Activation: Normalized linear activation L2-normalizes classifier weights and input features to reduce category-dependent scale variance in long-tailed data.A temperature factor controls the normalized logits, while the normalization is intended to balance predicted-probability distributions.
  • Objectness Branch: A separate objectness branch decouples foreground-background detection from C-class categorization because treating background as a classifier category would overly reduce foreground penalties.The branch predicts foreground versus background with normalized linear activation and cross-entropy loss.
  • Normalized Mask Prediction: Normalized mask prediction applies L2 normalization to the mask-head weights and spatial input features before sigmoid mask prediction.The formulation uses a temperature factor and normalizes each spatial feature vector.

4. Experiments

Experiments on LVIS and ImageNet-LT evaluate Seesaw Loss across frameworks, samplers, components, and training pipelines. Results show consistent gains over Cross-Entropy and competing long-tailed methods, while ablations support its design choices.

  • Benchmark Results: 6.0% AP and 2.1% AP gains over Cross-Entropy are obtained with Mask R-CNN using random sampling and RFS, respectively.Improvements occur across rare, common, and frequent categories.
  • Benchmark Results: Seesaw Loss outperforms EQL and BAGS under both samplers without sacrificing frequent-category AP as those methods do.The comparisons report larger gains against both alternatives and consistently higher APr, APc, and APf.
  • Benchmark Results: 6.4% AP and 2.3% AP gains are obtained with Cascade Mask R-CNN using random sampling and RFS, respectively.With Normalized Mask Prediction, the two sampler settings reach 29.6% AP and 30.1% AP.
  • Ablation Study: The mitigation factor raises AP from 23.7% to 25.1%, compensation adds 0.4% AP, and their combination reaches 25.7% AP.Normalized linear activation raises the baseline from 23.7% to 24.7% AP; all three components reach 26.4% AP.
  • Ablation Study: Online accumulation of category samples is preferred because samplers alter training frequencies, while end-to-end training outperforms the decoupled pipeline.On ImageNet-LT, Seesaw Loss improves CE top-1 accuracy from 44.4% to 50.4% with end-to-end training.

5. Conclusion

Seesaw Loss dynamically re-balances positive and negative gradients for each category to address long-tailed instance segmentation. Its mitigation and compensation mechanisms reduce excessive tail-class punishment while limiting false positives.

  • Seesaw Loss dynamically re-balances positive and negative gradients for each category using mitigation and compensation factors.The loss adjusts category penalties during training rather than relying on a fixed reweighting scheme.
  • The mitigation factor reduces punishment for tail categories according to cumulative training-instance ratios between categories.
  • The compensation factor increases penalties for misclassified instances to avoid false positives of tail categories.
  • The partial derivative with respect to Sij is always positive, so decreasing Sij reduces negative-sample gradients imposed by class i on class j.

Appendix B. How Seesaw Loss works

Seesaw Loss addresses gradient imbalance in Mask R-CNN on LVIS by dynamically adjusting category-specific penalties. The analysis compares gradient distributions under Cross-Entropy Loss and Seesaw Loss.

  • Mask R-CNN with Seesaw Loss significantly outperforms the Cross-Entropy Loss version on the LVIS dataset.
  • The gradient analysis uses Mask R-CNN with a ResNet-101 backbone and FPN, trained with a random sampler under a 2x schedule.
  • Under Cross-Entropy Loss, tail classes receive heavily imbalanced positive and negative gradients during training.
  • The overwhelming negative gradients under Cross-Entropy Loss bias classifier learning and produce low classification accuracy on tail classes.

Appendix C. Per-category Performance Comparison

Per-category AP comparisons show that Seesaw Loss performs strongly across categories with different frequencies and across multiple instance-segmentation frameworks.

  • Seesaw Loss consistently achieves strong performance across categories with different frequencies.The comparison includes Cross-Entropy Loss, Equalization Loss, and Balanced Group Softmax.
  • The per-category comparison covers Mask R-CNN and Cascade Mask R-CNN frameworks.

Appendix D. LVIS Challenge 2020

The MMDet LVIS Challenge 2020 entry adopts Seesaw Loss and reports a substantial gain over its strong baseline. It also introduces HTC-Lite as a lightweight HTC variant.

  • 6.9% AP improvement over the strong baseline was obtained on the LVIS v1 validation split.
  • The entry adopts Seesaw Loss for long-tailed instance segmentation.
  • HTC-Lite replaces HTC’s semantic segmentation branch with a global context encoder.
  • The challenge entry uses a single model without external data and annotations.

D.1. HTC-Lite

HTC-Lite is a lightweight HTC variant designed to accelerate training and inference while maintaining good performance. It replaces HTC’s unavailable semantic-segmentation branch with global context encoding and reduces mask heads, while outperforming mainstream cascading frameworks on LVIS.

  • HTC-Lite accelerates training and inference while maintaining good performance as a lightweight version of Hybrid Task Cascade.
  • Its two modifications replace semantic segmentation with global context encoding and reduce the number of mask heads.
  • Because LVIS lacks semantic segmentation annotations, the context encoder uses convolution, global average pooling, and an auxiliary fully connected layer for multi-label classification.
  • HTC-Lite is compared with Cascade Mask R-CNN and HTC using ResNet-50-FPN, repeat factor sampling, and a 1x schedule on LVIS v1.
  • The experiments show that HTC-Lite is more efficient than the compared frameworks and outperforms them.

D.2. Step by Step Results

The authors build their LVIS Challenge 2020 entry incrementally from a Mask R-CNN baseline by adding normalization, upsampling, HTC-Lite, TSD, classification-branch fine-tuning, and test-time augmentation. The completed system reaches 38.8% AP on validation and 38.92% AP on test-dev.

  • The baseline is Mask R-CNN with a ResNet-50-FPN backbone, multi-scale training, random sampling, and a 2x schedule.
  • The entry adds SyncBN, CARAFE upsampling, HTC-Lite, and TSD step by step.
  • A new classification branch is fine-tuned for each cascade stage on a fixed model using repeat factor sampling and a 1x schedule.
  • 38.8% AP is achieved on the val split and 38.92% AP on the test-dev split after adding the components and test-time augmentation.
Loading 2008.10032v4…