PaperlayerView exact revision on arXiv

Source-linked AI summary

The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation

Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, Jiashi Feng

arXiv:2007.11978v5cs.CV

TL;DR

Long-tail data challenge two-stage detection and segmentation models because proposal classification becomes biased toward head classes. The paper studies this failure, evaluates existing long-tail strategies, and introduces SimCal, which calibrates the classification head with bi-level class-balanced sampling. SimCal significantly improves tail-class instance segmentation on LVIS and COCO-LT and also enhances Mask R-CNN and HTC.

  • Problem

    Two-stage detection and segmentation models perform well on balanced benchmarks but face an unclear performance drop on realistic long-tail datasets, where tail-class recognition is important.

  • Method

    SimCal decouples classification-head calibration from standard training by retraining the head with bi-level class-balanced proposal sampling while freezing the other model parts.

  • Results

    SimCal significantly enhances Mask R-CNN and the state-of-the-art multi-stage HTC model, improving instance segmentation performance for tail classes on long-tail benchmarks.

  • Takeaways & Limitations

    Proposal classification is a central direction for improving region-based object detection and segmentation under long-tail distributions, and SimCal provides a simple baseline for that setting.

  • Takeaways & Limitations

    The study reports that re-weighting and related approaches can complicate deep-model optimization and lead to poor performance on head classes under extreme imbalance.

Abstract

from arXiv · show

Most existing object instance detection and segmentation models only work well on fairly balanced benchmarks where per-category training sample numbers are comparable, such as COCO. They tend to suffer performance drop on realistic datasets that are usually long-tailed. This work aims to study and address such open challenges. Specifically, we systematically investigate performance drop of the state-of-the-art two-stage instance segmentation model Mask R-CNN on the recent long-tail LVIS dataset, and unveil that a major cause is the inaccurate classification of object proposals. Based on such an observation, we first consider various techniques for improving long-tail classification performance which indeed enhance instance segmentation results. We then propose a simple calibration framework to more effectively alleviate classification head bias with a bi-level class balanced sampling approach. Without bells and whistles, it significantly boosts the performance of instance segmentation for tail classes on the recent LVIS dataset and our sampled COCO-LT dataset. Our analysis provides useful insights for solving long-tail instance detection and segmentation problems, and the straightforward \emph{SimCal} method can serve as a simple but strong baseline. With the method we have won the 2019 LVIS challenge. Codes and models are available at https://github.com/twangnh/SimCal.

1 Introduction

The paper investigates why two-stage detection and segmentation models degrade on long-tail data and identifies proposal classification, especially classification-head bias, as a major obstacle. It evaluates existing long-tail strategies and proposes SimCal, a calibration framework using bi-level class-balanced sampling to improve tail-class performance.

  • 1 Introduction: Long-tail datasets such as LVIS expose a performance-drop problem that is not clearly explained by prior evaluations on class-balanced benchmarks.The paper contrasts balanced benchmarks including Pascal VOC, COCO, and OpenImage with realistic long-tail settings.
  • 1 Introduction: Tail-class instances are exposed less frequently to the classifier than head-class instances under standard training, producing poor tail-class classification.This imbalance is described as classification-head bias.
  • 1 Introduction: Common long-tail classification strategies improve instance segmentation but provide limited tail gains and can sacrifice head-class performance.The considered strategies include loss re-weighting, adaptive loss adjustment, and data re-sampling.
  • 1 Introduction: SimCal calibrates a trained classification head through a decoupled learning scheme with bi-level class-balanced sampling of proposals.The method collects balanced samples using image-level and instance-level sampling, then calibrates the classification head while retaining the broader model structure.
  • 1 Introduction: The method is evaluated on LVIS, HTC, and COCO-LT, with reported improvements for tail classes and state-of-the-art single-model performance on LVIS.The experiments extend beyond Mask R-CNN to the multi-stage Hybrid Task Cascade model.
  • 1 Introduction: Classification is the primary obstacle preventing region-based detection and segmentation models from working well on long-tail data.The authors identify inaccurate classification of object proposals as a major cause of the observed degradation.

2 Related Works

Related work covers two-stage detection and segmentation frameworks and long-tail recognition methods. It emphasizes that long-tail detection and segmentation remain less explored than long-tail classification.

  • 2 Related Works: Two-stage detection frameworks such as Faster R-CNN and Mask R-CNN form a prevailing basis for object detection and instance segmentation.Mask R-CNN extends Faster R-CNN with a mask prediction head for region-based mask segmentation.
  • 2 Related Works: Long-tail recognition methods primarily use sampling or loss re-weighting to address imbalanced data.Sampling changes class exposure, while re-weighting assigns different weights to classes or training instances.
  • 2 Related Works: Existing long-tail research mainly focuses on classification, leaving long-tail object detection and segmentation comparatively unexplored.The paper positions its work as addressing this gap for proposal-based instance segmentation models.

3 Analysis: Performance Drop on Long-tail Distribution

The study finds that long-tail performance degradation in two-stage detection and segmentation is driven mainly by biased proposal classification, rather than class-agnostic box and mask heads.

  • Long-tail performance decline is investigated in popular two-stage instance detection and segmentation frameworks.
  • 0 box and mask AP are obtained for rare categories by the baseline Mask R-CNN model.The baseline performs especially poorly on the rare LVIS set.
  • 45.1% overall mask AP loss accompanies only an 8.8% proposal-recall loss when comparing LVIS with COCO.Mask AP falls from 32.8 to 18.0, while proposal recall falls from 55.9 to 51.0.
  • Assigning ground-truth proposal labels substantially increases tail-class box and mask AP, while class-agnostic box and mask heads remain less sensitive to imbalance.The improvement is especially large on rare and common sets.
  • The resulting diagnosis is classification head bias: poor proposal classification is the main cause of low tail-class performance.

4 Solutions: Alleviating Classification Bias

The paper evaluates existing long-tail classification strategies and proposes SimCal, which retrains only the classification head using bi-level class-balanced proposal sampling before combining calibrated and original predictions.

  • The paper adapts loss re-weighting, focal loss, class-aware margin loss, and data re-sampling to long-tail instance detection and segmentation.
  • Inverse-frequency re-weighting assigns class-dependent weights, giving tail samples higher training influence while clamping weights to [0.1, 10.0].
  • Focal loss adaptively reweights samples by prediction difficulty, using L = −(1 − p_i)^γ log(p_i).
  • Class-aware margin loss gives tail classes larger margins, while repeat sampling oversamples tail data to involve those classes more often in optimization.
  • These existing strategies provide limited tail improvement and can sacrifice head performance, overfit repeated tail samples, or increase computation cost.
  • SimCal retrains only the classification head with bi-level class-balanced proposal sampling while freezing the backbone, RPN, box head, and mask head.The calibrated head is initialized from the original head, and the procedure has negligible additional computation cost because only that head back-propagates gradients.
  • Bi-level sampling first samples classes uniformly, then images containing those classes, and collects proposals from the sampled classes plus background.
  • Dual-head inference combines the calibrated and original classifiers by selecting predictions for head and tail classes according to a training-frequency threshold.The calibrated head performs better on tail classes, while the original head performs better on head classes; inference adds small overhead.

5 Experiments

Experiments on LVIS and COCO-LT evaluate long-tail strategies and SimCal across Mask R-CNN, HTC, class-wise, and class-agnostic settings. SimCal consistently improves tail performance, while dual-head inference preserves much of the original head-class strength.

  • Datasets: Experiments use LVIS and a sampled COCO-LT dataset, reporting median overall AP over three runs.LVIS is the primary benchmark, while COCO-LT tests generalizability.
  • Baseline Strategies: 6.0, 6.2, and 7.7 AP1 gains are observed for loss re-weighting, focal loss, and image-level repeat sampling, respectively.These methods also reduce AP4 and APf by more than 2.0 for head classes.
  • SimCal on Mask R-CNN: More than 10 absolute mask and box AP improvement on tail bins is obtained for both class-agnostic and class-wise Mask R-CNN with calibration.The class-wise model drops from 23.7 to 21.9 on AP3 and from 29.6 to 25.3 on AP4.
  • Head Design: Class-agnostic heads perform better on tail classes, while class-wise heads perform better for many-shot classes.For r50-ag with calibration and dual inference, AP1 is 13.3 versus 10.2, while AP4 is 26.2 versus 28.7 for r50.
  • Decoupled Learning: 10.2 and 23.9 AP are achieved on AP1 and AP2 when competing imbalance methods use decoupled learning, confirming the bi-level sampling scheme’s effectiveness.The decoupled scheme improves tail recognition, while image-level sampling can harm box and mask learning.
  • Calibration Dynamics: With 10–20k calibration steps, AP across all bins and overall AP converge to steady values.Tail-bin AP increases while head-bin AP decreases during calibration.
  • Head Design: 4.1 and 4.3 AP gains on AP1 and AP2 occur when the 3-layer head is initialized from the original head.This setting outperforms a randomly initialized 3-layer head on tail bins.

6 Conclusions

The paper identifies proposal classification as the main source of long-tail performance degradation and proposes SimCal to calibrate the second-stage classifier. SimCal improves Mask R-CNN and HTC, while the authors note substantial room for further improvement.

  • 6 Conclusions: Proposal classification is identified as a major cause of performance drop for two-stage instance segmentation under long-tail distributions.The analysis attributes this to bias in the classification head.
  • 6 Conclusions: SimCal significantly enhances Mask R-CNN and the state-of-the-art multi-stage HTC model.The method is presented as a simple calibration approach for improving second-stage classification on tail classes.
  • 6 Conclusions: The authors state that substantial opportunity for improvement remains in long-tail instance detection and segmentation.They position the analysis and SimCal as a baseline for future research.

G Implementation Details

The implementation uses PyTorch and mmdetection, with SGD and momentum training on eight NVIDIA Tesla V100 GPUs.

  • G Implementation Details: The proposed method is implemented in PyTorch on top of mmdetection.All models use SGD with momentum 0.9 and training on 8 NVIDIA Tesla V100 GPUs.

G.1 Standard Model Training

Standard training samples eight images per minibatch, with separate Mask R-CNN and HTC schedules and a multi-scale augmentation setting for HTC. Calibration instead samples balanced class-image batches and trains for 12,000 steps.

  • G.1 Standard Model Training: Standard model training samples 8 images per mini-batch using random sampling.This is the normal whole-model training setting.
  • G.1 Standard Model Training: Mask R-CNN training uses learning rate 0.01, decays at epochs 8 and 11, and ends at epoch 12.Images use an 800-pixel short edge and a 1,333-pixel long-edge cap.
  • G.1 Standard Model Training: HTC training uses learning rate 0.01, decays at epochs 16 and 19, and ends at epoch 20 with multi-scale augmentation.The short edge is sampled from [400, 1,400] and the long edge is capped at 1,600 pixels.
  • G.1 Standard Model Training: Calibration samples 16 classes and one image per class per minibatch, collecting proposals from sampled classes and background.Foreground and background ROIs are sampled at a 1:1 ratio.
  • G.1 Standard Model Training: Calibration learning rates decay at steps 8,000 and 11,000, and training ends at step 12,000.The initial calibration learning rate is 0.01.
  • G.1 Standard Model Training: Hyperparameters for the adopted long-tail classification methods are tuned by grid search.The passage gives settings for re-weighting and focal loss, including γ = 3 and α = 0.5 for focal loss.

H Qualitative Results

Qualitative LVIS results show that calibration enables detection and segmentation of low-shot object instances that the original model misses, despite some false positives.

  • After calibration, the model detects and segments low-shot LVIS instances that the uncalibrated model misses.The visualization covers categories in the [1, 10) bin using the r50-ag model.

J How to Combine the Dual Heads

The proposed scheme combines calibrated and original classification heads through threshold selection and achieves the best reported overall result among tested alternatives. Ablations also favor calibrating the full classification head, while calibration learning rate was varied experimentally.

  • Combining calibrated and original heads: The evaluated alternatives include confidence-based selection, score scaling, and score normalization for combining head predictions.Score scaling addresses differing average scores, while normalization uses the summed score over classes.
  • Combining calibrated and original heads: 21.1 AP is achieved by the proposed combining scheme, outperforming the other tested alternatives.sel-thr, sel-scale, and sel-norm obtain similar but slightly lower performance.
  • Calibration learning rate: Calibration learning rates from 0.001 to 0.08 were tested while keeping the decay step and factor unchanged.Performance was measured using overall AP.
  • Calibration layers: The best result is obtained when calibration is applied to the full three-layer classification head rather than only its last one or two layers.The comparison uses overall AP for Mask R-CNN with a ResNet50-FPN backbone and class-agnostic box and mask heads.

M LVIS Mean and Std Analysis

Repeated LVIS experiments show greater AP variability for tail classes than for many-shot classes, which have more stable results because they have ample training instances.

  • AP variance is much larger for tail classes and smaller for many-shot classes across repeated LVIS experiments.The analysis repeats each experiment five times and reports mean and standard deviation for each metric.
  • The reported SimCal models include mean ± standard deviation values across multiple LVIS metrics.The table lists results for r50-ag-lvis and r50-lvis.
Loading 2007.11978v5…