PaperlayerView exact revision on arXiv

Source-linked AI summary

Imbalance Problems in Object Detection: A Review

Kemal Oksuz, Baris Can Cam, Sinan Kalkan, Emre Akbas

arXiv:1909.00169v3cs.CV

TL;DR

Object detection contains imbalance problems whose distributions can affect performance. This paper reviews them through a problem-based taxonomy, synthesizes solutions critically, and identifies open issues, including previously undiscussed imbalances.

  • Problem

    Imbalance problems in object detection can affect final detection performance when input-property distributions influence performance.

  • Method

    The paper introduces a problem-based taxonomy, reviews each imbalance problem and its solutions, and provides a unifying critical perspective.

  • Results

    The review organizes existing imbalance problems and solutions while identifying major open issues and previously undiscussed imbalance problems.

  • Takeaways & Limitations

    The accompanying webpage catalogs papers by taxonomy, enabling researchers to track newer studies on object-detection imbalance.

  • Takeaways & Limitations

    The effects of imbalance in object-orientation distributions still need investigation.

Abstract

from arXiv · show

In this paper, we present a comprehensive review of the imbalance problems in object detection. To analyze the problems in a systematic manner, we introduce a problem-based taxonomy. Following this taxonomy, we discuss each problem in depth and present a unifying yet critical perspective on the solutions in the literature. In addition, we identify major open issues regarding the existing imbalance problems as well as imbalance problems that have not been discussed before. Moreover, in order to keep our review up to date, we provide an accompanying webpage which catalogs papers addressing imbalance problems, according to our problem-based taxonomy. Researchers can track newer studies on this webpage available at: https://github.com/kemaloksuz/ObjectDetectionImbalance .

1 INTRODUCTION

Object detection estimates object categories and locations, but its performance is affected by multiple imbalance problems throughout the deep-learning training pipeline. This review organizes those problems, critically synthesizes their solutions, identifies open issues, and maintains a living paper repository.

  • Object detection simultaneously estimates object categories and locations, supporting applications including surveillance, autonomous driving, medical decision making, and robotics.
  • Imbalance occurs when an input-property distribution affects performance, and unaddressed imbalance adversely affects final detection performance.
  • Foreground-background imbalance can involve millions of negative examples versus only a few positives, greatly impairing detection accuracy when untreated.
  • The review identifies eight imbalance problems grouped into class, scale, spatial, and objective imbalance.
  • The paper proposes a taxonomy, critical literature review, open-issue discussion, cross-domain analysis, and continuously updated webpage organized by the taxonomy.
  • Unlike general detection surveys and machine-learning imbalance reviews, this work examines object detection’s recognition and search aspects across eight imbalance problems.

2 BACKGROUND, DEFINITIONS AND NOTATION

The paper briefly introduces modern object-detection pipelines, terminology, notation, and the three training phases used to locate imbalance problems. It distinguishes top-down and bottom-up systems and one-stage from two-stage top-down detectors.

  • Modern object detectors mainly follow top-down approaches, while bottom-up methods predict keypoints and group them into complete object instances.
  • Two-stage detectors reduce dense-anchor negatives through proposals, then process regions of interest to output boxes and class probabilities.
  • One-stage detectors predict results directly from anchors after feature extraction, using sampling and labeling within the common training pipeline.
  • The generic pipeline comprises feature extraction, detection and bounding-box matching, and labeling and sampling phases.
  • The paper defines core terms including backbone features, pyramidal features, detections, IoU, under-represented classes, and over-represented classes.
  • Regression methods may transform predictions into the log domain or directly predict bounding-box coordinates, so the review denotes the generic regression-loss input by x̂.

3 A TAXONOMY OF THE IMBALANCE PROBLEMS AND THEIR SOLUTIONS IN OBJECT DETECTION

The review defines imbalance as a performance-relevant distributional bias over an input property and organizes eight problems into four categories. The taxonomy maps problems and solutions onto stages of the detection pipeline.

  • The taxonomy groups eight imbalance problems into class, scale, spatial, and objective imbalance categories.
  • Class imbalance includes foreground-background and foreground-foreground imbalance and is usually addressed during sampling.
  • Scale imbalance concerns varied object scales and examples, including unbalanced contributions from different feature-abstraction levels.
  • Spatial imbalance covers regression-loss contributions, IoU distributions, and other bounding-box spatial properties.
  • Objective imbalance arises when multiple task losses have incompatible ranges or optima, requiring a strategy acceptable across objectives.
  • Some imbalance problems arise naturally from data, whereas objective, feature-level, and regression-loss imbalance can result from selected methods and may be avoidable.

4 IMBALANCE 1: CLASS IMBALANCE

Class imbalance occurs when object-detection classes are unevenly represented, with distinct foreground-background and foreground-foreground forms. The review illustrates these forms using anchor frequencies on MS-COCO and notes that they require different solution types.

  • Class imbalance occurs when one class has more examples than others and manifests as foreground-background or foreground-foreground imbalance.
  • Foreground-background imbalance contrasts numerous background anchors with positive object examples, while foreground-foreground imbalance often follows a long-tail class distribution.
  • The class-imbalance illustration applies RetinaNet’s default anchors to MS-COCO and counts foreground classes separately using IoU>0.5 and background anchors using IoU<0.4.
  • The two class-imbalance types have different characteristics and are addressed with different solution types, although some methods can target both.

4.1 Foreground-Background Class Imbalance

Foreground-background imbalance is inevitable because object detectors generate far more background than foreground boxes. The review organizes remedies into hard sampling, soft sampling, sampling-free, and generative methods.

  • Problem: Background boxes vastly outnumber foreground boxes because bounding-box matching labels most boxes as background.This imbalance occurs during training and does not depend on dataset background annotations.
  • Hard sampling: Hard sampling selects a desired subset of positive and negative boxes using binary weights, while discarded examples contribute nothing in that iteration.Each selected example contributes equally to the loss.
  • Hard sampling: Hard-example mining prioritizes examples with high losses instead of sampling randomly, based on the hypothesis that difficult examples improve detector training.SSD selects negative examples with the highest losses, while later methods consider positive and negative losses.
  • Soft sampling: Soft sampling retains every example but adjusts its loss contribution using weights, including constant coefficients or dynamically computed importance.Weights lie in [0, 1] for soft sampling, unlike binary hard-sampling weights.
  • Soft sampling: Focal Loss assigns larger weights to harder examples through estimated ground-truth probability, reducing to cross entropy when γ = 0.The reviewed study reports γ = 2 as a good trade-off for its architecture.
  • Soft sampling: GHM counts examples with similar gradient norms and suppresses gradients from easy positives and negatives, applying to classification and regression.Its balancing strategy is also relevant to imbalance in regression loss.

4.2 Foreground-Foreground Class Imbalance

Foreground-foreground imbalance concerns unequal representation among positive classes, arising at both dataset and batch levels. The review relates these imbalances to long-tailed datasets and uneven class composition within images or batches.

  • Definition: Foreground-foreground imbalance occurs when both over-represented and under-represented classes are foreground classes.The review divides its origins into dataset-level and batch-level imbalance.
  • Dataset-level imbalance: Natural differences in object frequency create significant gaps in class examples, making overfitting toward over-represented classes possible for naive approaches.Figure 5(a) illustrates the dataset-level gap in class examples.
  • Dataset-level imbalance: Ouyang et al. analyze long-tail effects using prediction accuracy and example counts, then cluster classes by visual similarity.Their similarity measure uses inner products of pretrained backbone features.
  • Batch-level imbalance: Uneven class distributions within batches can bias learning toward classes with more anchors.MS COCO statistics also show substantial variation in the numbers of objects and classes per image.

4.3 Comparative Summary

The comparative review reports gains from both early hard-sampling methods and newer approaches that retain or reweight examples. It also highlights positive-example sampling and localization-specific effects.

  • Hard sampling: 14.7% relative improvement over Fast R-CNN was reported for OHEM with a VGG-16 backbone on MS COCO 2015.OHEM was among the most effective early hard-sampling methods reviewed.
  • Soft sampling: 9.3% relative improvement over alpha-balanced cross entropy was reported for Focal Loss on RetinaNet, from 31.1 to 34.0 mAP5.The comparison uses the baseline and RetinaNet values stated in the review.
  • Soft sampling: 3.9% relative improvement over Focal Loss was reported for AP Loss, from 33.9 to 35.0 mAP.The review describes AP Loss as promising.
  • Positive sampling: 4.4% relative improvement was reported when PISA was applied to positives only, from 36.4 to 38.0 mAP.The review states that sampling also matters for positive examples.
  • Positive sampling: PISA’s reported improvement was concentrated in localization, with no AP@0.50 gain and up to 2.6% improvement at higher-IoU AP@0.75.The review attributes this pattern to a changed IoU distribution as a possible explanation.

4.4 Open Issues

The review identifies unresolved issues in foreground-example usefulness, foreground-foreground class imbalance, batch composition, sampling, and class-specific confidence ranking.

  • Sampling More Useful Examples: Soft sampling is increasingly favored over discarding samples, but which negative examples are most useful remains unresolved.Weighting examples has produced more performance improvement than discarding large portions of samples.
  • Sampling More Useful Examples: Positive-example selection remains contested because hard-example mining favors difficult samples, whereas IoU-based prime sampling favors easier positives.The review notes that the usefulness criteria may differ between positive and negative examples.
  • Foreground-Foreground Class Imbalance: Foreground-foreground class imbalance is less thoroughly studied, and the class with the fewest instances can still achieve among the best detection performance.This indicates that total instance counts alone do not explain performance differences between foreground classes.
  • Batch and Sampling Effects: A batch’s foreground-class distribution may differ from the overall dataset, while box sampling or weighting can introduce further class imbalance.Consequently, balancing the dataset or batch may not fully address foreground-foreground imbalance.
  • Objective-Level Issues: AP Loss ranks confidence scores across classes despite evidence that optimal confidence thresholds vary by class.The review suggests that class-specific confidence ranking warrants investigation.

5 IMBALANCE 2: SCALE IMBALANCE

Scale imbalance concerns skewed object sizes and feature representations across network levels. Reviewed solutions use multi-scale predictions, feature pyramids, or image pyramids, trading accuracy and efficiency under memory constraints.

  • Object/Box-Level Scale Imbalance: Scale imbalance arises when some object or bounding-box sizes are over-represented, with MS COCO showing distributions skewed toward smaller objects.The imbalance affects estimated RoI scales and overall detection performance.
  • Methods Using Backbone Features as a Basis: Independent predictions from different backbone levels provide multi-scale detection because network levels encode information at different object scales.SSD is an example of a one-stage detector that predicts from features at different layers.
  • Feature Imbalance: FPN combines different-scale features before prediction through a top-down pathway and lateral connections, addressing the limitations of single-level representations.However, straightforward feature combination can itself produce feature imbalance.
  • Image Pyramids in Deep Object Detectors: Image pyramids are generally less efficient than feature pyramids, while feature pyramids remain approximations constrained by the memory demands of image pyramids.Alleviating image-pyramid memory limitations remains an open problem.
  • Comparative Summary: 3.7% relative improvement on MS COCO testdev was reported by FPN with Faster R-CNN and ResNet-101, from 34.9 to 36.2.The same study reported over twofold faster inference than baseline Faster R-CNN with ResNet-50, at 150ms versus 320ms per image.
  • Comparative Summary: 10.2% relative improvement on MS COCO testdev was reported by NAS-FPN, from 40.1 to 44.2, with approximately twice as many parameters and 26% more inference time.The comparison used a ResNet-50 backbone and 1024 × 1024 images.
  • Comparative Summary: 19.5% relative improvement on MS COCO test-dev was reported by a multi-scale method, with 12ms inference versus 10ms for SSD300 and mAPs of 30.0 versus 25.1.The comparison used the same image size and backbone network.

6 IMBALANCE 3: SPATIAL IMBALANCE

Spatial imbalance includes uneven regression-loss contributions across examples and attributes such as bounding-box size, shape, location, and IoU. The review organizes regression-loss solutions around Lp-norm and IoU-based formulations, including methods that rebalance gradients or add geometric penalties.

  • Spatial attributes: Spatial attributes of bounding boxes include size, shape, location, and IoU, whose imbalance can affect training and generalization.Small positional shifts may produce large localization-loss changes when the loss function is unsuitable.
  • Imbalance in regression loss: Regression-loss imbalance occurs when individual examples contribute unevenly, with low-IoU hard examples dominating L2 more than L1.Figure 11 contrasts the relative contributions of yellow, green, and red prediction boxes under the two losses.
  • Lp-norm-based losses: Smooth L1 reduces outlier effects relative to L2 and is more stable for small errors than L1, while Balanced L1 increases inlier gradient contributions.Balanced L1 is motivated by the negative effect of outlier gradients on learning inliers with smaller gradients.
  • IoU-based losses: IoU-based losses directly optimize overlap-related objectives, while GIoU adds an enclosing-box term to provide non-zero gradients when boxes do not overlap.GIoU preserves IoU’s advantages and is reported to perform better than using IoU directly as a loss.
  • IoU-based losses: DIoU adds a center-distance penalty to 1 − IoU, and CIoU further adds an aspect-ratio inconsistency penalty.The enclosing-box diagonal normalizes the center-distance term, while CIoU extends the formulation with aspect-ratio information.

6.2 IoU Distribution Imbalance

IoU distribution imbalance arises because positive anchors are skewed toward lower IoUs, and regression affects anchors differently according to their starting IoU. The review reports degradation and false-positive patterns that motivate balancing or selectively avoiding regression.

  • Definition and observation: Positive anchors in RetinaNet have an IoU distribution skewed toward lower IoUs before regression.The review notes that the same imbalance has also been observed in two-stage detectors.
  • Regression effects: Regression degrades fewer anchors as their starting IoU approaches the regressor’s training threshold, while false positives increase toward the 0.5–0.6 IoU bin.Figure 12 summarizes these changes using before-regression and after-regression IoUs.
  • Regression effects: Around 5% of positive anchors are lost by regression in the 0.5–0.6 IoU bin.The reported loss refers to anchors becoming false positives after regression.
  • Regression effects: For starting IoUs of 0.8–0.9 and 0.9–1.0, average IoU error is lower without applying regression.The review therefore identifies unregressed anchors as preferable for these high-IoU intervals in the converged model examined.
  • Solutions: IoU-uniform R-CNN generates approximately uniform positive inputs for the regressor, whereas pRoI Generator trains both branches with generated RoIs.The reported performance improvement of pRoI Generator is not significant, possibly because of the different branch usage.

6.3 Object Location Imbalance

Object location imbalance arises because objects are non-uniformly distributed while dense anchors treat image regions equally. The review surveys location-related solutions and identifies unresolved spatial sampling and orientation questions.

  • Definition: Objects are non-uniformly distributed across images, whereas densely sampled anchors typically assign equal importance to every image region.This mismatch motivates location-aware anchor design.
  • Comparative Summary: Addressing spatial imbalance generally improves detection, with location-bias removal benefiting classification in addition to regression-focused gains.Regression-loss and IoU-distribution methods improve especially the regressor branch, while removing anchor-location bias also improves classification.
  • Comparative Summary: Cascaded structures regulate IoU distributions and improve detection, with Cascade R-CNN reporting an 18.2% relative gain over its Faster R-CNN+FPN baseline.The cited comparison reports 36.2 versus 42.8 on MS-COCO testdev.
  • Solutions: Guided anchoring increases average recall by 9.1% while using 90% fewer anchors.It learns anchor parameters to address location-related imbalance and reduce anchor count.
  • Open Issues: Designing optimal high-recall anchor sets has received limited attention despite recent progress in learning anchor scales and aspect ratios.The review identifies imbalance in object locations and scales as an open issue.
  • Open Issues: The effects of imbalance from overlapping-box sampling and object orientation distributions remain insufficiently explored.Some image regions may be over- or under-sampled, and detectors may overfit typical object orientations.

7 IMBALANCE 4: OBJECTIVE IMBALANCE

Objective imbalance occurs when classification and regression losses differ in gradients, ranges, optima, or learning pace. The review examines weighting and task-coupling strategies while emphasizing that task interdependence and architecture-general balancing remain open problems.

  • Definition: Objective imbalance arises because multi-task classification and regression losses can differ in gradient norms and value ranges, allowing one task to dominate training.The review illustrates classification dominating the overall gradient at initialization.
  • Solutions: Task weighting balances loss terms with additional hyperparameters, but increasing task count expands the weighting search space.Weights are commonly selected using a validation set.
  • Solutions: Loss-range differences complicate balancing, motivating strategies that first make the ranges of different objectives comparable.The review gives smooth L1, LAP, and GIoU as examples of differing ranges.
  • Solutions: CARL scales regression loss using classification confidence, allowing regression gradients to reach the classification branch and improving localization of high-quality examples.Its design assumes classification and regression are correlated.
  • Open Issues: No objective-imbalance method beyond linear weighting is suitable for all architectures, and inappropriate task weights can cause training divergence.The literature also lacks an in-depth analysis of objective imbalance.
  • Comparative Summary: CARL achieves a 1.6% relative improvement over prime sampling without CARL, from 37.9 to 38.5.Guided Loss instead dynamically weights classification and matches baseline RetinaNet performance without Focal Loss.
  • Open Issues: Task losses are interdependent and may learn at different speeds, so linear weighting may not fully address imbalance in values, gradients, or learning pace.The review reports classification loss decreasing faster than regression loss in RPN training and calls for further analysis.

8 IMBALANCE PROBLEMS IN OTHER DOMAINS

The review connects object-detection imbalance to methods from image classification, metric learning, and multi-task learning. It finds substantial transferable ideas, but notes that several approaches remain underused in object detection.

  • Image Classification: Image-classification strategies include resampling, transfer learning, loss weighting, hard-example selection, uncertainty-based training, and synthetic example generation.These approaches range from oversampling and undersampling to class-balanced loss, hard-example mining, curricular learning, and generative models.
  • Image Classification: Data-redundancy methods can balance training data while accelerating convergence by ignoring examples that contribute little during training.One study reported that around 10% of ImageNet and CIFAR-10 data was redundant during training.
  • Image Classification: Image-classification methods address foreground-foreground class imbalance and may extend to foreground-background imbalance in object detection.Object detection generally handles foreground-background imbalance through under-sampling, while advanced resampling and transfer-learning methods have not yet been adopted from a class-imbalance perspective.
  • Comparative Summary: Object-detection literature contains far fewer generative approaches for imbalance than image-classification literature.The review also notes that weak supervision and reducing over-represented samples may alleviate class imbalance.
  • Metric Learning: Metric-learning approaches use sampling, generative methods, or novel loss functions to alleviate imbalance, paralleling foreground-background imbalance methods in object detection.The review examines sampling and generative strategies because metric learning has its own loss-function configuration.
  • Metric Learning: Semi-hard and adaptive-hardness sampling are not considered by object detectors, although selecting the hardest triplets can lead to local minima.This indicates a specific gap between metric-learning sampling strategies and current object-detection practice.
  • Multi-Task Learning: Multi-task learning proposes dynamically balancing task weights so classification and localization-related tasks are trained at a similar pace.The review states that this aspect of object detectors has received little attention from the community.

9 OPEN ISSUES FOR ALL IMBALANCE PROBLEMS

The review identifies open issues spanning all imbalance types, especially their interdependence, measurement, labeling, and applicability to bottom-up detectors. Small changes in bounding-box quality can alter several imbalance properties simultaneously.

  • Open Issues: A unified approach is needed to address all imbalance problems while considering their interdependence.The review illustrates that changing a bounding box can affect class, hardness, labeling, and objective-related imbalance simultaneously.
  • Open Issues: A small bounding-box shift can change an example from a hard positive to a hard negative, altering class counts and the criteria used by imbalance methods.The same shift can also change whether prime sample attention uses IoU or classification score for that example.
  • Measuring and Identifying Imbalance: Imbalance research still lacks agreed answers about balanced distributions, desired distributions, and quantitative imbalance measures.The review notes that balanced and desired distributions may differ, and that Wasserstein distance may neglect the effect of a unit distribution change.
  • Labeling a Bounding Box as Positive or Negative: Object detectors use inconsistent IoU thresholds for positive and negative labeling, with Fast R-CNN, RetinaNet, and RPN using 0.5, 0.4, and 0.3 as negative-label thresholds.The review argues that consensus matters because labeling affects whether examples are treated as hard examples.
  • Labeling a Bounding Box as Positive or Negative: Hard IoU thresholding can produce noisy positives that are simultaneously hard for one class and misleading for another.The bicycle example includes occlusion and another object, while the person example contains mostly black pixels and little descriptive content.
  • Bottom-Up Detectors: Bottom-up detectors may share top-down imbalance problems, but dedicated research is needed to establish findings for their architectures.The review specifically calls for analysis of imbalance behavior in bottom-up object detectors.

10 CONCLUSION

The paper organizes object-detection imbalance problems and their solutions through a taxonomy, then uses that framework to review existing work. It also identifies open issues and previously undiscussed imbalance problems, providing a map for future research.

  • Conclusion: The review introduces a taxonomy of object-detection imbalance problems and their solutions, presenting each problem through a unified and critical perspective.The taxonomy is intended to make the literature more complete and coherent.
  • Conclusion: The paper identifies open issues for studied imbalance problems and new imbalance issues that had not previously been discussed.These issues are presented as critical areas requiring further attention in object detection.
  • Conclusion: The review and taxonomy provide a map for identifying the field’s current position and future research directions.The stated scope is developing better solutions to imbalance problems in object detection.
Loading 1909.00169v3…