PaperlayerView exact revision on arXiv

Source-linked AI summary

Focal Loss for Dense Object Detection

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár

arXiv:1708.02002v2cs.CV

TL;DR

Dense one-stage object detectors are faster and simpler but had lagged behind two-stage methods because of extreme foreground-background imbalance. This paper introduces Focal Loss and RetinaNet, achieving state-of-the-art accuracy and speed while surpassing existing two-stage detectors.

  • Problem

    Extreme foreground-background imbalance makes dense detector training inefficient and can overwhelm learning with easy negatives.

  • Method

    The paper proposes Focal Loss, a dynamically scaled cross-entropy loss that down-weights easy examples and focuses training on hard negatives, and evaluates it with RetinaNet.

  • Results

    39.1 vs. 33.2 COCO AP gives RetinaNet a 5.9-point advantage over the closest one-stage competitor while remaining faster.

  • Takeaways & Limitations

    Focal Loss enables a fully convolutional one-stage detector to achieve state-of-the-art accuracy and speed.

  • Takeaways & Limitations

    Addressing the high-frame-rate regime requires special network design and is beyond the scope of this work.

Abstract

from arXiv · show

The highest accuracy object detectors to date are based on a two-stage approach popularized by R-CNN, where a classifier is applied to a sparse set of candidate object locations. In contrast, one-stage detectors that are applied over a regular, dense sampling of possible object locations have the potential to be faster and simpler, but have trailed the accuracy of two-stage detectors thus far. In this paper, we investigate why this is the case. We discover that the extreme foreground-background class imbalance encountered during training of dense detectors is the central cause. We propose to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples. Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training. To evaluate the effectiveness of our loss, we design and train a simple dense detector we call RetinaNet. Our results show that when trained with the focal loss, RetinaNet is able to match the speed of previous one-stage detectors while surpassing the accuracy of all existing state-of-the-art two-stage detectors. Code is at: https://github.com/facebookresearch/Detectron.

1. Introduction

The introduction identifies extreme foreground-background imbalance as the main obstacle preventing dense one-stage detectors from matching two-stage accuracy. It proposes Focal Loss, a dynamically scaled cross-entropy loss that down-weights easy examples, and introduces RetinaNet to demonstrate the approach.

  • Motivation: Two-stage detectors achieve top COCO accuracy by generating sparse candidate locations and classifying them in a second stage.Proposal mechanisms reduce candidates to roughly 1–2k locations, while sampling heuristics further address imbalance.
  • Problem: Class imbalance is identified as the main obstacle preventing one-stage detectors from achieving state-of-the-art accuracy.The paper targets this barrier to match the COCO AP of more complex two-stage detectors.
  • Motivation: One-stage detectors densely enumerate roughly 100k locations, leaving training dominated by easily classified background examples.Their regular sampling spans spatial positions, scales, and aspect ratios, making conventional sampling heuristics inefficient.
  • Method: Focal Loss dynamically scales cross entropy so its contribution decays toward zero as confidence in the correct class increases.This automatically down-weights easy examples and focuses training on hard examples.
  • Evaluation: RetinaNet is a simple one-stage detector using an in-network feature pyramid and anchor boxes to evaluate Focal Loss.Its best ResNet-101-FPN model achieves a COCO test-dev AP of 39.1 while running at 5 fps.

2. Related Work

Related work contrasts dense one-stage detectors with the dominant two-stage paradigm, highlighting one-stage speed advantages but lower accuracy. It also identifies extreme class imbalance as a central training problem and distinguishes focal loss from robust losses by down-weighting easy examples.

  • Two-stage Detectors: Two-stage detectors generate sparse candidate proposals before classifying foreground and background, becoming the dominant modern object-detection paradigm.Selective Search pioneered proposal generation, while R-CNN improved the second-stage classifier with a convolutional network.
  • One-stage Detectors: Modern one-stage detectors such as SSD and YOLO prioritize speed, but their accuracy trails two-stage methods; SSD has a 10-20% lower AP.OverFeat was among the first modern deep-network one-stage detectors, followed by renewed interest from SSD and YOLO.
  • One-stage Detectors: RetinaNet follows prior dense-detector designs, including RPN anchors and SSD/FPN feature pyramids, while attributing its top results to the novel loss rather than network innovations.The passage specifically emphasizes shared design elements with previous dense detectors.
  • Class Imbalance: Dense detectors inherit a severe class imbalance because they evaluate 10^4-10^5 candidate locations per image while only a few contain objects.The resulting easy negatives make training inefficient and can overwhelm the training signal in aggregate.
  • Robust Estimation: Unlike robust losses that down-weight hard outliers, focal loss addresses class imbalance by down-weighting easy inliers so their aggregate contribution remains small.This distinction targets the large number of easy examples rather than examples with large errors.

3. Focal Loss

Focal Loss addresses dense detectors’ extreme foreground–background imbalance by down-weighting well-classified examples while preserving emphasis on hard examples. It extends cross entropy with a tunable focusing factor and can be combined with class balancing.

  • Class-balanced baseline: α-balanced cross entropy weights positive and negative classes but serves as an experimental baseline for focal loss.The weighting factor uses α for class 1 and 1−α for class −1, with α set by inverse frequency or cross validation.
  • Motivation: Extreme foreground–background imbalance in one-stage detection causes easy negatives to overwhelm cross entropy and dominate the gradient.The imbalance can be as large as 1:1000, and many small losses from easy examples can overwhelm the rare class.
  • Focal Loss: Focal Loss multiplies cross entropy by (1 − pt)^γ, where γ ≥0 controls how strongly easy examples are down-weighted.Misclassified examples with small pt retain nearly their original loss, whereas the loss approaches zero as pt approaches 1.
  • Focal Loss: γ = 0 makes Focal Loss equivalent to cross entropy, while increasing γ smoothly strengthens the suppression of easy examples.With γ = 2, pt = 0.9 receives 100× lower loss than under cross entropy, while pt ≈0.968 receives 1000× lower loss.
  • Practical formulation: The experiments use an α-balanced focal-loss variant, which yields slightly improved accuracy over the non-α-balanced form.The implementation combines sigmoid computation with the loss for greater numerical stability, and alternative focal-loss forms can be equally effective.

4. RetinaNet Detector

RetinaNet is a unified one-stage detector built from a ResNet-FPN backbone with separate subnetworks for anchor classification and box regression. It applies focal loss to all dense anchors during training and uses straightforward top-scoring prediction filtering and non-maximum suppression at inference.

  • Architecture: RetinaNet combines a backbone feature map with separate convolutional subnetworks for object classification and bounding-box regression.The backbone processes the entire input image, while the two task-specific subnetworks operate on its feature maps.
  • Feature Pyramid Network Backbone: The detector uses a ResNet-based Feature Pyramid Network with pyramid levels P3 through P7 and C = 256 channels at every level.FPN supplies multi-scale features, with each pyramid level supporting detection at a different object scale.
  • Anchors: RetinaNet uses A = 9 translation-invariant anchors per pyramid level, combining three aspect ratios with three additional anchor sizes for denser scale coverage.Anchor areas range from 32^2 to 512^2 across pyramid levels P3 to P7.
  • Task-Specific Subnetworks: The classification subnet predicts object presence for each anchor and class, while the parallel regression subnet predicts 4A offsets per spatial location.Both are small FCNs attached to each FPN level, but the classification and regression subnetworks do not share parameters.
  • Training and Inference: Focal loss is applied to all ∼100k anchors in each sampled image, with γ = 2 working well in practice and robustness across γ ∈[0.5, 5].At inference, RetinaNet retains at most 1k top-scoring predictions per FPN level after a 0.05 confidence threshold, then applies non-maximum suppression with a threshold of 0.5.

5. Experiments

Experiments on COCO show that focal loss enables stable and substantially more accurate dense detection than cross entropy, balanced cross entropy, OHEM, and prior one-stage methods. RetinaNet also matches or surpasses two-stage accuracy while maintaining competitive inference speed.

  • Network Initialization: Initializing the final layer with object prior probability π = .01 enables effective learning, while standard cross entropy training diverges; ResNet-50 reaches 30.2 AP.This result is on COCO with RetinaNet and the stated initialization.
  • Balanced Cross Entropy: Setting α = .75 for balanced cross entropy improves average precision by 0.9 points.The comparison covers multiple α values in Table 1a.
  • Focal Loss: With γ = 2, focal loss improves AP by 2.9 points over α-balanced cross entropy by discounting easy, low-loss examples.Increasing γ produces progressively larger gains over cross entropy, while γ = 0 is equivalent to cross entropy.
  • Focal Loss: Using γ = 2.0 and α = .25 as the default is effective; α = .5 performs only 0.4 AP lower.The best α values range from .25 to .75, and changing γ has the larger effect.
  • Analysis of the Focal Loss: For γ = 2, focal loss concentrates the vast majority of negative loss on a small fraction of hard negatives while largely discounting easy negatives.The effect on positive examples is minor, whereas increasing γ substantially shifts negative-loss weight toward hard examples.
  • Online Hard Example Mining (OHEM): 36.0 AP versus 32.8 AP shows focal loss outperforming the best tested OHEM setting by 3.2 AP in one-stage detection.The best OHEM configuration uses batch size 128, no 1:3 ratio, and NMS threshold .5.
  • Speed versus Accuracy: 122 ms versus 172 ms shows RetinaNet-101-600 matching Faster R-CNN accuracy while running faster on an Nvidia M40 GPU.Larger image scales allow RetinaNet to surpass the accuracy of all two-stage approaches while remaining faster.
  • Speed versus Accuracy: 39.1 vs. 33.2 yields a 5.9-point AP gap over the closest existing one-stage competitor, DSSD, while RetinaNet is faster.The comparison uses RetinaNet-101-800 with scale jitter and 1.5× longer training.

6. Conclusion

The conclusion identifies class imbalance as the primary obstacle to one-stage detectors surpassing two-stage methods and proposes focal loss to focus learning on hard negative examples. It reports that a fully convolutional one-stage detector demonstrates the approach’s efficacy.

  • Class imbalance is identified as the primary obstacle preventing one-stage object detectors from surpassing top-performing two-stage methods.
  • Focal loss applies a modulating term to cross entropy to focus learning on hard negative examples.
  • The focal-loss approach is demonstrated with a fully convolutional one-stage detector and extensive experimental analysis.
  • Both focal-loss variants reduce the relative loss assigned to well-classified examples with x_t > 0.

Appendix A: Focal Loss*

Appendix A presents FL* as an alternate focal-loss formulation whose parameters control the loss curve and whose selected settings reduce loss on well-classified examples. FL* achieves nearly the same AP as FL on RetinaNet-50-600, while parameter sweeps show that reducing weights for well-classified examples is effective.

  • Alternate formulation: FL* provides an alternate focal-loss instantiation with similar properties and comparable results, indicating that the exact loss form is not crucial.The appendix describes FL* as a reasonable practical alternative to FL.
  • Alternate formulation: The parameters γ and β control FL*’s loss-curve steepness and shift.Selected settings are plotted alongside cross entropy and FL.
  • Loss behavior: FL* with selected parameters, like FL, diminishes the loss assigned to well-classified examples.Correct classification corresponds to x_t > 0 and p_t > .5 under the appendix’s formulation.
  • Results: Nearly the same AP was achieved when RetinaNet-50-600 was trained with FL* instead of FL under identical settings.The appendix concludes that FL* works well in practice.
  • Results: AP over 33.5 identified effective converged models across a wide range of γ and β settings, with α = .25 used throughout.Effective losses reduce weights for well-classified examples (x_t > 0).

Appendix B: Derivatives

Appendix B gives the derivatives of CE, FL, and FL* with respect to x and compares their behavior across selected settings. Unlike CE, effective FL and FL* derivatives become small as soon as x_t > 0, while all losses approach -1 or 0 for high-confidence predictions.

  • Appendix B: Derivatives: The appendix provides derivatives for CE, FL, and FL* with respect to x.Selected settings are plotted in Figure 6.
  • Appendix B: Derivatives: For all loss functions, the derivative tends to -1 or 0 for high-confidence predictions.This behavior is shown for the selected settings in Figure 6.
  • Appendix B: Derivatives: Unlike CE, effective FL and FL* derivatives become small as soon as x_t > 0.The comparison identifies early derivative suppression as a distinguishing behavior of the effective FL settings.
Loading 1708.02002v2…