Source-linked AI summary
IoU-balanced Loss Functions for Single-stage Object Detection
Shengkai Wu, Jinrong Yang, Xinggang Wang, Xiaoping Li
TL;DR
Standard classification and localization losses can undermine localization accuracy in single-stage detectors. The paper proposes IoU-balanced losses that reweight examples according to IoU, and reports substantial performance gains, especially in localization, across several datasets.
Problem
Standard cross-entropy classification is independent of localization and can yield classification scores mismatched with localization accuracy.
Source: §1 · p. 2Method
The paper proposes IoU-balanced classification and localization losses that weight examples according to IoU to strengthen task correlation and suppress low-IoU localization gradients.
Source: §1 · pp. 3–4Results
Experiments on MS COCO, PASCAL VOC, and Cityscapes show substantial performance improvements for single-stage detectors, especially in localization.
Source: §5 · pp. 18–19Takeaways & Limitations
IoU-balanced losses improve single-stage detector performance without changing the detector architecture, according to the paper's reported conclusion.
Source: §5 · pp. 18–19
Abstract
from arXiv · showhide
Single-stage object detectors have been widely applied in computer vision applications due to their high efficiency. However, we find that the loss functions adopted by single-stage object detectors hurt the localization accuracy seriously. Firstly, the standard cross-entropy loss for classification is independent of the localization task and drives all the positive examples to learn as high classification scores as possible regardless of localization accuracy during training. As a result, there will be many detections that have high classification scores but low IoU or detections that have low classification scores but high IoU. Secondly, for the standard smooth L1 loss, the gradient is dominated by the outliers that have poor localization accuracy during training. The above two problems will decrease the localization accuracy of single-stage detectors. In this work, IoU-balanced loss functions that consist of IoU-balanced classification loss and IoU-balanced localization loss are proposed to solve the above problems. The IoU-balanced classification loss pays more attention to positive examples with high IoU and can enhance the correlation between classification and localization tasks. The IoU-balanced localization loss decreases the gradient of examples with low IoU and increases the gradient of examples with high IoU, which can improve the localization accuracy of models. Extensive experiments on challenging public datasets such as MS COCO, PASCAL VOC and Cityscapes demonstrate that both IoU-balanced losses can bring substantial improvement for the popular single-stage detectors, especially for the localization accuracy. On COCO test-dev, the proposed methods can substantially improve AP by $1.0\%\sim1.7\%$ and AP75 by $1.0\%\sim2.4\%$. On PASCAL VOC, it can also substantially improve AP by $1.3\%\sim1.5\%$ and AP80, AP90 by $1.6\%\sim3.9\%$.
1. Introduction
The introduction argues that standard classification and localization losses undermine localization accuracy, then proposes IoU-balanced losses to address both problems. Experiments are reported to improve single-stage detectors, especially localization, without sacrificing efficiency.
1. Introduction
- Standard cross-entropy can produce classification scores mismatched with localization accuracy, causing poorly localized detections to outrank better-localized ones.The mismatch can affect both non-maximum suppression and average-precision ranking.Source: §1 · pp. 2–3
- Smooth L1 bounds outlier gradients, but poorly localized examples can still dominate localization training and impede accurate localization.The authors propose further suppressing outlier gradients while increasing those of inliers.Source: §1 · pp. 3–4
- Across MS COCO, PASCAL VOC, and Cityscapes, IoU-balanced losses substantially improve single-stage detectors, especially localization, without sacrificing efficiency.The introduction also reports improvements for multi-stage detectors, but smaller ones.Source: §1 · p. 3, Figure 1 · p. 4
- IoU-balanced classification loss gives more weight to high-IoU positive examples, while localization loss up-weights high-IoU examples and suppresses low-IoU examples.The classification weighting is intended to strengthen the correlation between classification and localization tasks.Source: §1 · pp. 3–4
2. Related Work
Related work improves localization through architectural refinements, example selection, or stronger alignment between classification scores and localization quality. The proposed losses instead reweight examples by IoU without changing sampling or adding model branches.
Accurate object localization
- Localization methods such as Cascade R-CNN and RefineDet add stages or modules, whereas IoU-balanced losses aim to improve localization without changing model architecture.Multi-region detector [15] also uses iterative box refinement.Source: §2 · p. 5
Hard example mining
- Hard-example mining methods select or reweight examples through sampling rules, loss rankings, or gradient density; IoU-balanced losses instead weight positive examples by IoU without changing sampling.The related-work discussion notes that GHM-R's gradient-density computation can slow training.Source: §2 · pp. 5–7
Correlation between classification and localization task
- Fitness NMS [18], IoU-Net, IoU-aware RetinaNet, and MS R-CNN estimate localization quality for detection scoring, while IoU-balanced classification weights positives using their IoU directly.The related-work passage says this weighting does not sacrifice efficiency.Source: §2 · p. 6
Outliers during training localization subnetwork
- Smooth L1, GHM-R, and balanced L1 address outlier gradients differently; IoU-balanced localization instead weights positive examples by IoU.It up-weights high-IoU examples and down-weights low-IoU examples.Source: §2 · pp. 6–7
3. Method
The method reweights classification examples by regressed IoU and adjusts localization gradients to emphasize better-localized examples. It uses an upper bound of the IoU function to shape the localization loss.
3.1. Preliminaries
- Standard cross-entropy weights positive examples equally, while smooth L1 gradients from numerous poorly localized examples can dominate localization training.The method addresses these issues by adapting example weights according to localization accuracy.Source: §3.1 · p. 7
3.2. IoU-balanced Classification Loss
- Classification weights for positive examples increase with regressed IoU, up-weighting high-IoU examples and suppressing low-IoU examples.The parameter η controls the degree of reweighting, and normalization keeps the sum of positive-example classification loss unchanged from standard cross-entropy.Source: §3.2 · p. 8
3.3. IoU-balanced Localization Loss
- The localization loss uses an upper bound of the IoU function, whose relationship with coordinate differences is otherwise complex.The paper adopts the Bounded IoU idea to obtain this upper bound.Source: §3.3 · pp. 9–10
- The IoU-balanced localization loss increases gradient norms for inliers and reduces them for outliers relative to standard smooth L1 loss.The parameter λ controls this emphasis, while wloc is manually adjusted to keep the initial sum of localization loss unchanged.Source: §3.3 · pp. 9–10, Figure 2 · p. 11
4. Experiments
Experiments compare IoU-balanced losses with baselines across single-stage and two-stage detectors and multiple datasets. Results show gains across settings, particularly at higher IoU thresholds.
4.1. Experimental Settings
- Across COCO, PASCAL VOC, and Cityscapes, experiments evaluate anchor-based, anchor-free, and two-stage detectors using standard AP metrics and controlled baseline comparisons.For the main comparisons, only the loss functions are changed; the paper also specifies dataset splits, training schedules, image scales, and COCO-style metrics.Source: §4.1 · pp. 11–12
4.2. Main Results
- On COCO test-dev, IoU-balanced losses improve AP by 1.1% for RetinaNet, 1.7% for SSD300, 1.3% for SSD512, and 1.0% for FoveaBox; AP75 gains are 1.0%–2.4%.For RetinaNets with the same backbone, the method also surpasses Faster R-CNN by 0.4%–0.8% on AP and 0.2%–0.8% on AP75.Source: §4.2 · pp. 12–13, Table 1 · p. 13
4.3. Analysis
- On COCO val-2017, classification and localization losses improve AP by 0.7% and 0.8%, respectively, and combining them improves AP by 1.3%.Localization loss decreases AP50 by 0.2% while improving AP80 and AP90 by 1.8% and 1.9%, respectively.Source: §4.3 · pp. 13–14, Table 2 · p. 14
- On COCO val-2017, the best reported AP is 35.1% at η = 1.5 for classification loss and 35.2% at λ = 1.5 for localization loss.Classification loss raises average scores for high-IoU examples by 0.5%–2.1% and lowers scores for low-IoU examples by 2.7%–4.6%; localization loss raises the high-IoU detection percentage by 0.8%–4.8%.Source: §4.3 · pp. 14–15, Table 3 · p. 14, Figure 3 · p. 15
- Across RetinaNet backbones on COCO val-2017, IoU-balanced losses improve AP by 1.1%–1.4%, with AP80 and AP90 gains of 1.6%–2.6%.The reported AP50 and AP60 gains are 0.1%–0.8%.Source: Table 4 · p. 15, §4.3 · p. 16
- Visual examples show higher scores for well-localized detections, lower scores for poorly localized detections, and improved localization in selected cases.The paper illustrates these changes using baseline and IoU-balanced RetinaNet-ResNet50 detections on COCO val-2017 images.Source: §4.3 · p. 16, Figure 4 · p. 16
4.4. Generalization to Other Datasets
- On PASCAL VOC, IoU-balanced losses improve AP by 1.3%–1.5% for SSD and RetinaNet; on Cityscapes, they improve RetinaNet AP by 1.0%–1.2% across backbones.PASCAL VOC AP80 and AP90 gains are 1.6%–3.9%.Source: §4.4 · p. 17, Table 5 · p. 17, Table 6 · p. 18
4.5. Discussion
- For Faster R-CNN on COCO val-2017, classification and localization losses improve AP by 0.4% and 0.6%, respectively, but combining them adds no further improvement.The paper attributes the smaller gains versus single-stage detectors to more accurate RPN proposals, which alleviate score-localization mismatch and localization outliers.Source: §4.5 · pp. 17–18, Table 7 · p. 18
5. Conclusions
IoU-balanced classification and localization losses are proposed to improve single-stage detectors’ localization accuracy. Experiments across MS COCO, PASCAL VOC, and Cityscapes show substantial performance improvements, especially in localization accuracy.
5. Conclusions
- The proposed IoU-balanced loss functions target localization accuracy through separate classification and localization losses.Source: §5 · p. 18
- IoU-balanced classification loss enhances correlation between classification and localization tasks.Source: §5 · p. 18
- IoU-balanced localization loss decreases outlier gradient norms while increasing inlier gradient norms.Source: §5 · p. 18
- Experiments on MS COCO, PASCAL VOC, and Cityscapes show substantial improvement for single-stage detectors, especially in localization accuracy.Source: §5 · p. 19