Source-linked AI summary
Libra R-CNN: Towards Balanced Learning for Object Detection
Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, Dahua Lin
TL;DR
Object detector performance can be limited by imbalance in sampling, feature use, and objectives, despite advances in model architectures. Libra R-CNN revisits training and introduces three balancing components; on MS COCO, it achieves higher AP than FPN Faster R-CNN and RetinaNet.
Problem
Detector performance is limited by imbalance during training at the sample, feature, and objective levels.
Method
Libra R-CNN rebalances detector training with IoU-balanced sampling, a balanced feature pyramid, and balanced L1 loss.
Results
2.5 points higher Average Precision (AP) than FPN Faster R-CNN and 2.0 points higher AP than RetinaNet are achieved on MS COCO.
Takeaways & Limitations
Libra R-CNN generalizes to various backbones and improves both two-stage and single-stage detectors on MS COCO.
Abstract
from arXiv · showhide
Compared with model architectures, the training process, which is also crucial to the success of detectors, has received relatively less attention in object detection. In this work, we carefully revisit the standard training practice of detectors, and find that the detection performance is often limited by the imbalance during the training process, which generally consists in three levels - sample level, feature level, and objective level. To mitigate the adverse effects caused thereby, we propose Libra R-CNN, a simple but effective framework towards balanced learning for object detection. It integrates three novel components: IoU-balanced sampling, balanced feature pyramid, and balanced L1 loss, respectively for reducing the imbalance at sample, feature, and objective level. Benefitted from the overall balanced design, Libra R-CNN significantly improves the detection performance. Without bells and whistles, it achieves 2.5 points and 2.0 points higher Average Precision (AP) than FPN Faster R-CNN and RetinaNet respectively on MSCOCO.
1. Introduction
Libra R-CNN revisits detector training and identifies imbalance at the sample, feature, and objective levels as a limit on architecture utilization. It addresses these levels with three balanced-learning components and reports higher AP on MS COCO.
- Detector training depends on representative samples, fully utilized features, and an effective objective, but typical training is imbalanced across all three aspects.
- Sample-level imbalance occurs because random sampling is dominated by easy samples, while hard mining can be sensitive to noisy labels and costly.
- Libra R-CNN enforces balance through IoU-balanced sampling, a balanced feature pyramid, and balanced L1 loss at the sample, feature, and objective levels.
- 2.5 points higher Average Precision (AP) than FPN Faster R-CNN and 2.0 points higher AP than RetinaNet are achieved on MS COCO.
- The framework combines its three components and reports significant improvements over state-of-the-art single-stage and two-stage detectors on MS COCO.
2. Related Work
Related work addresses detector architectures and training imbalances involving samples, features, and objectives. Libra R-CNN is positioned against hard-sample mining, feature pyramids, and objective reweighting approaches.
- Two-stage and single-stage detector architectures have developed through methods including Faster R-CNN, FPN, and related enhancements.
- Libra R-CNN’s overview combines IoU-balanced sampling, a balanced feature pyramid, and balanced L1 loss to reduce imbalance at three training levels.
- OHEM selects hard samples but adds memory and speed costs and is sensitive to noisy labels, while focal loss generally brings little or no gain to two-stage detectors.
- FPN and PANet integrate multi-level features to enrich semantic information in shallow layers and low-level information in deep layers.
- Prior objective-level methods include loss weighting and IoU-related objectives aimed at improving multi-task balance or localization accuracy.
3. Methodology
Libra R-CNN addresses training imbalance at the sample, feature, and objective levels through an overall balanced design. Its components rebalance sampling, multi-level feature processing, and localization gradients.
- Libra R-CNN uses an overall balanced design to alleviate imbalance in detector training and exploit model architectures more fully.The framework targets imbalance across the training pipeline rather than changing the detector architecture.
- IoU-balanced sampling: More than 60% of hard negatives have IoU above 0.05, whereas random sampling provides only 30% samples above that threshold.This imbalance buries hard samples among many easy samples.
- IoU-balanced sampling: IoU-balanced sampling divides the IoU interval into K bins, distributes N negative samples equally across bins, and samples uniformly within each bin.The default K is 3, and higher-IoU samples consequently receive greater selection probability.
- Balanced feature pyramid: The balanced feature pyramid rescales, integrates, refines, and strengthens multi-level features using shared balanced semantic features.Reverse rescaling gives each resolution equal information from the others, and the procedure itself contains no parameters.
- Balanced L1 loss: The detector’s multi-task objective combines recognition and localization losses, with λ tuning their relative weight.Directly increasing localization weight can make training more sensitive to outliers because regression targets are unbounded.
- Balanced L1 loss: Balanced L1 loss promotes gradients from inliers while clipping outlier gradients at a maximum value of 1.0.Its factors independently control inlier-gradient promotion and the upper bound of regression errors.
4. Experiments
Experiments on MS COCO evaluate Libra R-CNN against detector baselines and isolate the effects of its three balancing components. The results show improvements in detection AP, proposal recall, and component-level ablations.
- Experimental Setup: Experiments use MS COCO with standard COCO-style AP metrics, including AP, AP50, AP75, and scale-specific APS, APM, and APL.Models are trained on train-2017, with ablations and final results reported on val-2017 and test-dev, respectively.
- Main Results: 2.5 points higher AP is achieved than FPN Faster R-CNN with Libra R-CNN using ResNet-50, while Libra RetinaNet achieves 37.8 AP and 2.0 points higher AP than RetinaNet.Libra R-CNN reaches 38.7 AP with ResNet-50; Libra RetinaNet integrates balanced feature pyramid and balanced L1 loss because RetinaNet has no sampling procedure.
- Main Results: Libra RPN improves proposal recall by 9.2 points in AR100, 6.9 points in AR300, and 5.4 points in AR1000 over RPN with ResNet-50.The reported gains indicate stronger proposal generation under balanced training.
- Ablation Experiments: IoU-balanced sampling adds 0.9 points of box AP over the ResNet-50 FPN Faster R-CNN baseline, selecting samples nearer regions of interest than random sampling.The complementary positive-sample balancing component alone provides only 0.2 points higher AP, while negative-sample results are close for K values of 2, 3, or 5.
- Ablation Experiments: Balanced feature pyramid raises box AP from 36.8 to 37.7, with consistent improvements across small, medium, and large object scales.A simple integration improves box AP by 0.4 points without extra convolutions, while Balanced PAFPN reaches 37.2 box AP, 0.9 points above PAFPN.
- Ablation Experiments: Balanced L1 loss raises box AP from 37.7 to 38.5, with most gains from AP75, which improves by 1.1 points over its corresponding baseline.With the best setting, balanced L1 loss reaches 37.2 AP, 1.3 points above the ResNet-50 FPN Faster R-CNN baseline; merely increasing loss weight improves results by 0.5 points before performance drops at weight 2.0.
5. Conclusion
Libra R-CNN addresses training-process imbalance through an overall balanced design and improves detection across backbones and detector types on MS COCO.
- Libra R-CNN balances detector training with IoU-balanced sampling, a balanced feature pyramid, and balanced L1 loss.These components target sample-, feature-, and objective-level imbalance, respectively.
- The framework produces significant improvements on the challenging MS COCO dataset.
- Libra R-CNN generalizes across various backbones for both two-stage and single-stage detectors.