Source-linked AI summary
Training Region-based Object Detectors with Online Hard Example Mining
Abhinav Shrivastava, Abhinav Gupta, Ross Girshick
TL;DR
Object-detector training must handle overwhelming numbers of easy background regions and costly sampling heuristics. OHEM selects diverse, high-loss regions online during SGD, simplifying region-based ConvNet training. It consistently improves detection accuracy and reaches 78.9% and 76.3% mAP on PASCAL VOC 2007 and 2012 with complementary additions.
Problem
Object detection training faces severe foreground–background imbalance and needs online hard-example selection without freezing deep ConvNet models during SGD.
Method
OHEM modifies SGD to sample diverse, high-loss candidate regions from region-based ConvNet detector mini-batches, using NMS to remove redundant overlapping regions.
Results
OHEM consistently improves detection accuracy, with combined additions reaching 78.9% mAP on PASCAL VOC 2007 and 76.3% mAP on PASCAL VOC 2012.
Takeaways & Limitations
OHEM eliminates several common training heuristics and hyperparameters while improving convergence and detection accuracy across standard benchmarks.
Takeaways & Limitations
OHEM adds 0.09s or 0.43s per training iteration and requires 1G or 2.3G more memory for VGGM or VGG16, respectively.
Abstract
from arXiv · showhide
The field of object detection has made significant advances riding on the wave of region-based ConvNets, but their training procedure still includes many heuristics and hyperparameters that are costly to tune. We present a simple yet surprisingly effective online hard example mining (OHEM) algorithm for training region-based ConvNet detectors. Our motivation is the same as it has always been -- detection datasets contain an overwhelming number of easy examples and a small number of hard examples. Automatic selection of these hard examples can make training more effective and efficient. OHEM is a simple and intuitive algorithm that eliminates several heuristics and hyperparameters in common use. But more importantly, it yields consistent and significant boosts in detection performance on benchmarks like PASCAL VOC 2007 and 2012. Its effectiveness increases as datasets become larger and more difficult, as demonstrated by the results on the MS COCO dataset. Moreover, combined with complementary advances in the field, OHEM leads to state-of-the-art results of 78.9% and 76.3% mAP on PASCAL VOC 2007 and 2012 respectively.
1. Introduction
Object detection training faces severe foreground–background imbalance and difficulty selecting useful examples online. OHEM addresses this by selecting diverse, high-loss regions during SGD, simplifying training and improving detection performance.
- Detection datasets can contain up to 100,000 background examples for every object example.
- Bootstrapping traditionally alternates between mining examples with a fixed model and training on the resulting active set.
- Purely online hard-example selection is needed because freezing deep ConvNet detectors during SGD would dramatically slow training.
- OHEM modifies SGD by sampling diverse, high-loss candidate regions from mini-batches containing thousands of examples.
- OHEM removes several common region-based ConvNet heuristics and hyperparameters.
- 78.9% and 76.3% mAP were achieved on PASCAL VOC 2007 and 2012 when OHEM was combined with complementary improvements.
2. Related work
Prior object detectors commonly used hard-example mining, but existing approaches were poorly matched to purely online SGD training of deep ConvNet detectors. OHEM extends online hard-example selection to region-based detection.
- Hard negative mining has long been used in object detection, especially with SVM-based detectors.
- Fast R-CNN and related detectors do not directly use these methods because they train deep ConvNets purely online with SGD.
- SVM hard-example mining alternates between optimizing a working set and updating it by removing easy examples and adding others.
- Non-SVM bootstrapping trains to convergence, harvests false positives from a larger dataset, and repeats training with them added.
- Concurrent work also selected hard examples by loss, but focused on image descriptors, triplet learning, or image classification rather than region-based detection.
3. Overview of Fast R-CNN
Fast R-CNN processes an image and object proposals through shared convolutional and RoI networks, producing classification and bounding-box outputs. Its training uses heuristic RoI sampling to address foreground–background imbalance, which OHEM can remove.
- Fast R-CNN takes an image and object-proposal regions as input, then processes them through convolutional and RoI network stages.
- During inference, RoI pooling converts each proposal into a fixed-length feature vector for fully connected layers.
- The network outputs a softmax distribution over object classes and background plus regressed bounding-box coordinates.
- FRCN trains with SGD using a classification log loss and a localization loss for each RoI.
- Mini-batches sample images first and then sample RoIs from each image, using N = 2 and B = 128 as effective settings.
- Foreground RoIs require IoU of at least 0.5, while background RoIs use the interval [bg lo, 0.5) with bg lo = 0.1.
- Removing or increasing the 1:3 foreground-to-background sampling ratio decreases accuracy by approximately 3 points mAP, whereas OHEM removes this hyperparameter without ill effect.
4. Our approach
OHEM adapts hard-example mining to online SGD by evaluating all RoIs, selecting the highest-loss examples, and updating only on those examples. Its implementation addresses redundant RoIs and avoids inefficient backward computation for discarded examples.
- Online hard example mining: OHEM modifies SGD by selecting training examples from the current loss distribution instead of using a heuristically sampled subset.The method exploits mini-batches containing thousands of candidate RoIs from one or two images.
- Online hard example mining: At each iteration, the RoI network evaluates all input RoIs, sorts them by loss, and selects the B/N highest-loss examples for training.Only the selected RoIs participate in the forward-backward update of the trainable network.
- Online hard example mining: Non-maximum suppression removes highly overlapping RoIs because co-located regions can have correlated losses and double-count the same feature-map area.The implementation uses a relaxed IoU threshold of 0.7.
- Online hard example mining: OHEM does not require a fixed foreground-background ratio because examples whose losses increase become more likely to be sampled.Depending on image content, a mini-batch may contain only background or only foreground RoIs.
- Implementation details: The proposed architecture uses a readonly RoI network to score all RoIs and a regular RoI network to compute gradients only for the selected hard examples.The readonly network processes all RoIs in a forward pass, while the regular network performs forward and backward passes on the selected subset.
- Implementation details: The architecture has similar memory usage to the alternative implementation but runs more than twice as fast under the stated settings.The reported configuration uses N = 2, approximately 4000 RoIs, and B = 128.
5. Analyzing online hard example mining
Experiments compare OHEM with heuristic sampling and all-RoI training on PASCAL VOC07. OHEM improves detection performance, achieves the lowest reported training loss, and adds measurable computational and memory overhead.
- Experimental setup: Experiments compare FRCN with OHEM against heuristic sampling and training on all available RoIs.The evaluations use VGGM and VGG16 architectures on PASCAL VOC07.
- Hyperparameters and robustness: 4.8 points: OHEM improves VGGM mAP over FRCN without the bg lo = 0.1 heuristic.Against FRCN using bg lo = 0.1, OHEM improves mAP by 2.4 points for VGGM.
- Hyperparameters and robustness: OHEM remains effective when training uses N = 1 image per batch, despite concerns that increased RoI correlation could destabilize gradients.The experiment was designed to test whether greater within-image correlation would break the method.
- Better optimization: OHEM achieves the lowest training loss among the evaluated FRCN training methods.Loss is measured as average loss over all RoIs on VOC07 trainval, independently of each method’s sampling procedure.
- Computational cost: 0.09s per iteration and 1G more memory: OHEM adds these costs for VGGM, while VGG16 requires 0.43s and 2.3G more memory.The measurements use an Nvidia Titan X GPU.
6. PASCAL VOC and MS COCO results
Experiments on PASCAL VOC and MS COCO show that OHEM consistently improves Fast R-CNN detection performance, with gains on larger and more challenging data.
- OHEM improves VOC07 mAP from 67.2% to 69.9% and from 70.0% to 74.6% with extra data.
- 4.1 points: OHEM raises VOC12 mAP from 65.7% to 69.8%.
- 3.5 points: with extra data, OHEM reaches 71.9% mAP versus 68.4% for FRCN on VOC12.
- 2.9 points: OHEM improves MS COCO AP from 19.7% to 22.6% on the standard evaluation metric.
- 6.6 points: OHEM boosts AP50 on MS COCO under the VOC overlap metric.
- 4.9 points: OHEM improves AP for medium-sized objects on the strict COCO AP metric.
7. Adding bells and whistles
OHEM remains complementary to multiscale testing and iterative bounding-box regression, producing stronger VOC and COCO results when combined with these additions.
- VOC 2007 and 2012 results: 78.9% mAP: with extra data, OHEM surpasses MR-CNN’s 78.2% mAP on VOC07.
- VOC 2007 and 2012 results: 76.3% mAP: with extra data, OHEM exceeds MR-CNN’s 73.9% mAP on VOC12.
- Ablation analysis: OHEM consistently yields higher mAP across variants using multiscale testing and iterative bounding-box regression.
- MS COCO results: 25.5% AP: training on the full MS COCO trainval set improves the method’s standard COCO performance.
8. Conclusion
The paper concludes that OHEM simplifies training region-based ConvNet detectors by automatically selecting hard examples and improves detection accuracy across standard benchmarks.
- OHEM eliminates several commonly used heuristics and hyperparameters through automatic hard-example selection.
- The experiments show better training convergence and consistent detection-accuracy improvements on standard benchmarks.
- OHEM produces state-of-the-art PASCAL VOC 2007 and 2012 results when combined with orthogonal additions.
- Although demonstrated with Fast R-CNN, OHEM can train any region-based ConvNet detector.