Source-linked AI summary
Soft-NMS -- Improving Object Detection With One Line of Code
Navaneeth Bodla, Bharat Singh, Rama Chellappa, Larry S. Davis
TL;DR
Hard-threshold NMS can suppress valid overlapping detections in object detection. Soft-NMS replaces zeroing with continuous overlap-based score decay, improving average precision across standard datasets and detectors without extra training. Its greedy, non-global rescoring remains a scope limitation.
Problem
Traditional NMS can miss an object when its detection overlaps a higher-scoring box beyond the predefined threshold.
Method
Soft-NMS decreases other detection scores as a continuous function of their overlap with the selected maximum-scoring box.
Results
Soft-NMS improves average precision across multiple overlap thresholds on standard datasets and detectors by changing only the NMS algorithm.
Takeaways & Limitations
Soft-NMS can be integrated into existing object detection pipelines without extra training and with the same O(N^2) complexity as traditional NMS.
Takeaways & Limitations
Soft-NMS remains greedy and does not find the globally optimal rescoring of detection boxes.
Abstract
from arXiv · showhide
Non-maximum suppression is an integral part of the object detection pipeline. First, it sorts all detection boxes on the basis of their scores. The detection box M with the maximum score is selected and all other detection boxes with a significant overlap (using a pre-defined threshold) with M are suppressed. This process is recursively applied on the remaining boxes. As per the design of the algorithm, if an object lies within the predefined overlap threshold, it leads to a miss. To this end, we propose Soft-NMS, an algorithm which decays the detection scores of all other objects as a continuous function of their overlap with M. Hence, no object is eliminated in this process. Soft-NMS obtains consistent improvements for the coco-style mAP metric on standard datasets like PASCAL VOC 2007 (1.7% for both R-FCN and Faster-RCNN) and MS-COCO (1.3% for R-FCN and 1.1% for Faster-RCNN) by just changing the NMS algorithm without any additional hyper-parameters. Using Deformable-RFCN, Soft-NMS improves state-of-the-art in object detection from 39.8% to 40.9% with a single model. Further, the computational complexity of Soft-NMS is the same as traditional NMS and hence it can be efficiently implemented. Since Soft-NMS does not require any extra training and is simple to implement, it can be easily integrated into any object detection pipeline. Code for Soft-NMS is publicly available on GitHub (http://bit.ly/2nJLNMu).
1. Introduction
Object detection pipelines use non-maximum suppression to remove redundant boxes, but hard suppression can miss objects with overlapping detections. Soft-NMS instead decreases scores according to overlap, improving detection performance without retraining or major pipeline changes.
- Object detection generates bounding boxes for specified categories and assigns classification scores, supporting applications including autonomous driving, indexing, and surveillance.
- Traditional NMS selects the highest-scoring box, removes it, and suppresses remaining boxes whose overlap exceeds threshold Nt.
- Hard suppression can set a neighboring detection's score to zero, causing a true object within the overlap threshold to be missed and reducing average precision.
- Soft-NMS decreases detection scores as an increasing function of overlap instead of setting them to zero.
- Soft-NMS uses continuous score decay so highly overlapping boxes receive lower scores while low-overlap boxes can retain their original scores.
- Soft-NMS improves average precision across multiple overlap thresholds and can be integrated without extra training or substantial implementation changes.
2. Related Work
NMS has long supported detection and related vision tasks, while alternative methods target narrower settings or require choices that do not transfer directly to generic object detection. Greedy NMS therefore remained favorable for generic detection metrics.
- NMS has been used for decades in edge detection, feature point detection, face detection, and object detection.
- In edge detection, NMS thins edges, while in feature point detection it performs local thresholding to obtain unique detections.
- Greedy NMS remained the best-performing approach under average precision for state-of-the-art detectors despite learning-based alternatives.
- Affinity-propagation alternatives select exemplars for object-class detection, but generic object detection evaluates all object instances across classes.
- Proposal subset optimization addresses salient-object detection, where finding salient objects differs from detecting all objects in generic detection.
3. Background
Modern object detectors generate region proposals and then classify and refine them, but multiple proposals can correspond to the same object. Class-wise NMS is applied afterward to produce final detections.
- A deep CNN generates region proposals through one branch and performs classification and regression through another branch using pooled RoI features.
- The proposal network assigns scores and regression offsets to multi-scale, multi-aspect-ratio anchor boxes, then selects the top K anchors.
- Classification and regression produce scores and refined positions for each proposal, but the network does not force unique RoIs for each object.
- Multiple proposals may correspond to one object, creating false positives from boxes beyond the first correct bounding box.
- NMS is applied independently to each class with a specified overlap threshold to obtain final detections.
4. Soft-NMS
Soft-NMS replaces NMS’s hard suppression threshold with overlap-dependent score decay, aiming to reduce misses at high evaluation overlaps while preserving practical efficiency.
- Motivation: Low NMS thresholds can suppress true detections near higher-scoring boxes, increasing miss-rate when evaluation requires high overlap.Evaluation averages precision across overlap thresholds from 0.5 to 0.95.
- Method: Soft-NMS revisits NMS pruning as a score-rescoring function rather than an irreversible removal rule.The method decays neighboring detection scores instead of suppressing every overlapping box.
- Method: The proposed linear rule leaves distant boxes unchanged and penalizes boxes more strongly as their overlap with M increases.Boxes above the threshold are decayed linearly according to overlap.
- Method: A continuous penalty function avoids abrupt ranking changes, with little penalty at low overlap and strong penalty as overlap approaches one.The Gaussian penalty function is introduced to satisfy these continuity and overlap-sensitivity requirements.
- Efficiency: Scores of all remaining detection boxes are updated at each iteration, yielding Soft-NMS complexity of O(N^2), the same as greedy NMS.The update step is O(N) per iteration, and pruning keeps the procedure practical.
- Limitation: Soft-NMS remains greedy and does not find the globally optimal re-scoring of detection boxes.Traditional NMS is presented as a discontinuous binary-weighting special case of the generalized framework.
5. Datasets and Evaluation
The evaluation uses PASCAL VOC and MS-COCO, with results reported for standard and deformable region-based detectors across benchmark test partitions.
- Datasets: Experiments use PASCAL VOC with 20 object categories and MS-COCO with 80 object categories.VOC 2007 test performance and MS-COCO minival sensitivity are evaluated.
- Datasets: MS-COCO results include a publicly available 5,000-image minival set and a 20,288-image test-dev partition.The minival set is used for sensitivity analysis, while test-dev results are also reported.
- Evaluation: Table 1 compares NMS and Soft-NMS on the MS-COCO test-dev set for R-FCN, Deformable-RFCN, and Faster-RCNN.Gaussian and linear weighting variants are identified as G and L, while MST denotes multi-scale testing.
- Evaluation: Table 2 reports COCO-style evaluation on the PASCAL VOC 2007 test set for off-the-shelf standard object detectors.The table compares detectors using traditional NMS with the proposed Soft-NMS method.
- Detectors: Evaluation covers Faster-RCNN, R-FCN, and Deformable-RFCN using publicly available or independently trained detector models.The detectors use a default NMS threshold of 0.3, with threshold variation included in sensitivity analysis.
6. Experiments
Experiments show that Soft-NMS improves detection performance across datasets, detectors, overlap thresholds, and object classes. Its gains are especially pronounced for higher-overlap localization and crowded-object categories, while remaining robust across parameter settings.
- Results: 1.3% and 1.1% mAP improvements are obtained on MS-COCO for R-FCN and Faster-RCNN, respectively, by changing only the NMS algorithm.On PASCAL VOC 2007, Soft-NMS improves performance by 1.7% for both detectors.
- Results: 3–6% AP improvements occur for herd animals such as zebra, giraffe, sheep, elephant, and horse, while gains are small for toaster, sports ball, and hair drier.The per-class comparison is reported for MS-COCO using the Gaussian weighting function.
- Sensitivity analysis: AP remains stable for Soft-NMS across σ values from 0.4 to 0.7, whereas traditional NMS is stable only between overlap thresholds 0.3 and 0.6.Soft-NMS is better than NMS across a broader range of tested settings.
- Sensitivity analysis: Soft-NMS outperforms traditional NMS across parameter settings, with approximately 2% better performance at higher overlap thresholds and no comparable drop at lower thresholds.A single Soft-NMS parameter setting performs well across multiple overlap thresholds, amplifying averaged performance gains.
- Precision versus recall: Soft-NMS improves precision at higher recall and at lower recall for higher overlap thresholds by rescoring neighboring boxes instead of suppressing them completely.Traditional NMS assigns zero scores to boxes exceeding its overlap threshold, causing missed detections.
- Qualitative results: Soft-NMS recovers true positives above a detection threshold of 0.45 in qualitative COCO-validation examples.The examples use R-FCN detections and illustrate cases involving false positives with small or low overlap with good detections.