Source-linked AI summary
Learning non-maximum suppression
Jan Hosang, Rodrigo Benenson, Bernt Schiele
TL;DR
Object detectors increasingly integrate their stages end to end, but NMS remains a hand-crafted post-processing step whose fixed suppression rule creates a precision–recall trade-off. The paper introduces Gnet, a pure neural NMS network that jointly processes scored detections, and reports improvements over GreedyNMS, especially for occluded cases and localization. In its current form, Gnet requires large amounts of training data.
Problem
NMS remains a disconnected, hand-crafted detector component whose fixed suppression behavior can trade precision against recall, especially for nearby objects.
Method
Gnet is a pure NMS network that jointly updates detection representations through stacked neighbour-communication blocks using scored detections without image content.
Results
Gnet is reported as a suitable replacement for GreedyNMS in single- and multi-class setups, with improvements particularly for occluded cases and localization.
Takeaways & Limitations
The approach opens the possibility of detectors trained without a separate NMS post-processing step and, potentially, fully end-to-end detectors.
Takeaways & Limitations
In its current form, Gnet requires large amounts of training data.
Abstract
from arXiv · showhide
Object detectors have hugely profited from moving towards an end-to-end learning paradigm: proposals, features, and the classifier becoming one neural network improved results two-fold on general object detection. One indispensable component is non-maximum suppression (NMS), a post-processing algorithm responsible for merging all detections that belong to the same object. The de facto standard NMS algorithm is still fully hand-crafted, suspiciously simple, and -- being based on greedy clustering with a fixed distance threshold -- forces a trade-off between recall and precision. We propose a new network architecture designed to perform NMS, using only boxes and their score. We report experiments for person detection on PETS and for general object categories on the COCO dataset. Our approach shows promise providing improved localization and occlusion handling.
1. Introduction
NMS remains a hand-crafted, disconnected post-processing step in otherwise increasingly end-to-end object detectors. The proposed pure NMS network learns to rescore detections jointly, aiming to replace GreedyNMS and improve performance across datasets and occlusion conditions.
- NMS merges detections belonging to the same object after windows are proposed and scored.
- GreedyNMS greedily keeps high-scoring detections and deletes nearby lower-confidence neighbours using a fixed suppression rule.It is simple, fast, and competitive with alternatives.
- End-to-end detector training generally ignores NMS, which remains disconnected post-processing after proposal generation, feature extraction, and classification have been integrated.
- A fixed suppression width forces a precision–recall trade-off: wide suppression can remove true positives in crowded scenes, while narrow suppression can retain false positives.
- The proposed pure NMS network uses detections without image content or another algorithm's decisions and makes final GreedyNMS unnecessary.
- Experiments on PETS pedestrians and COCO multi-class detection report improvements over GreedyNMS, and the work aims to enable true end-to-end detectors.
2. Related work
Prior NMS research explores clustering, image evidence, object co-occurrence, rescoring, graphs, and end-to-end detection, but existing approaches retain limitations or post-processing requirements. The proposed approach instead operates purely on detections.
- Alternative NMS methods include mean-shift, agglomerative, affinity-propagation, heuristic, and globally optimal clustering formulations.The passage states that several alternatives have not shown consistent gains, while globally optimal methods had not yet established the stated outcome.
- Methods linking detections to pixels or semantic labels use image evidence, segmentation, or labelling, whereas this system operates only on detections.
- Pair-based detection methods target strong occlusion but create a more complex NMS problem because single and double detections must both be handled.
- Auto-context methods jointly rescore detections using image information and improve detection quality, but still require NMS; this approach eliminates post-processing.
- Graph-based detection methods can require an ill-defined node ordering, while this work avoids that preprocessing requirement.
- End-to-end alternatives either include non-learnable GreedyNMS during training or generate sparse detections with patch-boundary duplication risks.
3. Detection and non-maximum suppression
Independent window scoring produces multiple high-confidence detections for one object, so detectors need learned mechanisms that penalize duplicates and process neighbouring detections jointly. The paper studies a pure NMS network built around these ingredients.
- Object detection: Object detection maps an image to one tightly enclosing box per object, but confidence-based evaluation permits uncertainty in detections.
- Object detection: Hypothesize-and-score detectors estimate class probabilities independently for candidate windows, causing overlapping windows covering one object to receive similar high scores.
- A detector producing one high-scoring detection per object must condition each detection on neighbouring detections and process repeated detections jointly.
- The two necessary ingredients are a loss penalizing double detections and joint processing of neighbouring detections.
- The proposed architecture is evaluated in isolation as a pure NMS network operating solely on scored detections without image features.
4. Doing NMS with a convnet
The Gnet reformulates NMS as joint, learnable rescoring: detections interact through repeated blocks, while training labels derive from benchmark matching so duplicate detections are penalized.
- Rescoring formulation: Gnet replaces hard suppression with rescoring, lowering detections covering already-detected objects and leaving final evaluation without post-processing.After rescoring, thresholding can reduce detections, although evaluation uses the full rescored set.
- Loss: Training uses benchmark-style matching labels: matched detections are positives, while unmatched detections become negatives for binary classification.The matching proceeds in descending confidence order, and surplus detections count as false positives.
- Loss: The per-detection loss is coupled through matching, and class-conditional weights counteract the detection task’s extreme class imbalance.The weighting is chosen so the expected positive-class weight equals γ.
- Loss: For multiclass detection, Gnet matches only same-class detections and can switch detections off without changing their predicted class.The resulting classification problem remains binary and uses the same loss.
- Gnet architecture: Gnet jointly processes detections through repeated blocks that let each detection access neighbours, update its representation, and generate a new score.The network alternates neighbour communication with per-detection representation updates.
- Gnet architecture: Each block forms detection-pair features independently, then uses global max-pooling to convert variable-sized neighbourhoods into fixed-size representations.This pooled context is passed through fully connected layers to update each detection representation.
5. Experiments
Experiments on PETS and COCO evaluate whether Gnet can replace GreedyNMS, including person detection under different occlusion levels and multi-class detection. Gnet consistently improves over tuned GreedyNMS, especially for highly occluded cases.
- 5.1. PETS: Pedestrian detection in crowds: PETS contains crowded pedestrian sequences with diverse occlusion levels, and the study uses the same detections and splits as prior work.An initial GreedyNMS threshold of 0.8 reduces detections so joint rescoring fits on one GPU.
- 5.1. PETS: Pedestrian detection in crowds: Gnet improves 4.8 AP over the best GreedyNMS performance on PETS and slightly outperforms Tnet.Tnet recombines GreedyNMS outputs across multiple thresholds, whereas Gnet does not access GreedyNMS decisions.
- 5.1. PETS: Pedestrian detection in crowds: Gnet improves 7.3 AP over the best GreedyNMS performance for highly occluded PETS cases, with gains mainly attributable to highly occluded examples.Performance for occlusion in [0, 0.5) is similar to overall performance.
- 5.2. COCO: Person detection: On COCO persons, Gnet gains approximately 1 AP point over GreedyNMS from two blocks onward, for both low- and high-occlusion cases.The analysis varies the number of blocks and reports AP0.95 0.5 on minival; one block performs on par with GreedyNMS.
- 5. Experiments: The experiments conclude that Gnet is a suitable replacement for GreedyNMS in single- and multi-class setups and provides improved localization, particularly under occlusion.The paper reports this conclusion with enough training data.
- 5.3. COCO multi-class: Across 80 COCO categories, Gnet improves on approximately 70 categories and reaches 24.3% mAP0.95 0.5 versus 23.5% for test-set tuned GreedyNMS.The same parameters and architecture are used for all categories, with per-class score vectors replacing scalar scores.
6. Conclusion
The Gnet is presented as a pure NMS network that can replace GreedyNMS in single- and multi-class detection, particularly improving occluded-case handling and localization. Its current limitation is a substantial training-data requirement.
- The approach identifies a loss penalizing double detections and joint detection processing as two necessary ingredients for learned NMS.
- The Gnet is a pure NMS network that performs NMS without image content or assistance from another algorithm.
- With enough training data, Gnet is a suitable replacement for GreedyNMS in both single- and multi-class setups.
- Gnet surpasses GreedyNMS particularly for occluded cases and provides improved localization.
- The current Gnet requires large amounts of training data, motivating data augmentation or synthetic-data pre-training.
B. Network details
The Gnet is trained directly for NMS and, at test time, maps detections to new scores without post-processing. Its pairwise context lets each detection update its representation using neighboring detections.
- Training and test architecture: During training, detections and object annotations produce matching labels for a logistic loss, while testing removes matching and loss layers.
- Training and test architecture: The training and test architectures are identical except for loss computation, and the test network has no post-processing.
- Pairwise detection context: Pairwise detection context combines detection features with pair features such as scores, overlap, and relative position.
- Pairwise detection context: Each detection accesses neighboring feature descriptors and updates its representation conditioned on those neighbors, enabling joint processing.
- Network architecture: The Gnet diagram uses 128-dimensional input and intermediate features and produces a scalar output.
C. Raw detections without post-processing
Raw detector outputs contain many duplicate and poorly localized detections, whereas unprocessed Gnet outputs make most alternatives nearly invisible through rescoring. The intended result is one clear high-scoring detection per person.
- Raw detector outputs contain severe numbers of superfluous detections, including repeated detections of people and poorly localized boxes.
- The Gnet only rescores detections, yet most alternatives become barely visible because their scores are very low.
- Unprocessed Gnet outputs typically show one clear high-scoring detection for each person.
- The Gnet changes a blobby score distribution into a peaky distribution with one high-scoring detection per object.
- Figure 10 displays raw detections without post-processing, with opacity selected by detection score.
D. Qualitative results
Qualitative comparisons evaluate Gnet and GreedyNMS on the same detections at 60% recall. Gnet suppresses false positives from partial or oversized detections and can increase the score of missed low-confidence objects.
- Both Gnet and GreedyNMS operate on the same detections and are shown at the same operating point of 60% recall.
- Gnet suppresses maxima that become high-scoring false positives under GreedyNMS, especially detections covering body parts or oversized boxes.
- Gnet provides an example of improved recall by increasing the score of an initially low-confidence object.
- Figures 11 and 12 present qualitative results for both detectors at 60% recall.
E. COCO persons mini-test results
On COCO minitest, increasing the number of Gnet blocks beyond one yields approximately a 1 AP-point gain over GreedyNMS. Figure 13 reports this trend separately for low- and high-occlusion cases, with variability shown across six runs.
- Two or more Gnet blocks provide an approximately 1 AP-point gain, while one block performs on par with GreedyNMS.The same trend was observed on the minival set.
- Figure 13 compares AP0.95 0.5 against the number of Gnet blocks for low- and high-occlusion cases.
- Error bars represent the standard deviation over six runs.
F. COCO multi-class results
The multi-class COCO minitest evaluation compares Gnet with GreedyNMS using per-class threshold tuning. Averaged across classes, Gnet achieves 24.3% mAP0.95 versus 23.5% for test-set-tuned GreedyNMS.
- 24.3% mAP0.95 is obtained by Gnet averaged across COCO classes.
- Gnet is compared against GreedyNMS after tuning the GreedyNMS threshold separately for each class.
- 23.5% is reported for a test-set-tuned GreedyNMS baseline.