Learning non-maximum suppression

1. Introduction

• Object detectors propose windows, score or refine them, then merge detections; current detectors train without accounting for NMS and apply GreedyNMS as disconnected post-processing.The paper frames learning NMS as a step toward true end-to-end detectors.Source: §1 · pp. 1–2
• GreedyNMS keeps high-scoring detections and deletes nearby lower-scoring ones, but its fixed suppression width forces a precision–recall trade-off for close objects.Wide suppression can remove true positives in crowded scenes, while narrow suppression can retain false positives.Source: §1 · pp. 1–2
• The paper proposes the first “pure NMS network,” which uses neither image content nor another algorithm’s decisions and makes final GreedyNMS unnecessary.The network is intended to adapt to the data distribution and could be incorporated into a detector.Source: §1 · p. 2
• The proposed network is reported to improve over GreedyNMS on both single-class PETS pedestrian detection and multi-class COCO.The introduction describes these results as promising but supplies no numerical comparisons.Source: §1 · p. 2

Clustering detections

• GreedyNMS has persisted from Viola&Jones through DPM and R-CNN [10] [21], while alternative clustering methods have not consistently improved it.Principled globally optimal formulations have also yet to surpass GreedyNMS; explored alternatives include mean-shift, agglomerative, affinity propagation [17], and heuristic methods [25].Source: §2 · p. 2

Linking detections to pixels

• Hough voting links detections to supporting image evidence, but its detectors have comparatively low overall performance.Other approaches combine detections with semantic labelling [5] or frame detection as labelling; these rely on image segmentation and labelling, unlike the proposed detections-only system.Source: §2 · p. 2

Co-occurrence

• Detecting object pairs can address strong occlusion, but complicates NMS by requiring single- and double-detection handling; crowd-density-based suppression is another approach.The proposed method uses neither image information nor pair-specific hand-crafted detection.Source: §2 · p. 2

Auto-context

• Image-informed local or global rescoring [31] reduces spread-out double detections and improves detection quality, but still requires NMS.The proposed approach also treats NMS as rescoring, but eliminates post-processing.Source: §2 · p. 2

Neural networks on graphs

• Graph-based detection methods [18] require preprocessing to define a node ordering, which is ill-defined for the proposed detection setting.Source: §2 · p. 2

End-to-end learning for detectors

• Earlier end-to-end approaches retain GreedyNMS or another final suppression step, whereas this network performs NMS without suppression alternatives or post-processing.Training with GreedyNMS [12] does not make it learnable; sparse LSTM detections [26] still need boundary post-processing, and a convnet combining GreedyNMS thresholds [13] retains GreedyNMS in the final decision.Source: §2 · p. 2

3. Detection and non-maximum suppression

• Objects can trigger several detections because highly overlapping windows receive high scores from nearly identical image content, although detection aims for one box per object.Detectors estimate class probabilities independently for candidate detections, which can vary in confidence according to how well each window covers the object.Source: §3 · p. 3
• GreedyNMS accepts the highest-scoring detection and rejects detections overlapping it beyond threshold ϑ, but crowded, occluded scenes create a trade-off between suppressing duplicates and retaining nearby objects.A suppression region wide enough to remove same-object detections may also remove detections of the next closest object.Source: §3 · p. 3

3.1. A future without NMS

• Exactly one detection per object requires penalizing double detections and jointly processing neighboring detections so the detector can recognize repeated detections of the same object.Independent image-window processing gives similar scores to overlapping detections, while typical classifier training encourages multiple high-scoring detections per object.Source: §3.1 · p. 3
• The proposed network design accommodates both ingredients and is studied in isolation as a pure NMS network operating on scored detections without image features.The study is intended to validate the ingredients and the network’s capacity to perform NMS without end-to-end learning with the detector.Source: §3.1 · p. 3

4. Doing NMS with a convnet

• Gnet reframes NMS as rescoring: it lowers scores for detections covering already-detected objects, then evaluates the full rescored set without post-processing.Simple thresholding can reduce the set when needed; the method avoids hard decisions that discard detections.Source: §4 · p. 3

4.1. Loss

• Training labels come from benchmark-style confidence-ordered matching, making matched detections positive and surplus unmatched detections negative examples.The resulting binary loss couples each detection’s score to the other detections through matching.Source: §4.1 · p. 4
• Class-imbalance weights balance positive examples; for multiple classes, detections match only same-class objects and can be switched off but not reassigned.One-hot score encoding and instance weights with uniform expected class-conditional weight account for mAP’s equal class weighting.Source: §4.1 · p. 4

4.2. “Chatty” windows

• Gnet stacks blocks that let each detection aggregate information from neighboring detections and update its representation, with learned pair features and pooling handling variable neighborhood sizes.Neighbors are selected using IoU > 0.2; global max-pooling converts the variable number of detection pairs into a fixed-size representation.Source: §4.2 · pp. 4–5

4.3. Remarks

• Gnet updates detections concurrently, uses only boxes and confidence scores, and is designed to jointly rescore detections with conditional dependence; its depth may require many training examples.The authors leave combining image or detector features with Gnet for future work and report that performance is relatively robust to parameter changes and improves with increasing depth.Source: §4.3 · p. 6

5.1. PETS: Pedestrian detection in crowds

• Gnet improves on the best GreedyNMS by 4.8 AP overall and 7.3 AP for highly occluded PETS cases, slightly outperforming Tnet.GreedyNMS’s fixed threshold trades recall against precision: raising it increases maximum recall but also admits more high-scoring false positives. PETS comparisons use the same detections across methods.Source: Figure 4 · p. 6, §5.1 · pp. 6–7, Figure 5 · p. 6

5.2. COCO: Person detection

• On COCO person minival, Gnet matches GreedyNMS with one block and gains about 1 AP from two blocks onward for both low- and high-occlusion cases.The authors report that deeper models reduce variance for highly occluded cases, while warning that too many parameters may eventually cause overfitting; minival results transfer well to minitest.Source: Figure 6 · p. 7, §5.2 · pp. 7–8, Table 1 · p. 7
• A 16-block Gnet processes an average of 67.3 person detections per image in 14 ms on a K40m GPU using unoptimised TensorFlow code.Source: §5.2 · p. 7

5.3. COCO multi-class

• 24.3% versus 23.5% mAP0.95: Gnet outperforms per-class tuned GreedyNMS across COCO’s 80 categories, improving on most classes without catastrophic failure.The multiclass model uses the same parameters and architecture as the person model, replacing the scalar score with a per-class score vector.Source: Figure 7 · p. 8, §5.3 · p. 8

6. Conclusion

• Detectors need a loss that penalizes double detections and joint processing of detections to support an NMS network.Source: §6 · p. 8
• Gnet is the first “pure” NMS network capable of performing NMS without image content or help from another algorithm.Source: §6 · p. 8
• Because Gnet can be incorporated into detectors and use image features, it could enable end-to-end detectors without post-processing.Source: §6 · p. 8
• With enough training data, Gnet is a suitable replacement for GreedyNMS in single- and multi-class setups, particularly improving occluded cases and localization.Source: §6 · p. 8
• In its current form, Gnet requires large amounts of training data; augmentation or synthetic-data pre-training may help, while image features could inform object counts.Source: §6 · p. 8

A. Supplementary material

• The supplementary material discusses training and test architectures, the detection context layer, raw and example detections, additional COCO person results, and per-class COCO results.Source: A. Supplementary material · p. 10

• During training, Gnet scores detections alone; fixed detection matching labels true and false positives against annotations, while logistic loss backpropagates through Gnet.Source: B. Network details · p. 11
• At test time, removing matching and loss leaves the network mapping detections to new scores.Source: B. Network details · p. 11
• Unlike detectors that define positives and negatives artificially during training and apply GreedyNMS at test, Gnet trains directly for the task and needs no test-time post-processing.Source: B. Network details · p. 11
• Each detection’s variable-size context combines its and its neighbors’ descriptors with pair features such as scores, overlap, and relative position, enabling neighbor-conditioned updates.Repeating the process amounts to joint processing of neighboring detections.Source: B. Network details · p. 11, Figure 9 · p. 11

C. Raw detections without post-processing

• Before Gnet rescoring, raw detections include many repeated detections of people and several poorly localized detections.The raw detections are those remaining after very low-scoring detections are discarded.Source: C. Raw detections without… · p. 12
• Without post-processing, Gnet rescoring leaves the detection count unchanged but gives each person one clear high-scoring detection, with most others barely visible.A few detections covering only a person’s upper body remain visible despite an existing successful detection; these are unproblematic if rare or sufficiently low-scoring.Source: C. Raw detections without… · p. 12
• Gnet intuitively transforms a “blobby” distribution of similar scores into a “peaky” one, with one high-scoring detection per object.Detection opacity is set directly to the detection score, without a manual transformation to make unwanted detections disappear perceptually.Source: C. Raw detections without… · p. 12, Figure 10 · p. 12

D. Qualitative results

• Gnet suppresses high-scoring false positives from detections on people’s parts or oversized boxes that GreedyNMS retains.The qualitative comparison uses the same detections for both methods at 60% recall.Source: D. Qualitative results · p. 13, Figure 11 · p. 13, Figure 12 · p. 14
• Gnet can improve recall by boosting a low-confidence object’s score so it clears a threshold that otherwise misses it.Figure 12 shows an example of this effect.Source: D. Qualitative results · p. 13, Figure 12 · p. 14

E. COCO persons mini-test results

• Two or more Gnet blocks provide a ∼1 AP point gain, while one block performs on par with GreedyNMS on minitest.Figure 13 plots AP0.95 0.5 versus block count for low and high occlusion; error bars show standard deviation over six runs.Source: E. COCO persons mini-test results · p. 15, Figure 13 · p. 15

F. COCO multi-class results

• Table 2 details the per-class improvement of multi-class Gnet over GreedyNMS after Gnet’s threshold is tuned per class.Results are on the COCO minitest set.Source: F. COCO multi-class results · p. 15, Table 2 · p. 16
• 24.3% mAP0.95 0.5 vs 23.5% for test-set-tuned GreedyNMS, averaged across classes on the COCO minitest set.Table 2 reports per-class AP0.95 0.5 and is sorted by improvement over GreedyNMS.Source: F. COCO multi-class results · p. 15, Table 2 · p. 16
• GreedyNMS’s threshold is selected optimally per class on the COCO minitest set for the Table 2 comparison.Source: Table 2 · p. 16