PaperlayerView exact revision on arXiv

Source-linked AI summary

GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection

Abhinav Kumar, Garrick Brazil, Xiaoming Liu

arXiv:2103.17202v1cs.CVcs.LG

TL;DR

Monocular 3D detectors typically use NMS only at inference, leaving training losses on boxes before suppression. GrooMeD-NMS makes NMS differentiable through a matrix formulation with unsupervised grouping and masking. It achieves state-of-the-art monocular 3D detection on KITTI and performs competitively with a video-based method.

  • Problem

    Most 3D detectors apply NMS only at inference, so losses are applied to boxes before NMS rather than the final boxes after it.

    Source: §1 · p. 1

  • Method

    GrooMeD-NMS formulates NMS as a matrix operation and uses unsupervised grouping and masking to obtain a closed-form differentiable operation for end-to-end training.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    GrooMeD-NMS achieves state-of-the-art monocular 3D detection on KITTI and performs competitively with video-based methods.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The method trains with loss on post-NMS boxes and is described as fairly generic for other object detection tasks.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Modern 3D object detectors have immensely benefited from the end-to-end learning idea. However, most of them use a post-processing algorithm called Non-Maximal Suppression (NMS) only during inference. While there were attempts to include NMS in the training pipeline for tasks such as 2D object detection, they have been less widely adopted due to a non-mathematical expression of the NMS. In this paper, we present and integrate GrooMeD-NMS -- a novel Grouped Mathematically Differentiable NMS for monocular 3D object detection, such that the network is trained end-to-end with a loss on the boxes after NMS. We first formulate NMS as a matrix operation and then group and mask the boxes in an unsupervised manner to obtain a simple closed-form expression of the NMS. GrooMeD-NMS addresses the mismatch between training and inference pipelines and, therefore, forces the network to select the best 3D box in a differentiable manner. As a result, GrooMeD-NMS achieves state-of-the-art monocular 3D object detection results on the KITTI benchmark dataset performing comparably to monocular video-based methods. Code and models at https://github.com/abhi1kumar/groomed_nms

1. Introduction

Monocular 3D detectors apply NMS only at inference, so training losses do not reflect the final post-NMS boxes. GrooMeD-NMS brings a differentiable, closed-form NMS into end-to-end training to address this mismatch.

1. Introduction

  • NMS is applied after training, leaving losses on pre-NMS boxes and creating a mismatch with the final predictions used at inference.The paper also reports a greater mismatch between classification and 3D localization than between classification and 2D localization.Source: §1 · pp. 1–2
  • Earlier efforts to include NMS in training focused on 2D detection, while 3D methods improving score-localization correlation remained non-end-to-end without differentiable NMS.The paper says the improvements from 2D training-time NMS were less visible and those from 3D correlation methods were limited.Source: §1 · p. 2
  • GrooMeD-NMS expresses NMS as a matrix operation, then uses soft pruning, unsupervised grouping, and masking to obtain a parallelizable closed-form operation.Grouping and masking remove the matrix inversion from the expression.Source: §1 · p. 2
  • Training with NMS gives the network gradients that encourage better-localized 3D boxes to rank higher.The method is designed to address both the training-inference mismatch and the mismatch between classification and 3D localization.Source: §1 · p. 2
  • GrooMeD-NMS achieves state-of-the-art monocular 3D detection on KITTI and performs comparably to monocular video-based methods.The paper identifies end-to-end training with loss on post-NMS boxes and unsupervised grouping and masking as its contributions.Source: §1 · p. 2

2. Related Work

Prior monocular 3D detection methods explored architectures, losses, confidence, augmentation, depth features, and temporal cues. The paper positions GrooMeD-NMS as a grouped, closed-form alternative to existing NMS approaches.

3D Object Detection

  • Monocular 3D detection infers 3D information from a single image, a setting described as ill-posed because of scale and depth ambiguity.Other approaches use sensors such as LiDAR, stereo, or radar; the passage notes limitations of LiDAR data and processing.Source: §2 · p. 2
  • Deep-learning monocular methods have varied architectures, losses, confidence modeling, augmentation, depth convolution, and temporal cues.The paper proposes incorporating NMS into monocular 3D detection training.Source: §2 · p. 2

Non-Maximal Suppression

  • Prior NMS methods span inference, optimization, and neural-network approaches, while GrooMeD-NMS uses a grouped closed-form approximation without multiple layers or message passing.The paper distinguishes its matrix-based formulation from the approaches reviewed in this section.Source: §2 · p. 2

3. Background

The background defines the boxes, scores, overlaps, groups, and pruning operation used to describe NMS. Classical and Soft-NMS share a greedy rescoring procedure and differ in whether pruning is hard or soft.

3.1. Notations

  • The notation defines boxes and their pre- and post-NMS scores, pairwise 2D IoUs, and a pruning function that rescores boxes from their overlaps.The set of retained boxes is denoted D; the IoU matrix is O.Source: §3.1 · p. 2

3.2. Classical and Soft-NMS

  • Classical and Soft-NMS greedily select a box index and update scores, using scores, the IoU matrix, and a pruning function as inputs.The algorithms also specify an NMS threshold and return the selected box index and post-NMS scores.Source: §3.1 · p. 3, §3.2 · p. 3

3.1. Notations

  • Groups partition the boxes, with α denoting the maximum group size; clipping, element-wise multiplication, and the identity matrix are also specified.The groups are disjoint subsets of the box set.Source: §3.1 · p. 3

3.2. Classical and Soft-NMS

  • Classical NMS hard-prunes boxes with high 2D IoU against selected boxes, whereas Soft-NMS makes pruning soft using temperature τ.The pruning function determines suppression from the 2D overlaps; the two procedures otherwise differ in their choice of pruning function.Source: §3.2 · p. 3

4. GrooMeD-NMS

GrooMeD-NMS replaces greedy, non-differentiable NMS with a matrix-based formulation that can be used during training. Sorting, grouping, masking, and soft pruning produce a simpler expression and useful gradients.

4. GrooMeD-NMS

  • Classical NMS is greedy and non-differentiable, motivating a smooth closed-form approximation for use in training.The method hard-sorts scores and overlaps; attempted soft sorting was too temperature-dependent and had unreliable gradients for training.Source: §4 · p. 3, §4.1.1 · pp. 3–4

4.1.1 Sorting

  • The method hard-sorts scores and permutes the overlap matrix accordingly, retaining hard sorting after soft sorting proved unreliable for training.The implemented soft-sorting approach was overly dependent on temperature τ to break out the ranks.Source: §4.1.1 · pp. 3–4

4.1.2 NMS as a Matrix Operation

  • The matrix formulation generalizes rescoring to account for suppressed and unsuppressed boxes, using a relaxed logical-OR operation to avoid recursion.The formulation uses a lower-triangular matrix with ones on its diagonal, which is always invertible.Source: §4.1.2 · p. 4

4.1.3 Grouping

  • Unsupervised grouping clusters boxes by 2D IoU overlap, limiting interactions to boxes within each group.Groups are formed around top-ranked boxes, with a maximum size α.Source: §4.1.3 · p. 4

4.1.4 Masking

  • Masking keeps only the top-scored box's column in each group, simplifying the matrix expression and bypassing matrix inversion.The resulting expression is the basis of GrooMeD-NMS.Source: §4.1.4 · p. 4

4.1.5 Pruning Function

  • The pruning function determines whether boxes are suppressed from their 2D IoU overlaps; the authors compare Linear, Exponential, and Sigmoidal choices.Their ablations find Linear pruning simplest and best; Exponential and Sigmoidal perform less well, possibly because of vanishing gradients near overlap zero or one.Source: §4.1.5 · p. 5

4.2. Differences from Existing NMS

GrooMeD-NMS differs from prior NMS methods by providing a matrix-based closed-form expression and being used in both training and inference. Its single-layer approach is parallelizable within groups, whose size is generally much smaller than the full box set.

4.2. Differences from Existing NMS

  • No method in Table 1 combines a matrix-based closed-form NMS expression with use during both training and inference.Classical, Soft-NMS [8], and Distance-NMS are inference-only; GrooMeD-NMS is used during training and inference.Source: §4.2 · p. 5, Table 1 · p. 5
  • Unlike neural-network-based NMS methods [31] [32], GrooMeD-NMS uses a single layer and requires neither multiple layers nor message passing.NN-NMS uses separate networks with multiple layers and/or message passing to approximate NMS.Source: §4.2 · p. 5
  • GrooMeD-NMS is parallelizable within groups, and the group size is generally much smaller than the full box set.Source: §4.2 · p. 5

4.3. Target Assignment and Loss Function

The method assigns positive targets to boxes using their 2D overlap and generalized 3D overlap with ground truth, then applies an image-wise AP loss after NMS. This loss supplements the pre-NMS losses and is designed to address the imbalance in best-box targets.

4.3. Target Assignment and Loss Function

  • The best box for a ground-truth object is selected by the product of IoU2D and gIoU3D, and is positive when that product exceeds threshold β.gIoU3D supplies a signal for non-intersecting boxes, for which ordinary IoU3D is zero.Source: §4.3 · p. 5
  • Because best-box targets are far rarer than foreground examples, the method uses modified AP-Loss after NMS to address class imbalance.Some ground-truth boxes may have no predicted box, leaving fewer best boxes than ground truths.Source: §4.3 · pp. 5–6
  • Imagewise AP-Loss ranks boxes separately per image, so gradients affect only images whose best boxes are not correctly ranked.The paper reports that this loss is better suited after NMS than classification loss.Source: §4.3 · p. 6
  • The overall objective adds the weighted post-NMS Imagewise AP-Loss to the losses computed before NMS.L = Lbefore + λLafter; the pre-NMS terms include classification, 2D and 3D regression, and confidence losses.Source: §4.3 · p. 6

5. Experiments

GrooMeD-NMS is evaluated on three KITTI splits, with results compared against image- and video-based detectors and alternative NMS methods. Across the reported evaluations, it performs strongly, while ablations examine the choices that shape the method.

5. Experiments

  • Experiments use KITTI and a Kinematic (Image) baseline for a fair comparison with the image-based GrooMeD-NMS method.The evaluation covers Test, Val 1, and Val 2 splits, focusing primarily on Cars and AP3D|R40 for the Moderate category.Source: §5 · p. 6

5.1. KITTI Test 3D Object Detection

  • GrooMeD-NMS outperforms M3D-RPN and several other methods on KITTI Test 3D and BEV evaluation, and beats Kinematic (Video) on the Hard set.It also performs competitively with Kinematic (Video) despite being image-based.Source: Table 2 · p. 6, §5.1 · p. 6

5.2. KITTI Val 1 3D Object Detection

  • On KITTI Val 1, GrooMeD-NMS is best in 6 of 12 cases and second-best in the rest, outperforming Kinematic (Image) at all reported depths and IoU3D thresholds.The largest improvements are reported on the Moderate and Hard sets, where objects are more distant and occluded.Source: §5.2 · p. 7, Table 3 · p. 7, Figure 3 · p. 7
  • Comparisons with classical NMS, Soft-NMS, and Distance-NMS indicate that training with GrooMeD-NMS benefits performance; replacing it with classical NMS at inference does not affect performance.The post-NMS score-IoU3D correlation is reported as 0.345, higher than the compared baselines.Source: §5.2 · p. 7, Table 4 · p. 7, Figure 4 · p. 7

5.3. KITTI Val 2 3D Object Detection

  • On KITTI Val 2, GrooMeD-NMS performs best in all reported cases, with improvements on the Moderate and Hard sets consistent with the Test and Val 1 results.Table 5 compares AP3D|R40 and APBEV|R40 on Val 2 Cars.Source: §5.1 · p. 6, §5.2 · p. 7, §5.3 · p. 8

5.4. Ablation Studies

  • Ablations find that grouping and masking help the model reach a better minimum, while Linear pruning performs better than Exponential and Sigmoidal pruning, which may have vanishing gradients near zero or one overlap.Imagewise AP loss performs better than Vanilla AP and BCE; combining self-balancing confidence and classification scores at inference is better than using either alone.Source: §5.4 · p. 8

6. Conclusions

GrooMeD-NMS integrates a differentiable NMS into end-to-end monocular 3D object detection, using a matrix formulation and unsupervised grouping and masking. The paper reports state-of-the-art results on KITTI.

  • GrooMeD-NMS trains end-to-end with a loss on boxes after NMS, addressing the mismatch between training and inference.It formulates NMS as a matrix operation and uses unsupervised grouping and masking to obtain a closed-form expression.Source: §6 · p. 8
  • On the KITTI benchmark, GrooMeD-NMS achieves state-of-the-art monocular 3D object detection results.The paper says the results are comparable to monocular video-based methods in its abstract, but that comparison is not stated in this passage.Source: §6 · p. 8

A1. Detailed Explanation of NMS as a Matrix Operation

The section recasts NMS rescoring as a matrix operation and approximates the recursive update to obtain a differentiable solution. The derivation uses sorted scores and a pruning matrix based on box overlaps.

  • Classical NMS rescoring is greedy and set-based, so the section generalizes the update to account for suppressed and unsuppressed boxes.The rescore for box i is described as a function of higher-scoring boxes; the generalized update also uses the current rescoring values.Source: A1. Detailed Explanation of NMS as a… · p. 12
  • The derivation sorts scores and overlaps in decreasing order, making higher-scoring box indices available without classical NMS’s hard argmax.The higher-scoring index set d<i contains the sampled box indices with scores above box i.Source: A1. Detailed Explanation of NMS as a… · p. 12
  • The product-based recursion is expanded and higher-order terms are dropped to obtain a closed-form recursion using addition and subtraction.Dropping a score factor in the second term preserves the relation r_i ≤ s_i because the pruning and rescoring terms lie in [0, 1].Source: A1. Detailed Explanation of NMS as a… · p. 12
  • The rescoring update is written using a prune matrix formed by applying the pruning function element-wise to the overlap matrix.The unclipped matrix solution is r ≈ (I + P)^−1 s; the section clips it because it is not necessarily bounded between 0 and 1.Source: A1. Detailed Explanation of NMS as a… · pp. 12–13

A2. Loss Functions

The training objective combines losses on boxes before NMS with an Imagewise AP loss on boxes after NMS. The stated loss weights preserve the other losses and weights used by the cited baseline.

  • Before NMS, the method applies 2D and 3D regression losses and confidence losses only to foreground boxes.The 3D loss uses Smooth-L1; the self-balancing confidence and orientation-bin terms are also described.Source: A2. Loss Functions · p. 13
  • After NMS, the loss is Imagewise AP-Loss.Source: A2. Loss Functions · p. 13
  • The overall loss weights the after-NMS term by λ, with λ = 0.05 and λa = 0.35.The passage states that the other losses and their weights match [12], except for LImagewise.Source: A2. Loss Functions · p. 13

A3. Additional Experiments and Results

These supplementary experiments examine oracle NMS scores, alternative NMS methods, parameter sensitivity, qualitative predictions, and a demo without temporal information. The reported comparisons include near-equal performance when classical NMS replaces GrooMeD-NMS at inference.

A3.1. Training

  • Val 1 and Val 2 training use 80k warmup and 50k full-training mini-batches; the Test Split uses 160k and 100k, respectively.Training uses random flipping with probability 0.5 and Adam with batch size 2.Source: A3.1. Training · p. 13

A3.2. KITTI Val 1 Oracle NMS Experiments

A3.3. KITTI Val 1 3D Object Detection

A3.4. KITTI Val 1 Sensitivity Analysis

  • The selected settings are optimal or near-optimal: Nt = 0.4, α = 100, and β = 0.3 are optimal, while v = 0.3 is close to optimal.The method has minor sensitivity to Nt, α, λ, and β, and is less sensitive to v; only α affects inference speed.Source: A3.4. KITTI Val 1 Sensitivity Analysis · pp. 14–15

A3.5. Qualitative Results

A3.6. Demo Video of GrooMeD-NMS

  • The demo runs independently on frames from three KITTI raw sequences, without temporal information; box orientations are reported as stable.None of those frames appear in the KITTI Val 1 training set.Source: A3.6. Demo Video of GrooMeD-NMS · p. 15

Related papers

Loading 2103.17202v1…