PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-end training of object class detectors for mean average precision

Paul Henderson, Vittorio Ferrari

arXiv:1607.03476v2cs.CV

TL;DR

Object detection traditionally trains window classification while evaluating ranked detections after NMS with mAP, leaving the training objective misaligned with the final task. The paper introduces gradient-like quantities for piecewise-constant losses and computes them efficiently to train Fast R-CNN directly for post-NMS mAP. On PASCAL VOC 2007 and 2012, this reaches equivalent performance to standard Fast R-CNN while using the same model and loss during training and testing.

  • Problem

    Traditional detectors optimize window classification accuracy but evaluate mAP after NMS, so training does not directly optimize the final detection task.

  • Method

    The method defines pseudo partial derivatives and pseudogradients for piecewise-constant functions and computes them efficiently for mAP with NMS.

  • Results

    Experiments on PASCAL VOC 2007 and 2012 show equivalent performance to traditional Fast R-CNN trained for window classification without NMS.

  • Takeaways & Limitations

    Training directly for post-NMS mAP makes the detector end-to-end, with identical model structure and loss at training and test time.

  • Takeaways & Limitations

    The approach uses transitivity approximations for NMS and has sparser, higher-variance, minibatch-biased optimization signals than softmax training.

Abstract

from arXiv · show

We present a method for training CNN-based object class detectors directly using mean average precision (mAP) as the training loss, in a truly end-to-end fashion that includes non-maximum suppression (NMS) at training time. This contrasts with the traditional approach of training a CNN for a window classification loss, then applying NMS only at test time, when mAP is used as the evaluation metric in place of classification accuracy. However, mAP following NMS forms a piecewise-constant structured loss over thousands of windows, with gradients that do not convey useful information for gradient descent. Hence, we define new, general gradient-like quantities for piecewise constant functions, which have wide applicability. We describe how to calculate these efficiently for mAP following NMS, enabling to train a detector based on Fast R-CNN directly for mAP. This model achieves equivalent performance to the standard Fast R-CNN on the PASCAL VOC 2007 and 2012 datasets, while being conceptually more appealing as the very same model and loss are used at both training and test time.

1 Introduction

Modern detectors train CNNs for window classification but evaluate ranked, post-NMS detections with mAP, creating a mismatch between training and testing. This work directly optimizes post-NMS mAP with new gradient-like quantities for piecewise-constant losses and reports equivalent performance on PASCAL VOC.

  • Motivation: Object detectors classify proposal windows independently, then apply NMS to retain locally highest-scored detections before mAP evaluation.The CNN assigns class and background scores to each window, while mAP evaluates detection-score rankings over classes.
  • Motivation: Traditional training optimizes window classification accuracy, whereas testing evaluates mAP on locally highest-scoring windows after NMS.Because NMS is omitted during training, this is training for a surrogate window-classification task rather than the final detection task.
  • Method: The proposed method trains object detectors directly with mAP computed after NMS as the loss.This aligns the training loss with the evaluation metric and includes the final detection operation during optimization.
  • Method: Post-NMS mAP is a non-convex, discontinuous, piecewise-constant structured loss over many windows, with cascading effects from NMS.The authors address this by defining gradient-like quantities for piecewise-constant functions and computing them efficiently for mAP following NMS.
  • Results: Experiments on PASCAL VOC 2007 and 2012 show equivalent performance to traditional Fast R-CNN training without NMS.The same model and loss are used during training and testing, and the proposed loss is described as a drop-in layer for existing frameworks.

2 Background

The background defines NMS, mAP, interpolated AP, and the Fast R-CNN baseline. It also illustrates how score ordering and perturbations affect AP, while contrasting the baseline’s separate training and testing pipelines.

  • Non-maximum suppression: NMS repeatedly retains the highest-scored window and suppresses sufficiently overlapping windows until all windows are marked retained or suppressed.The retained windows form the final detection set.
  • Mean average precision: mAP is the mean across classes of each class’s interpolated average precision, computed as the area under its precision/recall curve.The curve is built from detections ordered by decreasing score.
  • Mean average precision: Detections are mapped to sufficiently overlapping ground-truth instances, with the highest-scored detection per instance counted as a true positive and others as false positives.For PASCAL VOC, sufficient overlap is defined as more than 50% IoU.
  • Mean average precision: AP is the area under the interpolated precision/recall curve, using 11 recall samples for VOC 2007 and the true area for VOC 2012.Interpolation replaces each precision with the maximum precision at itself and higher recall levels.
  • Fast R-CNN: Fast R-CNN classifies proposal windows using convolutional and fully connected layers, then applies NMS only during testing before evaluating mAP.Its training loss is computed over windows, whereas the proposed architecture makes training and testing structurally identical.
  • AP sensitivity: In the AP illustration, score perturbations affect AP only when they change relevant true-positive/false-positive ordering on the filled-in curve.Some swaps leave AP unchanged, while others alter the filled-in curve across a wider recall range.

3 Related Work

Prior detection methods generally train window classifiers and omit NMS and mAP during training. Related approaches use structured prediction, AP-specific optimization, smooth ranking proxies, or training-time NMS, but differ from this paper’s objective and formulation.

  • Detection training: Most object detectors train window classifiers while ignoring NMS and mAP during training, whether windows come from dense grids or proposal methods.The paper positions its method as addressing this training-time omission.
  • Architectural comparison: Fast R-CNN is shown with separate training and testing architectures, whereas the proposed architecture is the same in both phases.This architectural contrast motivates the paper’s end-to-end formulation.
  • Structured detection losses: Structured SVM detection methods model object presence and bounding-box coordinates with IoU-based task losses rather than directly maximizing mAP.Their structured loss involves detection and ground-truth overlap but does not correspond to the paper’s target metric.
  • AP optimization: AP-specific structured-SVM methods optimize classification AP with linear models and require loss-augmented inference whose dynamic programming has quadratic complexity in training points.These approaches target classification rather than object detection with NMS.
  • Smooth ranking proxies: A ranking-loss approach smooths a nondifferentiable piecewise-constant loss by modeling predicted scores as Gaussian random variables and estimating rank distributions.The formulation is based on pairwise comparisons with other scores and does not specifically optimize AP.
  • Training-time NMS: Another object-detection method includes NMS during training but uses a deformable-parts model and a nonstandard NMS variant with a different structured loss.It therefore differs in both model formulation and training objective.

4 Proposed Method

The method trains Fast R-CNN directly on mAP after NMS by replacing unusable derivatives of piecewise-constant losses with pseudogradients and efficient step-location calculations.

  • Model: The proposed model removes Fast R-CNN’s softmax layer and uses last fully connected activations directly as window scores.The convolutional and fully connected architecture remains otherwise based on Fast R-CNN, with AlexNet or VGG16 backbones considered.
  • Gradients of mAP and NMS Layers: mAP is piecewise constant in detection scores, so conventional derivatives are almost everywhere zero or undefined and cannot guide gradient descent.NMS also requires a non-trivial Jacobian because score changes can affect which windows are retained.
  • Pseudogradients of General Piecewise-Constant Functions: Pseudo partial derivatives define gradient-like directions and magnitudes for piecewise-constant functions after a non-infinitesimal move reaches a change in function value.The resulting quantities extend to an analogue of the gradient and Jacobian.
  • Pseudogradients of General Piecewise-Constant Functions: Adaptive finite-difference estimators find the smallest perturbation that changes the function, while symmetric differences average positive and negative perturbations.The symmetric estimator reduces directional asymmetry but its magnitude depends on the distance from the current point to a step.
  • Application to mAP and NMS: For AP, two linear detection passes in descending and ascending score order locate the nearest score steps that can change AP.The method tracks the last-seen true- and false-positive detections while computing minimal score increases or decreases.
  • Application to mAP and NMS: NMS is handled jointly with AP using approximations that ignore cascaded long-distance interactions and assume shared ground-truth overlap in specified suppression cases.The training protocol also uses larger minibatches and gradient clipping to improve the gradient signal and numerical behavior.

5 Experiments

Experiments evaluate direct mAP training with NMS against Fast R-CNN and prior AP training on PASCAL VOC. The method reaches comparable performance while requiring specific training choices and exhibiting minibatch-related limitations.

  • Experimental setup: Experiments use Fast R-CNN with AlexNet or VGG16 on PASCAL VOC 2007 and 2012, comparing direct mAP training against standard window-classification training and prior AP training.Bounding-box regression is disabled for fair comparison, and VOC 2012 results compare against a method using separate class models without training-time NMS.
  • Training procedure: The overall loss combines the training procedure with mAP and NMS, using class scores, the total number of classes, and windows as its indexed quantities.The AP calculation used during training is matched to the evaluation procedure, with the VOC 2007 approximation or true AP used according to the test dataset.
  • VOC 2007 results: Comparable performance is achieved against Fast R-CNN on VOC 2007, while the union of VOC 2007 and 2012 training data increases mAP by up to 3.6%.For AlexNet, gains are 3.3% versus 1.8% for Fast R-CNN; for VGG16, gains are 3.6% versus 1.1%.
  • Ablation study: Ablations show that increasing the foreground fraction reduces mAP by 6.1, while disabling gradient clipping causes training to fail after fewer than 100 iterations.Halving batch size while doubling iteration count reduces mAP by 0.8.
  • VOC 2012 results: Comparable performance is achieved against the prior direct-AP method on VOC 2012, with MEE slightly outperforming SDE.The prior method trains separate models for each class and does not account for NMS during training.
  • Discussion: The authors attribute the lack of substantial overall improvement to sparse gradients, higher minibatch loss variance, and biased minibatch estimates of population mAP.Around 20% of scores have non-zero gradients on VOC 2007, compared with 100% for softmax loss.

6 Conclusions

The paper introduces pseudogradients for piecewise-constant functions and uses them to train Fast R-CNN directly with mAP and NMS. Performance matches Fast R-CNN while preserving the same model and loss at training and test time.

  • 6 Conclusions: Two pseudo-partial-derivative definitions enable training Fast R-CNN directly with mAP as the loss, including NMS during training.This makes training end-to-end for the final detection task rather than for window classification.
  • 6 Conclusions: The method achieves equivalent performance to Fast R-CNN and integrates into standard SGD frameworks through drop-in NMS and mAP loss layers.The approach does not require changing the minimisation algorithm or other model elements.
  • 6 Conclusions: The pseudogradient definitions may also support other piecewise-constant losses, ranking metrics, and piecewise-constant internal network layers.The approach is described as general because it does not require an efficient max-oracle.
Loading 1607.03476v2…