Towards Accurate One-Stage Object Detection with AP-Loss

1. Introduction

• Dense candidate boxes create foreground-background imbalance, allowing a detector to achieve high classification accuracy while performing poorly at detection.The paper notes that true negatives can dominate the classification metric.Source: §1 · p. 1, Figure 1 · p. 1
• The framework replaces classification with ranking using AP-loss, which models relationships among samples and is invariant to the positive-negative ratio.The authors argue AP is more consistent with the object-detection evaluation metric.Source: §1 · p. 2
• AP-loss is non-differentiable and non-decomposable, leaving standard gradient descent unsuitable and its optimization an open problem.Prior approaches described include structured-SVM methods, an upper-bound hinge loss, and approximate-gradient methods.Source: §1 · p. 2
• An error-driven update passes learning signals through AP-loss's non-differentiable activation, then backpropagation transfers them to network weights.The update signal for each variable is proportional to the error it makes.Source: §1 · p. 2
• The method reports notable gains over different classification losses on state-of-the-art one-stage detectors without changing their architectures.The paper also reports theoretical and experimental verification of its optimization algorithm.Source: §1 · p. 2

One-stage detectors

• One-stage detectors are relatively efficient but generally less accurate than two-stage approaches; RetinaNet and RefineDet have narrowed that gap, especially on COCO.Dense anchors contribute extreme foreground-background imbalance.Source: §2 · pp. 2–3
• OHEM and Focal Loss reduce easy-sample weights, but hand-crafted balance parameters may not generalize across datasets and anchor relationships remain poorly modeled.The paper identifies these as open challenges for existing one-stage detectors.Source: §2 · p. 3

AP as a loss for Object Detection

• Prior AP-optimization approaches include linear structured SVMs, an upper-bound structured hinge loss, approximate gradients, and reinforcement-learning fine-tuning.The paper contrasts these methods with its approach, which it describes as applicable to differentiable linear or nonlinear models and end-to-end detector training.Source: §2 · p. 3

3. Method

• The method changes the classification-loss branch to ranking and error-driven optimization while leaving the backbone and localization branch unchanged.The ranking procedure produces the primary AP-loss terms and label vector.Source: §3 · p. 3

3.1.1 Ranking Task

• Each anchor is replicated K times, with one scalar score per class-specific copy and labels assigned using the same IoU strategy.The ranking task requires positive boxes to rank above negative boxes and computes AP over scores from all classes.Source: §3.1.1 · p. 3

3.1.2 AP-Loss

• AP-loss is formed from pairwise score differences, pairwise ordering labels, and a non-differentiable activation that yields primary terms for the objective.The ranking is treated as proper by breaking ties arbitrarily; the resulting objective requires an alternative to standard gradient descent.Source: §3.1.2 · p. 4
• The authors argue AP-loss is more suitable than AUC-loss because AP matches the object-detection evaluation metric.AUC-loss is another ranking-based metric that can be used to design a ranking loss in this framework.Source: §3.1.2 · p. 4

3.2.1 Error-Driven Update

• The update for each input is driven by the difference between its desired and current activation output, with noncontributing terms left unchanged.The desired output is chosen to reduce AP-loss; terms with y_ij = 0 receive no update.Source: §3.2.1 · p. 4

3.2.2 Backpropagation

• The weight update is chosen to produce the desired input movement, then implemented by backpropagating through the score-difference transformation.The procedure regularizes weight changes with an L2 penalty.Source: §3.2.2 · pp. 4–5

3.3. Analyses

• Finite-step convergence is guaranteed for a linear model on linearly separable data, although the AP-loss is nonconvex even under those conditions.With mild modifications, accumulated AP-loss can also be bounded proportionally by the model's best performance on nonseparable data.Source: §3.3 · p. 5
• With softmax or a loss-augmented step activation, the update is equivalent to gradient descent on cross-entropy or hinge loss, respectively.The paper attributes this consistency to updates proportional to prediction errors.Source: §3.3 · p. 5

Minibatch Training

• Aggregating scores across images in a minibatch addresses score shifts that can make per-image AP-loss yield poor detection performance.The paper describes minibatch training as crucial for good convergence and performance.Source: §3.4 · p. 5

Piecewise Step Function

• A piecewise step function replaces the Heaviside function near zero, where small input changes can destabilize updates.Its width δ controls the decision boundary and relates closely to weight decay.Source: §3.4 · pp. 5–6, Figure 4 · p. 6

Interpolated AP

• Interpolated AP provides a smooth approximation of actual AP that stabilizes gradients and reduces the impact of ranking wiggles on update signals.Interpolation makes precision at successive positive samples monotonically increasing.Source: §3.4 · p. 6

4.1. Experimental Settings

• Evaluation used RetinaNet on PASCAL VOC and MS COCO; VOC models were trained for 160 epochs, while COCO networks used trainval35k and were trained for 100 epochs.Source: §4.1 · p. 6

4.2.2 Comparison on Different Losses

• AP-loss outperformed all compared losses on VOC2007 and COCO; focal loss did not outperform CE-loss with OHEM at VOC AP50.The authors attribute AP-loss's advantage over AUC-loss to greater penalties for misordering objects higher in the predicted ranking.Source: Table 2 · p. 7, §4.2.2 · p. 7

4.2.1 Comparison on Different Parameter Settings

• Batch size 8 performed best, δ = 1 avoided unstable or distorted training, and interpolated AP offered marginal benefits over standard AP.The authors associate larger batches with reduced cross-image score shift and more robust AP-loss gradient calculation.Source: §4.2.1 · p. 7

4.2.3 Comparison on Different Optimization Methods

• Approximate-gradient optimization did not converge, while structured hinge loss stabilized near 0.8, below the asymptotic limit reached by the error-driven update scheme.The authors suggest the hinge method's upper-bound objective and hand-picked AUC-like discriminant may contribute to its optimization variability.Source: §4.2.3 · p. 7, Figure 5 · p. 7

4.3. Benchmark Results

• Across VOC2007, VOC2012, and COCO, the AP-loss detector outperformed all listed methods in single- and multi-scale tests; its COCO gain over RetinaNet500 was 3.0% (37.4% vs. 34.4%).It retained RetinaNet500's detection speed of approximately 11 fps on one TitanX GPU.Source: Table 3 · p. 8, §4.3 · p. 8

• The paper addresses one-stage detector class imbalance by replacing the classification sub-task with ranking optimized using AP-loss.Source: §5 · p. 8
• An error-driven update scheme based on perceptron learning optimizes the non-differentiable, non-convex AP-loss, supported by theoretical analysis.Source: §5 · p. 8
• Experiments show that the approach significantly improves state-of-the-art one-stage detectors.Source: §5 · p. 8

• The convergence proof generalizes the original perceptron-learning proof.Source: A1. Convergence · p. 10
• Finite-step convergence is guaranteed when the learning model is linear and the training data is linearly separable.Source: A1. Convergence · p. 10
• At each step, the algorithm randomly selects a training sample on which it makes an error and updates the model weights.Source: A1. Convergence · p. 10

• In the example, gradient descent makes both parameters increase while maintaining θ1 > θ2, and the objective approaches 1/6.Source: A2. An Example of Gradient Descent… · pp. 10–11
• The data are separable because the two positive samples score above the negative sample when 0 < θ1 < 1.Source: A2. An Example of Gradient Descent… · pp. 10–11
• The global minimum is 0, reached only if θ1 → +∞ and θ2 − 3θ1 → +∞; gradient descent in this example does not converge to it.Source: A2. An Example of Gradient Descent… · p. 11

A3. Inseparable Case

• The accumulated AP-loss bound depends on the best performance of the learning model, so the analysis treats inseparable training data explicitly.The paper frames this analysis using online-learning bounds.Source: A3. Inseparable Case · p. 11

A3.1. Preliminary

• The modified update introduces a non-zero decision margin for pairwise scores; the practical strategy also addresses the denominator’s discontinuity.The theoretical modification changes H(·) to eH(·) in the numerator, while the paper says the Section 3.4.2 strategy is practically necessary for good performance.Source: A3.1. Preliminary · p. 11

A3.2. Bound of Accumulated Loss

• Under convexity of the surrogate loss in the weights and a bounded model Jacobian, the accumulated AP-loss is bounded in terms of comparator performance.The stated conditions hold for a linear learning model.Source: A3.1. Preliminary · p. 11, A3.2. Bound of Accumulated Loss · p. 12
• If some comparator has zero surrogate loss at every step, accumulated AP-loss is bounded by a constant and finite-step convergence follows; otherwise, the bound depends on comparator loss.The paper says the bound is meaningful when a sufficiently good solution still exists in the inseparable case.Source: A3.2. Bound of Accumulated Loss · p. 12

A3.3. Offline Setting

• In the offline setting, the paper derives a simpler bound whose active form depends on whether Z(u) is small.When Z(u) is not small, the alternative bound is active.Source: A3.3. Offline Setting · p. 12

Consistency

• With softmax activation, the optimization algorithm is equivalent to gradient descent on cross-entropy loss.The analysis considers multi-class outputs and a ground-truth class label.Source: A4. Consistency · p. 12
• With a loss-augmented step function, the algorithm is equivalent to gradient descent on hinge loss.The setup is binary classification with outputs (−x, x) and a ground-truth label in {1, 2}.Source: A4. Consistency · pp. 12–13
• For hinge loss, the update is error-driven for the correct class and zero for the other class, matching the proposed update scheme.For cross-entropy, the gradient is likewise described as error-driven, using desired output 1_y=i and current output L_i.Source: A4. Consistency · pp. 12–13

A5.1. Experimental Settings

• SSD models are trained on VOC2007 and VOC2012 trainval and tested on VOC2007 test using VGG-16 and 512×512 input images.The focal-loss evaluation uses γ = 1 and α = 0.25, which the authors report performed better than the original γ = 2 and α = 0.25 settings.Source: A5.1. Experimental Settings · p. 13

A5.2. Results

• AP-loss outperforms the other losses at final and snapshot times on SSD; across RetinaNet and SSD, it also performs better on PASCAL VOC and MS COCO.Figure 7 reports detection accuracy (mAP) on the VOC2007 test set; the paper describes the results as showing robustness and strong generalization.Source: Figure 7 · p. 13, A5.2. Results · p. 13