Localization Recall Precision (LRP): A New Performance Metric for Object Detection

1 Introduction

• AP can assign the same score to detectors with very different recall-precision curves and does not directly measure bounding-box localization accuracy.The paper notes that qualitative box examples can be limited and biased, while AP at different IoU thresholds still does not directly measure localization accuracy.Source: §1 · pp. 2–3
• The paper identifies localization, false-positive rate, and false-negative rate as the three key factors a detection metric should capture.It also argues that metrics should reveal detector strengths and recall-precision characteristics to inform threshold selection.Source: §1 · p. 3
• LRP combines components related to localization, precision, and recall; Optimal LRP selects the minimum-error detector configuration and supports confidence-score thresholding.The proposed thresholding method aims to decrease the number of detections in an optimal manner.Source: §1 · p. 3
• Optimal LRP provides richer, more discriminative information than AP for state-of-the-art detectors, while class-specific thresholds improve performance over a general threshold in online detection.The online detector uses RetinaNet as its still-image backbone.Source: §1 · p. 3

2 Related Work

• Multi-target tracking has component-based metrics such as OSPA that combine localization and cardinality errors, but analogous measures are lacking in object detection.Cardinality error includes both false positives and false negatives.Source: §2 · p. 4
• Prior classifier-threshold research centers on F-measure, but the reviewed work does not directly incorporate localization error for object detection.The paper therefore focuses on evaluation and class-specific threshold optimization for deep-learning object detectors.Source: §2 · pp. 4–5

3 Average Precision: an analysis and its deficiencies

• AP evaluates precision across the entire recall domain, favoring overall precision rather than detectors whose recall-precision curves are nearer the top-right corner.The paper characterizes AP as measuring overall capability rather than comparing the maximum.Source: §3 · p. 5
• AP is insensitive to confidence-score values when detection order is unchanged and does not recommend a threshold for practical detection filtering.The section notes filtering may be needed because of response-time limits.Source: §3 · p. 5
• AP interpolation between neighboring recall values is especially problematic for small classes, such as toaster with nine validation instances.The cited example is from the MSCOCO 2017 validation set.Source: §3 · p. 5

4 Localization-Recall-Precision (LRP) Error

• LRP is evaluated on detections that pass confidence threshold s, with IoU threshold τ also specified.The text gives 0 ≤ s ≤ 1 and 0 ≤ τ < 1.Source: §4 · p. 5
• After filtering, detections in Ys are assigned to ground-truth boxes in X using the assignment procedure described for AP.Source: §4 · pp. 5–6
• The assignment yields counts of true positives, false positives, and false negatives, which are used to define LRP error.Source: §4 · p. 6

Z (wIoULRPIoU(X, Ys) + wF P LRPF P (X, Ys) + wF NLRPF N(X, Ys)) ,

• The weights make each component interpretable and prevent the total error from becoming undefined when an individual component's denominator is zero.Source: Z (wIoULRPIoU(X, Ys) + wF P LRPF P (X,… · p. 6
• LRP's IoU component measures mean localization error among valid detections; one minus that component is their average IoU.Source: Z (wIoULRPIoU(X, Ys) + wF P LRPF P (X,… · p. 6
• The false-positive and false-negative components represent the precision and recall characteristics of the thresholded detections, respectively.Source: Z (wIoULRPIoU(X, Ys) + wF P LRPF P (X,… · p. 6
• LRP normalizes localization penalties by 1 − τ, assigns unit upper-bound penalties to FP and FN, and averages errors over TP + FP + FN.The resulting average error lies in [0, 1], with lower values indicating better performance.Source: Z (wIoULRPIoU(X, Ys) + wF P LRPF P (X,… · p. 6
• Lowering confidence threshold s moves along the recall-precision curve toward higher recall, while higher IoU threshold τ requires tighter boxes.Together, τ sets the overlap requirement and s selects the curve point evaluated by LRP.Source: Z (wIoULRPIoU(X, Ys) + wF P LRPF P (X,… · p. 7

• At τ = 0.5, oLRP is the minimum achievable LRP error and is parameter independent.Source: §5 · p. 7
• For ablations or practical requirements, τ may vary, with oLRP@τ denoting the optimal error at that value.Source: §5 · p. 7
• oLRP searches confidence scores for the best balance among precision, recall, and IoU, locating the detector's balanced setting on the RP curve.The authors call one RP curve sharper when its top-right peak is nearer to the (1, 1) pair.Source: §5 · p. 7
• oLRP separates mean box tightness from FP and FN components that together describe RP-curve sharpness.The sharpness point can be represented by 1 − oLRP_FP and 1 − oLRP_FN.Source: §5 · p. 7
• moLRP is the detector-level metric formed from class-specific oLRP values, with mean localization, FP, and FN components.Source: §5 · p. 7
• Mean FP and FN components need not lie on the average RP curves across classes, but still provide information about curve sharpness.Source: §5 · p. 8
• For time-limited processing or deployments that discard confidence scores, oLRP supplies class-specific thresholds; its search exhaustively tests confidence scores at 0.01 intervals.Accepting larger LRP errors permits higher class-specific confidence thresholds for limited-time processing.Source: §5 · p. 8

6 Experimental Evaluation

• The evaluation tests Faster R-CNN, RetinaNet, and SSD variants on MS COCO validation 2017, covering 80 classes and 5,000 images.The online-video threshold experiment separately tests 346 ImageNet VID validation videos and 15 classes.Source: §6 · p. 8, §6.3 · p. 12

6.1 Analyzing Parameters s and τ

• As s rises, localization error changes little except at large s, while opposing FP and FN trends shape a second-order oLRP curve whose minima balance precision and recall.Fig. 2 plots the components and total error against s for Faster R-CNN (X101+FPN).Source: Figure 2 · p. 8, §6.1 · p. 9
• Raising τ reduces localization error but rapidly increases FP and FN components and total oLRP; evaluating multiple τ values supports application-specific overlap requirements and ablations.The reported extreme is τ = 1, where hardly any detections remain valid and oLRP is approximately 1.Source: Figure 3 · p. 9, §6.1 · p. 9

6.2 Evaluating Common Image Object Detectors

• Across detector variants, added FPN, depth, or input size lowers moLRP; its ordering generally tracks mAP, except Faster R-CNN (R101+FPN) outranks equal-mAP RetinaNet (X101+FPN).The comparison covers mAP, mAP@0.50, moLRP, and class-specific threshold ranges.Source: Table 1 · p. 9, §6.2 · p. 10
• RetinaNet (X101+FPN) has the tightest boxes (average IoU 0.839), but equal-mAP Faster R-CNN (R101+FPN) has lower mean oLRP and a sharper RP curve.The reported localization tightness and RP-curve sharpness distinguish the detectors despite equal mAP.Source: Table 1 · p. 9, Figure 4 · p. 10, §6.2 · pp. 11–12
• RetinaNet’s longer, shallower tails can lift AP despite less-sharp curves: on zebra it scores 0.899 versus Faster R-CNN’s 0.880, while oLRP favors Faster R-CNN.The paper observes this shallow-curve, longer-tail pattern in more than 50 classes.Source: Figure 5 · p. 11, §6.2 · p. 11

6.3 Better Threshold, Better Performance

• Compared with G, class-specific S improves mAP by 2.3% and moLRP by 0.6%; on bus (s*=0.27), S gains 8.7% AP and 4.1% oLRP.The class-specific threshold effect is most visible for classes with unusually low or high optimal thresholds.Source: Figure 6 · p. 12, Table 2 · p. 13, §6.3 · pp. 13–14

7 Discussion and Conclusion

• Optimal LRP captures both recall-precision behavior and bounding-box localization, addressing detector strengths and weaknesses that AP does not show.The paper describes it as more suitable for ablation studies.Source: §7 · p. 14
• By identifying the best confidence score, Optimal LRP removes LRP’s dependence on a chosen threshold and compares practical detector configurations.In the illustrated example, total errors are 0.5, 0.5, and 0.93 for detectors (a), (b), and (c), respectively.Source: §7 · p. 14
• The quantity 1−oLRP_IoU represents mean box localization accuracy, offering an alternative to estimating AP at multiple IoU thresholds.Source: §7 · p. 14
• Optimal LRP can evaluate detection results on a single image without interpolation.The accompanying supplement includes the proof that LRP is a metric.Source: §7 · p. 14

Supplementary Material for Localization Recall Precision (LRP): A New Performance Metric for

• The supplement contains a detailed AP definition, confidence-threshold distribution results, an online-detector description, and a proof that LRP is a metric.Source: Supplementary Material for… · p. 18

1 Definition of AP

• AP is the area under the recall-precision curve obtained by varying the confidence threshold and calculating recall and precision at each setting.A prediction is a true positive when its IoU with a ground-truth box exceeds the specified threshold; otherwise it is a false positive.Source: §1 · p. 18

2 Analysis of Threshold Distributions

• SSD-512’s most confident settings cluster around 0.3, while Faster R-CNN’s cluster around 0.7, illustrating detector-specific threshold distributions.Source: §2 · pp. 18–19, Figure 1 · p. 19
• A single detector-wide threshold can rarely produce class-appropriate outcomes: lower-threshold classes may rarely appear, while higher-threshold classes may generate many false positives.Source: §2 · pp. 18–19
• The analysis concludes that confidence thresholds should be set separately for each class because their optimal values vary substantially.Source: §2 · pp. 18–19

3 A Simple Online Video Object Detector

• The two online detectors differ in thresholding: G uses a general 0.5 threshold, while S uses class-specific thresholds.At each time interval, detections from previous and current frames are associated using the Hungarian algorithm [1].Source: §3 · p. 19
• Associated boxes use a linking score based on neighboring-frame confidence-distribution differences and tracker-prediction overlap, and their current confidence scores are updated using Bayes’ theorem.The prior is the previous frame’s updated tubelet score, and the likelihood is the currently associated high-confidence detection; the box-linking score is inspired by tube linking.Source: §3 · p. 19
• A box is called a “dominant object” when its updated score increases by 0.2, and its disappearance triggers tracker-based location prediction and class-wise object-versus-background classification.The update scores converge to 1 quickly, which is bad for lower recall, while precision increases in larger recall portions.Source: §3 · p. 19

4 Proof that LRP is a Metric

• The proof establishes 1−IoU(x, y) as a metric, with the triangle inequality applying to arbitrary bounding boxes x, y, and z.Identity and symmetry are described as trivial.Source: §4 · p. 20
• LRP is a metric, derived from DASA by setting p=1, c=1−τ, and d(x_i, y_xi)=1−IoU(x_i, y_xi).The derivation uses the fact that 1−IoU is a metric, and the IoU predicate defines the validation limit for true positives.Source: §4 · p. 20
• LRP’s component weights make its localization, false-positive, and false-negative terms interpretable, and normalization by 1−τ sets its upper bound to 1.The proof notes that dividing by a constant does not violate metricity.Source: §4 · p. 20