Prime Sample Attention in Object Detection

1 Introduction

• Because most sampled regions are background, feeding all regions or a random subset through the network and optimizing average loss is not necessarily effective.The introduction notes that random sampling and hard mining may not be optimal for training an effective detector.Source: §1 · p. 1
• The study finds that attending to prime samples is more effective for detector performance than attending to hard samples.The figure contrasts a precisely located prime sample with a hard sample and shows RoC curves for different sampling strategies.Source: §1 · p. 1, Figure 1 · p. 1
• Samples compete to cover objects, so high-IoU positive samples are more important than treating all positives as equally valuable.The study also links classification and localization: samples precisely located around ground-truth objects need high-confidence classification.Source: §1 · pp. 1–2
• PISA uses Hierarchical Local Rank to prioritize high-IoU positives and high-score negatives, then reweights them and applies classification-aware regression loss.The loss suppresses samples with large regression loss, reinforcing attention to prime samples.Source: §1 · p. 2
• PISA improves Faster R-CNN, Mask R-CNN, RetinaNet, and SSD by 2.0%, 1.5%, 1.8%, and 2.1%, respectively.The reported test-dev results use MSCOCO and a ResNet-101-32x4d backbone for the first three detectors.Source: §1 · p. 2

Region-based object detectors

• Two-stage detectors sample proposals for classification and regression, whereas single-stage detectors predict scores and box offsets directly; PISA applies to both paradigms.Two-stage detectors first generate proposals; single-stage detectors omit that step.Source: §2 · p. 2

Sampling strategies in object detection

• Random sampling is widely used, while hard mining and Focal Loss emphasize hard samples; PISA instead focuses on prime samples, which are not necessarily hard.The paper contrasts hard mining’s goal of improving average classifier performance with PISA’s biased performance across samples.Source: §2 · p. 2

Improvement of NMS with localization confidence

• IoU-Net uses predicted localization confidence for NMS, whereas PISA reweights prime samples and adds a loss correlating two branches because high IoU alone does not imply training importance.PISA also identifies the relative ranking among proposals around objects as important.Source: §2 · p. 2

A Revisit to mAP

• For positive samples, the highest-IoU box for each ground truth matters most for recall, while higher-IoU boxes across objects matter more for precision as the IoU threshold rises.Source: §3 · p. 3

A Revisit to False Positives

• A negative sample’s importance depends on whether it survives NMS: within an overlapping local region, the highest-score negative matters most, and higher-scoring regional leaders harm precision more.Lower-scoring overlapping negatives are discarded after NMS, making them less important even when their own scores are high.Source: §3 · p. 3

Hierarchical Local Rank (HLR)

• IoU-HLR and Score-HLR rank samples hierarchically by local and global relationships, using final regressed positions because mAP evaluates those positions.IoU-HLR groups samples by nearest ground truth, ranks by IoU within groups, then sorts samples sharing a local rank; Score-HLR applies a similar procedure to negatives grouped by NMS clusters using their maximum foreground-class score.Source: §3 · p. 3
• With a 10% total-loss-reduction budget, increasing scores for top5 and top25 IoU-HLR samples shows that focusing on top samples is better than attending to more samples equally.The study increases scores for those ranked samples to simulate reducing the total loss by 10%.Source: §3 · p. 3, Figure 2 · p. 3
• Prime and hard samples differ in their loss and IoU distributions: prime positives tend toward high IoU and low loss, while prime negatives span low losses and more divergent IoUs.Hard positives have high classification losses and a wider IoU range; hard negatives tend toward high losses and high IoUs.Source: §3 · pp. 3–4, Figure 4 · p. 4

4.1 Importance-based Sample Reweighting

• ISR assigns positive and negative samples different loss weights according to their importance, using IoU-HLR for positives and Score-HLR for negatives.It linearly maps within-class ranks to importance values, then uses an exponential function to convert importance to weights; the weights are normalized to keep the total loss unchanged.Source: §4.1 · pp. 4–5

4.2 Classification-Aware Regression Loss

• CARL propagates regression-branch gradients to classification scores, suppressing unimportant samples and reinforcing attention to prime samples.Greater regression loss produces larger classification-score gradients; regression loss reflects localization quality and approximately estimates sample rank, with top-ranked samples generally having low regression loss.Source: §4.2 · p. 5

5.1 Experimental Setting

• Experiments use MS COCO 2017 train for training and report validation and test-dev performance with COCO-style AP averaged over IoUs from 0.5 to 0.95 in 0.05 increments.Backbones include ResNet-50, ResNeXt-101-32x4d, and VGG16.Source: §5.1 · p. 5

5.2 Results

• On MS COCO, PISA improves Faster R-CNN, Mask R-CNN, and RetinaNet by 2.1%, 1.8%, and 1.4% with ResNet-50, and yields gains above 2.0% on SSD300 and SSD512.Improvements also appear with ResNeXt-101-32x4d; PISA adds no parameters and leaves inference time unchanged.Source: §5.2 · pp. 5–6, Table 1 · p. 6
• On PASCAL VOC, PISA outperforms baselines under both the 0.5-IoU metric and the COCO metric averaging multiple IoU thresholds.The authors report that PISA is especially beneficial at high IoU and improves localization precision.Source: §5.2 · p. 6, Table 2 · p. 6
• On COCO val, PISA outperforms random sampling and hard mining for positive, negative, and combined samples, with gains mainly in AP75.For positive samples, gains are 1.6% over random sampling and 2.0% over hard mining; for negative samples, 0.9% and 0.4%; for both, 2.1% and 1.7%. Qualitatively, PISA produces fewer false positives and higher scores for prime positive samples.Source: §5.2 · p. 6, Table 3 · p. 6

5.3 Analysis

• ISR-P, ISR-N, and CARL improve mAP by 0.7%, 0.9%, and 1.0%, respectively; all three components yield a 2.1% total gain.ISR (ISR-P + ISR-N) gains 1.5%, while applying PISA only to positive samples (ISR-P + CARL) gains 1.6%.Source: §5.3 · p. 6, Table 4 · p. 6
• With selected settings γP = 2.0, γN = 0.5, βP = βN = 0, k = 1.0, and b = 0.2, performance is not very sensitive to these hyperparameters.For positive samples, prime samples have high IoUs and low losses, unlike hard samples; for negatives, PISA has an intermediate preference between random sampling and hard mining and maintains sample diversity.Source: §5.3 · p. 7, Figure 6 · p. 7
• ISR-P raises scores for top-ranked positive samples and lowers them for lower-ranked positives, while ISR-N lowers negative-sample scores, especially for top-ranked samples.CARL couples classification scores to regression loss, boosting scores for high-IoU samples and decreasing scores for low-IoU samples relative to the FPN baseline.Source: §5.3 · p. 7, Figure 7 · p. 7, Figure 8 · p. 7

• The paper studies which samples are most important for training an object detector and establishes the notion of prime samples.Source: §6 · p. 7
• PISA is a simple and effective sampling and learning strategy designed to highlight important samples.Source: §6 · p. 7
• On MS COCO and PASCAL VOC, PISA achieves consistent improvements over random sampling and hard-mining counterparts.Source: §6 · p. 7

Appendix A: Derivative of CARL

• The argument assumes a large batch, so v_i << S.The passage motivates this assumption by noting that batch size is usually large.Source: Appendix A · p. 8
• The derivation also uses 0 ≤ b < 1 and k > 0.Source: Appendix A · p. 8
• The derivation reports that L(d_i, ˆd_i) terms are positively correlated.This result is presented in the discussion of the CARL derivative.Source: Appendix A · p. 8

Appendix B: Implementation details

• SSD training uses 120 epochs, 64 images per minibatch, and a learning rate reduced by 0.1 after epochs 80 and 110.The initial learning rate is 0.001.Source: Appendix B · p. 8
• Other methods use ResNet-50 or ResNeXt-101-32x4d backbones with FPN by default, training for 12 epochs with batch size 16.The initial learning rate is 0.02, reduced by 0.1 after epochs 8 and 11; 512 RoIs are sampled from 2,000 proposals at a 1:3 positive-to-negative ratio.Source: Appendix B · p. 8
• PISA combines ISR-P, ISR-N, and CARL, but omits ISR-N in single-stage models because their larger number of negative samples would add significant training overhead.Source: Appendix B · p. 8