PaperlayerView exact revision on arXiv

Source-linked AI summary

Prime Sample Attention in Object Detection

Yuhang Cao, Kai Chen, Chen Change Loy, Dahua Lin

arXiv:1904.04821v2cs.CV

TL;DR

Object detectors often treat samples equally, although samples in a mini-batch are neither independent nor equally important for mAP. The paper defines prime samples and introduces PISA, which ranks and reweights them during training. PISA consistently improves detectors over random sampling and hard mining across datasets and detector types, including gains around 2% on MSCOCO.

  • Problem

    The paper asks which samples most influence object-detection mAP because random sampling and hard mining are not necessarily optimal.

  • Method

    PISA identifies prime samples using HLR rankings for positive and negative regions and directs training toward them through reweighting and joint classification-regression learning.

  • Results

    PISA consistently outperforms random sampling and hard mining on MSCOCO and PASCAL VOC across two-stage and single-stage detectors, with MSCOCO gains around 2%.

  • Takeaways & Limitations

    Focusing on samples most consequential to detection performance is often more effective than focusing on hard samples.

Abstract

from arXiv · show

It is a common paradigm in object detection frameworks to treat all samples equally and target at maximizing the performance on average. In this work, we revisit this paradigm through a careful study on how different samples contribute to the overall performance measured in terms of mAP. Our study suggests that the samples in each mini-batch are neither independent nor equally important, and therefore a better classifier on average does not necessarily mean higher mAP. Motivated by this study, we propose the notion of Prime Samples, those that play a key role in driving the detection performance. We further develop a simple yet effective sampling and learning strategy called PrIme Sample Attention (PISA) that directs the focus of the training process towards such samples. Our experiments demonstrate that it is often more effective to focus on prime samples than hard samples when training a detector. Particularly, On the MSCOCO dataset, PISA outperforms the random sampling baseline and hard mining schemes, e.g., OHEM and Focal Loss, consistently by around 2% on both single-stage and two-stage detectors, even with a strong backbone ResNeXt-101.

1 Introduction

Object detectors commonly optimize average loss over sampled regions, but samples differ in importance because detection decisions compete and localization quality is tied to classification confidence. PISA focuses training on prime samples identified as especially influential for detection performance.

  • Region selection is critical because most candidate regions lie in background areas, making exhaustive or random average-loss optimization ineffective.
  • Hard mining improves detector performance by emphasizing high-loss samples through explicit selection or loss reweighting.
  • Prime samples are well-localized positive regions with high IoU that should receive high classification confidence.
  • Samples within a mini-batch are competing rather than independent, so the highest-IoU positive samples can matter more than equally weighting all positives.
  • PISA ranks positive and negative samples with IoU-HLR and Score-HLR, then reweights training toward highly important regions and adds classification-aware regression loss.
  • PISA improves Faster R-CNN, Mask R-CNN, RetinaNet, and SSD by 2.0%, 1.5%, 1.8%, and 2.1%, respectively, on MSCOCO with ResNet-101-32x4d.

2 Related Work

Object detection uses region-based classification and regression in two-stage and single-stage frameworks, with sampling strategies shaping which candidate regions drive training. Random sampling and hard mining are established alternatives, while PISA is designed to apply across both detector paradigms.

  • Region-based object detectors: Region-based detectors formulate detection as bounding-box classification and regression, using either two-stage proposals or single-stage processing.
  • Sampling strategies in object detection: PISA is not detector-specific and can be applied to both two-stage and single-stage detection paradigms.
  • Sampling strategies in object detection: Random sampling selects a subset of candidates, often enforcing a fixed positive-to-negative ratio because negative regions greatly outnumber positive ones.
  • Sampling strategies in object detection: Hard mining selects samples with larger losses, while methods such as Focal Loss implement a soft form of emphasis without explicit sampling.

3 Prime Samples

Prime-sample importance follows how candidate boxes affect mAP: high-IoU positives and high-score negatives are especially consequential. HLR captures these local and global relationships to rank samples for training.

  • Prime Samples: Prime samples are regions with greater influence on object-detection performance, assessed by revisiting each sample’s contribution to mAP.
  • A Revisit to mAP: Among positives overlapping one object, the highest-IoU box matters most for recall; across objects, higher-IoU leaders matter more for precision as thresholds increase.
  • A Revisit to False Positives: Among overlapping negative regions, the highest-score sample is most important locally, while higher scores across regions affect precision earlier.
  • Hierarchical Local Rank: HLR ranks positive and negative samples hierarchically, reflecting relationships within local object or region groups and across the image or mini-batch.
  • Hierarchical Local Rank: IoU-HLR groups positives by nearest ground-truth object, ranks within groups by IoU, then orders samples across equal local ranks.
  • Hierarchical Local Rank: Score-HLR groups negative samples into NMS-based clusters, uses maximum foreground-class score, and applies the analogous ranking procedure.
  • Sample distributions: Prime positives tend to combine high IoU with low classification loss, unlike hard positives, which have high losses across a wider IoU range.

4 Learn Detectors via Prime Sample Attention

PISA shifts detector training from equal treatment toward prime samples, using importance-based reweighting and classification-aware regression to align classification with localization quality.

  • PISA biases training toward prime samples rather than treating all samples evenly.Prime samples are those considered more important for detection performance.
  • Importance-based Sample Reweighting: Importance-based Sample Reweighting assigns sample loss weights using IoU-HLR for positives and Score-HLR for negatives.The strategy is designed to fit existing detection frameworks through soft sampling.
  • Importance-based Sample Reweighting: HLR ranks samples hierarchically by grouping them by nearest ground-truth object and ordering within groups by importance.IoU-HLR uses descending IoU and aggregates samples by local rank across groups.
  • Importance-based Sample Reweighting: The exponential weighting function increases preference for important samples while using β to set the minimum weight and γ to control preference strength.The mapping is monotone increasing in sample importance.
  • Classification-Aware Regression Loss: CARL jointly optimizes classification and regression so localization quality influences classification scores for prime samples.The method propagates regression-related gradients toward classification and uses predicted probabilities to form sample weights.

5 Experiments

Experiments across COCO and VOC evaluate PISA on multiple detector architectures and sampling configurations. PISA consistently improves mAP, with gains linked to emphasizing high-IoU positive samples and suppressing false positives.

  • Overall results: PISA improves mAP across two-stage and single-stage detectors on MS COCO, including models with different backbones.The evaluation uses COCO train/validation/test-dev data and the standard COCO-style AP metric.
  • Overall results: PISA improves Faster R-CNN, Mask R-CNN, and RetinaNet by 2.1%, 1.8%, and 1.4%, respectively, with a ResNet-50 backbone.These are reported COCO test-dev improvements.
  • Overall results: More than 2.0% gains are observed on SSD300 and SSD512, while PISA adds no parameters and preserves baseline inference time.The reported efficiency result concerns inference rather than training.
  • Overall results: On PASCAL VOC, PISA performs better under both the 0.5-IoU evaluation and the multi-threshold COCO metric, especially at high IoU thresholds.The reported COCO-metric gains indicate more accurate predictions for precisely located samples.
  • Component analysis: ISR-P, ISR-N, and CARL improve mAP by 0.7%, 0.9%, and 1.0%, respectively; together, all three components produce a 2.1% gain.ISR combines ISR-P and ISR-N and yields a 1.5% gain; ISR-P plus CARL yields 1.6%.
  • Sampling analysis: Hard mining favors positive samples with high loss and low IoU, whereas PISA favors positive samples with high IoU and low loss.For negative samples, PISA has an intermediate preference between random sampling and hard mining.
  • Component analysis: ISR-P raises scores for top-ranked positive samples and lowers scores for lower-ranked ones, while ISR-N suppresses negative scores, especially for top-ranked samples.CARL boosts high-IoU scores and decreases low-IoU scores by linking classification scores to regression loss.

6 Conclusion

The paper establishes prime samples as especially influential for object detection and introduces PISA to emphasize them during sampling and learning. Across MS COCO and PASCAL VOC, PISA consistently improves over random sampling and hard mining.

  • 6 Conclusion: Prime samples are samples with greater influence on object-detection performance, and PISA highlights them through sampling and learning.The conclusion frames PISA as a simple and effective strategy.
  • 6 Conclusion: PISA achieves consistent improvements over random sampling and hard-mining counterparts on both MS COCO and PASCAL VOC.The conclusion reports this pattern across both datasets.

Appendix A: Derivative of CARL

The appendix establishes a positive relationship between CARL’s regression loss and classification-score gradients, linking localization quality to classification supervision.

  • CARL’s regression loss and the gradient of the classification score are positively correlated.
  • Because batch size is usually large, each sample’s weight is much smaller than the batch weight sum.
  • The derivative analysis assumes 0 ≤ b < 1 and k > 0.
  • The derivative result indicates that the relevant loss terms are positively correlated.

Appendix B: Implementation details

The experiments use specified GPU, backbone, batch-size, training-duration, and learning-rate settings, with ISR-N omitted for single-stage models because its negative-sample count would add training overhead.

  • Training settings: Experiments use 8 Tesla V100 GPUs, with SSD trained for 120 epochs using minibatches of 64 images.SSD uses 8 images per GPU; its learning rate starts at 0.001 and drops by 0.1 after 80 and 110 epochs.
  • Training settings: Other methods use ResNet-50 or ResNeXt-101-32x4d backbones with FPN by default.
  • Training settings: Other methods are trained for 12 epochs with batch size 16 and an initial learning rate of 0.02.The batch size is 2 images per GPU, and the learning rate is decreased by 0.1 after 8 and 11 epochs.
  • Method-specific setting: ISR-N is excluded from single-stage models because their larger number of negative samples would introduce significant training-time overhead.
Loading 1904.04821v2…