Source-linked AI summary
An Analysis of Scale Invariance in Object Detection - SNIP
Bharat Singh, Larry S. Davis
TL;DR
The paper investigates why object detection struggles with extreme scale variation, particularly the prevalence of very small objects. It compares scale-specific and scale-invariant designs and proposes SNIP, which selectively trains object instances at suitable image-pyramid scales. SNIP improves detection performance, including a 69.7% mAP result for an ensemble on COCO at 50% overlap.
Problem
Object detection must handle extreme scale variation and very small objects, while CNN robustness to changes in scale remains limited.
Method
SNIP trains detectors on an image pyramid and back-propagates only object instances whose resolution is close to the pretrained CNN's resolution.
Results
69.7% mAP at 50% overlap was achieved by an ensemble with a Deformable-RFCN backbone, a 7.4% improvement over the COCO state of the art.
Takeaways & Limitations
Image pyramids and scale normalization are important for object detection, and SNIP can be incorporated into pipelines for tasks with large object-scale variation.
Takeaways & Limitations
Feature-pyramid methods remain limited for extremely small or large objects because pretrained high-level features may not be useful at those scales.
Abstract
from arXiv · showhide
An analysis of different techniques for recognizing and detecting objects under extreme scale variation is presented. Scale specific and scale invariant design of detectors are compared by training them with different configurations of input data. By evaluating the performance of different network architectures for classifying small objects on ImageNet, we show that CNNs are not robust to changes in scale. Based on this analysis, we propose to train and test detectors on the same scales of an image-pyramid. Since small and large objects are difficult to recognize at smaller and larger scales respectively, we present a novel training scheme called Scale Normalization for Image Pyramids (SNIP) which selectively back-propagates the gradients of object instances of different sizes as a function of the image scale. On the COCO dataset, our single model performance is 45.7% and an ensemble of 3 networks obtains an mAP of 48.3%. We use off-the-shelf ImageNet-1000 pre-trained models and only train with bounding box supervision. Our submission won the Best Student Entry in the COCO 2017 challenge. Code will be made available at \url{http://bit.ly/2yXVg4c}.
1. Introduction
Object detection is harder than image classification partly because COCO contains extreme object-scale variation, especially very small objects. The paper studies scale-specific and scale-invariant training and proposes SNIP to reduce scale variation during training while retaining available instances.
- COCO object instances have a median relative scale of 0.106, compared with 0.554 for ImageNet classification.
- The smallest and largest 10% of COCO objects have relative scales of 0.024 and 0.472, spanning almost 20 times.
- Architectural solutions address small and large objects using multi-layer features, dilated or deformable convolutions, independent multi-resolution predictions, context, and image pyramids.
- The study asks whether upsampling is necessary, whether low-resolution pretraining can help, and whether training should restrict object resolutions or use all available scales.
- Controlled ImageNet and COCO experiments compare scale-specific detectors with a single detector trained across a wide scale range.
- SNIP reduces scale variation through an image pyramid while selectively back-propagating gradients for object instances near the pretrained CNN's resolution.
- 69.7% mAP at 50% overlap was obtained by an ensemble with a Deformable-RFCN backbone, improving 7.4% over the COCO state of the art.
2. Related Work
Prior work uses architectural features, image pyramids, and scale-specific processing to handle objects at different resolutions. The paper argues that feature pyramids remain limited for extremely small or large objects and that scale-specific filters can hurt detection.
- Scale-space methods motivate learning representations invariant to scale for detection, pose estimation, and instance segmentation.
- Modern detectors use dilated or atrous convolutions to increase feature-map resolution while preserving pretrained weights and receptive fields.
- Feature pyramids combine shallow and deep features, but a 25x25 object remains only 50x50 pixels after 2x upsampling.
- For such small objects, high-level conv5 features pretrained at 224x224 may not provide useful semantic information, limiting feature-pyramid alternatives to image pyramids.
- A face-detection approach using scale-specific filters has limitations for object detection because object-detection data and appearance variation are broader.
- Figure 3 contrasts CNN-B, CNN-S, and CNN-B-FT under high-resolution, low-resolution, and fine-tuned upsampled inputs using ResNet-101.
3. Image Classification at Multiple Scales
ImageNet experiments examine domain shift and classifier behavior across input scales. They find that pretrained classifiers degrade when tested at unseen resolutions, while fine-tuning on upsampled low-resolution images outperforms a specialized low-resolution architecture.
- Training at 800x1200 while inferring across an image pyramid introduces domain shift between training and testing resolutions.
- As the difference between training and testing resolution increases, the pretrained ResNet-101's Top-1 accuracy decreases.
- Resolution-specific classifiers modify network architecture and stride to process low-resolution images.
- CNN-S performs significantly better than CNN-B on low-resolution images in the reported experiments.
- Fine-tuned CNN-B-FT performs better than CNN-S on upsampled low-resolution images.
- The analysis supports upsampling images and using a classifier pretrained on high-resolution images rather than using a specialized small-object architecture.
4. Background
The experiments use Deformable-RFCN as the detector baseline and examine its deformable convolutions, position-sensitive pooling, and proposal-generation setup.
- Deformable-RFCN is selected because it reports the best single-model COCO results and is publicly available.
- Deformable convolutions in conv5 adapt receptive fields by predicting spatial offsets for convolutional sub-filters.
- The baseline extracts proposals at 800x1200 using a ResNet-101 Deformable-RFCN with five RPN anchor scales.
5. Data Variation or Correct Scale?
Training detectors at a correct object scale must be balanced against preserving appearance and pose variation. Multi-scale training exposes objects to varied resolutions but still suffers from extreme object sizes, while higher-resolution training offers only marginal gains.
- Conclusion: The experiments conclude that detectors should use appropriately scaled objects while preserving as much appearance and pose variation as possible.The study evaluates detector behavior on 1400x2000 images, focusing on COCO objects smaller than 32x32 pixels.
- Training at different resolutions: Training and testing at 1400x2000 marginally outperformed 800x1400 training because the resolutions matched.At the higher resolution, medium-to-large objects became too large to classify correctly.
- Scale specific detectors: Ignoring objects larger than 80 pixels at 1400x2000 performed worse than 800all because it removed about 30% of appearance and pose variation.Eliminating extreme-scale objects helped less than retaining the variation provided by medium-to-large instances.
- Multi-Scale Training (MST): Multi-scale training degraded on extremely small and large objects and performed similarly to 800all.Although MST exposes instances to many resolutions, extreme object sizes remain difficult to classify.
6. Object Detection on an Image Pyramid
SNIP trains detectors on an image pyramid while retaining instances whose resolutions are near the pre-training scale and ignoring extreme-scale instances at each resolution. During inference, valid detections from each scale are combined after rescaling.
- Object Detection on an Image Pyramid: SNIP combines maximal appearance and pose variation with a restricted, reasonable object-scale range.The method is designed to retain variation without exposing the detector to objects that are too large or too small.
- Scale Normalization for Image Pyramids: SNIP uses object instances near the typical 224x224 pre-training resolution and ignores the remainder during back-propagation.Each instance can appear at multiple image-pyramid scales, allowing suitable appearances to contribute training signals.
- Scale Normalization for Image Pyramids: During training, proposals and anchors associated with invalid object-size ranges are excluded, while valid ground-truth boxes still assign labels.At inference, out-of-range detections are discarded and detections from multiple resolutions are combined with soft-NMS.
- Scale Normalization for Image Pyramids: SNIP matches pooled RoI resolution to the pre-trained network, preserving positional correspondence for position-sensitive filters.This alignment is especially important for R-FCN-style methods that divide RoIs into sub-parts.
- Scale Normalization for Image Pyramids: High-resolution training uses 1000x1000 chips selected to cover small objects while reducing computation in regions without them.Randomly positioned chips are sampled and selected iteratively until all objects are covered.
7. Datasets and Evaluation
The evaluation uses COCO with multiple image resolutions, proposal-recall measurements, and detector comparisons to assess SNIP across classification, proposal generation, and detection. Results indicate that multi-scale training and inference improve performance, particularly for small objects.
- Datasets and evaluation: COCO evaluation uses 123,000 images, with 118,000 for training and a 5,000-image minival set for proposal recall.Small, medium, and large objects are defined as less than 32x32, 32x32–96x96, and greater than 96x96 pixels, respectively.
- Datasets and evaluation: The detector is trained at three resolutions: (480, 800), (800, 1200), and (1400, 2000), with separate RPN and RCN models.The first value denotes the shorter image side and the second limits the maximum side length.
- Datasets and evaluation: Only 30% of COCO ground-truth boxes match an RPN anchor above 0.7 IoU at 800x1200, rising to 58% when the threshold is lowered to 0.5.Consequently, more than 40% of ground-truth boxes receive a positive or ignored anchor with overlap below 0.5.
- Results: SNIP improves recall at 50% overlap by 2.9% overall and by 6.3% for objects smaller than 25 pixels.Recall for objects larger than 100 pixels at 50% overlap is already close to 100%, limiting the value of further improvement there.
- Results: 5% overall and 8.7% for small objects are the gains from multi-scale training and inference over the single-scale deformable R-FCN baseline.The comparison uses deformable position-sensitive filters and Soft-NMS, with three scales and DPN-92 proposals unless otherwise noted.
- Results: SNIP improves deformable-convolution Faster-RCNN performance by 1.3%, while a single-head Faster-RCNN reaches 44.4% mAP without a feature pyramid.This detector is described as more robust to scale and stronger on large objects because it lacks position-sensitive filters.
8. Conclusion
The paper analyzes object detection under extreme scale variation and identifies shortcomings in the existing training pipeline. It then proposes SNIP to address the scale spectrum of training instances and reduce domain shift for pretrained classification networks.
- Conclusion: The paper analyzes techniques for recognizing and detecting objects under extreme scale variation.The analysis exposes shortcomings in the current object detection training pipeline.
- Conclusion: SNIP is proposed to tackle the wide scale spectrum of object instances participating in training.
- Conclusion: SNIP is also intended to reduce domain shift for the pretrained classification network.