Source-linked AI summary
Augmentation for small object detection
Mate Kisantal, Zbigniew Wojna, Jakub Murawski, Jacek Naruniec, Kyunghyun Cho
TL;DR
Small-object detection and segmentation lag behind large-object performance because small objects are sparsely represented in MS COCO and receive fewer matched anchors during training. The paper addresses these issues by oversampling small-object images and copy-pasting small objects, achieving relative gains on both tasks.
Problem
Small-object detection underperforms large-object detection because MS COCO contains fewer small-object images, limited small-object area, and fewer matched anchors.
Method
The method oversamples images containing small objects and augments them by copy-pasting small objects during preprocessing.
Results
9.7% relative improvement for small-object instance segmentation and 7.1% for small-object detection are reported versus Mask R-CNN on MS COCO.
Takeaways & Limitations
Oversampling and copy-paste augmentation improve small-object performance, while oversampling can reduce large-object performance depending on the chosen ratio.
Abstract
from arXiv · showhide
In recent years, object detection has experienced impressive progress. Despite these improvements, there is still a significant gap in the performance between the detection of small and large objects. We analyze the current state-of-the-art model, Mask-RCNN, on a challenging dataset, MS COCO. We show that the overlap between small ground-truth objects and the predicted anchors is much lower than the expected IoU threshold. We conjecture this is due to two factors; (1) only a few images are containing small objects, and (2) small objects do not appear enough even within each image containing them. We thus propose to oversample those images with small objects and augment each of those images by copy-pasting small objects many times. It allows us to trade off the quality of the detector on large objects with that on small objects. We evaluate different pasting augmentation strategies, and ultimately, we achieve 9.7\% relative improvement on the instance segmentation and 7.1\% on the object detection of small objects, compared to the current state of the art method on MS COCO.
1 Introduction
Small-object detection remains substantially weaker than large-object detection in MS COCO, motivating targeted data augmentation. The paper attributes this gap to limited small-object representation and proposes oversampling plus copy-pasting small objects.
- Small-object AP is 2-3 times lower than large-object AP in top MS COCO instance-segmentation submissions.
- Small-object detection matters for applications including self-driving, medical imaging, and industrial inspection.
- Mask R-CNN misses most small objects, including detecting only six birds out of hundreds in one example.
- The paper identifies fewer small-object images and limited small-object occurrence within images as dataset-related issues.
- The proposed method oversamples images containing small objects and copy-pastes small objects repeatedly within them.The approach avoids overlap with existing objects and ultimately reports 9.7% relative improvement for small-object instance segmentation and 7.1% for small-object detection versus Mask R-CNN.
2 Related Work
Related work addresses scale variation through detector architectures, higher-resolution features, generated features, and contextual cropping. The paper instead emphasizes copy-pasting small objects while preserving image context.
- Object Detection: Faster R-CNN, R-FCN, and SSD differ in how and where region proposals are attached.
- Small Objects: The proposed augmentation copies small objects into images, with same-image pasting providing a high chance of appropriate surrounding context.
- Object Detection: Faster R-CNN variants merge differently cropped proposals into one resolution, but imperfect alignment may hurt practical performance.
- Instance Segmentation: Instance-segmentation methods predict object masks using cascades, position-sensitive maps, pixel embeddings, or Mask R-CNN mask branches.
- Small Objects: Small-object methods increase resolution, fuse features, generate scale-invariant features, shift features, or add context during proposal cropping.
3 Identifying issues with detecting small objects
The analysis links poor small-object detection to MS COCO’s uneven object representation and anchor-matching process. Small objects occupy few pixels and receive fewer positive anchors than large objects.
- 3.1 MS COCO: MS COCO contains 118,287 training images, 5,000 validation images, 40,670 test images, and annotations across 80 categories.
- 3.1 MS COCO: Average precision is computed across categories and IoU thresholds, with separate metrics for small, medium, and large objects.
- 3.2 Mask R-CNN: Mask R-CNN uses an FPN region proposal network with fifteen anchors spanning five scales and three aspect ratios.
- 3.3 Small object detection by Mask R-CNN on MS COCO: Only about half of training images contain small objects, although small objects comprise 41.43% of annotated objects.
- 3.3 Small object detection by Mask R-CNN on MS COCO: Small objects occupy only 1.23% of annotated pixels, compared with 10.18% for medium objects and 82.28% for large objects.
- 3.3 Small object detection by Mask R-CNN on MS COCO: Anchor assignment favors large objects because they overlap many anchors, while small objects may match only one anchor at low IoU.
4 Oversampling and Augmentation
The method targets MS COCO’s small-object imbalance with preprocessing that oversamples relevant images and copy-pastes small objects into new, non-overlapping locations.
- Oversampling and Augmentation: The approach oversamples images containing small objects and augments them with copy-pasted small objects to focus detectors on this scale.Although evaluated with Mask R-CNN, both operations are data preprocessing and are generally usable with other detection frameworks.
- Oversampling and Augmentation: Copy-pasting increases the number of small objects and matched anchors, strengthening their contribution to the RPN loss during training.The copies are placed at different positions to increase location diversity while retaining image context.
- Oversampling and Augmentation: Pasted objects receive random scale and rotation transformations, remain non-overlapping, and must be at least five pixels from image boundaries.Only non-occluded objects are used for pasting.
- Oversampling and Augmentation: The proposed augmentation is illustrated as increasing the number of matched anchors during training and improving small-object detection.
5 Experimental Setup
Experiments vary oversampling and copy-pasting configurations to study their effects on small-, medium-, and large-object performance and on pasting design choices.
- Experimental Setup: Oversampling ratios of two, three, and four are evaluated using offline copies of images containing small objects.The experiments examine effects across small, medium, and large object detection.
- Experimental Setup: One experiment copy-pastes all small objects once per image and examines how this augmentation interacts with oversampling.
- Experimental Setup: Augmentation is tested by replacement, duplication of augmented images, or retaining both original and augmented images.The final setting corresponds to oversampling small-object images by two while augmenting duplicated copies.
- Copy-Pasting Strategies: Copy-pasting strategies compare repeating one object, pasting many objects once, and repeating all small objects in random locations while retaining original images.
- Experimental Setup: Oversampling helps small-object detection across ratios, while the ratio mediates a trade-off between small- and large-object performance.
- Pasting Algorithms: Pasting algorithms vary overlap handling and Gaussian boundary blurring to test whether either design choice improves performance.The experiments compare non-overlapping placement with alternatives and blur filters of varying sizes.
6 Result and Analysis
Results show that oversampling and carefully configured copy-pasting improve small-object performance, but excessive augmentation can overfit pasted artifacts or reduce overall and large-object performance.
- Oversampling: 1% AP gain for small objects, corresponding to an 8.85% relative improvement, is observed with 3× oversampling.Large-object detection and segmentation consistently suffer from oversampling, so the ratio should reflect the relative importance of object scales.
- Augmentation: 38% increase in small-object augmentation performance on an augmented validation set indicates overfitting to pasted objects rather than necessarily to original objects.The reported augmented-validation performance is 0.161; pasting artifacts may make synthetic objects easier for the network to recognize.
- Augmentation: The best augmentation performance uses original and copy-pasted images together, with augmentation probability p = 0.5 and a 2:1 original-to-augmented object ratio.This setting outperforms oversampling alone.
- Copy-Pasting Strategies: Copy-pasting one object improves small-object performance over two-times oversampling, with a small drop on large images and gains peaking at one or two pastes.
- Copy-Pasting Strategies: Copy-pasting multiple small objects performs better than copying only one, with benefits from pasting each object up to three times.
- Copy-Pasting Strategies: Copy-pasting all small objects performs best for both tasks when every object is augmented once, but does not outperform repeating selected objects.More copies can reduce the original-to-pasted ratio and create a mismatch between training and test images.
- Pasting Algorithms: Random pasting that permits overlap produces inferior small-object performance, supporting non-overlapping placement.Gaussian edge blurring provides no improvement, favoring unprocessed pasted objects unless more sophisticated fusion is used.
7 Conclusion
The paper addresses poor small-object detection by augmenting training data through image oversampling and copy-pasting. These strategies improve small-object performance while trading off prediction quality between small and large objects.
- The authors identify insufficient small-object representation in training data as a factor behind poor average precision.Mask R-CNN requires enough objects for predicted anchors to match during training.
- Oversampling images containing small objects improves small-object performance during training.
- Copy-pasting small objects provides a second augmentation strategy for addressing limited small-object representation.
- 9.7% relative improvement was achieved for instance segmentation of small objects compared to the current state of the art on MS COCO.
- 7.1% relative improvement was achieved for object detection of small objects compared to the current state of the art on MS COCO.
- The augmentation methods trade off prediction quality between small and large objects.Experiments verified this trade-off.