PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Large-Scale Small Object Detection: Survey and Benchmarks

Gong Cheng, Xiang Yuan, Xiwen Yao, Kebing Yan, Qinghua Zeng, Xingxing Xie, Junwei Han

arXiv:2207.14096v4cs.CV

TL;DR

Small object detection remains difficult because small targets provide limited, distorted information and large-scale benchmarks are scarce. This paper reviews SOD, constructs driving and aerial benchmarks, and evaluates representative detectors, with RoI Transformer reaching 36.0% AP on SODA-A.

  • Problem

    Small object detection trails normal-sized object detection, while limited representations and scarce large-scale datasets hinder comprehensive benchmarking.

  • Method

    The paper reviews SOD and constructs SODA-D and SODA-A, using defined object-size categories and horizontal or oriented bounding-box annotations.

  • Results

    Benchmarking representative detectors shows substantial variation across methods and scenarios; RoI Transformer achieves 36.0% AP on SODA-A.

  • Takeaways & Limitations

    SODA-D and SODA-A provide large-scale, exhaustively annotated benchmarks for comparing small object detection methods.

  • Takeaways & Limitations

    Context-modeling approaches select contextual regions heuristically, so their constructed representations are not guaranteed to be interpretable for detection.

Abstract

from arXiv · show

With the rise of deep convolutional neural networks, object detection has achieved prominent advances in past years. However, such prosperity could not camouflage the unsatisfactory situation of Small Object Detection (SOD), one of the notoriously challenging tasks in computer vision, owing to the poor visual appearance and noisy representation caused by the intrinsic structure of small targets. In addition, large-scale dataset for benchmarking small object detection methods remains a bottleneck. In this paper, we first conduct a thorough review of small object detection. Then, to catalyze the development of SOD, we construct two large-scale Small Object Detection dAtasets (SODA), SODA-D and SODA-A, which focus on the Driving and Aerial scenarios respectively. SODA-D includes 24828 high-quality traffic images and 278433 instances of nine categories. For SODA-A, we harvest 2513 high resolution aerial images and annotate 872069 instances over nine classes. The proposed datasets, as we know, are the first-ever attempt to large-scale benchmarks with a vast collection of exhaustively annotated instances tailored for multi-category SOD. Finally, we evaluate the performance of mainstream methods on SODA. We expect the released benchmarks could facilitate the development of SOD and spawn more breakthroughs in this field. Datasets and codes are available at: \url{https://shaunyuan22.github.io/SODA}.

1 INTRODUCTION

The introduction frames small object detection as an important but underdeveloped task because small targets suffer weak, noisy representations and remain difficult even for leading detectors. It motivates a comprehensive SOD review and introduces two large-scale, scenario-specific benchmarks for evaluation and future progress.

  • Motivation: Small object detection remains substantially harder than generic detection, with leading detectors showing a large performance gap between small and normal-sized objects.DyHead achieves only 28.3% mAP for small objects on the COCO test-dev set.
  • Challenges: Limited object size, downsampling, and contamination from background or neighboring instances produce low-quality feature representations for small objects.Prevailing feature extractors downsample feature maps to reduce spatial redundancy and learn high-dimensional features, which can extinguish tiny-object representations.
  • Data limitations: Existing small-object datasets are too small for supervised CNN training, while general datasets do not provide impartial benchmarks tailored to multi-category SOD.Several datasets contain many small objects but are either designed for other purposes or cannot support customized SOD model training and evaluation.
  • Contributions: The authors construct two large-scale SOD benchmarks for driving and aerial scenarios and evaluate representative detection methods with quantitative and qualitative analyses.SODA-D contains 24828 high-quality driving images, while the benchmarks are presented as the first large-scale datasets tailored specifically for SOD.
  • Contributions: The paper provides a comprehensive, cross-domain review of small object detection that analyzes prior methods more rigorously than existing partial or overly broad reviews.Its taxonomy covers sample-oriented, scale-aware, attention-based, feature-imitation, context-modeling, and focus-and-detect approaches.

2 REVIEW ON SMALL OBJECT DETECTION

Small object detection is difficult because small targets lose information, yield noisy representations, tolerate little localization error, and provide inadequate training samples. Existing research addresses these issues through scale handling, feature enrichment, contextual cues, and adaptive focus-and-detect processing, although each strategy retains limitations.

  • Challenges: SOD primarily suffers from object information loss, noisy feature representation, low tolerance for bounding-box perturbation, and inadequate training samples.These challenges compound generic detection issues such as intra-class variation, inaccurate localization, and occlusion.
  • Challenges: Subsampling in generic backbones reduces feature-map resolution, causing small-object information to disappear before detection decisions are made.Small objects also have low-quality appearance and regional features contaminated by background and other instances, weakening discrimination.
  • Challenges: A 6-pixel diagonal box deviation can reduce small-object IoU from 100% to 32.5%, making localization substantially more sensitive than for larger objects.Small instances also overlap weakly with anchors or points, challenging conventional positive-negative label assignment.
  • Scale handling: Scale-specific architectures and feature-fusion approaches target scale variation and bridge spatial-semantic gaps, but heuristic scale assignment can confuse detectors.Scale variation is especially severe in traffic and remote-sensing images, where objects have disparate detection difficulties.
  • Feature and context enhancement: Feature enrichment, contextual priming, and focus-and-detect methods improve small-object cues or reduce computation, but context selection and focus localization remain unresolved.Adaptive crops and zoom-in operations process smaller objects at higher resolution while filtering empty regions, reducing background interference and inference costs.

3 REVIEW OF DATASETS FOR SMALL OBJECT DE-

Existing object-detection benchmarks provide validation platforms, but few are specifically designed for small object detection. The reviewed datasets cover diverse natural, traffic, pedestrian, and aerial settings while retaining important domain-specific limitations.

  • Existing benchmarks: WiderFace contains 32203 images and 393703 face instances across substantial variation in scale, pose, occlusion, expression, appearance, and illumination.It also provides occlusion, pose, and event-category attributes alongside bounding boxes.
  • Existing benchmarks: 72561 persons in 1610 images make TinyPerson a dedicated tiny-scale seaside pedestrian benchmark with tiny and small subsets.It assigns ignore labels to regions that cannot be certainly recognized.
  • Existing benchmarks: 30000 traffic-sign instances in 100000 images make TT100K a realistic benchmark spanning 45 Chinese traffic-sign classes and considerable small instances.Each sign has precise bounding-box and instance-level mask annotations, and images vary in weather and illumination.
  • Existing benchmarks: VisDrone provides 10209 drone-captured images at 2000 × 1500 pixels with 542K instances across 10 common object categories.The dataset covers urban and suburban areas in 14 Chinese cities and supports image and video detection plus tracking tasks.
  • Existing benchmarks: 1793658 instances in 11268 DOTA images span 18 categories, but its numerous small objects concentrate in only a few categories, especially small-vehicle.DOTA uses horizontal and oriented bounding-box annotations for overhead-view objects.

4 BENCHMARKS

This section introduces SODA-D and SODA-A, large-scale benchmarks tailored to small object detection in driving and aerial scenes. It defines object-size and ignore criteria, describes dataset construction and splits, and highlights their scale, diversity, resolution, density variation, and orientations.

  • Object definition and annotation: SODA defines Small objects by absolute pixel area, separates them into extremely Small, relatively Small, and generally Small subsets, and treats 1024–2000-pixel objects as Normal.Objects larger than 2000 pixels or excessively small and heavily occluded instances may receive ignore labels and do not affect evaluation.
  • Dataset construction: SODA-D and SODA-A use train, validation, and test splits of approximately 50%:20%:30% and 40%:25%:35%, respectively.SODA-D draws mainly from MVD, self-shooting, and Internet imagery, whereas SODA-A uses Google Earth imagery from hundreds of cities.
  • Benchmark characteristics: SODA-D offers diverse locations, weather, periods, views, and scenarios, together with an average image resolution of 3407 × 2470.Its 153976 well-annotated ignore regions are intended to stabilize training and evaluation.
  • Benchmark characteristics: 1 to 11134 instances per image occur in SODA-A, whose average is 347.02, while annotated orientations span −π/2 to π/2.These dense, arbitrarily oriented instances and well-annotated oriented boxes support challenging localization scenarios.

5 EXPERIMENTS

The experiments evaluate representative detectors on SODA-D and SODA-A using COCO-style AP metrics, patch-based training, and standardized open-source implementations. Results show that cascade or proposal-based methods perform strongly, while extremely small, imbalanced, occluded, low-contrast, and densely packed objects remain challenging.

  • Evaluation protocol: AP averages performance across 10 IoU thresholds from 0.5 to 0.95, with AP50, AP75, APeS, APrS, APgS, and APN also reported.The area-specific metrics emphasize performance on size-limited objects.
  • Implementation: 800 × 800 patches with stride 650 are extracted from high-resolution SODA images and resized to 1200 × 1200 for training and testing.Experiments use mmdetection for SODA-D and mmrotate for SODA-A because directly processing full-resolution images exceeds GPU memory limits.
  • SODA-D results: 31.2% AP and 27.8% AP75 make Cascade RCNN the strongest reported SODA-D detector, while RFLA reaches 29.7% AP but loses 0.7 APeS points.RCNN scores 28.9% AP, RetinaNet 28.2% AP, YOLOX 26.7% AP and 13.6% APeS, Sparse RCNN 24.2% AP, and Deformable DETR 19.2%.
  • SODA-D analysis: Rider, bicycle, motor, and traffic-camera achieve inferior category-wise AP on SODA-D because of class imbalance and limited object area.The bicycle category contains only 2560 samples, while nearly half of traffic-camera objects occupy less than 256 pixels.
  • Qualitative analysis: Visualizations show that tiny, occluded, background-like, low-contrast, and densely packed objects are frequently missed or confused by current detectors.SODA-A category performance is also affected by limited helicopter instances and elongated large-vehicle and container structures that challenge regression.
  • SODA-A results: 36.0% AP makes RoI Transformer the top SODA-A method, attributed to rotated proposals from its RRoI Learner that provide high recall for small objects.Rotated Faster RCNN scores 32.5% AP, while S2A-Net reaches 28.3% AP but declines 3.3 points on AP75 versus Rotated RetinaNet.

6 CONCLUSION AND OUTLOOK

The paper concludes with a systematic review of small object detection and two large-scale benchmarks, SODA-D and SODA-A. It identifies key directions involving feature extraction, hierarchical representation, label assignment, and evaluation metrics.

  • Contributions: The study reviewed hundreds of SOD papers and constructed SODA-D and SODA-A for driving and aerial scenes.SODA-D contains 278433 horizontally boxed instances, while SODA-A contains 872069 objects with oriented boxes.
  • Outlook: Effective feature extractors should preserve small-object information while limiting computational cost, since deeper backbones may produce poorer representations.The paper identifies backbone design as a priority for balancing feature extraction capability, computation, and information loss.
  • Outlook: FPN architectures remain suboptimal because heuristic pyramid assignment leaves few samples at higher levels and weakens high-level optimization and feature fusion.In benchmark experiments, only P2 was responsible for detection, while higher layers were optimized implicitly and indirectly.
  • Outlook: Label assignment should provide sufficient positive samples for extremely small instances, which current overlap-based and distribution-based schemes often mishandle.The paper argues that improved assignment can stabilize training and further improve performance.
  • Outlook: SOD evaluation should recognize objects and rough locations without relying excessively on stringent multiple-IoU-threshold criteria.The paper considers standard generic-detection metrics impractical for extremely small objects in scenarios where precise localization is less important.

APPENDIX A BENCHMARKS · A.1 Scene Selection

The benchmark selects driving and aerial scenes to gather abundant, identifiable small instances under realistic viewing conditions. These scenarios support fundamental detection evaluations and address demanding Small Object Detection applications.

  • APPENDIX A BENCHMARKS: The benchmark construction first explains scene-selection considerations, followed by data cleaning and instance-level annotation details.
  • A.1 Scene Selection: Driving and aerial scenes are carefully chosen to acquire vast collections of small instances for training robust deep models.
  • A.1 Scene Selection: Increasing shooting distance simulates real environments while preserving object identifiability through natural sizes and distributions.
  • A.1 Scene Selection: Intensive occurrence of similarly sized instances provides sufficient and valuable small objects for the benchmarks.
  • A.1 Scene Selection: The benchmarks use Horizontal Bounding Box and Oriented Bounding Box annotations, corresponding to horizontal and oriented object detection tasks.
  • A.1 Scene Selection: These annotation types make the benchmarks suitable for most Small Object Detection algorithms to perform evaluation and comparison.
  • A.1 Scene Selection: Autonomous driving needs reliable real-time scene understanding, but distant or occluded objects have limited sizes and challenge perception systems.
  • A.1 Scene Selection: Overhead-view image analysis urgently requires small-object handling because of large flying altitudes and varied shooting views.

A.2 Data Cleaning

The authors cleaned SODA-D and SODA-A by removing images whose artifacts or blur would impede reliable annotation, while also eliminating duplicates in SODA-D.

  • A.2 Data Cleaning: SODA-D excludes images affected by artifacts, lens flare, strong motion blur, or other annotation-impeding factors, and removes duplicates collected from different websites.SODA-A excludes images with noticeable blur and artifacts.

A.4 Data Annotation

SODA-D uses horizontal bounding boxes, while SODA-A uses polygons, with zoom-enabled annotation tools supporting fine-grained labeling. SODA-A further employs category-specific strategies, including crisscross annotation for cruciform objects and enclosed polygons for other classes.

  • Annotation tools: SODA-D objects are annotated with horizontal bounding boxes, whereas SODA-A instances use polygons.The two benchmarks therefore use different annotation types.
  • Annotation tools: LabelImg and LabelMe support annotation through high-degree zoom-in operations for fine-grained labeling.LabelImg is used for SODA-D and LabelMe for SODA-A.
  • Instance-level annotation: For SODA-D, annotators identify instances in predefined categories and draw tight bounding boxes around them.This procedure follows general detection-benchmark annotation practice.
  • Instance-level annotation: For SODA-A, airplane and helicopter instances use crisscross annotation defined by four extreme points, suited to cruciform structures.This category-specific method is introduced to improve labeling efficiency.
  • Instance-level annotation: Storage-tank and windmill instances use horizontal boxes, while the remaining SODA-A classes use enclosed polygons created with LabelMe.The annotation strategy is tailored to different categories.

APPENDIX B BENCHMARKING OF SODA-D

This appendix describes the training details for SODA-D and discusses how label assignment and loss designs affect small object detection.

  • The appendix presents training details for SODA-D.
  • It discusses the effects of label assignment on small object detection.
  • It also discusses the effects of loss designs on small object detection.

B.1 Training Details

The SODA-D benchmarking experiments use standardized optimization and scheduling settings, with model-specific learning rates and pyramidal features. High-resolution images require special handling because direct input exceeds GPU memory limits.

  • Training hyperparameters: Most detectors use SGD, while Sparse RCNN and Deformable DETR use AdamW.
  • Training hyperparameters: A 1× schedule trains most models for 12 epochs, decaying the learning rate by 10 at epochs 9 and 12.
  • Training hyperparameters: Weight decay is 0.0001 for all baseline detectors, and warmup linearly increases the learning rate to its predefined initial value.
  • Training hyperparameters: SODA-D images have very high resolutions, making direct input to deep models infeasible under GPU memory limitations.
  • Model-wise settings: CornerNet, CenterNet, Deformable-DETR, and YOLOX use tailored necks for multi-level representations, so they are excluded from the pyramidal-feature settings.
  • Model-wise settings: Pyramidal levels P2–P6 correspond to feature strides of 4, 8, 16, 32, and 64, respectively.

B.2 The Effect of Label Assignment

The section examines how label-assignment strategies affect small object detection using Faster R-CNN, RetinaNet, and FCOS as reference methods. Reported results indicate that current strategies struggle with extremely small instances, while densely arranged priors remain competitive and larger objects benefit substantially.

  • B.2 The Effect of Label Assignment: 0.8 percentage points: RFLA improves Faster R-CNN’s overall AP but deteriorates APeS.The authors attribute this deterioration to inadequate sample assignment for extremely small instances under a Gaussian Receptive Field-based scheme with only one prior.
  • B.2 The Effect of Label Assignment: FreeAnchor incorporates recall into optimization so size-limited objects obtain at least one prediction.It models training as a maximum likelihood estimation problem.
  • B.2 The Effect of Label Assignment: PAA jointly considers classification and localization in label assignment, optimization, and post-processing.It formulates anchor assignment probabilistically by calculating detector-based anchor scores and maximizing their likelihood for a probability distribution.
  • B.2 The Effect of Label Assignment: Prevailing label-assignment strategies struggle with extremely small instances but substantially improve performance on large objects.The section also concludes that densely arranged priors remain predominant compared with competing paradigms.

B.3 The Effect of Loss Function · APPENDIX C BENCHMARKING OF SODA-A

The paper evaluates how loss designs affect small object detection using RetinaNet and ATSS baselines, and introduces training details for SODA-A. GHM slightly improves overall AP and APeS over RetinaNet, while GFL substantially surpasses ATSS across metrics.

  • B.3 The Effect of Loss Function: RetinaNet and ATSS serve as baseline and reference models for studying loss designs on small objects.The experiments compare generic detection losses because no loss functions are tailored specifically to SOD.
  • B.3 The Effect of Loss Function: GHM decays weights for samples the model cannot handle, reducing attention to outliers during training.GHM is motivated by limitations of vanilla Focal Loss under imbalance.
  • B.3 The Effect of Loss Function: 28.4% overall AP is achieved by GHM, slightly ahead of RetinaNet.The passage reports GHM’s overall AP as 28.4% points.
  • B.3 The Effect of Loss Function: 0.6% points higher APeS is achieved by GHM than RetinaNet.The passage associates this gain with extremely small objects’ distribution, though the explanation is speculative.
  • B.3 The Effect of Loss Function: Table B3 reports the effect of loss designs on SOD with ResNet-50 and a ’1×’ training schedule.The table is the stated source for the loss-design comparison.
  • B.3 The Effect of Loss Function: GFL surpasses ATSS by a substantial margin at all metrics in the SOD experiments.The reported advantage is attributed to GFL’s ability to capture uncertain boundaries of small instances.
  • B.3 The Effect of Loss Function: The remarkable AP75 provides further evidence for GFL’s handling of uncertain small-instance boundaries.GFL models detection with a distribution rather than a common Dirac delta distribution under complex scenes.
  • APPENDIX C BENCHMARKING OF SODA-A: SODA-A training details and proposal parameter settings are discussed in the appendix.The supplied passage identifies this appendix subsection as covering those settings.

C.1 Training Details · C.2 Number of Proposals

The SODA-A baselines use training settings largely matching SODA-D, with patch-wise NMS adjusted to an IoU threshold of 0.1. Because SODA-A contains densely distributed objects, proposal counts must balance recall, computational cost, and detection accuracy.

  • C.1 Training Details: SODA-A baseline training hyperparameters are generally similar to those used for SODA-D.The stated exception is the patch-wise NMS IoU threshold.
  • C.1 Training Details: 0.1 is the patch-wise NMS IoU threshold used for SODA-A, following mmrotate’s default setting.This is the only training-hyperparameter difference identified relative to SODA-D.
  • C.1 Training Details: Learning rates and pyramidal features for each SODA-A baseline detector are reported in Table C1.The table specifies learning rates at batch size 4 and identifies pyramidal levels P2–P6 by their feature strides.
  • C.1 Training Details: The detailed architecture designs of the SODA-A baseline detectors are provided through the corresponding configurations in the released code.The code is available at https://shaunyuan22.github.io/SODA.
  • C.2 Number of Proposals: Dense object distributions in SODA-A require deliberate proposal-parameter settings.The section frames proposal selection as a balance between proposal count, detection accuracy, computational consumption, and recall.
  • C.2 Number of Proposals: Excessive proposals can preserve recall but incur massive computation, whereas inadequate proposals hinder overall performance.The authors therefore seek an optimal proposal-count choice rather than maximizing proposals unconditionally.
  • C.2 Number of Proposals: Table C2 compares AP and speed for Oriented RCNN under different proposal counts per patch on the SODA-A test set.Speed is measured in FPS on a single RTX 2080Ti GPU.
Loading 2207.14096v4…