Source-linked AI summary
You Only Look Twice: Rapid Multi-Scale Object Detection In Satellite Imagery
Adam Van Etten
TL;DR
Small objects in large satellite images challenge conventional detection because imagery is vast and targets may occupy only a few pixels. YOLT addresses this with a multi-scale, native-resolution pipeline for arbitrary-size imagery, achieving strong detection scores and rapid processing across sensors and object scales.
Problem
Small-object detection in large satellite imagery remains challenging because targets are tiny and images span enormous pixel and geographic areas.
Method
YOLT combines finer-grained detection, multi-scale classifiers, sliding-window evaluation, and post-processing for arbitrary-size satellite images.
Results
F1 scores of approximately 0.6–0.9 were achieved depending on category, while objects of at least 5 pixels achieved F1 > 0.85.
Takeaways & Limitations
YOLT processes vehicles and buildings at approximately 30 km2 per minute and airports at approximately 6000 km2 per minute, with cross-sensor application demonstrated.
Takeaways & Limitations
The airplane, airport, and watercraft training datasets are small by computer-vision standards, and larger datasets may improve inference performance.
Abstract
from arXiv · showhide
Detection of small objects in large swaths of imagery is one of the primary problems in satellite imagery analytics. While object detection in ground-based imagery has benefited from research into new deep learning approaches, transitioning such technology to overhead imagery is nontrivial. Among the challenges is the sheer number of pixels and geographic extent per image: a single DigitalGlobe satellite image encompasses >64 km2 and over 250 million pixels. Another challenge is that objects of interest are minuscule (often only ~10 pixels in extent), which complicates traditional computer vision techniques. To address these issues, we propose a pipeline (You Only Look Twice, or YOLT) that evaluates satellite images of arbitrary size at a rate of >0.5 km2/s. The proposed approach can rapidly detect objects of vastly different scales with relatively little training data over multiple sensors. We evaluate large test images at native resolution, and yield scores of F1 > 0.8 for vehicle localization. We further explore resolution and object size requirements by systematically testing the pipeline at decreasing resolution, and conclude that objects only ~5 pixels in size can still be localized with high confidence. Code is available at https://github.com/CosmiQ/yolt.
1 INTRODUCTION
Satellite imagery challenges conventional object detection because targets are tiny, variably oriented, sparsely labeled, and embedded in enormous images. YOLT addresses these issues while exploiting consistent object pixel sizes from satellite viewing geometry.
- Challenges: Satellite images contain small, densely clustered objects unlike the large subjects common in ImageNet data.Cars may occupy only approximately 15 pixels even in 30 cm GSD imagery.
- Challenges: Objects viewed from overhead can have arbitrary orientations, requiring complete rotation invariance.Ships may appear at any heading between 0 and 360 degrees.
- Challenges: Training data for satellite object detection is relatively scarce compared with major computer-vision datasets.The paper notes efforts such as SpaceNet to improve this situation.
- Challenges: Images often contain hundreds of megapixels, so downsampling them to standard network input sizes would discard essential object detail.A 416 × 416 sliding-window approach produces approximately 1500 cutouts for one 8 × 8 km DigitalGlobe image.
- Approach: YOLT addresses these issues separately while leveraging the approximately 400 km sensor-to-object distance and nadir-facing sensors to obtain consistent object pixel sizes.Its contributions include finer-grained detection, native-resolution processing of arbitrary-size images, and augmentation for rotation and limited labeled data.
2 RELATED WORK
Standard ground-based detection architectures struggle with overhead imagery because downsampling loses tiny targets, dense scenes exceed coarse grids, and full-image processing is memory-intensive. YOLT responds with finer-scale, multi-scale processing over manageable native-resolution chips.
- Limitations: Standard convolutional detectors are often poorly suited to satellite object sizes, orientations, and hundreds-of-megapixel images.Their multiple downsampling layers produce coarse features for objects only a few pixels in extent.
- Response: Rotation and augmentation of training images make the classifier agnostic to object heading and more robust to appearance variation.The method rotates images around the unit circle and randomly scales HSV values.
- Limitations: Whole-image inference is infeasible on typical hardware because a 256 megapixel image exceeds the memory capacity of a 12 GB GPU.Although global context improves background differentiation, the corresponding memory footprint is impractical.
- Limitations: Downsampling a 4000 × 4000 test image to 416 × 416 caused the standard YOLO model to miss all 1142 cars.A 416 × 416 cutout also produced many false negatives when dense cars could not be differentiated by the 13 × 13 grid.
- Limitations: Sliding-window search across satellite images can become computationally intractable because multiple window sizes are needed for different object scales.The paper estimates more than one million cutouts for a 10 meter boat in a DigitalGlobe image.
- Response: YOLT combines local interpolation on approximately 200 meter chips with a multi-scale ensemble of detectors.This design targets both tiny overhead objects and the broad scale range found in satellite imagery.
3 YOU ONLY LOOK TWICE
YOLT is an overhead-imagery detection framework combining a finer-grained network with sliding-window testing and global post-processing. It trains and processes labeled imagery from multiple sensors and object categories.
- 3 YOU ONLY LOOK TWICE: YOLT extends Darknet into an object-detection framework optimized for overhead imagery.The framework integrates geospatial-image analysis with Python-based preprocessing and post-processing.
- 3.1 Network Architecture: The network uses 22 layers and downsamples by 16, producing a 26 × 26 prediction grid from a 416 × 416 input.The denser grid is intended for small, densely packed objects such as cars and buildings.
- 3.1 Network Architecture: Its final layer predicts bounding boxes and classes with size N_f = N_boxes × (N_classes + 5).N_boxes denotes boxes per grid and N_classes denotes object classes.
- 3.2 Test Procedure: At test time, YOLT partitions arbitrary-size images into manageable cutouts using a sliding window with user-defined bin sizes and 15% overlap by default.Each cutout is tracked by its image, row, column, height, and width.
- 3.3 Post-Processing: Training data comprises labeled chips from DigitalGlobe, Planet, and aerial imagery covering airplanes, boats, buildings, cars, and airports.The paper uses different detectors at different scales for objects such as airplanes and airports.
- 3.3 Post-Processing: The pipeline restores each cutout prediction to global image coordinates and applies non-maximal suppression to merge detections overlapping at chip boundaries.This recombines hundreds or thousands of chips into one final image strip.
4 TRAINING DATA
YOLT training data combines imagery from DigitalGlobe, Planet, and aerial platforms across five object categories, using scale-specific preparation for large infrastructure. Several datasets are small by computer-vision standards.
- Training data comes from DigitalGlobe satellites, Planet satellites, and aerial platforms, covering airplanes, boats, building footprints, cars, and airports.
- The COWC dataset provides high-quality annotated overhead car imagery at 15 cm GSD, while labels are converted into 3 meter bounding boxes for training.
- 230 airplanes and 556 boats are labeled from DigitalGlobe imagery, while airports are labeled in 37 Planet training images.
- Airport imagery is downsampled by a factor of four because runways can exceed 1000 pixels in low-resolution Planet images.
- The raw airplane, airport, and watercraft training datasets are small by computer-vision standards, and larger datasets may improve inference performance.
- Training uses stochastic gradient descent with 5 boxes per grid, a 10^-3 initial learning rate, 0.0005 weight decay, and 0.9 momentum.
5 TEST IMAGES
Test images are geographically separated from training examples to evaluate robustness across regions and object categories. The test set includes large labeled collections for cars and buildings but smaller sets for airports, airplanes, and boats.
- All test images come from geographic regions different from the training examples.
- The test split contains 19,807 cars, 73,778 building footprints, 74 airplanes, 771 boats, and ten airport images.
6 OBJECT DETECTION RESULTS
YOLT shows strong detection performance across several satellite-imagery tasks, but a universal classifier confuses airports with similarly scaled linear features. Scale-specific classifiers address this issue while retaining rapid inference over large images.
- Universal Classifier Object Detection Results: The universal classifier produces spurious airport detections because downsampled runways resemble highways at the wrong scale.
- Scale Confusion Mitigation: YOLT uses separate classifiers for vehicles and buildings versus airports, applying them at 200m and 2500m scales before merging overlapping detections.
- Car Detection: F1 = 0.95 is achieved on a 600 × 600 meter Salt Lake City image containing 1389 cars, with GPU processing time below 1 second.
- Building Footprints: YOLT scores 0.61 on SpaceNet building-footprint detection, placing it in the top three, despite not being optimized for footprint extraction.
- Multi-category Results: Airplane, boat, and airport detection on a SpaceNet DigitalGlobe image yields F1 scores of 0.83, 0.84, and 1.0, respectively.
- Speed: GPU inference runs at approximately 50 frames per second, translating to under 6 minutes for vehicles and under 2 seconds for airports across Washington DC-sized areas.
- Detailed Performance Analysis: YOLT achieves a COWC per-scene F1 of 0.90±0.09 and a predicted-to-ground-truth car-count fraction of 0.95 ± 0.05.
7 RESOLUTION PERFORMANCE STUDY
The resolution study finds that object detection remains strong as car-sized objects shrink to roughly five pixels, but performance drops sharply for one-pixel objects. Models trained at each resolution outperform a single high-resolution model across resolutions.
- F1 declines from 0.92 for 20-pixel objects to 0.27 for 1-pixel objects, with a mean error of 0.09.
- Shrinking objects from 20 to 5 pixels reduces F1 by only approximately 5%.
- Objects at least 5 pixels in size achieve F1 > 0.85 for overhead cars.
- A series of models trained at each resolution performs better than a single model trained at high resolution.
8 CONCLUSIONS
YOLT adapts object detection to overhead imagery by using scale-specific classifiers and a fully convolutional pipeline for large, arbitrarily sized images. It achieves category-dependent F1 scores of approximately 0.6–0.9, detects objects as small as five pixels with F1 > 0.85, and supports rapid processing across sensors.
- YOLT uses a fully convolutional pipeline to localize vehicles, buildings, and airports in satellite imagery.
- Training dual classifiers at different scales improved results by separating buildings and vehicles from infrastructure.
- F1 scores range from ≈0.6–0.9 depending on object category.
- F1 > 0.85 is achieved for overhead cars whose objects are ≥5 pixels in size.
- At least for cars viewed overhead, models trained on one sensor can be applied to imagery from another sensor.
- The pipeline processes vehicles and buildings at ≈30 km2 per minute and airports at ≈6,000 km2 per minute.It evaluates satellite and aerial images of arbitrary input size at native resolution.