Source-linked AI summary
Towards Multi-class Object Detection in Unconstrained Remote Sensing Imagery
Seyed Majid Azimi, Eleonora Vig, Reza Bahmanyar, Marco Körner, Peter Reinartz
TL;DR
Unconstrained remote-sensing imagery challenges multi-class object detection because sensors, scales, orientations, categories, resolutions, and backgrounds vary widely. The paper addresses this with joint image and feature pyramids, rotational proposal and ROI networks, and geometric loss enforcement, achieving large-margin DOTA gains and competitive cross-dataset results.
Problem
Multi-class object detection in remote-sensing imagery must accommodate diverse sensors, ground sampling distances, object scales, orientations, categories, and complex backgrounds for applications including traffic monitoring and disaster management.
Method
The method combines image cascade and feature pyramid networks with deformable and multi-size convolutions, rotational R-RPN and R-ROI modules, rotational NMS, and losses for HBB, OBB, and rectangular geometry.
Results
The method achieves large-margin improvements over published algorithms on DOTA and competitive results on NWPU VHR-10 and UCAS-AOD, including when trained on DOTA.
Takeaways & Limitations
The approach supports multi-class object detection in unconstrained aerial and satellite imagery across varying image and object scales and orientations.
Abstract
from arXiv · showhide
Automatic multi-class object detection in remote sensing images in unconstrained scenarios is of high interest for several applications including traffic monitoring and disaster management. The huge variation in object scale, orientation, category, and complex backgrounds, as well as the different camera sensors pose great challenges for current algorithms. In this work, we propose a new method consisting of a novel joint image cascade and feature pyramid network with multi-size convolution kernels to extract multi-scale strong and weak semantic features. These features are fed into rotation-based region proposal and region of interest networks to produce object detections. Finally, rotational non-maximum suppression is applied to remove redundant detections. During training, we minimize joint horizontal and oriented bounding box loss functions, as well as a novel loss that enforces oriented boxes to be rectangular. Our method achieves 68.16% mAP on horizontal and 72.45% mAP on oriented bounding box detection tasks on the challenging DOTA dataset, outperforming all published methods by a large margin (+6% and +12% absolute improvement, respectively). Furthermore, it generalizes to two other datasets, NWPU VHR-10 and UCAS-AOD, and achieves competitive results with the baselines even when trained on DOTA. Our method can be deployed in multi-class object detection applications, regardless of the image and object scales and orientations, making it a great choice for unconstrained aerial and satellite imagery.
1 Introduction
Remote-sensing object detection must handle diverse sensors, resolutions, object scales, orientations, and complex backgrounds for applications such as traffic monitoring and disaster management. The paper proposes an end-to-end CNN combining image and feature pyramids, deformable and multi-size convolutions, rotational proposal and ROI networks, and geometric loss enforcement for accurate HBB and OBB localization.
- Remote-sensing object detection supports applications including traffic monitoring, parking utilization, disaster management, urban management, search and rescue, and maritime monitoring.
- Different imaging modalities and ground sampling distances from 10 cm to 30 m, together with varying object scales, sizes, and orientations, make detection challenging.
- The proposed end-to-end CNN is designed for images spanning scales, aspect ratios, GSDs, and complex backgrounds, while using oriented bounding boxes for localization.
- The method combines a joint image cascade and feature pyramid network, deformable inception network, multi-scale rotational R-RPN and R-ROI, and rotational NMS.
- Its training objective jointly predicts categories, HBB and OBB coordinates, and rectangular OBB geometry by constraining quadrilateral edge angles to 90 degrees.
- The approach achieves significant improvements over published methods on three challenging datasets and is reported as robust across spatial resolutions from airborne and space-borne platforms.
2 Proposed Method
The proposed pipeline combines image and feature pyramids with deformable, multi-scale feature extraction and rotation-aware proposal and ROI modules. It jointly predicts horizontal and oriented boxes while adding a rectangularity constraint for oriented detections.
- Image Cascade, Feature Pyramid, and Deformable Inception Subnetworks: ICN resizes inputs to 1.5×, 1×, 0.75×, and 0.5×, then processes them with weight-sharing ResNet subnetworks.Different cascade levels use different subsets of residual blocks, limiting added computation beyond input resizing.
- Image Cascade, Feature Pyramid, and Deformable Inception Subnetworks: FPN combines strong top-level semantics with weaker bottom-level features through top-down upsampling and lateral connections.Its bottom-up hierarchy is supplied by ICN residual-block outputs C_i, i ∈{1, 2, 3, 4, 5}.
- Image Cascade, Feature Pyramid, and Deformable Inception Subnetworks: DIN uses deformable convolutions to accommodate geometric transformations and localize objects outside a fixed kernel range in remote-sensing images.The added offset layer is trained from scratch to adapt the pretrained backbone to the new domain.
- Rotation Region Proposal Network: R-RPNs generate rotated proposals at 0°, 45°, 90°, and 135° across FPN levels using K-means++-clustered anchor scales and aspect ratios.Anchors are assigned at each level from P2 through P6; P6 is used for large-object proposals, while P1 is omitted for memory reasons.
- Rotated Region of Interest Network: R-ROI rotates proposals to axis alignment, pools fixed-length features, and predicts classes, refined HBB and OBB positions, and OBB angles.The module uses sequential fully connected layers followed by four sibling output layers.
- Rotated Region of Interest Network: The final multi-task loss jointly predicts classes, HBB coordinates, OBB coordinates, and rectangular OBB geometry.The rectangularity term penalizes angles between adjacent quadrilateral edges that differ from 90°.
3 Experiments and Discussion
Experiments evaluate the method on three multi-class remote-sensing datasets, ablate its components, and analyze localization and false positives. The method benefits from image cascades, DIN, rotated proposals, and task-specific loss choices, while performance varies with object scale and shape.
- Datasets and evaluation: Experiments use DOTA, UCAS-AOD, and NWPU VHR-10, with DOTA providing the main ablation and benchmark evaluation.DOTA contains 2,806 diverse images and 15 categories; UCAS-AOD and NWPU VHR-10 provide additional multi-class tests.
- Ablation studies: Adding a 1.5× image-cascade level increased validation mAP by around 2%, whereas further 1.75× and 2× levels degraded accuracy.The degradation was attributed to missing annotations for very small objects, while downsampled levels yielded smaller gains.
- Ablation studies: Replacing 1 × 1 convolutions with DIN augmented mAP by more than 2%, with stronger gains when DIN was applied after lower-level residual blocks.Using DIN after multiple blocks improved performance further, and deformable convolutions had a smaller positive impact.
- Ablation studies: Rotated proposals improved mAP by 1.4%, while clustered rotated-anchor initialization added 0.7% mAP.The rotated setup used 18 anchors at four angles, for 18×4 total anchors.
- Ablation studies: Joint HBB and OBB loss increased OBB prediction by 0.81% mAP but reduced HBB mAP by 3.98% relative to HBB-only regression.The authors therefore omitted the OBB loss for the HBB benchmark.
- Error analysis: False positives were dominated by background confusion, while similar-class confusion was smaller and particularly affected long objects and vehicle subclasses.Overall proposal AR on DOTA was 61.25%; very small and very large objects had the lowest localization recall.
- State-of-the-art comparison: The method significantly outperformed published methods on DOTA and improved upon baselines on NWPU VHR-10 and UCAS-AOD, remaining competitive when trained only on DOTA.Training on DOTA trainval added 2–4% mAP over training on train alone, although helicopter, bridge, and SBF mAPs were lower than the baseline.
4 Conclusions
The paper concludes with a multi-component detector for unconstrained remote-sensing imagery that combines image and feature pyramids, rotation proposals, and geometric shape enforcement. It reports large gains on DOTA and robustness across imagery from different spatial resolutions and platforms.
- Conclusions: The proposed algorithm combines image cascades, feature pyramids, rotation proposals, and geometric shape enforcement for multi-class detection in unconstrained remote-sensing imagery.It was evaluated on three challenging datasets.
- Conclusions: The method outperforms published algorithms on DOTA by a large margin and remains robust across spatial resolutions and airborne or space-borne platforms.The conclusion specifically cites comparisons with methods [30, 32].
- Cross-dataset results: The UCAS-AOD results include 95.67 mAP for the proposed method trained and tested on UCAS-AOD, and 86.13 mAP when trained on DOTA and tested on UCAS-AOD.The baseline Xia et al. result on UCAS-AOD is 89.41 mAP.