Source-linked AI summary
R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object
Xue Yang, Junchi Yan, Ziming Feng, Tao He
TL;DR
Practical rotation detection remains difficult for varied orientations, large aspect ratios, and dense distributions. R3Det uses progressive coarse-to-fine refinement with feature alignment and an approximate SkewIoU loss, and experiments across four public datasets show its effectiveness. Its feature-refinement gains are reported for rotating datasets but were not satisfactory on COCO.
Problem
Rotation detection must handle objects with arbitrary orientations, large aspect ratios, and dense distributions in practical image settings.
Method
R3Det combines coarse-to-fine anchor refinement, pixel-wise feature interpolation for feature reconstruction and alignment, and a derivable approximate SkewIoU loss.
Results
FRM increased overall performance by 2.79% to 66.31% in the reported ablation, with additional gains on ICDAR2015, HRSC2016, and UCAS-AOD.
Takeaways & Limitations
The combined approach achieves state-of-the-art performance with considerable speed across DOTA, HRSC2016, UCAS-AOD, and ICDAR2015.
Takeaways & Limitations
FRM produced unsatisfactory results on COCO, although it achieved considerable gains on multiple rotating datasets.
Abstract
from arXiv · showhide
Rotation detection is a challenging task due to the difficulties of locating the multi-angle objects and separating them effectively from the background. Though considerable progress has been made, for practical settings, there still exist challenges for rotating objects with large aspect ratio, dense distribution and category extremely imbalance. In this paper, we propose an end-to-end refined single-stage rotation detector for fast and accurate object detection by using a progressive regression approach from coarse to fine granularity. Considering the shortcoming of feature misalignment in existing refined single-stage detector, we design a feature refinement module to improve detection performance by getting more accurate features. The key idea of feature refinement module is to re-encode the position information of the current refined bounding box to the corresponding feature points through pixel-wise feature interpolation to realize feature reconstruction and alignment. For more accurate rotation estimation, an approximate SkewIoU loss is proposed to solve the problem that the calculation of SkewIoU is not derivable. Experiments on three popular remote sensing public datasets DOTA, HRSC2016, UCAS-AOD as well as one scene text dataset ICDAR2015 show the effectiveness of our approach. Tensorflow and Pytorch version codes are available at https://github.com/Thinklab-SJTU/R3Det_Tensorflow and https://github.com/SJTU-Thinklab-Det/r3det-on-mmdetection, and R3Det is also integrated in our open source rotation detection benchmark: https://github.com/yangxue0827/RotationDetection.
1. Introduction
Rotation detection must handle objects with varied orientations, large aspect ratios, dense layouts, and feature misalignment while preserving speed. R3Det addresses these challenges through coarse-to-fine refinement, feature reconstruction, and an approximate SkewIoU loss.
- Rotation detectors are needed because practical images contain objects in varied orientations, including scene text, retail, and remote sensing settings.
- Large aspect ratios make SkewIoU highly sensitive to angular changes, while dense object layouts complicate detection.
- R3Det uses horizontal anchors initially for higher recall and speed, then rotating anchors in refinement stages for dense scenarios.
- The feature refinement module interpolates features around refined anchors to reconstruct aligned feature maps and reduce refined bounding boxes.
- An approximate SkewIoU loss preserves SkewIoU amplitude while approximating its gradient direction because direct SkewIoU calculation is not derivable.
2. Related Work
Related work spans two-stage and single-stage object detectors, with rotation-specific methods developed for remote sensing, scene text, and retail imagery. Two-stage systems can be accurate but may incur speed bottlenecks from more complex structures.
- Two-stage detectors generate region proposals before extracting region features and applying category-specific classification and regression.
- Rotation detection methods have been developed for remote sensing, scene text, and retail scenes, including both two-stage and single-stage designs.
3. The Proposed Method
R3Det is a refined single-stage rotation detector that progressively refines boxes while reconstructing aligned features and using a derivable approximate SkewIoU loss. Its refinement design combines anchor flexibility, feature refinement, and stage-wise optimization for rotating-object detection.
- Refined Rotation RetinaNet: R3Det adds repeatable refinement stages to RetinaNet, progressively improving predicted rotating bounding boxes and regression accuracy.The refined detector can use multiple refinement stages, with stage-specific losses and IoU thresholds.
- Rotation RetinaNet: The detector uses five parameters—x, y, w, h, and θ—to represent arbitrary-oriented rectangles and predicts an angular offset.The angle θ is defined relative to the x-axis, while the regression targets encode center, size, and angle offsets relative to anchors.
- Rotation RetinaNet: Because smooth L1 is inconsistent with SkewIoU for large-aspect-ratio objects, R3Det introduces a derivable approximate SkewIoU loss.The loss preserves SkewIoU magnitude while approximating its gradient direction, allowing differentiable optimization.
- Feature Refinement Module: FRM comprises box filtering, a large-kernel convolution, and feature reconstruction, while sampling five feature points rather than RoI Align’s default 196.Box filtering retains the highest-scoring box per feature point, enabling feature reconstruction and reducing sampling overhead.
- Feature Refinement Module: The Feature Refinement Module reconstructs feature maps by interpolating position information from refined boxes to corresponding feature points.FRM uses bilinear interpolation and replaces selected feature vectors before adding the reconstructed map to the original feature map.
4. Experiments
Experiments evaluate R3Det across DOTA, HRSC2016, UCAS-AOD, and ICDAR2015 using baseline, ablation, and state-of-the-art comparisons. The results show gains from progressive refinement, feature refinement, and approximate SkewIoU loss, alongside competitive accuracy and speed.
- Baseline methods: Progressive regression combines horizontal anchors for recall and speed with rotating refined anchors for dense-scene detection.The baseline R3Det* achieves 63.52%, exceeding RetinaNet-H and RetinaNet-R, whose overall mAP values are 62.79% and 62.76%.
- Ablation study: 2.79%: FRM raises performance from the feature-refinement baseline to 66.31% on DOTA.Across ICDAR2015, HRSC2016, and UCAS-AOD, FRM-based methods improve by 1.69%, 0.12% (1.03)%, and 1.14%, respectively, under the same configuration.
- Ablation study: Three or more refinement stages add no further overall improvement, while ensembling multi-stage outputs can improve results.The ablation study examines the relationship between refinement-stage count and performance on DOTA.
- State-of-the-art comparison: 73.79%: R3Det achieves the best performance among published single-stage DOTA methods without multi-scale training and testing.With a stronger backbone and multi-scale training and testing, R3Det reaches about 76.47%; on UCAS-AOD it achieves 96.17%.
- State-of-the-art comparison: 86.67% at 20fps: R3Det maintains accuracy and speed with MobileNetv2 on 600×600 HRSC2016 inputs.RoI Transformer reaches 86.20% mAP but runs at about 6fps without post-processing operations.
5. Conclusion
The paper presents an end-to-end refined single-stage detector for rotating objects with large aspect ratios, dense distributions, and arbitrary orientations. Its feature refinement module and approximate SkewIoU loss support state-of-the-art accuracy with high efficiency.
- Conclusion: R3Det targets rotating objects with large aspect ratios, dense distributions, and arbitrary orientations common in aerial, retail, and scene-text images.The detector is evaluated on DOTA, HRSC2016, UCAS-AOD, and ICDAR2015.
- Conclusion: FRM reconstructs and aligns features by re-encoding refined bounding-box position information through pixel-wise feature interpolation.The method addresses feature misalignment in refined single-stage detection.
- Conclusion: An approximate SkewIoU loss addresses the nondifferentiability of SkewIoU for more accurate rotation estimation.The loss preserves SkewIoU amplitude while approximating its gradient direction.
- Conclusion: Experiments demonstrate state-of-the-art detection accuracy with high efficiency across multiple rotation-detection datasets.Figure 7 reports performance versus speed on HRSC2016 and states that the algorithm surpasses competitors in accuracy while running very fast.
6. Supplementary Material
Supplementary experiments examine speed, feature sampling, loss stability, and cross-domain performance. The results include strong ICDAR2015 detection performance, while FRM behaves differently in horizontal and rotation detection.
- Results on ICDAR2015: 84.96% accuracy at 13.5fps is achieved on ICDAR2015, while R3Det‡ reaches 89.21% at 9fps under heavier settings.The heavier comparison uses extra training data and larger test images.
- Feature Refinement Module: FRM samples tighter points around rotating objects than around horizontally detected objects.The horizontal-detection corner points may lie farther from the object, whereas rotation-detection points remain close.
- Feature Refinement Module: FRM produces inaccurate sampled features for horizontal detection but achieves considerable gains across many rotating datasets.Experiments on COCO were unsatisfactory, while rotating-dataset results were substantially better.
- Loss Analysis: Approximate SkewIoU loss makes training more stable than Smooth L1 loss according to the variance and mean of their loss curves.Figure 9 compares the two training-loss curves.
- Visualization: Detection results are visualized on remote sensing and scene text datasets.The visualizations are provided in Figures 10, 11, and 12.
7. Broad Societal Implications
The paper highlights broad applications for rotation detection while acknowledging that more accurate and efficient systems may increase privacy risks. It therefore calls for additional privacy-protection measures during development.
- Broad Societal Implications: More accurate and efficient rotation detection may put individual privacy at risk, motivating additional protective measures.The stated application areas include face, remote sensing, scene text, and retail scenes.