PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Loss

Xue Yang, Junchi Yan, Qi Ming, Wentao Wang, Xiaopeng Zhang, Qi Tian

arXiv:2101.11952v4cs.CVcs.AI

TL;DR

Rotated detection regression is challenged by metric–loss inconsistency, boundary discontinuity, and square-like behavior. The paper represents rotated boxes as 2-D Gaussians and uses GWD to approximate a differentiable rotational IoU-induced loss. Experiments across five datasets and different detectors show the approach's effectiveness.

  • Problem

    Angle-regression detectors face metric–loss inconsistency, boundary discontinuity, and square-like problems that remain without a unified solution.

  • Method

    The method converts rotated bounding boxes into 2-D Gaussian distributions and uses Gaussian Wasserstein Distance to approximate the indifferentiable rotational IoU-induced loss for gradient-based learning.

  • Results

    Experiments on five public datasets and two popular detectors show the effectiveness of the GWD-based approach.

  • Takeaways & Limitations

    GWD aligns model learning with detection accuracy and resolves boundary discontinuity and square-like problems regardless of bounding-box definition.

  • Takeaways & Limitations

    The method requires a differentiable learning objective, and GWD alone can be sensitive to large errors.

Abstract

from arXiv · show

Boundary discontinuity and its inconsistency to the final detection metric have been the bottleneck for rotating detection regression loss design. In this paper, we propose a novel regression loss based on Gaussian Wasserstein distance as a fundamental approach to solve the problem. Specifically, the rotated bounding box is converted to a 2-D Gaussian distribution, which enables to approximate the indifferentiable rotational IoU induced loss by the Gaussian Wasserstein distance (GWD) which can be learned efficiently by gradient back-propagation. GWD can still be informative for learning even there is no overlapping between two rotating bounding boxes which is often the case for small object detection. Thanks to its three unique properties, GWD can also elegantly solve the boundary discontinuity and square-like problem regardless how the bounding box is defined. Experiments on five datasets using different detectors show the effectiveness of our approach. Codes are available at https://github.com/yangxue0827/RotationDetection and https://github.com/open-mmlab/mmrotate.

1. Introduction

Rotated detection regression suffers from metric–loss inconsistency, boundary discontinuity, and square-like behavior. The paper models rotated boxes as 2-D Gaussians and uses GWD to provide a differentiable, definition-independent loss addressing these issues.

  • Rotated object detection spans aerial images, scene text, faces, 3D objects, and retail scenes but remains relatively early-stage with open problems.
  • Angle-regression detectors introduce metric–loss inconsistency, boundary discontinuity, and the square-like problem, which can hurt performance at parameterization boundaries.
  • The proposed method converts each rotated bounding box into a 2-D Gaussian and approximates indifferentiable rotational IoU-induced loss with Gaussian Wasserstein Distance.
  • GWD-based detectors align model learning with detection accuracy and are immune to boundary discontinuity and square-like problems regardless of bounding-box definition.
  • Experiments on five public datasets and two popular detectors show the effectiveness of the proposed approach.

2. Related Work

Related work extends horizontal detection to rotated boxes, predominantly through angle regression, but existing methods address different failure modes separately. The paper presents GWD as a unified approach decoupled from bounding-box definitions.

  • Rotated object detection: Rotated detection research extends classical horizontal detectors with rotated bounding boxes, while regression-based detectors remain dominant across applications.
  • Boundary discontinuity and square-like problems: Existing methods target boundary discontinuity or square-like behavior through loss smoothing, angle classification, or long-edge-specific designs.
  • Approximate differentiable rotating IoU loss: Rotating IoU is indifferentiable, so prior approximate losses use pixel counting, projection, or combined IoU and Smooth L1 formulations.
  • Bounding-box definitions: OpenCV and long-edge protocols exchange edge order and differ by 90° in angle when representing the same box.
  • The proposed method addresses the interleaved problems in a unified manner and is decoupled from the specific bounding-box definition.

3. Rotated Object Regression Detector Revisit

Angle-regression detectors face metric–loss inconsistency, boundary discontinuity, and square-like problems that are coupled to bounding-box parameterization. The section motivates Gaussian Wasserstein Distance as a differentiable, unified alternative for rotated-box regression.

  • Bounding Box Definition: OpenCV and long-edge protocols exchange edge order and differ by 90° when representing the same bounding box.They use different angle ranges and edge conventions, creating distinct parameterizations of equivalent boxes.
  • Motivation: Rotated detectors commonly regress angles, introducing metric–loss inconsistency, boundary discontinuity, and square-like problems.These issues can substantially hurt performance, especially near boundary positions.
  • Inconsistency between Metric and Loss: Smooth L1 is convex with angle differences, but its loss remains constant across aspect ratios while IoU loss changes drastically.Center-point shifts also produce poor consistency between the compared loss functions.
  • Proposed Direction: The proposed approach models rotated boxes as two-dimensional Gaussians and uses Wasserstein distance to replace indifferentiable rotational IoU loss.The method is presented as a unified solution that is decoupled from the specific bounding-box definition.
  • Boundary Discontinuity and Square-Like Problem: At parameter boundaries, equivalent regression paths can incur sharply different losses because angle periodicity and edge exchangeability create discontinuities.A direct rotation may leave the defined angle range, forcing a more complex alternative involving larger rotation and scale changes.
  • Boundary Discontinuity and Square-Like Problem: Square-like boxes can have IoU close to 1 yet high regression loss when angle parameters are inconsistent.This mismatch can encourage the model to rotate farther rather than choose the small-angle prediction.

4. The Proposed Method

The method represents rotated bounding boxes as 2-D Gaussian distributions and uses Gaussian Wasserstein Distance to construct a differentiable approximation of rotational IoU loss. Its properties address loss–metric inconsistency, boundary discontinuity, square-like ambiguity, and dependence on box-definition conventions.

  • Gaussian representation: The proposed loss converts each rotating bounding box B(x, y, w, h, θ) into a 2-D Gaussian distribution N(m, Σ).The covariance uses a rotation matrix R and a diagonal matrix S of eigenvalues.
  • Wasserstein loss: Gaussian Wasserstein Distance provides a differentiable approximation of the IoU-induced loss for rotating bounding boxes.The method follows a Wasserstein-distance formulation and uses a nonlinear transformation f(·) with hyperparameter τ to make the loss smoother and more expressive.
  • Wasserstein loss: For horizontal boxes, the Wasserstein formulation becomes approximately equivalent to the l2-norm loss, supporting its use as a regression loss.This equivalence includes an additional denominator of 2 for width and height.
  • Wasserstein loss: GWD remains informative for non-overlapping boxes and produces a loss curve more consistent with IoU than smooth L1 loss.The paper states that GWD can measure distances when IoU=0, while the corresponding rotating IoU losses are indifferentiable.
  • Representation robustness: Gaussian covariance properties make equivalent box definitions and square-like representations produce the same or nearly the same Gaussian distribution.The method is therefore not bound to a specific box definition, although angle definitions and regression ranges can still affect final performance.
  • Detector formulation: The detector uses RetinaNet with five-parameter rotated boxes (x, y, w, h, θ) and a multi-task loss combining classification and regression terms.The regression targets normalize center coordinates, logarithmically transform width and height, and subtract anchor angle; λ1 and λ2 control loss-term trade-offs.

5. Experiments

Experiments across five datasets and different detectors evaluate GWD against alternative losses, box definitions, training strategies, and detection settings. Results report gains across aerial and scene-text benchmarks, high-IoU evaluation, and comprehensive comparisons, while noting fairness limits in cross-method comparisons.

  • Loss ablation: With f(·) = sqrt and τ = 2, the model achieves the best performance at about 68.93%, while the regression loss d2 reaches only 49.11%.Using sqrt outperforms log by about 0.98±0.3%.
  • Ablation results: GWD improves performance under both tested box definitions, increasing results by 2.14% and 3.20% while avoiding dependence on a specific definition for boundary or square-like problems.Under smooth L1, the reported Dle-based accuracy is 64.17% versus 65.73% for the comparison definition.
  • Ablation results: GWD improves RetinaNet by 1.28%, 0.88%, 3.20%, and 2.14% across HRSC206, UCAS-AOD, and DOTA evaluations, and gains 6.16% and 4.51% on MLT and ICDAR2015.The method also improves R3Net on DOTA by 0.90% and retains gains of 1.31% to 1.56% with augmentation or a stronger detector.
  • Training strategies and tricks: Multi-scale training and testing increases performance by 0.9%, 1.09%, and 0.58%, while longer schedules add 0.77% and 1.22%.SWA adds 0.99%, 1.20%, and 1.13% on DOTA; multi-scale cropping improves GTF and SBF by 6.43% and 6.14%.
  • Further and overall comparison: At AP75 on HRSC2016, GWD improves results by 11.89% and 22.46% for two detectors, and reaches 38.68% mAP75 and 38.71% mAP50:95 on DOTA.The paper reports about 80.23% overall performance on DOTA and about 89.85% and 97.37% on HRSC2016 under two evaluation metrics.
  • Comprehensive overall comparison: The paper cautions that DOTA comparisons are not absolutely fair because methods use different resolutions, network structures, training strategies, and tricks.The analysis emphasizes large-aspect-ratio and square-like categories as containing many corner cases.

6. Conclusion

The paper presents GWD as a loss for modeling deviation between rotating bounding boxes. It directly aligns with detection accuracy, supports efficient back-propagation, addresses boundary discontinuity and square-like problems independently of box definition, and achieves state-of-the-art performance on public benchmarks.

  • 6. Conclusion: GWD models deviation between two rotating bounding boxes and can be efficiently learned through back-propagation.The loss is designed to align model learning with detection accuracy.
  • 6. Conclusion: GWD addresses boundary discontinuity and square-like problems regardless of how the bounding box is defined.The conclusion attributes these properties to the design of the Gaussian Wasserstein distance-based loss.
  • 6. Conclusion: Experiments on extensive public benchmarks report state-of-the-art performance for the proposed detector.

Appendix

The appendix outlines the Gaussian Wasserstein distance formulation and its proof strategy, then evaluates alternative GWD-based regression-loss forms on DOTA.

  • Proof of Gaussian Wasserstein distance: The Wasserstein coupling distance is defined over joint random vectors with prescribed marginal distributions, yielding a formula for Gaussian distributions.The proof reduces to centered Gaussians, restricts the optimization to Gaussian couplings, and applies a covariance constraint.
  • Proof of Gaussian Wasserstein distance: In the commutative covariance case, the Gaussian Wasserstein formula reduces to a Frobenius-norm expression that approximately matches horizontal-detector l2 loss.The width and height terms include an additional denominator of 2, supporting Wasserstein distance as a regression loss.
  • Proof of Gaussian Wasserstein distance: The proof derives the Gaussian Wasserstein formula by minimizing over admissible covariance couplings or, alternatively, by constructing an optimal linear transportation map.Both routes identify N(m2, Σ2) as the image law of N(m1, Σ1) under a linear map.
  • Improved GWD-based regression loss: Figure 6 compares different forms of the GWD-based regression loss curve, while Table 9 reports their DOTA ablation results with RetinaNet.The compared forms are d2, 1 − 1/(τ+f(d2)), and f(d2).
  • Improved GWD-based regression loss: 49.11% results from directly using d2, whereas 1 − 1/(τ+f(d2)) improves performance by fitting IoU loss; sqrt with τ = 2 reaches 68.93%.Using sqrt outperforms log by about 0.98±0.3%, while directly using f(d2) with sqrt reaches 54.27%.
Loading 2101.11952v4…