PaperlayerView exact revision on arXiv

Source-linked AI summary

AO2-DETR: Arbitrary-Oriented Object Detection Transformer

Linhui Dai, Hong Liu, Hao Tang, Zhiwei Wu, Pinhao Song

arXiv:2205.12785v1cs.CV

TL;DR

AOOD requires detecting arbitrarily oriented, densely arranged objects despite the limitations of anchor- and point-based pipelines. AO2-DETR uses oriented proposal generation, adaptive proposal refinement, and rotation-aware set matching in an end-to-end transformer detector. It reports state-of-the-art performance on several oriented-detection datasets while requiring longer training convergence and greater computation than CNN-based methods.

  • Problem

    AOOD involves arbitrary orientations and cluttered arrangements, while existing approaches rely on redundant candidates and complicated hand-designed processing.

  • Method

    AO2-DETR combines oriented proposal generation, adaptive oriented proposal refinement, and rotation-aware set matching for direct oriented-box prediction.

  • Results

    AO2-DETR achieves state-of-the-art performance compared with recent anchor-free and single-stage methods on DOTA, HRSC2016, and SKU110K-R.

  • Takeaways & Limitations

    The framework eliminates hand-designed components and complex pre/post-processing by transforming oriented proposals into object predictions through an encoder-decoder architecture.

  • Takeaways & Limitations

    Compared with CNN-based methods, AO2-DETR has longer training convergence time and requires relatively larger computation cost.

Abstract

from arXiv · show

Arbitrary-oriented object detection (AOOD) is a challenging task to detect objects in the wild with arbitrary orientations and cluttered arrangements. Existing approaches are mainly based on anchor-based boxes or dense points, which rely on complicated hand-designed processing steps and inductive bias, such as anchor generation, transformation, and non-maximum suppression reasoning. Recently, the emerging transformer-based approaches view object detection as a direct set prediction problem that effectively removes the need for hand-designed components and inductive biases. In this paper, we propose an Arbitrary-Oriented Object DEtection TRansformer framework, termed AO2-DETR, which comprises three dedicated components. More precisely, an oriented proposal generation mechanism is proposed to explicitly generate oriented proposals, which provides better positional priors for pooling features to modulate the cross-attention in the transformer decoder. An adaptive oriented proposal refinement module is introduced to extract rotation-invariant region features and eliminate the misalignment between region features and objects. And a rotation-aware set matching loss is used to ensure the one-to-one matching process for direct set prediction without duplicate predictions. Our method considerably simplifies the overall pipeline and presents a new AOOD paradigm. Comprehensive experiments on several challenging datasets show that our method achieves superior performance on the AOOD task.

I. INTRODUCTION

AOOD is difficult because objects may have arbitrary orientations, dense layouts, and cluttered backgrounds, while conventional detectors rely on redundant candidates and hand-designed processing. AO2-DETR addresses these issues with oriented proposals, adaptive refinement, and rotation-aware matching in an end-to-end transformer detector.

  • AOOD is challenging because objects can have arbitrary orientations, dense arrangements, and highly complex backgrounds.
  • Conventional oriented detectors use rotated boxes or dense points, producing redundant predictions and requiring anchor generation, transformations, and NMS reasoning.
  • Directly extending transformers with horizontal object queries causes misalignment, cluttered features, and limited matching for rotated objects.
  • AO2-DETR generates oriented proposals as object queries, providing positional priors that improve feature pooling and cross-attention for arbitrary-oriented objects.
  • The framework combines adaptive oriented proposal refinement with rotation-aware set matching to reduce feature misalignment and support one-to-one prediction.
  • Experiments on DOTA-v1.0, DOTA-v1.5, SKU110K-R, and HRSC2016 report state-of-the-art performance among anchor-free and single-stage methods.

II. RELATED WORKS

Related oriented detectors use anchors, dense points, or keypoints, but these approaches involve manual design, redundant candidates, or specialized assignment strategies. AO2-DETR instead uses an end-to-end transformer design with oriented proposals and one-to-one matching.

  • A. Anchors in Oriented Object Detection: Anchor-based detectors place rotated anchors with varied angles, scales, and aspect ratios, incurring substantial computation and memory costs.
  • A. Anchors in Oriented Object Detection: Keypoint-based methods construct oriented boxes from object keypoints and introduce specialized sampling, alignment, or representation mechanisms.
  • A. Anchors in Oriented Object Detection: AO2-DETR directly predicts absolute object positions without manual preset boxes or complicated hand-designed components.
  • B. Label Assignment Strategy for Oriented Object Detection: Anchor-based methods commonly use dataset-dependent thresholds, whereas transformer detectors assign one positive sample per ground-truth box through bipartite matching.
  • C. Transformer Network and its Application: Transformers provide permutation-invariant processing, adaptive receptive fields, and global-dependency modeling for oriented objects.
  • C. Transformer Network and its Application: AO2-DETR differs from prior transformer detectors through oriented proposals and an adaptive refinement module, offering fewer inductive biases for AOOD.

III. METHOD

AO2-DETR combines a CNN backbone, deformable encoder and decoder, oriented proposal generation, adaptive proposal refinement, and rotation-aware matching for arbitrary-oriented detection.

  • AO2-DETR processes images with a CNN backbone and transformer encoder to produce multi-scale feature maps with positional and scale-level embeddings.
  • Oriented proposal generation creates region proposals from encoder memory, providing positional priors for pooling features and modulating decoder cross-attention.
  • Adaptive oriented proposal refinement reconstructs feature maps, refines initial proposals, and selects top-k refined proposals as decoder object queries.
  • The deformable decoder transforms object queries through self-attention and multi-scale deformable attention before independently predicting classes and oriented boxes.
  • Rotation-aware matching loss enforces correct one-to-one matching during training, while the feed-forward network can predict either detections or “no object.”

C. Oriented Proposals Generation Mechanism

The proposal-generation and refinement design addresses orientation-induced feature misalignment by generating oriented priors and adaptively reconstructing aligned features.

  • Oriented Proposal Generation: The oriented proposal generation mechanism learns each proposal’s center, size, and angle, producing more accurate oriented object queries than horizontal proposals.
  • Motivation: Fixed axis-aligned convolutional receptive fields can misalign extracted features with arbitrarily oriented objects, especially under dense arrangements.
  • Adaptive Proposal Refinement: The refinement module combines three-way convolutions with a feature alignment module to exploit contextual information and adjust oriented proposals.
  • Feature Alignment: Feature alignment re-encodes initial proposal positions at corresponding feature points and reconstructs the feature map using pixel-wise bilinear interpolation.
  • Adaptive Proposal Refinement: The reconstructed feature map captures proposal geometry and surrounding context, then feeds the deformable encoder and proposal-generation module to produce refined proposals.

E. Deformable Transformer Decoder

AO2-DETR’s deformable decoder attends around oriented reference proposals, iteratively refining queries and boxes to better align attention with arbitrarily oriented objects.

  • The decoder uses multi-scale deformable attention to extract features across feature maps, encoding broader context and learning oriented receptive fields.
  • Unlike horizontal-proposal attention, the module samples a small fixed number of key points around each oriented region proposal.
  • Sampling offsets and attention weights are defined per feature level, attention head, and sampled key within the multi-scale deformable attention operation.
  • Refined oriented proposals make decoder attention correlate more strongly with predicted boxes, avoiding background or neighboring-object information in dense scenes.
  • Each decoder layer iteratively refines oriented boxes, and the detection head outputs class labels and box coordinates without imposing an ordering on predictions.

F. Set Matching and Loss Function

AO2-DETR formulates oriented detection as one-to-one matching between predicted and ground-truth sets, using classification and box similarity costs. Its rotation-aware IoU term accounts for oriented-box geometry during matching, while regression and classification losses train the final predictions.

  • Set Matching: AO2-DETR predicts a fixed-sized set of N oriented boxes and matches it to a padded ground-truth set using a minimum-cost permutation.The optimal bipartite assignment is computed with the Hungarian algorithm.
  • Set Matching: The matching cost combines class predictions with the similarity between predicted and ground-truth oriented boxes.Ground-truth boxes encode center coordinates, height, width, and angle relative to image size.
  • Rotation-aware Matching: The rotation-aware set matching loss Lriou is added to ensure correct one-to-one matching between predicted oriented boxes and ground truth.It extends rotated IoU loss into the Hungarian matching process.
  • Rotation-aware Matching: Rotated IoU is computed from the four vertices of each oriented box, the overlap polygon, and its intersection area.The overlap vertices are ordered anticlockwise before the intersection area is calculated.
  • Loss Function: The training objective uses L1 regression, rotated IoU, and Focal classification losses, with matching-loss weights normalized by the number of batch objects.This normalization helps avoid complex post-processing steps.

A. Datasets

The experiments use four oriented-detection datasets spanning aerial imagery, densely packed retail objects, and ships. Dataset-specific image scales, augmentation, evaluation protocols, and benchmark tables define the comparison settings.

  • Datasets: Experiments cover DOTA-v1.0, DOTA-v1.5, SKU110K-R, and HRSC2016 for oriented object detection.The datasets represent aerial images, commodity detection, and ship detection settings.
  • DOTA: DOTA-v1.0 contains 2,806 aerial images with 15 object categories and image sizes ranging from about 800×800 to 4000×4000.The dataset includes oriented objects such as planes, ships, vehicles, and sports fields.
  • DOTA: DOTA images are cropped into 1024 × 1024 patches with stride 824 and augmented using three scales and five random rotation angles.The reported rotation angles are 30°, 60°, 90°, 120°, and 150°.
  • SKU110K-R: SKU110K-R contains densely packed supermarket commodities, averaging 154 objects per image and reaching up to 718 objects.Evaluation uses COCO-style AP75, mAP, and AR300 metrics.
  • HRSC2016: HRSC2016 contains 1,061 harbor images in one ship category, with image sizes from 300 × 300 to 1500 × 900.Training uses the training and validation sets, while testing uses the test set.

B. Implementation Details

AO2-DETR is implemented on MMRotate using Deformable DETR with a ResNet-50 backbone and 300 object queries. The paper also defines benchmark and ablation-study settings for evaluating methods and matching costs.

  • Model Setup: AO2-DETR is implemented on MMRotate with Deformable DETR and a ResNet-50 backbone pretrained on ImageNet.Multi-scale feature maps are extracted from conv3 through conv5, using the Deformable DETR encoder-decoder architecture.
  • Evaluation Settings: Table IV compares oriented detectors on HRSC2016, distinguishing VOC2012 evaluation from the VOC2007 metrics used by other methods.The table marks the best and second-best results in red and blue, respectively.
  • Ablation Studies: Table V reports ablations of AO2-DETR modules on DOTA-v1.0, including the Deformable DETR-O baseline with predicted angles added.Bold results indicate the best performance.
  • Ablation Studies: Table VI studies the set-matching cost on DOTA-v1.0 and marks performance gains in blue.Bold results indicate the best performance.
  • Training: The implementation uses 300 object queries and trains for 50 epochs with AdamW.The learning rate is 1e−4 for 40 epochs and 1e−5 for the remaining 10 epochs.

C. State-of-the-art Comparison

AO2-DETR is evaluated against state-of-the-art oriented detectors across four datasets. It reports strong results among anchor-free and single-stage methods, with state-of-the-art performance on DOTA-v1.5, SKU110K-R, and HRSC2016 under the reported settings.

  • DOTA-v1.0: 77.73% mAP with single-scale data and 79.22% mAP with multi-scale data are achieved on DOTA-v1.0.With ResNet-50, the method outperforms R3Det by 1.26% and KLD by 0.9% under the reported comparisons.
  • DOTA-v1.0: AO2-DETR achieves the best result among the compared single-stage and anchor-free DOTA-v1.0 methods using a single model without bells and whistles.The comparison includes R3Det, KLD, CFA, SASM, and DARDet.
  • DOTA-v1.5: 66.26% mAP with single-scale data and 75.89% mAP with multi-scale data are reported on DOTA-v1.5.The paper states that these results outperform RetinaNet OBB, Faster R-CNN OBB, Mask R-CNN OBB, and HTC by a large margin.
  • SKU110K-R: 58.0% AP is achieved on SKU110K-R, improving on the compared state-of-the-art anchor-free methods by 1.0%.The reported comparison is 58.0% versus 57.0%.
  • Proposal Analysis: Visualization shows that learned proposal sampling points concentrate inside objects, focusing deformable attention on object regions.The figure displays the top-300 proposals per image on DOTA.
  • HRSC2016: 92.68% under VOC 2007 and 97.87% under VOC 2012 metrics are achieved on HRSC2016.The paper describes the result as state of the art without using a more complicated architecture.

D. Ablation Studies

Ablation studies show that AO2-DETR’s oriented proposals, proposal refinement, iterative refinement, and rotation-aware matching each contribute to performance, while query count trades speed against accuracy.

  • Oriented Proposal Generation Mechanism: OPG improves mAP by 29.44% (71.30% vs. 41.86%), indicating stronger object localization with less feature interference.The authors attribute this gain to better locating objects and reducing interference from other objects or backgrounds.
  • Adaptive Oriented Proposal Refinement Module: OPR improves performance by 4.93% (76.23% vs. 71.30%) and helps mitigate misalignment and cluttered features.The refinement modules provide more precise features for oriented object detection.
  • Adaptive Oriented Proposal Refinement Module: Removing iterative refinement reduces performance from 77.73% to 70.24%, supporting its role in modeling decoder-context relationships.The authors state that iterative refinement enhances the relationship between the transformer decoder and contextual information.
  • Set Matching Loss: Rotation-aware set matching improves performance from 56.65% to 77.73% and ensures correct one-to-one matching in AO2-DETR.The authors report that this addresses limited matching in arbitrary-oriented detection.
  • Number of Queries: Redundant object queries can interfere with dense objects, causing slight performance declines at 350 and 400 queries.This observation explains the reported performance fluctuations at higher query counts.
  • Number of Queries: Increasing queries from 150 to 400 lowers FPS from 14.2 to 13.5 on DOTA-v1.0 and from 16.3 to 15.4 on HRSC2016.The authors select 300 queries as a speed-performance balance, yielding 14.0 FPS on DOTA-v1.0 and 16.0 FPS on HRSC2016.
  • Summary: Across the ablations, oriented proposal generation, adaptive proposal refinement, iterative box refinement, and rotation-aware matching loss all play important roles.The conclusion summarizes the component-level ablation findings.

E. Qualitative Results

Qualitative results show that AO2-DETR detects objects across orientations, sizes, aspect ratios, and dense arrangements, while remaining vulnerable to severe occlusion, tiny objects, and low contrast.

  • Qualitative Results: AO2-DETR handles multi-oriented, small, large-aspect-ratio, and densely packed objects across DOTA, HRSC2016, and SKU110K-R examples.The qualitative results include multi-category detections on DOTA-v1.0 and large-aspect-ratio and densely arranged objects in Fig. 10.
  • Limitations: Qualitative failure cases remain when objects are heavily occluded, extremely small, or similar in color to the background.The authors identify these cases as targets for future transformer-based AOOD methods.
  • Qualitative Results: The conclusion describes AO2-DETR as an end-to-end transformer detector with oriented proposal generation, adaptive proposal refinement, and rotation-aware set matching.Its encoder-decoder architecture uses oriented proposals as object queries and removes hand-designed components and complex pre/post-processing.
  • Limitations: Compared with CNN-based methods, the main limitation is longer training convergence time and higher computational cost for transformers.The authors attribute the computational cost to the relatively larger computation required for transformer performance.
Loading 2205.12785v1…