PaperlayerView exact revision on arXiv

Source-linked AI summary

Align Deep Features for Oriented Object Detection

Jiaming Han, Jian Ding, Jie Li, Gui-Song Xia

arXiv:2008.09397v3cs.CV

TL;DR

S2A-Net addresses misalignment between oriented aerial objects, anchors, and axis-aligned convolutional features, which contributes to inconsistency between classification scores and localization accuracy. It combines feature alignment and orientation-aware detection, and experiments report state-of-the-art performance on DOTA and HRSC2016 while maintaining high efficiency.

  • Problem

    Existing aerial object detectors suffer misalignment between anchor boxes and axis-aligned convolutional features, creating inconsistency between classification scores and localization accuracy.

  • Method

    S2A-Net combines a Feature Alignment Module using ARN and AlignConv with an Oriented Detection Module using active rotating filters and orientation-sensitive and orientation-invariant features.

  • Results

    S2A-Net achieves state-of-the-art performance on DOTA and HRSC2016 while keeping high efficiency.

  • Takeaways & Limitations

    The method provides full feature alignment and alleviates inconsistency between regression and classification while supporting a better speed–accuracy trade-off for large-size images.

Abstract

from arXiv · show

The past decade has witnessed significant progress on detecting objects in aerial images that are often distributed with large scale variations and arbitrary orientations. However most of existing methods rely on heuristically defined anchors with different scales, angles and aspect ratios and usually suffer from severe misalignment between anchor boxes and axis-aligned convolutional features, which leads to the common inconsistency between the classification score and localization accuracy. To address this issue, we propose a Single-shot Alignment Network (S$^2$A-Net) consisting of two modules: a Feature Alignment Module (FAM) and an Oriented Detection Module (ODM). The FAM can generate high-quality anchors with an Anchor Refinement Network and adaptively align the convolutional features according to the anchor boxes with a novel Alignment Convolution. The ODM first adopts active rotating filters to encode the orientation information and then produces orientation-sensitive and orientation-invariant features to alleviate the inconsistency between classification score and localization accuracy. Besides, we further explore the approach to detect objects in large-size images, which leads to a better trade-off between speed and accuracy. Extensive experiments demonstrate that our method can achieve state-of-the-art performance on two commonly used aerial objects datasets (i.e., DOTA and HRSC2016) while keeping high efficiency. The code is available at https://github.com/csuhan/s2anet.

I. INTRODUCTION

Aerial object detectors must handle crowded objects with large scale variations and arbitrary orientations, but anchor-feature misalignment limits one-stage accuracy. S2A-Net addresses this through refined rotated anchors, aligned features, and orientation-aware detection.

  • Aerial images contain crowded objects with large scale variations and arbitrary orientations, challenging object detection.
  • One-stage detectors are efficient but use densely sampled anchors and often lag behind two-stage detectors in accuracy.
  • Heuristic anchors may fail to cover objects such as bridges, causing anchor-object misalignment and aggravating foreground-background class imbalance.
  • Axis-aligned convolutional features remain misaligned with arbitrarily oriented objects, making classification scores inconsistent with localization accuracy.
  • S2A-Net combines a Feature Alignment Module and an Oriented Detection Module to generate high-quality anchors, align features, and encode orientation information.
  • Alignment Convolution alleviates axis-aligned feature misalignment with negligible extra time and can be embedded into detectors with little modification.

II. RELATED WORKS

Prior aerial-image detectors address orientation and scale challenges through two-stage, rotated-anchor, or specialized architectures, often trading efficiency for accuracy. S2A-Net instead refines horizontal anchors into rotated anchors and aligns features convolutionally.

  • One-stage detectors avoid region proposal generation and usually run faster, but their performance commonly lags behind two-stage detectors because of class imbalance.
  • Aerial objects are crowded, vary in scale, and appear at arbitrary orientations, so horizontal anchors can contain multiple instances.
  • Rotated-anchor methods alleviate misalignment but incur heavy computation from anchor transforms and ground-truth matching.
  • S2A-Net uses a Feature Alignment Module to refine horizontal anchors into high-quality rotated anchors and align features fully convolutionally.

B. Feature Alignment in Object Detection

Feature alignment concerns matching convolutional features to anchor boxes or regions of interest, while classification and regression can remain inconsistent when they share misaligned features. The proposed approach adjusts sampling locations using anchor guidance.

  • Feature alignment matches convolutional features with anchor boxes or regions of interest and is important for both two-stage and one-stage detectors.
  • RoIPooling, RoIAlign, and Deformable RoIPooling extract region features with progressively more adaptive handling of sampling or alignment.
  • RoI operators can involve substantial region-wise operations such as feature warping and interpolation.
  • Guided Anchoring and Align-Det align features in one-stage detection, whereas S2A-Net adjusts sampling locations for arbitrarily oriented objects using anchor boxes.
  • Shared backbone features support classification and bounding-box regression, yet high classification scores may accompany low localization accuracy and suppress better-localized detections during NMS.

A. RetinaNet as Baseline

RetinaNet provides a fully convolutional single-shot baseline for oriented detection by replacing horizontal-box regression with oriented-box regression. The section also defines Alignment Convolution, which shifts sampling according to anchor geometry to transform axis-aligned features into oriented ones.

  • Baseline: RetinaNet uses an FPN backbone and two fully convolutional subnetworks for classification and bounding-box regression.Focal loss addresses foreground-background imbalance during training.
  • Baseline: RetinaNet is adapted for oriented detection by replacing horizontal bounding-box regression with oriented bounding-box regression while retaining other settings.The oriented box uses center, width, height, and angle parameters.
  • Alignment Convolution: Standard convolution samples a regular grid, whereas AlignConv adds an offset field to shift sampling locations according to anchor boxes.The offset is computed from anchor-based sampling locations relative to regular sampling locations.
  • Alignment Convolution: AlignConv uses anchor size and orientation to transform axis-aligned convolutional features into arbitrary oriented features.Unlike DeformConv, its offset field is inferred directly from anchor boxes rather than learned with weak supervision.

C. Feature Alignment Module (FAM)

The Feature Alignment Module combines an Anchor Refinement Network with an Alignment Convolution Layer. It refines one squared anchor per feature-map location and uses the resulting anchor geometry to extract aligned features efficiently.

  • FAM: FAM consists of an Anchor Refinement Network and an Alignment Convolution Layer.The ARN refines anchors, while the alignment layer uses them to adjust feature sampling.
  • Anchor Refinement Network: The ARN uses parallel classification and regression branches, refining horizontal anchors into high-quality rotated anchors.One squared anchor is preset for each feature-map location; the classification branch is discarded during standard inference.
  • Alignment Convolution Layer: The Alignment Convolution Layer decodes each anchor prediction into an absolute box and computes an offset field for AlignConv.For each 5-dimensional anchor, 9 sampled points produce an 18-dimensional offset field.
  • Alignment Convolution Layer: Figure 4 contrasts regular convolution, DeformConv, and AlignConv sampling for horizontal and rotated anchor boxes.Green dots mark regular locations, blue dots mark deformable locations or offset vectors, and orange rectangles mark anchors.

D. Oriented Detection Module (ODM)

The Oriented Detection Module addresses classification-localization inconsistency by encoding orientation explicitly, then separating orientation-sensitive regression features from orientation-invariant classification features.

  • Orientation Encoding: ODM applies active rotating filters to encode orientation information in feature maps.An active rotating filter produces multiple orientation channels by actively rotating the filter during convolution.
  • Task-Specific Features: Orientation-sensitive features support bounding-box regression, while orientation-invariant features support object classification.The two feature types are produced from the same orientation-encoded representation.
  • Task-Specific Features: Orientation-invariant features are obtained by selecting the orientation channel with the strongest response.Pooling reduces an H×W×256 feature map with 8 orientation channels to H×W×32.

E. Single-Shot Alignment Network

S^2A-Net combines FAM and ODM in a fully convolutional single-shot detector, using refined anchors, oriented regression targets, IoU matching, and multi-task losses. During inference, it produces refined aligned features, orientation-aware predictions, and final detections through top-k selection and NMS.

  • Architecture: S^2A-Net is formed by combining the Feature Alignment Module and Oriented Detection Module with RetinaNet as the baseline.The model is trained and evaluated as a single-shot detector.
  • Training: Regression targets are parameterized from ground-truth and anchor box geometry, with angle normalization enforced through an integer k.FAM uses horizontal anchors, while ODM recomputes targets after decoding FAM outputs.
  • Training: FAM and ODM assign positives and negatives using oriented-box IoU, with foreground and background thresholds of 0.5 and 0.4.The same thresholds are used in both modules.
  • Training: Training uses a multi-task loss combining classification and regression losses for both FAM and ODM.Focal loss is used for classification and smooth L1 loss for regression.
  • Inference: During inference, pyramid features pass through FAM and ODM before top-k selection and NMS produce final detections.The detector selects the top 2000 predictions before applying NMS.

IV. EXPERIMENTS AND ANALYSIS

Experiments evaluate S2A-Net on DOTA and HRSC2016, two aerial-image datasets containing oriented objects across varied image sizes and categories.

  • DOTA: DOTA contains 2,806 aerial images ranging from 800 × 800 to 4,000 × 4,000 pixels, with 188,282 instances across 15 categories.The categories include planes, vehicles, ships, bridges, fields, harbors, swimming pools, and helicopters.
  • DOTA: DOTA training and validation images are jointly used for training, while the testing set is reserved for evaluation.Images are cropped into 1024 × 1024 patches with stride 824, with horizontal flipping and random rotation used for augmentation.
  • HRSC2016: HRSC2016 contains 1,061 high-resolution ship images with oriented bounding-box annotations, spanning 300 × 300 to 1,500 × 900 pixels.The dataset uses 436 training images, 181 validation images, and 444 testing images; images are resized to 800 × 512 without changing aspect ratio.

B. Implementation Details

Experiments evaluate S2A-Net’s alignment components, network designs, and computational trade-offs against RetinaNet and alternative convolution methods. AlignConv and the joint use of alignment with active rotating filters provide the main reported gains, while deeper designs can hurt small-object detection.

  • Baseline and efficiency: 68.05% mAP is obtained by the RetinaNet baseline with 215.92 GFLOPs and 36.42 M parameters.Reducing the RetinaNet head depth from four to two layers lowers mAP by 0.41% while reducing computation and parameters.
  • Effectiveness of AlignConv: AlignConv improves mAP by about 3% over standard convolution while adding only 1.41 GFLOPs.It also improves performance across most categories, especially bridges, densely distributed vehicles, and helicopters.
  • Effectiveness of AlignConv: 71.71% and 71.33% mAP are achieved by DeformConv and GA-DeformConv, respectively, below AlignConv.The comparison keeps other settings unchanged while replacing the convolution method.
  • Effectiveness of ARN and ARF: 74.12% mAP is achieved when ACL and ARF are combined, improving from 73.24%.Without ARN, ACL, and ARF, the method reaches 68.26% mAP, while ARF alone or with ARN reaches 68.35% and 71.11%.
  • Large-image inference: 1024 × 1024 cropping with stride 512 raises mAP from 71.20% to 74.62% but increases chip count from 8143 to 20012 and inference time by about 135%.The large-image experiments compare mAP and overall inference time across input sizes and cropping strides.
  • Network design: S2A-Net remains effective and efficient across tested network designs, while deeper networks can reduce performance for small objects.The authors hypothesize that deeper networks’ larger receptive fields hinder small-object detection.

D. Detecting on large-size images

Large aerial images create substantial computation and memory demands, motivating alternatives to exhaustive chip-based detection. The experiments compare cropping settings with direct inference to seek a better speed–accuracy trade-off.

  • Motivation: Aerial images often span thousands to tens of thousands of pixels, increasing computation and memory requirements.Prior methods use chip-based detection, even when chips contain no objects.
  • Inference procedure: The large-image approach uses the same network for the original image and 1024 × 1024 chips, without resizing.The chips use a stride of 824 in the qualitative comparison, and corresponding instances share the same number.
  • Cropping settings: 1024 × 1024 chips with stride 512 improve mAP from 71.20% to 74.62% but increase chip images from 8143 to 20012.The corresponding overall inference time increases by about 135%.
  • Direct large-image detection: Direct detection on original large-size images reduces inference time by 50% compared with the reported cropped-image setting.The passage reports the speed reduction but does not provide the full corresponding mAP value.

E. Comparisons with the State-of-the-Art

S2A-Net achieves strong accuracy and efficiency across DOTA and HRSC2016, including state-of-the-art results under the reported settings. Its performance is demonstrated with both single-scale and multi-scale evaluations.

  • 74.01% mAP at 22.6 FPS is achieved on DOTA2 using ResNet50-FPN without data augmentation.FPS is averaged over inference on 10833 chip images.
  • 76.11% mAP on DOTA2 with ResNet101-FPN outperforms all reported two-stage and one-stage methods.
  • 79.42% and 79.15% mAP are obtained in multi-scale experiments with ResNet-50-FPN and ResNet-101-FPN, respectively.
  • S2A-Net achieves the best results in 10 of 15 DOTA2 categories, including bridge, soccer-ball field, swimming pool, and helicopter.Compared with RetinaNet, it also produces fewer false predictions for densely distributed objects with large scale variations.
  • 90.17% and 95.01% mAP are achieved on HRSC2016 under VOC2007 and VOC2012 metrics, respectively, outperforming other methods.The dataset contains objects with large aspect ratios and arbitrary orientations.
  • The proposed Feature Alignment and Oriented Detection Modules support state-of-the-art performance on DOTA and HRSC2016 while maintaining high efficiency.The paper also explores detection on large-size images for a better speed-accuracy trade-off.
Loading 2008.09397v3…