PaperlayerView exact revision on arXiv

Source-linked AI summary

Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection

Maoxun Yuan, Yinyan Wang, Xingxing Wei

arXiv:2209.13801v1cs.CV

TL;DR

Aerial RGB-IR detection faces weak cross-modal misalignment in object position, size, and angle, limiting assumptions of direct multimodal fusion. The paper analyzes its causes and proposes TSRA, which aligns modality features through translation, scale, and rotation prediction with modality selection. TSFADet achieves state-of-the-art accuracy on DroneVehicle and reduces the effect of cross-modal misalignment.

  • Problem

    Aerial RGB-IR object pairs exhibit coupled position, size, and angle deviations, while existing approaches commonly assume perfect alignment.

  • Method

    TSRA calibrates two-modality feature maps by predicting translation, scale, and rotation deviations, using Modality-Selection and Multi-task Jitter within TSFADet.

  • Results

    TSFADet achieves state-of-the-art accuracy on the DroneVehicle dataset and reduces the effect of cross-modal misalignment.

  • Takeaways & Limitations

    The proposed detector provides an end-to-end approach for RGB-IR object detection in aerial images and is reported to generalize to other multispectral detection tasks.

Abstract

from arXiv · show

Integrating multispectral data in object detection, especially visible and infrared images, has received great attention in recent years. Since visible (RGB) and infrared (IR) images can provide complementary information to handle light variations, the paired images are used in many fields, such as multispectral pedestrian detection, RGB-IR crowd counting and RGB-IR salient object detection. Compared with natural RGB-IR images, we find detection in aerial RGB-IR images suffers from cross-modal weakly misalignment problems, which are manifested in the position, size and angle deviations of the same object. In this paper, we mainly address the challenge of cross-modal weakly misalignment in aerial RGB-IR images. Specifically, we firstly explain and analyze the cause of the weakly misalignment problem. Then, we propose a Translation-Scale-Rotation Alignment (TSRA) module to address the problem by calibrating the feature maps from these two modalities. The module predicts the deviation between two modality objects through an alignment process and utilizes Modality-Selection (MS) strategy to improve the performance of alignment. Finally, a two-stream feature alignment detector (TSFADet) based on the TSRA module is constructed for RGB-IR object detection in aerial images. With comprehensive experiments on the public DroneVehicle datasets, we verify that our method reduces the effect of the cross-modal misalignment and achieve robust detection results.

1 Introduction

Aerial RGB-IR detection must handle weak cross-modal misalignment because arbitrary object orientations create coupled position, scale, and angle deviations. The paper introduces TSRA and TSFADet to align modality features and improve aerial detection.

  • Motivation: Aerial images contain arbitrarily oriented objects, while visible-only detectors cannot address limited illumination such as nighttime conditions.
  • Motivation: RGB and infrared images provide complementary information, but few methods and paired datasets target RGB-IR detection in aerial imagery.
  • Problem: Cross-modal aerial misalignment affects object position, scale, and angle, with these deviations coupled during alignment.
  • Approach: The paper attributes weak misalignment mainly to hardware errors and annotation errors, then proposes TSRA to calibrate two-modality proposal features.
  • Approach: TSRA predicts translation, scale, and rotation deviations, while Modality-Selection chooses reference annotations before fused features support classification and regression.
  • Evaluation: TSFADet evaluates TSRA as an end-to-end two-stream detector, and experiments report robust results on the DroneVehicle dataset.

2 Related Work

Prior oriented detection and cross-modal alignment methods address aerial rotation or modality registration separately. The paper argues that translation-only alignment cannot resolve the coupled angle and size deviations in aerial RGB-IR imagery.

  • Oriented Object Detection: Oriented aerial detectors include proposal-based, one-stage, and anchor-free frameworks, with Gaussian Wasserstein and KL-divergence methods optimizing box localization.
  • Cross-Modal Image Alignment: Cross-modal alignment methods use area-based or feature-based registration, but alignment adds time and cannot completely remove weak misalignment.
  • Cross-Modal Image Alignment: End-to-end alignment methods such as RFA and IAFA align modality features before detection, but they consider translation only.
  • Cross-Modal Image Alignment: TSRA predicts position, size, and angle offsets to address the more complex misalignment found in aerial images.

3 Methodology and Analysis

The methodology analyzes weak cross-modal misalignment in aerial RGB-IR imagery and motivates region-wise alignment for deviations caused by hardware and annotation issues.

  • 3 Methodology and Analysis: The proposed methodology combines modality selection, alignment, and multi-task jitter within the TSFADet framework.
  • 3.1 Analysis: Weak misalignment commonly arises in aerial cross-modal images collected by different sensors.
  • 3.1 Analysis: Hardware errors can produce modality-dependent differences in object scale, position, and appearance through radiation distortions and sensor clock skews.
  • 3.1 Analysis: Annotation errors occur because different annotators may label corresponding objects inconsistently across visible and infrared images.
  • 3.1 Analysis: Hardware and annotation errors may occur together, so affine transformation alone cannot address the resulting deviation.

3.2 Translation-Scale-Rotation Alignment Module

The TSRA module aligns RGB and IR region features by predicting translation, scale, and rotation deviations, while modality selection chooses the more reliable reference annotation.

  • 3.2 Translation-Scale-Rotation Alignment Module: TSRA comprises an alignment process, modality-selection strategy, and multi-task jitter augmentation.
  • Alignment Process: The alignment process subtracts reference and sensed region features, predicts position, angle, and size deviations, then re-pools the corrected sensed features.
  • Alignment Process: Three sets of fully connected layers predict deviations in position, size, and angle during alignment.
  • Modality-Selection Strategy: Modality selection chooses the better-annotated bounding box as the reference instead of always selecting infrared.
  • Modality-Selection Strategy: The evaluation score combines object coverage and bounding-box occupancy, ranges from 0 to 1, and approaches 1 for ideal annotations.

Multi-task Jitter.

Multi-task Jitter models sensed proposal uncertainty by perturbing location, width, height, and angle.

  • Multi-task Jitter: The jitter variables are sampled from Gaussian distributions for translation, scale, and rotation deviations.
  • Multi-task Jitter: Multi-task Jitter represents proposal perturbations for location, width, height, and angle.

3.3 TSRA-based Oriented Detector

TSFADet is a two-stage, two-stream oriented detector that generates proposals before applying TSRA to align RGB-IR features for final prediction.

  • 3.3 TSRA-based Oriented Detector: TSFADet combines a two-stream backbone, oriented RPN, oriented R-CNN head, and TSRA module.
  • 3.3 TSRA-based Oriented Detector: The first stage generates oriented proposals with an oriented RPN, while the second stage aligns oriented features with TSRA.
  • 3.3 TSRA-based Oriented Detector: Feature maps from RGB and IR modalities are aggregated, modality selection and alignment predict offsets, and aligned ROI features support classification and regression.

Overall Architecture

The section introduces the loss function used to measure predicted deviation accuracy.

  • The loss function measures the accuracy of predicted deviation.
  • It is used to evaluate deviation predictions within the model.
  • The formulation provides the objective for the deviation-accuracy calculation.

Loss Function.

The loss combines position, size, and angle deviation errors with classification and regression objectives. Ground-truth deviations are defined relative to reference bounding boxes.

  • Deviation loss: Deviation loss averages smooth L1 errors for predicted position, size, and angle deviations over positive proposals.
  • Deviation loss: Positive proposals are identified by g_i = 1, while N_deviation counts the total positive proposals.
  • Deviation targets: Ground-truth deviations are computed from sensed and reference bounding-box location, width, height, and angle values.
  • Total loss: The total objective is L = L_cls + L_reg + L_rpn + λL_deviation.
  • Total loss: The implementation sets λ = 1 so the average gradient of each loss remains at the same scale.

Implementation Details

The network uses a ResNet-50 backbone and standard training augmentations and optimization settings in a unified MMDetection-based implementation.

  • Implementation Details: The implementation modifies MMDetection and uses ImageNet-pretrained ResNet-50 as the backbone.
  • Implementation Details: Horizontal and vertical flipping are used for training augmentation.
  • Implementation Details: Training uses SGD with momentum 0.9 and weight decay 0.0001.
  • Implementation Details: TSFADet is trained for a maximum of 20 epochs.

4 Experimental Results

Experiments on DroneVehicle evaluate the detector, its alignment components, and speed–accuracy trade-offs. The results show that modeling cross-modal deviations and selecting reference modalities improve detection performance.

  • Dataset: DroneVehicle contains 28,439 RGB-Infrared pairs across urban roads, residential areas, parking lots, and other scenarios.
  • Dataset: The final training and validation sets contain 17,990 and 1,469 image pairs, respectively, with mAP evaluated at IoU 0.5.
  • Misalignment statistics: More than 20% of bounding boxes show position or size offsets of 3 pixels or angle offsets of 3 degrees.
  • Ablation studies: The alignment-only TSFADet variant reaches 71.13% mAP, nearly 3% above the two-stream Oriented R-CNN baseline.
  • Ablation studies: Adding Position and Size prediction branches increases mAP from 68.2% to 70.71%.
  • Ablation studies: The Modality-Selection strategy raises the no-MJ variant to 72.36% mAP and outperforms alternative reference-modality choices.
  • Quantitative comparison: TSFADet achieves 73.06% mAP, while combining it with Cascade R-CNN reaches 73.90% mAP.
  • Speed–accuracy comparison: TSFADet runs at 18.6 FPS with 73.06% mAP, 1.8 FPS slower than Halfway Fusion.

5 Conclusions

The paper analyzes weak cross-modal misalignment in multispectral aerial detection and presents TSFADet with TSRA-based alignment to improve robustness.

  • TSFADet uses a TSRA module to alleviate weak misalignment in multispectral aerial detection.The detector is trained end-to-end and achieves state-of-the-art accuracy on the DroneVehicle dataset.
  • The alignment process predicts position, size, and angle deviations, while Modality Selection addresses human-caused misalignment.The method also uses Multi-task Jitter to improve TSRA robustness.
  • The proposed method is intended to generalize to other multispectral detection tasks and support potential applications.
Loading 2209.13801v1…