PaperlayerView exact revision on arXiv

Source-linked AI summary

$\mathbf{C}^2$Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection

Maoxun Yuan, Xingxing Wei

arXiv:2306.16175v3cs.CVcs.MM

TL;DR

RGB-IR detectors are limited by modality miscalibration and fusion imprecision, despite infrared improving robustness. C2Former combines Inter-modality Cross-Attention with Adaptive Feature Sampling, is embedded through detector backbones, and delivers robust results across DroneVehicle and KAIST evaluations.

  • Problem

    RGB-IR object detection still faces modality miscalibration and fusion imprecision, limiting effective use of complementary multimodal information.

  • Method

    C2Former combines ICA for cross-modal calibration and complementary fusion with AFS for reducing the computational cost of global attention.

  • Results

    C2Former-based one-stage and two-stage detectors achieve robust results on the DroneVehicle and KAIST RGB-IR datasets.

  • Takeaways & Limitations

    The feature-domain C2Former can be embedded through backbone networks and applied with different RGB-IR object-detector frameworks.

Abstract

from arXiv · show

Object detection on visible (RGB) and infrared (IR) images, as an emerging solution to facilitate robust detection for around-the-clock applications, has received extensive attention in recent years. With the help of IR images, object detectors have been more reliable and robust in practical applications by using RGB-IR combined information. However, existing methods still suffer from modality miscalibration and fusion imprecision problems. Since transformer has the powerful capability to model the pairwise correlations between different features, in this paper, we propose a novel Calibrated and Complementary Transformer called $\mathrm{C}^2$Former to address these two problems simultaneously. In $\mathrm{C}^2$Former, we design an Inter-modality Cross-Attention (ICA) module to obtain the calibrated and complementary features by learning the cross-attention relationship between the RGB and IR modality. To reduce the computational cost caused by computing the global attention in ICA, an Adaptive Feature Sampling (AFS) module is introduced to decrease the dimension of feature maps. Because $\mathrm{C}^2$Former performs in the feature domain, it can be embedded into existed RGB-IR object detectors via the backbone network. Thus, one single-stage and one two-stage object detector both incorporating our $\mathrm{C}^2$Former are constructed to evaluate its effectiveness and versatility. With extensive experiments on the DroneVehicle and KAIST RGB-IR datasets, we verify that our method can fully utilize the RGB-IR complementary information and achieve robust detection results. The code is available at https://github.com/yuanmaoxun/Calibrated-and-Complementary-Transformer-for-RGB-Infrared-Object-Detection.git.

I. INTRODUCTION

RGB-IR detection improves robustness beyond RGB-only detection but remains challenged by modality miscalibration and imprecise feature fusion. C2Former addresses both through cross-attention and adaptive sampling.

  • RGB-only detectors struggle in nighttime and adverse weather, motivating infrared as an additional modality for robust detection.
  • Weak RGB-IR alignment arises from differing sensors, fields of view, and imaging times, while existing fusion can produce features worse than infrared features.
  • Existing methods often address miscalibration and fusion imprecision separately, leaving complementary RGB-IR information underexploited.
  • C2Former uses Inter-modality Cross-Attention to relate feature points across modalities, incorporating global context as complementary features.
  • Adaptive Feature Sampling predicts cross-modal foreground offsets before ICA, reducing feature-map dimensions while preserving important regions affected by miscalibration.
  • The proposed network combines ICA and AFS end-to-end and is evaluated through RGB-IR detectors on challenging datasets, with reported state-of-the-art performance.

II. RELATED WORK

Related work spans multimodal transformers, aerial object detection, and multispectral pedestrian detection. C2Former targets RGB-IR-specific miscalibration and complementarity while supporting varied detector architectures.

  • A. Multi-modal Transformers: Prior multimodal transformers model interactions among words, images, text, video, point clouds, and other modality features.
  • A. Multi-modal Transformers: Some prior methods address feature misalignment, but they do not further exploit modality complementarity during fusion.
  • A. Multi-modal Transformers: C2Former formulates RGB-T modality miscalibration and fusion imprecision jointly in a transformer framework, addressing both simultaneously.
  • B. Aerial Object Detection: Aerial detection commonly uses rotated bounding boxes, with methods such as R-RPN, R2CNN, RoI Transformer, R3Det, and S2A-Net improving localization or efficiency.
  • B. Aerial Object Detection: DroneVehicle provides an RGB-IR vehicle-detection benchmark, supporting multispectral fusion research for aerial detection.
  • C. Multispectral Pedestrian Detection: Multispectral pedestrian detection uses RGB and IR fusion, while ARCNN predicts cross-modal offsets but has poor generalization limited to two-stage detectors.

III. PROPOSED METHOD

C²Former combines Adaptive Feature Sampling and Inter-modality Cross-Attention to reduce feature dimensions, align RGB-IR features, and generate complementary fused representations. Modality Normalization transforms feature distributions before cross-correlation alignment and soft-attention fusion.

  • Inter-modality Cross-Attention: ICA generates query, key, and value descriptors with convolution layers, reshapes them into feature tensors, and computes cross-modal correlations.The descriptors use 1 × 1 convolutions and reshaping, while cross-correlation uses matrix multiplication followed by normalization.
  • Modality Normalization: Modality Normalization instance-normalizes one modality and injects predicted mean and variance parameters to transform its feature distribution.For IR features, convolution layers predict learnable parameters that are combined with statistics from the RGB features before transformation.
  • Adaptive Feature Sampling: AFS samples features to reduce the input dimension and preserve coarse alignment before ICA processing.The module predicts cross-modal offsets and samples according to them, using deformable-convolution-based feature sampling.
  • Detection Framework: The C²Former framework first reduces feature dimensions with AFS, then uses ICA to output aligned and fused features for downstream backbone integration.The outputs are added to opposite ResNet-50 backbones, and stage outputs are combined into multi-scale features for the FPN.
  • Soft-Attention Fusion: Column-normalized similarity matrices weight value descriptors to dynamically complement features with contextual information.The resulting representations are reshaped to the original feature format and processed with modality-specific convolution layers.

B. Adaptive Feature Sampling (AFS)

AFS reduces ICA’s computational burden by sampling lower-dimensional, offset-aligned RGB and IR features before cross-attention.

  • B. Adaptive Feature Sampling (AFS): AFS samples RGB and IR features before ICA to reduce the dimensionality required for cross-attention.It predicts coarse inter-modality offsets using a deformable-convolution-based design.
  • B. Adaptive Feature Sampling (AFS): The module concatenates both modalities, reduces channels with a 1 × 1 convolution, and predicts inter-modality offsets with a deviation network.The predicted offsets are used during subsequent feature sampling.
  • B. Adaptive Feature Sampling (AFS): Reference grids are downsampled by stride s and normalized before offset-based bilinear interpolation samples the modality features.Sampling locations shift the RGB grid by the predicted offset while retaining the IR reference grid.
  • B. Adaptive Feature Sampling (AFS): The sampled features have dimensions R^C×H_s×W_s and become ICA inputs, while 1 × 1 deconvolutions restore output resolution.The upsampling stride equals the AFS stride s.

C. C2Former-based Object Detectors

C2Former is embedded in the backbone to exchange complementary RGB-IR features before a detection head, supporting both one-stage and two-stage detectors.

  • C. C2Former-based Object Detectors: C2Former receives corresponding RGB and IR backbone-stage outputs and adds complementary features into the opposite modality’s backbone.The resulting sum of the two modality features is passed to the detection head.
  • C. C2Former-based Object Detectors: The framework instantiates one-stage S2A-Net and two-stage Cascade R-CNN detectors for oriented and horizontal detection, respectively.ResNet-50 is used as the baseline backbone, with convolutional layers reducing RGB-IR feature channels.

IV. EXPERIMENTS

The experiments investigate computational cost and AFS stride through ablations, then compare C2Former’s cost with a state-of-the-art method.

  • IV. EXPERIMENTS: The experimental section includes ablation studies, computational-cost analysis, intermediate-result visualization, and comparisons with related detectors.These analyses are used to investigate the method and its computational behavior.
  • IV. EXPERIMENTS: Table III compares the computational cost of C2Former with a state-of-the-art method.The comparison is presented as a computational-cost evaluation.
  • IV. EXPERIMENTS: Figure 6 evaluates the effect of different downsampled strides s in AFS.The figure focuses on stride as an ablation variable.

A. Datasets and Evaluation Metric

Experiments use DroneVehicle and KAIST RGB-IR datasets, evaluating vehicle detection with mAP and pedestrian detection with MR−2 across stated conditions.

  • A. Datasets and Evaluation Metric: DroneVehicle contains 56,878 images and 953,087 vehicle instances across scenarios from day to night.It provides oriented bounding-box annotations for five vehicle categories and reports results on a held-out test set.
  • A. Datasets and Evaluation Metric: KAIST contains 95,328 RGB-IR image pairs, 103,128 pedestrian annotations, and 1,182 unique pedestrians from driving environments.The dataset covers campus, street, and countryside scenes.
  • A. Datasets and Evaluation Metric: DroneVehicle results report mAP in percent for detectors locating and classifying vehicles with oriented bounding-box heads.The table also includes feature addition using S2A-Net.
  • A. Datasets and Evaluation Metric: KAIST pedestrian detection uses miss rate MR−2 averaged over false positives per image from 10^-2 to 100.Evaluation includes the All condition and six all-day subsets defined by pedestrian distance and occlusion.

B. Implementation Details

The evaluation uses C2Former with S2A-Net on DroneVehicle and Cascade R-CNN on KAIST, while visualizing ICA intermediate features.

  • Implementation Details: DroneVehicle uses the single-stage oriented S2A-Net detector with initial anchors and ratio 1.0.
  • Implementation Details: KAIST uses the two-stage horizontal Cascade R-CNN detector with anchors and ratio 2.43.
  • Implementation Details: ICA visualizations show cross-modal similarity maps locating calibration positions and complementary features generated from RGB and IR inputs.

C. Ablation Studies

Ablations show that ICA improves detection but is computationally expensive, while AFS reduces this cost and preserves performance; placement in later stages is most effective.

  • Ablation on each component: +2.3% improvement follows from adding ICA, but FLOPs increase by about 40G.
  • Ablation on each component: AFS reduces computational complexity from 139G to 100G and improves model performance by 0.4%.
  • Ablation on each component: C2Former increases mAP by 2.7% while adding only 3G FLOPs overall.
  • C2Former at different stages: Adding C2Former at the second and third stages achieves 74.2% overall accuracy, whereas adding it at the first stage adds parameters without improving accuracy.
  • Computational Cost Comparison: Compared with TSFADet, the C2Former detector significantly outperforms the competitor in all three reported metrics.

E. Visualization Results

Visualizations indicate that ICA calibrates cross-modal features and complements weak modality representations, while C2Former produces stronger object-background contrast and handles complex scenes.

  • ICA intermediate results: ICA similarity maps localize unaligned positions, while complementary features enhance obscure RGB and IR representations.
  • Feature map visualization: C2Former feature maps show vehicles with more contrast against backgrounds than MBNet across daytime, nighttime, and dark-night conditions.
  • Complex scenarios: C2Former visual detections cover occlusion, low resolution, and angle-change scenarios on DroneVehicle.
  • DroneVehicle evaluation: C2Former-S2A-Net achieves 74.2% mAP on DroneVehicle, exceeding the 67.0% and 67.5% mAP reported for Oriented R-CNN and S2A-Net.
  • Visual comparison: Compared methods can misidentify or miss objects under modality miscalibration or fusion imprecision, whereas ICA provides aligned and complementary features.

G. Evaluation on KAIST Dataset

On KAIST, C2Former-Cascade R-CNN is evaluated with miss-rate curves and visual comparisons, achieving strong results across illumination, distance, and occlusion conditions.

  • Detector construction: The paper constructs C2Former-CascadeRCNN for KAIST and reports superiority of its C2Former-based detectors across the evaluated datasets.
  • Evaluation protocol: The evaluation includes miss-rate versus FPPI curves at IoU thresholds 0.5 and 0.75, plus RGB and infrared visual detections.
  • KAIST evaluation: C2Former-Cascade R-CNN obtains 28.48 MR, 26.67 MR, and 28.39 MR on the Day, Night, and All subsets at IoU 0.5.
  • KAIST evaluation: At IoU 0.75, all detectors experience a significant performance drop, while the proposed method is about 2% lower than MBNet.
  • Distance and occlusion subsets: On the Far subset, C2Former-Cascade R-CNN is about 8% and 5% lower than MBNet at IoU thresholds 0.5 and 0.75.
Loading 2306.16175v3…