PaperlayerView exact revision on arXiv

Source-linked AI summary

$\mathbf{C}^2$Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection

Maoxun Yuan, Xingxing Wei

arXiv:2306.16175v3cs.CVcs.MM

TL;DR

RGB-IR detectors are limited by modality miscalibration and fusion imprecision, despite infrared improving robustness. C2Former combines Inter-modality Cross-Attention with Adaptive Feature Sampling, is embedded through detector backbones, and delivers robust results across DroneVehicle and KAIST evaluations.

  • Problem

    RGB-IR object detection still faces modality miscalibration and fusion imprecision, limiting effective use of complementary multimodal information.

    Source: §I · p. 1

  • Method

    C2Former combines ICA for cross-modal calibration and complementary fusion with AFS for reducing the computational cost of global attention.

    Source: §I · p. 2

  • Results

    C2Former-based one-stage and two-stage detectors achieve robust results on the DroneVehicle and KAIST RGB-IR datasets.

    Source: §V · pp. 10–11

  • Takeaways & Limitations

    The feature-domain C2Former can be embedded through backbone networks and applied with different RGB-IR object-detector frameworks.

    Source: C. C2Former-based Object Detectors · p. 6, §V · p. 10

Abstract

from arXiv · show

Object detection on visible (RGB) and infrared (IR) images, as an emerging solution to facilitate robust detection for around-the-clock applications, has received extensive attention in recent years. With the help of IR images, object detectors have been more reliable and robust in practical applications by using RGB-IR combined information. However, existing methods still suffer from modality miscalibration and fusion imprecision problems. Since transformer has the powerful capability to model the pairwise correlations between different features, in this paper, we propose a novel Calibrated and Complementary Transformer called $\mathrm{C}^2$Former to address these two problems simultaneously. In $\mathrm{C}^2$Former, we design an Inter-modality Cross-Attention (ICA) module to obtain the calibrated and complementary features by learning the cross-attention relationship between the RGB and IR modality. To reduce the computational cost caused by computing the global attention in ICA, an Adaptive Feature Sampling (AFS) module is introduced to decrease the dimension of feature maps. Because $\mathrm{C}^2$Former performs in the feature domain, it can be embedded into existed RGB-IR object detectors via the backbone network. Thus, one single-stage and one two-stage object detector both incorporating our $\mathrm{C}^2$Former are constructed to evaluate its effectiveness and versatility. With extensive experiments on the DroneVehicle and KAIST RGB-IR datasets, we verify that our method can fully utilize the RGB-IR complementary information and achieve robust detection results. The code is available at https://github.com/yuanmaoxun/Calibrated-and-Complementary-Transformer-for-RGB-Infrared-Object-Detection.git.

I. INTRODUCTION

RGB-IR detection improves robustness beyond RGB-only detection but remains challenged by modality miscalibration and imprecise feature fusion. C2Former addresses both through cross-attention and adaptive sampling.

  • RGB-only detectors struggle in nighttime and adverse weather, motivating infrared as an additional modality for robust detection.Source: §I · p. 1
  • Weak RGB-IR alignment arises from differing sensors, fields of view, and imaging times, while existing fusion can produce features worse than infrared features.Source: §I · p. 1, Figure 1 · p. 1
  • Existing methods often address miscalibration and fusion imprecision separately, leaving complementary RGB-IR information underexploited.Source: §I · pp. 1–2
  • C2Former uses Inter-modality Cross-Attention to relate feature points across modalities, incorporating global context as complementary features.Source: §I · p. 2
  • Adaptive Feature Sampling predicts cross-modal foreground offsets before ICA, reducing feature-map dimensions while preserving important regions affected by miscalibration.Source: §I · p. 2
  • The proposed network combines ICA and AFS end-to-end and is evaluated through RGB-IR detectors on challenging datasets, with reported state-of-the-art performance.Source: §I · p. 2

II. RELATED WORK

Related work spans multimodal transformers, aerial object detection, and multispectral pedestrian detection. C2Former targets RGB-IR-specific miscalibration and complementarity while supporting varied detector architectures.

A. Multi-modal Transformers

B. Aerial Object Detection

  • Aerial detection commonly uses rotated bounding boxes, with methods such as R-RPN, R2CNN, RoI Transformer, R3Det, and S2A-Net improving localization or efficiency.Source: B. Aerial Object Detection · p. 3
  • DroneVehicle provides an RGB-IR vehicle-detection benchmark, supporting multispectral fusion research for aerial detection.Source: B. Aerial Object Detection · p. 3

C. Multispectral Pedestrian Detection

III. PROPOSED METHOD

C²Former combines Adaptive Feature Sampling and Inter-modality Cross-Attention to reduce feature dimensions, align RGB-IR features, and generate complementary fused representations. Modality Normalization transforms feature distributions before cross-correlation alignment and soft-attention fusion.

Inter-modality Cross-Attention

  • ICA generates query, key, and value descriptors with convolution layers, reshapes them into feature tensors, and computes cross-modal correlations.The descriptors use 1 × 1 convolutions and reshaping, while cross-correlation uses matrix multiplication followed by normalization.Source: A. Inter-modality Cross-Attention (ICA) · pp. 3–5

Modality Normalization

  • Modality Normalization instance-normalizes one modality and injects predicted mean and variance parameters to transform its feature distribution.For IR features, convolution layers predict learnable parameters that are combined with statistics from the RGB features before transformation.Source: Figure 3 · p. 4, A. Inter-modality Cross-Attention (ICA) · p. 4

Adaptive Feature Sampling

  • AFS samples features to reduce the input dimension and preserve coarse alignment before ICA processing.The module predicts cross-modal offsets and samples according to them, using deformable-convolution-based feature sampling.Source: Figure 4 · p. 4

Detection Framework

Soft-Attention Fusion

  • Column-normalized similarity matrices weight value descriptors to dynamically complement features with contextual information.The resulting representations are reshaped to the original feature format and processed with modality-specific convolution layers.Source: A. Inter-modality Cross-Attention (ICA) · p. 5

B. Adaptive Feature Sampling (AFS)

AFS reduces ICA’s computational burden by sampling lower-dimensional, offset-aligned RGB and IR features before cross-attention.

B. Adaptive Feature Sampling (AFS)

  • AFS samples RGB and IR features before ICA to reduce the dimensionality required for cross-attention.It predicts coarse inter-modality offsets using a deformable-convolution-based design.Source: B. Adaptive Feature Sampling (AFS) · p. 5
  • The module concatenates both modalities, reduces channels with a 1 × 1 convolution, and predicts inter-modality offsets with a deviation network.The predicted offsets are used during subsequent feature sampling.Source: B. Adaptive Feature Sampling (AFS) · p. 5
  • Reference grids are downsampled by stride s and normalized before offset-based bilinear interpolation samples the modality features.Sampling locations shift the RGB grid by the predicted offset while retaining the IR reference grid.Source: B. Adaptive Feature Sampling (AFS) · pp. 5–6
  • The sampled features have dimensions R^C×H_s×W_s and become ICA inputs, while 1 × 1 deconvolutions restore output resolution.The upsampling stride equals the AFS stride s.Source: B. Adaptive Feature Sampling (AFS) · p. 6

C. C2Former-based Object Detectors

C2Former is embedded in the backbone to exchange complementary RGB-IR features before a detection head, supporting both one-stage and two-stage detectors.

C. C2Former-based Object Detectors

  • C2Former receives corresponding RGB and IR backbone-stage outputs and adds complementary features into the opposite modality’s backbone.The resulting sum of the two modality features is passed to the detection head.Source: C. C2Former-based Object Detectors · p. 6
  • The framework instantiates one-stage S2A-Net and two-stage Cascade R-CNN detectors for oriented and horizontal detection, respectively.ResNet-50 is used as the baseline backbone, with convolutional layers reducing RGB-IR feature channels.Source: C. C2Former-based Object Detectors · p. 6

IV. EXPERIMENTS

The experiments investigate computational cost and AFS stride through ablations, then compare C2Former’s cost with a state-of-the-art method.

IV. EXPERIMENTS

  • The experimental section includes ablation studies, computational-cost analysis, intermediate-result visualization, and comparisons with related detectors.These analyses are used to investigate the method and its computational behavior.Source: §IV · p. 6
  • Table III compares the computational cost of C2Former with a state-of-the-art method.The comparison is presented as a computational-cost evaluation.Source: Table III · p. 6
  • Figure 6 evaluates the effect of different downsampled strides s in AFS.The figure focuses on stride as an ablation variable.Source: Figure 6 · p. 6

A. Datasets and Evaluation Metric

Experiments use DroneVehicle and KAIST RGB-IR datasets, evaluating vehicle detection with mAP and pedestrian detection with MR−2 across stated conditions.

A. Datasets and Evaluation Metric

  • DroneVehicle contains 56,878 images and 953,087 vehicle instances across scenarios from day to night.It provides oriented bounding-box annotations for five vehicle categories and reports results on a held-out test set.Source: A. Datasets and Evaluation Metric · p. 6
  • KAIST contains 95,328 RGB-IR image pairs, 103,128 pedestrian annotations, and 1,182 unique pedestrians from driving environments.The dataset covers campus, street, and countryside scenes.Source: A. Datasets and Evaluation Metric · p. 6
  • DroneVehicle results report mAP in percent for detectors locating and classifying vehicles with oriented bounding-box heads.The table also includes feature addition using S2A-Net.Source: Table IV · p. 7
  • KAIST pedestrian detection uses miss rate MR−2 averaged over false positives per image from 10^-2 to 100.Evaluation includes the All condition and six all-day subsets defined by pedestrian distance and occlusion.Source: A. Datasets and Evaluation Metric · p. 7

B. Implementation Details

The evaluation uses C2Former with S2A-Net on DroneVehicle and Cascade R-CNN on KAIST, while visualizing ICA intermediate features.

Implementation Details

  • DroneVehicle uses the single-stage oriented S2A-Net detector with initial anchors and ratio 1.0.Source: B. Implementation Details · p. 7
  • KAIST uses the two-stage horizontal Cascade R-CNN detector with anchors and ratio 2.43.Source: B. Implementation Details · p. 7
  • ICA visualizations show cross-modal similarity maps locating calibration positions and complementary features generated from RGB and IR inputs.Source: Figure 7 · p. 7

C. Ablation Studies

Ablations show that ICA improves detection but is computationally expensive, while AFS reduces this cost and preserves performance; placement in later stages is most effective.

Ablation on each component

C2Former at different stages

  • Adding C2Former at the second and third stages achieves 74.2% overall accuracy, whereas adding it at the first stage adds parameters without improving accuracy.Source: C. Ablation Studies · p. 8

Computational Cost Comparison

E. Visualization Results

Visualizations indicate that ICA calibrates cross-modal features and complements weak modality representations, while C2Former produces stronger object-background contrast and handles complex scenes.

ICA intermediate results

  • ICA similarity maps localize unaligned positions, while complementary features enhance obscure RGB and IR representations.Source: E. Visualization Results · p. 8

Feature map visualization

  • C2Former feature maps show vehicles with more contrast against backgrounds than MBNet across daytime, nighttime, and dark-night conditions.Source: E. Visualization Results · p. 8

Complex scenarios

DroneVehicle evaluation

Visual comparison

G. Evaluation on KAIST Dataset

On KAIST, C2Former-Cascade R-CNN is evaluated with miss-rate curves and visual comparisons, achieving strong results across illumination, distance, and occlusion conditions.

Detector construction

Evaluation protocol

KAIST evaluation

Distance and occlusion subsets

Related papers

Loading 2306.16175v3…