PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Modality Fusion Transformer for Multispectral Object Detection

Fang Qingyun, Han Dapeng, Wang Zhaokui

arXiv:2111.00273v4eess.IVcs.CV

TL;DR

Multispectral detection must exploit complementary RGB and thermal information under varied conditions, while existing CNN-based approaches provide limited cross-modality interaction. CFT inserts Transformer-based fusion into the backbone to model global dependencies and jointly fuse modalities, achieving state-of-the-art performance across three datasets. Its main computational limitation is the large attention memory requirement at high image resolutions.

  • Problem

    Multispectral detection needs robust use of complementary RGB and thermal information, but effective cross-modality fusion remains challenging for CNN-based approaches.

  • Method

    CFT embeds Transformer modules in a two-stream backbone, using self-attention to integrate global context and jointly fuse intra-modality and inter-modality features.

  • Results

    78.5, 97.5, and 85.3 mAP50 are achieved on FLIR, LLVIP, and VEDAI, respectively, with improvements across one-stage and two-stage detectors.

  • Takeaways & Limitations

    CFT provides a Transformer-based multispectral fusion approach that exploits modality complementarity and is reported effective across multiple datasets and detector types.

  • Takeaways & Limitations

    At 640 × 640 inputs, the QK^T attention matrix can exceed 2.4G elements after two downsamplings, making memory access unacceptable for ordinary computers.

Abstract

from arXiv · show

Multispectral image pairs can provide the combined information, making object detection applications more reliable and robust in the open world. To fully exploit the different modalities, we present a simple yet effective cross-modality feature fusion approach, named Cross-Modality Fusion Transformer (CFT) in this paper. Unlike prior CNNs-based works, guided by the transformer scheme, our network learns long-range dependencies and integrates global contextual information in the feature extraction stage. More importantly, by leveraging the self attention of the transformer, the network can naturally carry out simultaneous intra-modality and inter-modality fusion, and robustly capture the latent interactions between RGB and Thermal domains, thereby significantly improving the performance of multispectral object detection. Extensive experiments and ablation studies on multiple datasets demonstrate that our approach is effective and achieves state-of-the-art detection performance. Our code and models are available at https://github.com/DocF/multispectral-object-detection.

1. Introduction

Multispectral detection combines visible and thermal information to improve robustness in challenging environments, but effective cross-modality integration remains difficult. CFT addresses this challenge with Transformer-guided fusion that incorporates global context and modality interactions.

  • Multispectral imaging combines visible and thermal information to improve detection perceptibility, reliability, and robustness under rain, fog, occlusion, poor lighting, and low resolution.
  • Multispectral data introduces the challenge of integrating representations and designing cross-modality fusion mechanisms that maximize performance gains.
  • Thermal images can clarify pedestrian contours and reveal occluded pedestrians at night, whereas visible images provide richer edges, textures, and colors during daytime.
  • Prior multispectral detectors mainly use CNN-based fusion, although extending CNNs from intra-modality reasoning to cross-modality interaction is non-trivial.
  • CFT embeds Transformer modules in the feature-extraction backbone to integrate global contextual information from different modalities.
  • The proposed contributions include a two-stream backbone, a CFT module that fuses intra- and inter-modality features, and state-of-the-art results on three public datasets.

2. Related work

Multispectral fusion research distinguishes where features are fused from how the fusion function is constructed. Existing approaches vary from architectural fusion stages to direct operations that merge RGB and thermal feature maps.

  • Multispectral fusion represents fused features as a function of RGB and thermal inputs, their extracted feature maps, and a fusion function.
  • Prior work can therefore be organized into approaches focused on input features at the macro level and approaches focused on fusion functions at the micro level.
  • Macro level: The macro level selects where in the network to fuse modalities by designing alternative network architectures and fusion stages.
  • Micro level: The micro level constructs the fusion function that combines complementary modality features through operations such as concatenation, addition, averaging, maximum, or cross product.

3. Methodology

The Cross-Modality Fusion Backbone combines two-stream RGB and thermal feature extraction with three CFT modules. CFT uses Transformer self-attention to model global, intra-modality, and inter-modality relationships while controlling computation through feature-map downsampling.

  • Cross-Modality Fusion Backbone: The Cross-Modality Fusion Backbone redesigns YOLOv5 with separate RGB and thermal streams plus embedded CFT modules for multimodal fusion.The backbone comprises a two-stream feature extractor and three Cross-Modality Fusion Transformer modules.
  • Cross-Modality Fusion Transformer: CFT concatenates flattened RGB and thermal feature maps into a positional-embedded sequence before Transformer processing.Each modality contributes HW tokens with C channels, producing an input sequence of shape 2HW × C.
  • Cross-Modality Fusion Transformer: Self-attention computes scaled query-key correlations and applies them to values, followed by multi-head attention and a feed-forward network.The scaling factor 1/√D_K prevents softmax gradients from becoming extremely small as dot-product magnitudes grow.
  • Computational considerations: Global average pooling reduces feature maps to H = W = 8 before the Transformer, and bilinear upsampling restores the original resolution afterward.Without this reduction, a 640 × 640 input after two downsamplings would produce a QK^T matrix exceeding 2.4G elements.
  • Cross-Modality Fusion Transformer: The correlation matrix contains two intra-modality blocks and two inter-modality blocks, enabling simultaneous within-modal and cross-modal fusion.The four blocks correspond to RGB-RGB, thermal-thermal, and the two cross-modal relationships.

4.1. Datasets

The evaluation uses three multispectral detection datasets spanning aligned ground-view imagery, low-light pedestrian pairs, and aerial vehicle imagery.

  • Datasets: Experiments evaluate the method on FLIR, LLVIP, and VEDAI multispectral benchmark datasets.These datasets cover different imaging conditions and detection settings.
  • FLIR: The aligned FLIR dataset contains 5,142 image pairs, with 4,129 for training and 1,013 for testing across person, car, and bicycle categories.The study uses the manually aligned version because unaligned pairs complicate training.
  • LLVIP: LLVIP contains 16,836 strictly spatio-temporally aligned visible-infrared pairs, mostly captured in low-light environments.The dataset includes 33,672 individual images for low-light pedestrian detection.
  • VEDAI: VEDAI provides RGB and infrared aerial imagery with nine vehicle classes, more than 3,700 annotated targets, and over 1,200 images at two resolutions.The resolutions are 1024 × 1024 and 512 × 512.

4.2. Experimental Settings

Training combines bounding-box regression, classification, and confidence losses, using SGD and COCO-pretrained YOLOv5 initialization with Mosaic augmentation.

  • Loss Function: The overall training loss sums bounding-box regression, classification, and confidence losses.The component losses are denoted L_box, L_cls, and L_conf.
  • Loss Function: Generalized Intersection over Union loss is used for predicted box regression.The paper describes GIoU loss as preferable to IoU loss for box regression.
  • Loss Function: Classification uses cross-entropy, while confidence loss contains squared-error terms for object and non-object predictions.The confidence components are L_noobj and L_obj.
  • Implementation: Models use SGD with learning rate 1e-2, momentum 0.937, weight decay 0.0005, batch size 32, and 200 training epochs.Training uses two NVIDIA TITAN RTX GPUs.
  • Implementation: COCO-pretrained YOLOv5 initializes the weights, and Mosaic augmentation combines four training images into one.The baseline is YOLOv5 extended with an additional thermal-feature branch.

4.3. Evaluation Metrics

Evaluation follows MS-COCO object-detection metrics, including mAP, mAP50, and mAP75, with mAP as the primary and stricter metric.

  • Evaluation Metrics: Models are evaluated using mean Average Precision, mAP50, and mAP75.These are the three object-detection metrics introduced with MS-COCO.
  • Evaluation Metrics: mAP50 averages category AP values at IoU = 0.50, while mAP75 averages them at IoU = 0.75.AP is the integral of the precision-recall curve for each category.
  • Evaluation Metrics: mAP averages AP over IoU thresholds from 0.50 to 0.95 in increments of 0.05 and is stricter than mAP50 and mAP75.The metric is identified as the primary challenge metric.

4.4. Ablation Study

The ablation comparisons evaluate CFT across datasets, detector settings, and qualitative detection outcomes. CFT improves detection performance over the baseline and produces fewer visible errors in challenging scenes.

  • 4.3%, 2.6%, and 2.1% are the CFT improvements over the Faster R-CNN baseline for mAP50, mAP75, and mAP, respectively.
  • CFT qualitatively detects densely obscured objects that baselines miss or misclassify with false positives and false negatives.

4.5. Visual Interpretation

The visual analyses inspect modality-specific correlations, feature attention, qualitative detections, and residual CFT features. They show how RGB and thermal information are related and how CFT adds a comparatively small enhancement to existing features.

  • The correlation matrix contains RGB and thermal intra-modality blocks plus two inter-modality blocks, with some symmetry between the inter-modality blocks.
  • Thermal features focus more accurately on targets at night, while daytime visual features have stronger target focus and fewer distractions.
  • Figures 4–6 compare color and thermal inputs with ground truth, baseline detections, and CFT detections across FLIR, LLVIP, and VEDAI.Red inverted triangles mark false negatives; Figure 6 also uses blue inverted triangles for false positives.
  • CFT features are introduced residually to enhance mono-spectral features rather than directly altering the visual or thermal features.The residual formulation is described as H(x) = F(x) + x, with identity mapping when F(x) equals 0.
  • The CFT feature visualization ranges from −1.7 to −1.2, while the original feature visualization ranges from −45 to 100.The paper reports that these ranges differ by an order of magnitude.

4.6. Comparison with State-of-the-art Methods

CFT achieves state-of-the-art multispectral detection performance across FLIR, LLVIP, and VEDAI, outperforming competing mono- and multimodal methods on reported metrics.

  • FLIR: 5.8% to 7.5% mAP50 improvements distinguish CFT from other multimodal networks on FLIR.Against GAFF with ResNet18, CFT gains 5.8% mAP50, 2.6% mAP75, and 2.7% mAP.
  • Evaluation scope: The evaluation compares CFT with competing methods on FLIR, LLVIP, and VEDAI datasets.The referenced tables are organized as performance comparisons for each dataset.
  • LLVIP: 97.5 mAP50, 72.9 mAP75, and 63.6 mAP are reported for CFT on LLVIP.The passage attributes the result to interacting and fusing complementary features from different modalities.

5. Conclusion

The conclusion presents CFT as a Transformer-based fusion approach that enhances two-stream detectors by learning long-range dependencies and integrating global context. Across three datasets and detector types, it reports clear performance gains and broad applicability.

  • Conclusion: CFT learns long-range dependencies and integrates global contextual information to enhance two-stream CNN representations for multispectral detection.The modules are densely inserted into the backbone to exploit complementarity between modalities.
  • Conclusion: 78.5, 97.5, and 85.3 mAP50 are reported on FLIR, LLVIP, and VEDAI, respectively.These are presented as state-of-the-art performances for the proposed method.
  • Conclusion: CFT improves multispectral object detection with YOLOV5, YOLOv3, and Faster R-CNN by a clear margin.The conclusion states that the improvement holds with both one-stage and two-stage detectors.
  • Conclusion: CFT may also apply to RGB-LiDAR, RGB-D, and stereo image super-resolution tasks.The conclusion characterizes the approach as simple yet effective when suggesting these additional computer vision applications.
Loading 2111.00273v4…