PaperlayerView exact revision on arXiv

Source-linked AI summary

ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection

Jifeng Shen, Yifei Chen, Yue Liu, Xin Zuo, Heng Fan, Wankou Yang

arXiv:2308.07504v1cs.CV

TL;DR

Multispectral feature fusion must address misalignment and limited local interaction while remaining computationally efficient. ICAFusion combines dual cross-attention transformers with iterative parameter sharing, and it reports superior detection accuracy and running speed on three datasets. Its stated limitation concerns false positives caused by visual similarity between traffic signs or trees and persons.

  • Problem

    Existing convolution-based fusion is vulnerable to misalignment from limited local-range interaction, while stacking transformer blocks increases computational cost.

  • Method

    ICAFusion uses cross-modal feature enhancement with dual cross-attention and iterative interaction that shares parameters among block-wise multimodal transformers.

  • Results

    The method outperforms other state-of-the-art methods in detection accuracy and running speed on KAIST, FLIR, and VEDAI datasets.

  • Takeaways & Limitations

    The framework is presented as a general feature-fusion method that can be integrated with different detection frameworks and backbones.

  • Takeaways & Limitations

    False positives remain when traffic signs or trees visually resemble persons.

Abstract

from arXiv · show

Effective feature fusion of multispectral images plays a crucial role in multi-spectral object detection. Previous studies have demonstrated the effectiveness of feature fusion using convolutional neural networks, but these methods are sensitive to image misalignment due to the inherent deffciency in local-range feature interaction resulting in the performance degradation. To address this issue, a novel feature fusion framework of dual cross-attention transformers is proposed to model global feature interaction and capture complementary information across modalities simultaneously. This framework enhances the discriminability of object features through the query-guided cross-attention mechanism, leading to improved performance. However, stacking multiple transformer blocks for feature enhancement incurs a large number of parameters and high spatial complexity. To handle this, inspired by the human process of reviewing knowledge, an iterative interaction mechanism is proposed to share parameters among block-wise multimodal transformers, reducing model complexity and computation cost. The proposed method is general and effective to be integrated into different detection frameworks and used with different backbones. Experimental results on KAIST, FLIR, and VEDAI datasets show that the proposed method achieves superior performance and faster inference, making it suitable for various practical scenarios. Code will be available at https://github.com/chanchanchan97/ICAFusion.

1. INTRODUCTION

Multispectral detection exploits complementary RGB and thermal information, but conventional fusion is vulnerable to misalignment and limited local interaction. ICAFusion uses dual cross-attention and parameter-sharing iterative refinement to model complementary global interactions while controlling complexity.

  • Motivation: RGB provides detailed appearance under good illumination, while thermal images preserve perceptible contours in poor illumination, motivating multimodal fusion.These complementary characteristics make effective feature fusion critical for multispectral object detection.
  • Limitations of Prior Fusion: Convolution-based fusion mainly captures local information, limiting long-range feature relationships because CNNs have restricted receptive fields.This limitation contributes to vulnerability to image misalignment in traditional concatenation or addition-based fusion.
  • Proposed Fusion: The proposed dual cross-attention transformer aggregates RGB and thermal features from local and global perspectives while borrowing complementary information through auxiliary-modality queries.Unlike methods computing attention from all modality tokens, it computes cross-modal correlation using queries from the auxiliary modality.
  • Efficient Refinement: An iterative learning strategy refines inter-modal and intra-modal representations through repeated bidirectional feature interaction while sharing parameters across blocks.The strategy is designed to balance model performance and complexity rather than stacking independently parameterized transformer blocks.
  • Scope and Contributions: The fusion method is designed to integrate complementary RGB and thermal information, work with different backbones and detection frameworks, and support different input modalities.The contribution list also describes a feasible setting when one modality is missing or has poor quality.
  • Reported Results: Experiments report state-of-the-art results on KAIST, FLIR, and VEDAI while also achieving fast inference.The introduction presents the method as suitable for practical scenarios.

2. Related work

Prior multispectral detection work explored semantic, attention-based, illumination-aware, alignment, uncertainty, and local/global fusion strategies. This paper introduces iterative cross-attention to use local and global information across modalities while addressing limitations of simpler Transformer integration.

  • Multispectral Object Detection: Other approaches cyclically refine features, compare common-mode and differential-mode information, or strengthen aggregated and modality-specific features.These methods pursue complementary information and robustness across RGB and thermal modalities.
  • Multispectral Object Detection: Recent methods improve multispectral detection through semantic segmentation, mask-guided mutual attention, score fusion, and illumination-aware weighting.These approaches target discriminative features, accuracy-speed trade-offs, or day-night variation.
  • Multispectral Object Detection: Alignment-oriented methods estimate feature offsets, model uncertainty, or guide RGB and thermal representations toward similarity.These methods specifically address modality misalignment and cross-modal representation consistency.
  • Multispectral Object Detection: CFT and LGADet add local and global attention to improve fusion, but existing methods may not fully exploit complementary global information.The cited limitation concerns integrating Transformer or non-local components without fully using cross-modal complementarity.
  • Attention Mechanism: Attention research includes channel, spatial, local cross-channel, class-specific, and cross-modal mechanisms for adaptive feature refinement.The surveyed modules range from lightweight channel interactions to attention mechanisms that enhance mono-modality features with auxiliary information.
  • Transformer for Multimodal Learning: Transformer studies demonstrate multimodal effectiveness, motivating its use here to harvest complementary RGB and thermal information globally.The paper positions its method within prior Transformer applications to image-text, LiDAR-camera, tracking, RGB-D, and other multimodal tasks.

3. The proposed method

The method uses a dual-branch architecture with cross-modal fusion and detection stages, combining dual cross-attention with iterative feature enhancement to model global interactions while reducing complexity.

  • Architecture: The architecture extracts RGB and thermal features independently before cross-modal fusion and detection-neck processing.Multi-scale branch features are fused and passed to the detector head for classification and regression.
  • Architecture: A dual cross-attention feature fusion transformer models the cross-modal fusion function.The transformer is designed to capture long-range dependencies and complementary information between modalities.
  • DMFF: DMFF comprises Spatial Feature Shrinking, Iterative Cross-modal Feature Enhancement, and bimodal NIN fusion modules.SFS compresses feature maps, ICFE refines cross-modal features, and bimodal fusion performs local feature fusion.
  • CFE: Dual CFE modules use auxiliary-modality queries to retrieve complementary information and enhance each modality despite imperfect RGB-thermal alignment.The modules use separate parameters for the RGB and thermal branches and perform global cross-modal interaction.
  • CFE: The proposed cross-attention design uses two improved transformers instead of concatenating modalities in a single encoder, with fewer parameters and lower computational complexity.The comparison is made against CFT, which directly concatenates modality tokens and computes cross-modal correlations with one transformer encoder.
  • ICFE: ICFE iteratively refines multimodal features while sharing parameters across iterations, reducing the complexity and computation associated with stacked enhancement blocks.The iterative strategy is presented as a parameter-sharing approach for balancing model performance and complexity.
  • Generalization: The CFE module supports dual- and single-modality inputs, and can still produce satisfactory features when one modality is missing or has poor image quality.This supports use with different input image modalities and detection settings.

4.1. Datasets and Evaluation metrics

The evaluation uses KAIST, FLIR, and VEDAI multispectral datasets, with log-average miss rate for KAIST and AP or mAP as object-detection metrics.

  • Datasets: KAIST is a multispectral pedestrian-detection dataset evaluated using weakly aligned RGB-thermal image pairs and sanitized annotations.The experiments use the FLIR-aligned version because the original images are misaligned.
  • Datasets: FLIR contains aligned multispectral image pairs covering daytime and night scenes, with separate training and testing partitions.The dataset includes 5,142 pairs, of which 4,129 are used for training and 1,013 for testing.
  • Datasets: VEDAI is an aerial-imagery dataset for small-target detection containing more than 3,700 annotated targets across 1,268 RGB-infrared image pairs.It includes nine vehicle categories and uses 1024 × 1024 images for training and testing.
  • Metrics: KAIST performance is reported with log-average miss rate, averaged over 9 FPPI values sampled logarithmically from 10^-2 to 1.Lower MR^-2 indicates better performance.
  • Metrics: AP averages detection precision under an IoU threshold, while mAP averages AP across categories; higher values indicate better performance.The usual IoU threshold described is 0.5.

4.2. Implementation Details

The implementation trains the model in PyTorch with SGD, cosine-annealed learning rates, augmentation, and YOLOv5 or FCOS detector settings.

  • Environment: The model is implemented in PyTorch 1.7.1 on Ubuntu 18.04 using an RTX 3090 GPU.The reported server also includes an Intel i7-9700 CPU and 64G memory.
  • Optimization: Training runs for 60 epochs with batch size 8 and SGD initialized at 1.0×10^-2 learning rate with momentum 0.937.Weight decay is 0.0005 and cosine annealing controls learning-rate decay.
  • Evaluation setup: YOLOv5 is used for ablation studies, while NIN fusion serves as the default comparison baseline.The implementation adopts the detector-specific default settings for the evaluation framework.

4.3. Ablation Study

The ablations show that learnable parameters on both residual branches improve detection without significantly increasing computational cost, while iterative interaction is more efficient than stacking modules. CFE benefits both modalities, but excessive stacking or iteration offers limited gains and can amplify background interference.

  • Learnable parameters on both residual branches reduce KAIST MR from 7.86% to 7.63% and increase FLIR mAP50 from 77.1% to 77.5%.
  • CFE improves mono-modality detection, with thermal enhancement gaining 0.59% on KAIST and 1.20% on FLIR over thermal-only detection.
  • RGB features are stronger than thermal features on KAIST, whereas thermal features are stronger than RGB features on FLIR.
  • Stacking 10 modules increases parameters and GPU memory by more than 4×, reduces speed from 40.5 Hz to 17.3 Hz, and yields only a 0.70% mAP benefit.
  • Iterative learning reduces KAIST MR from 7.63% to 7.17% and raises FLIR mAP50 from 77.50% to 79.20%, with one iteration achieving the best result.
  • More iterations share parameters without extra parameter or memory costs, but can enhance background noise and degrade performance.
  • Mixed pooling produces MR of 7.17% on KAIST and mAP50 of 79.20% on FLIR, so it is used to reduce feature-map complexity.

4.4. Comparison with State-of-the-art Methods

Across KAIST, FLIR, and VEDAI comparisons, the method reports strong performance against existing approaches and maintains fast inference, though stricter KAIST evaluation leaves it below two fusion methods.

  • The method runs at 38.46 Hz on an RTX 3090 and is reported as suitable for high-speed object detection scenarios.
  • On FLIR, the approach is reported to outperform all existing methods and achieve state-of-the-art performance.
  • On KAIST, the method achieves 79.20% mAP50, 36.9% mAP75, and 41.4% mAP, with category scores of 66.90%, 89.00%, and 81.60%.
  • The method outperforms the paper's CFT-based Ours* comparison on mAP50, mAP75, and mAP.
  • On VEDAI, the method exceeds the baseline by 1.96% mAP and achieves a reported mAP of 76.62%.
  • On KAIST under stricter mAP evaluation, the method is 0.28% and 1.37% lower than Input Fusion and Mid Fusion, respectively.

4.5. Qualitative Analysis

Qualitative visualizations indicate that multispectral aggregation helps identify objects under poor illumination and complex traffic scenes, while attention maps show more discriminative spatial focus than the baseline.

  • Under poor illumination, aggregating RGB and thermal images enables the method to identify and locate objects that are difficult to see in RGB images alone.
  • In complex urban traffic scenes, the auxiliary modality helps distinguish different object categories.
  • The baseline attends to disparate image regions and produces more false positives, whereas the proposed method uses global spatial and inter-object correlation information.
  • Figure 7 compares RGB and thermal ground truth with heatmaps from NIN fusion and the proposed method across daytime and nighttime examples.

4.6. Limitations

The reported failure cases identify errors caused by visual similarity, low image quality, and occlusion across the KAIST, FLIR, and VEDAI datasets.

  • On KAIST, traffic signs or trees can be misidentified as people because of visual similarity and low image quality.
  • On FLIR, occlusion between overlapping pedestrians can produce false negatives.
  • On VEDAI, rooftop devices may be misidentified as cars because their shapes and colors appear similar across modalities.
  • Figure 8 marks false positives and false negatives for KAIST, FLIR, and VEDAI failure cases.

5. Conclusions

The paper presents a cross-modal feature-fusion framework for multispectral object detection that addresses existing methods’ focus on local correlations. It combines global complementary information with iterative learning, improving detection accuracy and running speed across multiple datasets.

  • The fusion framework is presented as generalizable across detection frameworks and backbones, with experiments on KAIST, FLIR, and VEDAI.
  • The proposed framework addresses existing multispectral detection methods’ focus on local feature correlations between modalities.
  • The cross-modal feature enhancement module improves each modality’s feature representation by leveraging global information from the complementary modality.
  • Iterative learning refines complementary information and improves model performance without adding extra parameters.
  • The method outperforms other state-of-the-art methods in detection accuracy and running speed.
  • Future work will explore more efficient and lightweight cross-modal fusion and extend the approach to other multimodal tasks.
Loading 2308.07504v1…