Source-linked AI summary
Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery
Qingyun Fang, Zhaokui Wang
TL;DR
Multispectral object detection must determine how to fuse information from two modalities under varying illumination, weather, resolution, and occlusions. CMAFF is a lightweight fusion approach that processes different features separately, and the paper reports improved detector performance while keeping overhead low.
Problem
Multispectral object detection faces the core problem of how to fuse information from two modalities under varying illumination, weather, resolution, and occlusions.
Method
CMAFF is a novel, lightweight multispectral fusion approach that processes different modality features separately and uses joint common- and differential-modality attention.
Results
CMAFF effectively utilizes shared and modality-specific information, improving detector performance while keeping overhead low.
Takeaways & Limitations
The plug-in CMAFF module can be embedded in detectors to enhance or select features from different modalities.
Takeaways & Limitations
The thermal-modality branch lacks large-scale infrared datasets and therefore has no pre-trained models or can only use limited alternatives.
Abstract
from arXiv · showhide
Cross-modality fusing complementary information of multispectral remote sensing image pairs can improve the perception ability of detection algorithms, making them more robust and reliable for a wider range of applications, such as nighttime detection. Compared with prior methods, we think different features should be processed specifically, the modality-specific features should be retained and enhanced, while the modality-shared features should be cherry-picked from the RGB and thermal IR modalities. Following this idea, a novel and lightweight multispectral feature fusion approach with joint common-modality and differential-modality attentions are proposed, named Cross-Modality Attentive Feature Fusion (CMAFF). Given the intermediate feature maps of RGB and IR images, our module parallel infers attention maps from two separate modalities, common- and differential-modality, then the attention maps are multiplied to the input feature map respectively for adaptive feature enhancement or selection. Extensive experiments demonstrate that our proposed approach can achieve the state-of-the-art performance at a low computation cost.
1. Introduction
Multispectral detection addresses difficult conditions such as nighttime imagery, but simple fusion can miss complementary information and worsen modality imbalance. CMAFF separates modality-specific and shared features, enhancing the former and selecting the latter, and is integrated into YOLOFusion.
- Visible-band detectors struggle under varying illumination, weather, resolution, and occlusion, particularly at night.
- Fusing complementary RGB and thermal IR modalities can improve algorithm perceptibility under difficult conditions.
- Simple operations such as concatenation, addition, and cross product cannot fully exploit modality complementarity and may aggravate modality imbalance.
- CMAFF retains and enhances modality-specific features while selecting modality-shared features to mitigate redundancy.
- CMAFF uses joint common-modality and differential-modality attentions as a lightweight multispectral fusion approach.
- YOLOFusion combines CMAFF with YOLOv5-small as a unified end-to-end multispectral detection network.
2. Related work
Related work covers advanced CNN detectors and four-stage multispectral fusion strategies. Prior studies favor middle fusion, while subsequent methods refine feature interaction through attention, gating, and channel-wise weighting.
- Advanced detection algorithms: YOLO variants are emphasized for concise architecture, fast running speed, and strong detection performance.
- Multispectral fusion approaches: Multispectral fusion is categorized as early, middle, late, or score fusion according to where RGB and IR information are combined.
- Multispectral fusion approaches: Middle fusion has been reported to outperform the other three fusion forms across single-stage and two-stage detector frameworks.
- Multispectral fusion approaches: Existing modules learn modality combinations using interactive attention, context enhancement, gated fusion, adaptive intra- and inter-modality attention, or channel-wise attention.
- Multispectral fusion approaches: A common fusion goal is retaining reliable features while suppressing redundant and degraded information.
3. Method
YOLOFusion combines a two-stream RGB/IR detector with CMAFF, which separately enhances modality-specific features and selects modality-shared features. CMAFF is designed as a plug-and-play fusion module for two-stream CNN detectors.
- Overall YOLOFusion architecture: YOLOFusion combines YOLOv5-small with a two-stream backbone, CMAFF fusion, a PANet-like pyramid, and a detection head with NMS.The two streams extract RGB and IR feature maps before fusion; the pyramid aggregates multi-scale features.
- CMAFF design: CMAFF divides fusion into differential enhancement of modality-specific features and selective recombination of modality-shared features.The design addresses both differences and commonalities between RGB and IR modalities.
- Fusion arrangement: After both modules refine the inputs, their informative feature maps are added for final fusion, and CMAFF can be embedded in two-stream CNN detectors.The module can operate in parallel or sequential arrangements, while fusion occurs at three stages for multi-scale semantic and spatial information.
- Differential Enhancive Module: The differential enhancive module subtracts RGB and thermal features, derives channel attention from the difference, and adds refined modality features before summation.Average and max pooling provide the global receptive field used by the attention mechanism.
- Common Selective Module: The common selective module sums RGB and thermal features, uses softmax channel attention, and remixes selected modality features while filtering redundancy.Its attention maps are multiplied with the respective inputs before the outputs are added.
4. Experiment
The experiments evaluate YOLOFusion on VEDAI, analyze predicted attention masks to interpret improvements, and examine inference speed and parameter counts.
- Experimental protocol: Experiments use the Vehicle Detection in Aerial Imagery (VEDAI) dataset to evaluate the proposed method.The study also visualizes predicted attention masks and reports speed and parameter analyses.
4.1. Dataset
VEDAI contains aligned RGB and IR imagery for small-vehicle detection across varied backgrounds, with nine vehicle categories and more than 3,700 annotated targets. The experiments use 1024 × 1024 images and 10-fold cross-validation.
- Dataset characteristics: VEDAI provides aligned RGB and IR images of the same scenes for multispectral remote-sensing detection.Images depict vehicles across woods, cities, roads, parking lots, construction sites, and fields.
- Dataset scale: The dataset covers nine vehicle categories and more than 3,700 annotated targets in over 1,200 images per resolution.Available resolutions are 1024 × 1024 and 512 × 512.
- Target distribution: Vehicles average 5.5 instances per image and occupy about 0.7% of image pixels, making the targets small relative to the scene.Spatial resolutions are 12.5 cm for 1024 × 1024 images and 25 cm for 512 × 512 images.
4.2. Evaluation metrics
The evaluation uses Precision, Recall, AP0.5, and mAP0.5. AP0.5 is computed at IoU = 0.50, while mAP0.5 averages category AP values at that threshold.
- Metrics: The evaluation reports Precision, Recall, AP0.5, and mAP0.5 for measuring CMAFF and comparing methods.These indicators are adopted to measure detection performance.
- Metric definitions: Precision and Recall distinguish correct detections from false positives and missed true targets using an IoU threshold.A predicted box meeting the ground-truth IoU threshold is a true positive; otherwise it is a false positive, while an undetected true target is a false negative.
- Average precision: AP is the integral of the Precision-Recall Curve for each category, and AP0.5 denotes AP at IoU = 0.50.mAP0.5 averages AP values across all categories at IoU = 0.50.
- Strict evaluation: mAP0.5:0.95 averages mAP across IoU thresholds from 0.50 to 0.95 in steps of 0.05 and is stricter than mAP0.5.The passage identifies mAP0.5:0.95 as the primary challenge metric.
4.3. Implementation details
The implementation uses paired RGB and IR inputs with matched augmentation, fixed 640×640 images, and Mosaic augmentation to address VEDAI’s imbalance and small targets.
- YOLOFusion trains with YOLOv5-small pretrained on MS COCO, SGD, a 1e-2 learning rate, momentum 0.937, 400 epochs, and batch size 32.
- 640×640 images are used during both training and testing, with warmup to stabilize early optimization.
- Mosaic augmentation mixes four training images and exposes objects outside their normal context.
- RGB and IR images must undergo the same augmentation and remain aligned to avoid training failure.
- VEDAI has category imbalance and predominantly small targets; Mosaic makes target positions more random, uniform, and dense.
4.4. Ablation study
Ablation studies compare modality baselines, fusion strategies, attention-module combinations, module arrangements, and qualitative feature behavior on VEDAI.
- Necessity of CMAFF module: RGB-only and thermal-only detectors reached 74.3% and 74.0% mAP, respectively, while YOLOFusion was compared with four multimodal fusion methods.
- Necessity of CMAFF module: 78.6% mAP was achieved by YOLOFusion, versus 74.4% for the worst-performing RGB-only, thermal-only, and summation-based Input Fusion I models.
- Effect of cross-modality modules: Using either CSM or DEM alone improves performance by nearly 1%, with CSM slightly outperforming DEM; jointly using both produces more informative and robust fused features.
- Attention arrangement: Parallel attention-map generation infers finer maps than sequential processing, while DEM outperforms DEM(Concat) and CSM-first ordering beats DEM-first ordering.
- Attention arrangement: All combined attention arrangements outperform using only CSM or DEM, and the best arrangement further improves performance.
- Attention mask interpretation: Visualizations show CSM filtering redundant features and DEM enlarging and intensifying target-focused regions.
- Attention mask interpretation: Thermal features form foreground and background clusters, while color features are described as foreground-focused and thermal features as background-focused.
4.5. Speed and parameter analysis
CMAFF adds limited computational overhead: its three modules contribute 0.55M parameters and 0.1ms inference time while retaining a lightweight design.
- The analysis reports PyTorch inference testing without TensorRT on an NVIDIA TITAN RTX platform, while TensorRT is identified as a future acceleration option.
- CMAFF uses one sigmoid operation, one softmax layer, and several learnable convolutional layers, with other components parameter-free.
- 0.55M parameters and 0.1ms inference time are added by the three CMAFF modules.
- The average additional parameter cost is 0.183M per CMAFF module.
4.6. Comparison with state-of-the-art object detection methods
YOLOFusion achieves state-of-the-art VEDAI detection performance against mono-modality and multimodality methods while using substantially fewer parameters than YOLOv3.
- 2.86% higher mAP0.5 was achieved than the best mono-modality network, YOLO-fine.
- 4.52% higher mAP0.5∶0.95 was achieved than YOLOv3 with mid-level fusion.
- YOLOFusion has approximately 12.5M parameters, compared with approximately 61.6M for the original YOLOv3.
5. Conclusion
CMAFF separates modality-shared and modality-specific processing to exploit complementary information in multispectral features. Experiments and visualizations support its efficacy, while its plug-in design keeps added computation small.
- CMAFF combines common selective and differential enhancive sub-modules to enhance and select fused features.The two sub-modules process different feature types in distinct ways.
- The module improves detector performance while keeping computational overhead small.
- Parallel and sequential arrangements of the two sub-modules are explored to boost the representation power of fused features.
- Extensive experiments on YOLOFusion verify the efficacy of CMAFF, with visualizations interpreting how it fuses two modal branch features.
- As a separate plug-in, CMAFF can be embedded in any two-stream CNN-based detector with little additional computation cost.