Source-linked AI summary
Fusion-Mamba for Cross-modality Object Detection
Wenhao Dong, Haodong Zhu, Shaohui Lin, Xiaoyan Luo, Yunhang Shen, Xuhui Liu, Juan Zhang, Guodong Guo, Baochang Zhang
TL;DR
RGB and infrared modalities offer complementary information, but existing fusion methods inadequately handle disparities between their representations. Fusion-Mamba uses gated Mamba-based hidden-state interaction through SSCS and DSSF, and experiments report new state-of-the-art performance on three public RGB-IR datasets.
Problem
Existing cross-modality fusion methods inadequately account for modality disparities, limiting consistent interaction between RGB and infrared features.
Method
Fusion-Mamba uses an improved gated Mamba hidden state space with SSCS for shallow fusion and DSSF for deeper cross-modal feature interaction.
Results
Extensive experiments on three public RGB-IR datasets achieve new state-of-the-art performance for cross-modal object detection.
Takeaways & Limitations
The work establishes a baseline for Mamba-based cross-modal fusion and supports its potential for RGB-IR feature fusion.
Abstract
from arXiv · showhide
Cross-modality fusing complementary information from different modalities effectively improves object detection performance, making it more useful and robust for a wider range of applications. Existing fusion strategies combine different types of images or merge different backbone features through elaborated neural network modules. However, these methods neglect that modality disparities affect cross-modality fusion performance, as different modalities with different camera focal lengths, placements, and angles are hardly fused. In this paper, we investigate cross-modality fusion by associating cross-modal features in a hidden state space based on an improved Mamba with a gating mechanism. We design a Fusion-Mamba block (FMB) to map cross-modal features into a hidden state space for interaction, thereby reducing disparities between cross-modal features and enhancing the representation consistency of fused features. FMB contains two modules: the State Space Channel Swapping (SSCS) module facilitates shallow feature fusion, and the Dual State Space Fusion (DSSF) enables deep fusion in a hidden state space. Through extensive experiments on public datasets, our proposed approach outperforms the state-of-the-art methods on $m$AP with 5.9% on $M^3FD$ and 4.9% on FLIR-Aligned datasets, demonstrating superior object detection performance. To the best of our knowledge, this is the first work to explore the potential of Mamba for cross-modal fusion and establish a new baseline for cross-modality object detection.
1. Introduction
Cross-modality RGB-IR detection benefits from complementary information but is hindered by modality disparities that existing fusion methods do not adequately address. Fusion-Mamba uses gated hidden-state interaction through FMB to reduce these disparities and improve fused-feature consistency.
- FMB heatmaps show more effective fusion and stronger target-focused localization than the illustrated single-modality, CNN-fusion, and Transformer-fusion methods.
- RGB and infrared images provide complementary information: infrared captures thermal structure despite luminance, while visible images provide texture but are affected by lighting.
- Existing CNN- and Transformer-based fusion methods inadequately model cross-modality disparities and correlations between targets.
- Fusion-Mamba maps cross-modal features into a hidden state space using improved Mamba and gating to reduce disparities during fusion.
- The Fusion-Mamba block combines SSCS for shallow feature fusion with DSSF for deeper hidden-state feature association and complementarity.
- Extensive experiments on three public RGB-IR datasets show state-of-the-art performance and establish a new cross-modal object-detection baseline.
2. Related Works
Related work spans pixel-level and feature-level multimodal detection, while Mamba has mainly been extended to vision tasks such as detection, segmentation, and video understanding. This work introduces Mamba specifically for multimodal feature fusion in a hidden state space.
- Multi-modality Object Detection: Multimodal object detection research primarily follows pixel-level image fusion or feature-level fusion strategies.
- Mamba: Mamba-based vision research has addressed object detection, segmentation, medical imaging, and video understanding through task-specific architectures.
- Mamba: Fusion-Mamba differs from prior methods by applying Mamba to multimodality feature fusion in a hidden state space.
3. Method
The method uses Fusion-Mamba blocks to fuse RGB and infrared features through shallow channel interaction and deeper hidden-state-space association. Its scan-based, gated design reduces modality disparities while retaining local information, and it offers lower fusion complexity than Transformer-based alternatives.
- Preliminaries: State Space Models process input sequences through implicit states, while Mamba selectively propagates or forgets information based on the input.Mamba parameterizes the SSM using input-dependent Δ, A, B, and C.
- Preliminaries: SS2D expands an image into four sequences, processes them independently with S6 blocks, and merges the outputs into a 2D feature map.This adapts selective scanning to spatial visual data.
- Architecture: The backbone extracts RGB and infrared features with dual streams and applies three Fusion-Mamba blocks before the detection neck and head.FMB associates cross-modal features in a hidden state space to reduce disparities and improve representation consistency.
- SSCS module: The SSCS module performs shallow fusion by splitting RGB and infrared channels, swapping complementary parts, and applying a VSS block.The resulting shallow fused features enhance cross-modality interaction.
- DSSF module: The DSSF module projects both modalities into a hidden state space and uses dual gating to perform deeper cross-modal feature fusion.Gating outputs modulate hidden-state features, while projection back to the original space and residual connections produce complementary features.
- Compared to Transformer-based fusion: Fusion-Mamba scans features in four directions, maps patches into hidden space, and uses gated attention to suppress redundancy and capture complementary information.Replacing Transformer-based fusion saves 7−19ms inference time on one paired image, while Mamba has O(N) rather than Transformer attention’s O(N^2) complexity.
4. Experiments
Experiments evaluate Fusion-Mamba on three visible-infrared benchmarks against state-of-the-art detectors and through ablations of its fusion components. The method achieves strong detection performance while reducing inference time and parameter count relative to transformer-based fusion.
- Experimental setup: Fusion-Mamba is evaluated on LLVIP, M 3FD, and FLIR-Aligned using mAP and mAP50, with inference time also reported.LLVIP contains paired low-light pedestrian images; M 3FD covers varied driving environments; FLIR-Aligned includes day and night scenes.
- LLVIP results: 97.0% mAP50 and 64.3% mAP are achieved by Fusion-Mamba with the YOLOv8 backbone on LLVIP, establishing state-of-the-art performance.With YOLOv5, Fusion-Mamba gains 0.9% mAP over IR-only YOLOv5 and 1.5% over RSDet.
- M 3FD results: 3% mAP50 and 4.4% mAP increases are obtained with the YOLOv8 backbone on M 3FD, while YOLOv5 Fusion-Mamba exceeds SuperFusion by 1.5% on both metrics.The YOLOv8 model achieves new state-of-the-art results for People, Bus, Motorcycle, and Truck categories.
- FLIR-Aligned results: 5.6% and 4.9% improvements over CrossFormer are reported in mAP50 and mAP for YOLOv8 on FLIR-Aligned.YOLOv5 Fusion-Mamba also surpasses CrossFormer by 5% mAP50 and 2.3% mAP.
- Efficiency: 7–19ms inference-time savings and about 100M fewer parameters are reported against transformer-based fusion with the YOLOv5 backbone.The YOLOv8 version adds about 40M parameters relative to YOLOv5 but increases mAP by 2.6%.
- Ablation study: Removing SSCS, DSSF, or both reduces performance, with joint removal causing drops of 4.8% mAP50 and 7.6% mAP.The {P3, P4, P5} FMB placement provides the best performance–computation trade-off, and eight DSSF modules perform best.
5. Conclusion
Fusion-Mamba combines shallow channel swapping with deeper hidden-state interaction through SSCS and DSSF. Experiments on three public RGB-IR datasets show state-of-the-art performance and higher inference efficiency than Transformers.
- Fusion-Mamba uses SSCS for shallow infrared-visible channel exchange and DSSF for deeper multimodal interaction in a Mamba-based hidden state space.Gated attention suppresses redundant features during fusion.
- Experiments on three public RGB-IR datasets demonstrate new state-of-the-art performance with higher inference efficiency than Transformers.
- The results support Mamba’s potential for cross-modal fusion and motivate further research on Mamba for cross-modal tasks.
6. More Heatmap Visualization Results
Heatmap visualizations across LLVIP, M 3FD, and FLIR-Aligned show Fusion-Mamba focusing more on targets than CNN- and Transformer-based fusion methods. The visualizations also indicate effective modeling of correlations between targets across modalities.
- Across LLVIP, M 3FD, and FLIR-Aligned, Fusion-Mamba heatmaps focus more on targets than CNN- and Transformer-based fusion methods.
- The heatmaps indicate that Fusion-Mamba effectively models correlations between targets in different modalities.
7. Visualization of Object Detection
Detection visualizations across LLVIP, M 3FD, and FLIR-Aligned show Fusion-Mamba reducing missed detections and improving mAP over several state-of-the-art methods. The examples span low lighting, occlusion, adverse weather, distant small targets, and dense scenes.
- Fusion-Mamba significantly reduces missed detections and improves mAP compared with several state-of-the-art methods.
- Under insufficient lighting and severe occlusion in LLVIP, Fusion-Mamba detects more target objects than other methods.The passage attributes this to hidden-space interaction integrating infrared information.
- In harsh weather with small, distant targets in M 3FD, Fusion-Mamba detects more diverse target objects than other methods.The passage attributes this to gated attention combining features.
- Fusion-Mamba achieves the best detection performance in complex scenes involving low lighting, adverse weather, high occlusion, target density, and small targets.