PaperlayerView exact revision on arXiv

Source-linked AI summary

FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba

Xinyu Xie, Yawen Cui, Tao Tan, Xubin Zheng, Zitong Yu

arXiv:2404.09498v3cs.CV

TL;DR

Multimodal fusion needs both global context and detailed, cross-modal feature interaction, while existing CNN and Transformer approaches have complementary limitations. FusionMamba enhances Mamba with dynamic convolution, channel attention, and a dynamic feature fusion module. It reports state-of-the-art performance across multimodal image-fusion tasks and downstream experiments.

  • Problem

    Existing fusion methods struggle with global context, local detail, channel redundancy, and effective inter-modal interaction.

  • Method

    FusionMamba integrates dynamic convolution and channel attention into Mamba and combines DFEM with CMFM in a dynamic feature fusion module.

  • Results

    FusionMamba achieves state-of-the-art performance across various multimodal image-fusion tasks and downstream experiments.

  • Takeaways & Limitations

    The framework provides an efficient and versatile Mamba-based alternative for multimodal image fusion across IR-VIS, medical, and biomedical tasks.

Abstract

from arXiv · show

Multimodal image fusion aims to integrate information from different imaging techniques to produce a comprehensive, detail-rich single image for downstream vision tasks. Existing methods based on local convolutional neural networks (CNNs) struggle to capture global features efficiently, while Transformer-based models are computationally expensive, although they excel at global modeling. Mamba addresses these limitations by leveraging selective structured state space models (S4) to effectively handle long-range dependencies while maintaining linear complexity. In this paper, we propose FusionMamba, a novel dynamic feature enhancement framework that aims to overcome the challenges faced by CNNs and Vision Transformers (ViTs) in computer vision tasks. The framework improves the visual state-space model Mamba by integrating dynamic convolution and channel attention mechanisms, which not only retains its powerful global feature modeling capability, but also greatly reduces redundancy and enhances the expressiveness of local features. In addition, we have developed a new module called the dynamic feature fusion module (DFFM). It combines the dynamic feature enhancement module (DFEM) for texture enhancement and disparity perception with the cross-modal fusion Mamba module (CMFM), which focuses on enhancing the inter-modal correlation while suppressing redundant information. Experiments show that FusionMamba achieves state-of-the-art performance in a variety of multimodal image fusion tasks as well as downstream experiments, demonstrating its broad applicability and superiority.

1 Introduction

Multimodal fusion combines complementary imaging information, but existing CNN and Transformer approaches face limitations in global context, spatial adaptation, and inter-modal interaction. FusionMamba addresses these issues with dynamic Mamba-based feature enhancement and fusion modules, reporting leading performance across multiple fusion tasks.

  • Motivation: Multimodal image fusion integrates information from different imaging modalities into one image for medical imaging, remote sensing, security, autonomous driving, and computer vision.Infrared-visible fusion combines low-light robustness with visible-image detail, while medical fusion combines modalities such as PET, CT, and MRI.
  • Challenges: Static CNN convolutions have constrained receptive fields and process pixels uniformly, limiting global context and sensitivity to spatial differences.These limitations are problematic because fusion inputs and spatial content vary across images.
  • Challenges: Current fusion strategies often overlook inter-modal relationships or fail to preserve texture detail, motivating stronger modal connections and emphasis on key image information.Element-wise fusion, deeper feature extraction, and existing cross-modal methods each leave interaction or detail limitations.
  • Mamba Motivation: Mamba offers long-range dependency modeling with linear scalability and efficient parallel processing, providing a route toward balancing global receptive fields and computational efficiency.Its recursive state-space formulation and parallel scanning support efficient training on modern hardware.
  • Proposed Approach: FusionMamba integrates dynamic convolution and channel attention into Mamba, reducing channel redundancy while enhancing local feature extraction and retaining global modeling capabilities.The framework also introduces DFFM, combining DFEM for texture and differential-feature enhancement with CMFM for inter-modal correlation.
  • Results: FusionMamba achieves leading performance across IR-VIS, multimodal medical, and biomedical image fusion tasks.The reported evaluation emphasizes the framework’s efficiency and versatility across multiple fusion settings.

2 Related Work

Related work progresses from CNN-based fusion toward Transformer and state-space approaches. CNNs offer efficient local processing but limited global context, while SSMs and Mamba provide linear-scaling long-range modeling and have expanded into vision applications.

  • Deep Learning Fusion: Deep learning image-fusion frameworks include CNNs, GANs, autoencoders, Transformer-based networks, and related architectures.
  • CNN-Based Methods: Early CNN-based methods commonly used simple element-wise fusion rules, later improving results through dense connections, pyramid structures, concatenation, and task-specific losses.
  • Transformer-Based Methods: Transformers address CNNs’ limited global feature extraction, while attention and adaptive localization methods target global and local features in fusion tasks.Some Transformer-based methods still face local-feature accuracy limitations.
  • State Space Models: SSMs provide linear scalability with sequence length for long-range dependency modeling by mapping input sequences through implicit states to outputs.S4 and Mamba are presented as structured state-space sequence models based on continuous systems.
  • State Space Models: SSM parameters are discretized using a timescale parameter and commonly the zero-order hold rule before practical deep-learning implementation.The state and projection matrices define the continuous formulation and its discrete counterpart.
  • State Space Models: The discretized state-space formulation can be equivalently represented in CNN form using a structured convolution kernel over the input sequence.The kernel length corresponds to the input-sequence length.
  • Mamba in Vision: Mamba’s data-dependent mechanisms, hardware efficiency, and language performance have motivated applications in image classification, video understanding, biomedical segmentation, and other vision tasks.

3 Methodology

FusionMamba uses a U-Net-based architecture with dynamic visual state-space modules for feature extraction and reconstruction, and a DFFM for modality-aware fusion. Its modules combine long-range dependency modeling with dynamic local texture enhancement, channel redundancy reduction, difference perception, and cross-modal correlation modeling.

  • Overall framework: FusionMamba separates feature extraction, feature fusion, and feature reconstruction within a U-Net-based architecture.The network uses four layers with patch merging and expanding operations; DVSS modules support extraction and reconstruction.
  • Dynamic Vision State Space Module: DVSS combines ESSM, dynamic local convolution, and efficient channel attention to model long-range dependencies while enhancing local features and reducing channel redundancy.LDC restores neighborhood similarity and learns intricate textures, while ECA selects critical channels.
  • Dynamic Feature Enhancement Module: DFEM adaptively enhances modality-specific texture details and perceives inter-modal differences using dynamic convolution, coarse-grained fusion, and difference-weighted feature maps.The two modality features are processed with LDC, differenced, globally pooled, and passed through sigmoid weighting before application to fused features.
  • Cross-Modal Fusion Mamba Module: CMFM performs fine-grained fusion by applying depthwise convolution, hybrid feature generation, efficient 2D scanning, and channel attention to cross-modal features.The module uses ES2D for long-term spatial dependencies and ECA to reduce channel redundancy.
  • Loss Function: Training uses intensity, texture, and structural losses, with gradient loss preserving fine-grained image details.The total loss combines Lint, Ltext, and Lssim, while the gradient term compares fused-image gradients with the maximum source-image gradients.

4.1 Setup

The experiments cover visible-infrared, medical, and cellular multimodal image fusion using public datasets, augmentation, standard hardware settings, and six fusion-quality metrics. FusionMamba is compared with traditional, CNN-based, and Transformer-based methods.

  • Datasets: The KAIST dataset provides 70,000 training pairs of infrared and visible-light images for visible-infrared fusion.Scenes include schoolyards, streets, countryside traffic, people, cars, bicycles, and road signs across daytime and nighttime conditions.
  • Datasets: Harvard provides 166 CT-MRI, 329 PET-MRI, and 539 SPECT-MRI pairs, with 30,000 augmented pairs and 21 randomly selected test pairs.The medical images are 256×256 pixels and are used independently for training and testing.
  • Datasets: The GFP dataset contains 148 registered GFP-PC image pairs of Arabidopsis thaliana cells, augmented to 30,000 pairs with 20 random test pairs.Each image is 358 × 358 pixels.
  • Implementation: Training uses Adam with learning rate 0.0002, batch size 2, and loss weights α1=100, α2=10, and α3=1 on an RTX 3090 GPU.The implementation uses PyTorch with an Intel Core i9-9900K CPU.
  • Evaluation: Fusion quality is evaluated using SCD, MS-SSIM, QAB/F, FMI, and VIF, which measure structural, similarity, edge, information, and visual-fidelity properties.The study compares FusionMamba with traditional CSMCA, CNN-based methods, and Transformer-based methods using public codes and corresponding parameter settings.

4.2 Infrared and Visible Image Fusion

Across TNO, MSRS, and RoadScene infrared-visible fusion datasets, FusionMamba consistently preserves target details and textures while reducing thermal contamination and texture blurring.

  • FusionMamba consistently outperforms comparison methods across the TNO, MSRS, and RoadScene datasets, especially in preserving critical target details and texture information.
  • FusionMamba mitigates thermal target degradation, spectral contamination, and texture blurring by integrating complementary and shared information across modalities.
  • The efficient Mamba framework provides global context awareness that improves feature extraction and reconstruction.
  • FusionMamba achieves the highest QAB/F score and significantly outperforms competing methods in VIF, while FMI and MS-SSIM preserve texture and structural similarity.
  • The SCD metric confirms that FusionMamba minimizes texture distortion better than other approaches and delivers well-rounded fusion performance.

4.3 Multimodal Medical Image Fusion

FusionMamba is evaluated on CT-MRI, PET-MRI, and SPECT-MRI fusion tasks using qualitative comparisons and quantitative metrics on the Harvard public medical dataset.

  • The medical evaluation covers CT-MRI, PET-MRI, and SPECT-MRI fusion tasks on the Harvard public medical dataset.
  • In CT-MRI fusion, FusionMamba preserves dense CT information and MRI texture details, whereas FusionGAN and U2Fusion show unsatisfactory brightness and clarity.
  • In PET-MRI and SPECT-MRI fusion, FusionMamba preserves clear edges and textures with a color distribution more similar to PET images.
  • FusionMamba achieves optimal average results for VIF, SCD, QAB/F, MS-SSIM, and FMI in CT-MRI fusion, with similar superiority across PET-MRI and SPECT-MRI metrics.

4.4 Multimodal Biomedical Image Fusion

FusionMamba is tested for GFP-PC biomedical image fusion, where it preserves both texture and color information in comparison with existing methods.

  • GFP images provide protein-distribution information, while PC images contain structural details such as nuclei and mitochondria.
  • Existing methods show trade-offs between color preservation and texture detail, with some retaining color but losing texture and others introducing color distortion.
  • SwinFusion and FusionMamba effectively preserve both texture and color information in GFP-PC fusion images.

4.5 Computational Cost Analysis

The computational cost analysis evaluates operational efficiency using FLOPs and running time, including comparisons with a traditional Matlab implementation.

  • Table 8 measures operational efficiency through floating-point operations per second and running time.
  • The evaluation tests the first dataset image in an infrared-visible fusion scenario and includes CSMCA to compare deep-learning and traditional methods.

5 Ablation Experiments

Ablation experiments show that FusionMamba’s dynamic state-space and feature-fusion components improve fusion quality, while the joint loss and its weighting balance detail, texture, and structure.

  • Network Structure: Replacing DVSS with a Transformer decreases metrics, while EVSS is satisfactory but less effective than DVSS.The authors attribute DVSS’s advantage to stronger feature extraction capability.
  • Network Structure: Removing DFFM decreases QAB/F, removing DFEM produces similar results, and removing CMFM degrades VIF or MS-SSIM.These ablations support the roles of dynamic feature enhancement and cross-modal integration in fusion.
  • Loss Function: Removing SSIM loss decreases MS-SSIM, removing texture loss significantly decreases QAB/F, and excluding intensity loss diminishes visual impact.The complete loss function performs best across the evaluated metrics.
  • Parameter Setting: Increasing δ1 improves VIF but can overemphasize details, increasing δ2 reduces SCD but can smooth textures, and increasing δ3 enhances MS-SSIM but can reduce natural appearance.The selected setting δ1 = 20, δ2 = 10, and δ3 = 10 achieves the best overall performance across VIF, SCD, MS-SSIM, and FMI.

6 Application of Downstream Tasks

FusionMamba is evaluated in downstream object detection and multimodal brain-tumor segmentation, where it outperforms competing fusion approaches on reported detection and segmentation measures.

  • Object Detection: FusionMamba outperforms seven state-of-the-art methods in object detection precision, recall, and mean accuracy (mAP).The evaluation uses pre-trained YOLOv5s models on fused, infrared, and visible images from MSRS.
  • Multimodal Segmentation: FusionMamba fusion improves segmentation accuracy over single-modality MR-Flair and MR-T2 inputs and the SwinFusion pipeline.The study uses Unet with Dice coefficient and Sensitivity on BRATS19 MR-Flair and MR-T2 data.
  • Multimodal Segmentation: FusionMamba refines image details while preserving contrast, supporting the reported segmentation improvement.This explanation is stated for the multimodal tumor segmentation results.

7 Conclusion

The paper concludes that FusionMamba combines Mamba with dynamic feature enhancement and DFFM to achieve state-of-the-art multimodal image-fusion performance and reported generalization across tasks.

  • Conclusion: FusionMamba integrates an efficient Mamba model with dynamic convolution, channel attention, DFEM, and CMFM.The design targets global modeling, local feature extraction, texture and disparity enhancement, and inter-modal correlation.
  • Conclusion: DFFM contains DFEM and CMFM, which enhance texture and difference perception while improving modality correlation and suppressing redundant information.These functions are described as part of the proposed dynamic feature fusion design.
  • Conclusion: The method demonstrates state-of-the-art performance in various multimodal image-fusion tasks and reported downstream experiments.The authors present these results as validating FusionMamba’s generalization ability.
  • Future Work: Future work will examine real-time deployment, resource-constrained devices, more diverse datasets, and emerging fusion methods.These directions define the stated scope for extending practical evaluation and deployment.
Loading 2404.09498v3…