Source-linked AI summary
Pan-Mamba: Effective pan-sharpening with State Space Model
Xuanhua He, Ke Cao, Keyu Yan, Rui Li, Chengjun Xie, Jie Zhang, Man Zhou
TL;DR
Pan-sharpening must fuse low-resolution multispectral and high-resolution panchromatic information, while existing approaches face challenges in global modeling and multimodal fusion. Pan-Mamba introduces Mamba-based blocks for efficient global feature extraction and cross-modal exchange, and reports state-of-the-art fusion results with a lightweight model.
Problem
Pan-sharpening requires combining complementary LRMS and PAN imagery, while existing methods face global-information limitations and Mamba remains insufficiently explored for multimodal image fusion.
Method
Pan-Mamba uses Mamba, channel-swapping Mamba, and cross-modal Mamba blocks for global feature extraction and efficient cross-modal information exchange.
Results
Pan-Mamba achieves state-of-the-art fusion results on publicly available remote-sensing datasets with robust spectral accuracy and texture preservation.
Takeaways & Limitations
Pan-Mamba establishes an application of Mamba to pan-sharpening with efficient global feature extraction and linear-complexity cross-modal information exchange.
Abstract
from arXiv · showhide
Pan-sharpening involves integrating information from low-resolution multi-spectral and high-resolution panchromatic images to generate high-resolution multi-spectral counterparts. While recent advancements in the state space model, particularly the efficient long-range dependency modeling achieved by Mamba, have revolutionized computer vision community, its untapped potential in pan-sharpening motivates our exploration. Our contribution, Pan-Mamba, represents a novel pan-sharpening network that leverages the efficiency of the Mamba model in global information modeling. In Pan-Mamba, we customize two core components: channel swapping Mamba and cross-modal Mamba, strategically designed for efficient cross-modal information exchange and fusion. The former initiates a lightweight cross-modal interaction through the exchange of partial panchromatic and multi-spectral channels, while the latter facilities the information representation capability by exploiting inherent cross-modal relationships. Through extensive experiments across diverse datasets, our proposed approach surpasses state-of-the-art methods, showcasing superior fusion results in pan-sharpening. To the best of our knowledge, this work is the first attempt in exploring the potential of the Mamba model and establishes a new frontier in the pan-sharpening techniques. The source code is available at \url{https://github.com/alexhe101/Pan-Mamba}.
1 Introduction
Pan-sharpening fuses complementary low-resolution multispectral and high-resolution panchromatic imagery, but existing methods struggle with global information and efficient cross-modal modeling. Pan-Mamba addresses these challenges with Mamba-based components and reports state-of-the-art results across multiple datasets.
- Pan-sharpening combines low-resolution multispectral images with texture-rich high-resolution panchromatic images to produce high-resolution multispectral imagery.
- Existing methods face global-information challenges because Transformers can be computationally costly or restrict receptive fields, while Fourier approaches create spatial-frequency information gaps.
- Pan-Mamba uses Mamba as its core module, with channel-swapping and cross-modal Mamba components for global information extraction and efficient feature fusion.
- Channel-swapping Mamba exchanges partial PAN and LRMS channels to initiate lightweight cross-modal interaction, while cross-modal Mamba exploits relationships between modalities for fusion.
- The proposed method demonstrates state-of-the-art results in qualitative and quantitative assessments across multiple datasets.
2 Related Work
Related work spans traditional rule-based and deep learning-based pan-sharpening methods, followed by state space models that efficiently model global information. Their application to multimodal image fusion remains insufficiently explored.
- Traditional pan-sharpening methods rely on manually designed priors, including component substitution, multi-resolution analysis, and variational optimization algorithms.
- Deep learning methods began with PNN and later adopted residual blocks, multi-scale convolution, and progressive designs for pan-sharpening.
- State space models evolved from S4 through S5 and H3, with later models reducing complexity and competing with Transformers in global-information modeling.
- Visual state space models such as Vision Mamba achieved strong classification and segmentation results, but multimodal image-fusion applications remained insufficiently explored.
3 Methods
Pan-Mamba combines Mamba-based long-range feature extraction with channel-swapping and cross-modal Mamba blocks for shallow and deep fusion of PAN and LRMS features.
- Preliminaries: The state space model maps an input sequence x(t) through hidden state h(t) to output y(t), with A, B, and C as evolution and projection parameters.The model uses a hidden space of dimension N and projects the state to the output.
- Preliminaries: Mamba discretizes continuous parameters A and B using timescale ∆ and zero-order hold, then computes recurrent states and outputs through the discrete system.The discrete representation uses h_t = Āh_{t-1} + B̄x_t and y_t = Ch_t.
- Network Architecture: The architecture contains Mamba, channel-swapping Mamba, and cross-modal Mamba blocks for long-range extraction and cross-modal fusion.Mamba processes PAN and LRMS features, while the other two blocks explore relationships between modalities.
- Network Architecture: Convolutional projection and flattening convert PAN and LRMS images into tokens, which independent Mamba blocks process for global feature extraction.The resulting token sequences are denoted T^ms_i and T^pan_i, with ψ_mi and ψ_pi representing modality-specific Mamba blocks.
- Mamba Block: The Mamba block normalizes tokens, projects and convolves them, computes SSM outputs, gates them with z, and adds them residually to the input.Its computational complexity is expressed as 3N(2D)C + N(2D)C, with D and C fixed.
- Channel Swapping Mamba: Channel swapping divides each modality’s features into channel portions, processes exchanged portions with Mamba, and adds the results to enhance cross-modal feature diversity.The exchanged features incorporate information from distinct channels before subsequent fusion.
- Cross Modal Mamba: Cross-modal Mamba projects both modalities into a shared space, uses gating to learn complementary features and suppress redundancy, then applies depth-wise convolution to enhance locality.The fused features are reshaped and passed through convolution layers and residual connections to produce the final output.
- Loss Function: Training uses L1 loss between the predicted output Mout and corresponding ground truth G.The loss is written as L = ||G - Mout||_1.
4 Experiment
Experiments use WV2, GF2, and WV3 datasets spanning diverse resolutions and scenes, with Wald-protocol generation where ground truth is unavailable.
- Datasets: Experiments use WorldView-II, Gaofen-2, and WorldView-III datasets containing industrial, natural, mountainous, riverine, road, and urban scenes.The datasets provide diverse resolutions and a broad spectrum of scenes.
- Datasets: When ground truth is absent, the dataset generation process follows the Wald protocol.This protocol is used for the experimental dataset construction described by the authors.
- Implementation: Training runs on an Nvidia V100 GPU using PyTorch with N=32 feature channels.The implementation and training procedures use the PyTorch framework.
- Training Settings: The initial learning rate is 5e-4 and decays to 5e-8 after 500 epochs with cosine scheduling; Adam optimization uses gradient clipping of 4.Training lasts 200 epochs for WV2 and 500 epochs for GF2 and WV3.
4.3 Comparison with State of Arts Methods
Pan-Mamba outperforms state-of-the-art methods across quantitative and visual evaluations on multiple datasets, with reported gains in key image-quality measures.
- 0.38 and 0.29 PSNR improvements are reported on the WorldView-II and WorldView-III datasets, respectively.The authors interpret these gains as closer alignment with ground truth.
- WV2 spectral similarity surpasses state-of-the-art methods, while WV3 and GF2 show comparable outcomes.
- ERGAS validates the method’s overall superior performance across each spectral band.
- Pan-Mamba consistently produces the smallest errors in representative WorldView-II and WorldView-III visual comparisons.Brighter regions in the error maps indicate greater discrepancies from ground truth.
- The method extracts high-frequency information and preserves spectral details, producing clearer textures in pan-sharpened results.
Evaluation on Full-Resolution Scene
Full-resolution WV2 experiments evaluate Pan-Mamba under real-world degradation without ground truth, using non-reference metrics and visual comparisons against other methods.
- The full-resolution WV2 visual results showcase the method’s superior image-fusion capabilities.
- Figure 2 compares the approach with nine other methods on the WorldView-II dataset.
- Figure 3 compares the approach with nine other methods on the WorldView-III dataset.
- Figure 4 compares the approach with four other methods on the full-resolution WV2 dataset.
4.4 Ablation Study
The ablation study evaluates Pan-Mamba’s core operator and module contributions using controlled experiments on the WorldView-II dataset.
- Ablations are trained and reported exclusively on WorldView-II with consistent configurations for fair comparison.
- The second ablation part removes or substitutes three proposed modules to assess their individual functionality.
- The first ablation part compares the state space model with prevalent low-level vision operators.Substitutions include convolution, transposed attention, window attention, and self-attention.
Effectiveness of the State Space Model.
Ablations support the effectiveness of the state space model and Mamba-based blocks for feature extraction and cross-modal fusion in Pan-Mamba.
- The state space model uses the fewest parameters and least computational resources among the compared core operators, while achieving superior WV2 performance.
- Removing the Mamba block causes a decline in model performance, supporting its role in modeling long-range feature dependencies.
- Removing channel swapping and its associated Mamba block causes a noticeable performance decline despite the module’s low computational cost.The module is designed for shadow feature fusion and channel-feature diversity.
Effectiveness of the cross mmodal Mamba Block.
The cross-modal Mamba block performs deep LRMS–PAN fusion, using gating to reduce redundant features; removing it substantially worsens results.
- Removing the cross-modal Mamba block causes a substantial decline in fusion results.The ablation retains only Channel Swapping Mamba for feature fusion.
- Table 4 compares methods using FLOPs and parameter numbers.
- Figure 5 compares performance and efficiency among operators and the proposed model.
4.5 Comparison of Efficiency
Pan-Mamba offers favorable efficiency across operators and benchmark methods, combining the best FLOPs performance with competitive memory and inference complexity.
- The efficiency analysis compares computational complexity between Pan-Mamba, ablation operators, and benchmark methods.
- Pan-Mamba achieves the best FLOPs metric among self-attention, transposed attention, window attention, and convolution across input resolutions from 128 × 128 to 1024 × 1024.GPU memory usage and inference time are comparable to transposed attention and lower than window attention and self-attention.
- Pan-Mamba has computational complexity comparable to benchmark methods while achieving the best results.The comparison uses similar parameter and computational complexity.
5 Conclusion
Pan-Mamba is a lightweight pan-sharpening network built around Mamba blocks for efficient global feature extraction and cross-modal exchange with linear complexity. It outperforms state-of-the-art methods on public remote-sensing datasets while preserving spectral accuracy and texture information.
- Pan-Mamba incorporates Mamba, channel swapping Mamba, and cross-modal Mamba blocks.
- The network achieves efficient global feature extraction and cross-modal information exchange with linear complexity.
- On publicly available remote-sensing datasets, Pan-Mamba outperforms state-of-the-art methods with a lightweight model.The conclusion highlights robust spectral accuracy and preservation of texture information.