Pan-Mamba: Effective pan-sharpening with State Space Model

• Pan-sharpening combines low-resolution multispectral images with texture-rich high-resolution panchromatic images to produce high-resolution multispectral imagery.Source: §1 · p. 1
• Existing methods face global-information challenges because Transformers can be computationally costly or restrict receptive fields, while Fourier approaches create spatial-frequency information gaps.Source: §1 · p. 2
• Pan-Mamba uses Mamba as its core module, with channel-swapping and cross-modal Mamba components for global information extraction and efficient feature fusion.Source: §1 · p. 2
• Channel-swapping Mamba exchanges partial PAN and LRMS channels to initiate lightweight cross-modal interaction, while cross-modal Mamba exploits relationships between modalities for fusion.Source: §1 · pp. 2–3
• The proposed method demonstrates state-of-the-art results in qualitative and quantitative assessments across multiple datasets.Source: §1 · p. 3

• Traditional pan-sharpening methods rely on manually designed priors, including component substitution, multi-resolution analysis, and variational optimization algorithms.Source: §2.1 · p. 3
• Deep learning methods began with PNN and later adopted residual blocks, multi-scale convolution, and progressive designs for pan-sharpening.Source: §2.1 · p. 3
• State space models evolved from S4 through S5 and H3, with later models reducing complexity and competing with Transformers in global-information modeling.Source: §2.2 · p. 4
• Visual state space models such as Vision Mamba achieved strong classification and segmentation results, but multimodal image-fusion applications remained insufficiently explored.Source: §2.2 · p. 4

Preliminaries

• The state space model maps an input sequence x(t) through hidden state h(t) to output y(t), with A, B, and C as evolution and projection parameters.The model uses a hidden space of dimension N and projects the state to the output.Source: §3.1 · p. 4
• Mamba discretizes continuous parameters A and B using timescale ∆ and zero-order hold, then computes recurrent states and outputs through the discrete system.The discrete representation uses h_t = Āh_{t-1} + B̄x_t and y_t = Ch_t.Source: §3.1 · pp. 4–5

Network Architecture

• The architecture contains Mamba, channel-swapping Mamba, and cross-modal Mamba blocks for long-range extraction and cross-modal fusion.Mamba processes PAN and LRMS features, while the other two blocks explore relationships between modalities.Source: §3.2 · p. 5, Figure 1 · p. 5
• Convolutional projection and flattening convert PAN and LRMS images into tokens, which independent Mamba blocks process for global feature extraction.The resulting token sequences are denoted T^ms_i and T^pan_i, with ψ_mi and ψ_pi representing modality-specific Mamba blocks.Source: §3.2 · p. 6

Mamba Block

• The Mamba block normalizes tokens, projects and convolves them, computes SSM outputs, gates them with z, and adds them residually to the input.Its computational complexity is expressed as 3N(2D)C + N(2D)C, with D and C fixed.Source: §3.3 · pp. 6–7

Channel Swapping Mamba

• Channel swapping divides each modality’s features into channel portions, processes exchanged portions with Mamba, and adds the results to enhance cross-modal feature diversity.The exchanged features incorporate information from distinct channels before subsequent fusion.Source: Channel Swapping Mamba Block · pp. 7–8

Cross Modal Mamba

• Cross-modal Mamba projects both modalities into a shared space, uses gating to learn complementary features and suppress redundancy, then applies depth-wise convolution to enhance locality.The fused features are reshaped and passed through convolution layers and residual connections to produce the final output.Source: §3.2 · p. 6, Cross modality Mamba Block · p. 8

Loss Function

• Training uses L1 loss between the predicted output Mout and corresponding ground truth G.The loss is written as L = ||G - Mout||_1.Source: §3.4 · p. 8

Datasets

• Experiments use WorldView-II, Gaofen-2, and WorldView-III datasets containing industrial, natural, mountainous, riverine, road, and urban scenes.The datasets provide diverse resolutions and a broad spectrum of scenes.Source: §4.1 · p. 9
• When ground truth is absent, the dataset generation process follows the Wald protocol.This protocol is used for the experimental dataset construction described by the authors.Source: §4.1 · p. 9

Implementation

• Training runs on an Nvidia V100 GPU using PyTorch with N=32 feature channels.The implementation and training procedures use the PyTorch framework.Source: §4.2 · p. 9

Training Settings

• The initial learning rate is 5e-4 and decays to 5e-8 after 500 epochs with cosine scheduling; Adam optimization uses gradient clipping of 4.Training lasts 200 epochs for WV2 and 500 epochs for GF2 and WV3.Source: §4.2 · p. 9

• 0.38 and 0.29 PSNR improvements are reported on the WorldView-II and WorldView-III datasets, respectively.The authors interpret these gains as closer alignment with ground truth.Source: Evaluation on Reduced-Resolution Scene · p. 9
• WV2 spectral similarity surpasses state-of-the-art methods, while WV3 and GF2 show comparable outcomes.Source: Evaluation on Reduced-Resolution Scene · p. 10
• ERGAS validates the method’s overall superior performance across each spectral band.Source: Evaluation on Reduced-Resolution Scene · p. 10
• Pan-Mamba consistently produces the smallest errors in representative WorldView-II and WorldView-III visual comparisons.Brighter regions in the error maps indicate greater discrepancies from ground truth.Source: Evaluation on Reduced-Resolution Scene · p. 10
• The method extracts high-frequency information and preserves spectral details, producing clearer textures in pan-sharpened results.Source: Evaluation on Reduced-Resolution Scene · p. 10

• The full-resolution WV2 visual results showcase the method’s superior image-fusion capabilities.Source: Evaluation on Full-Resolution Scene · p. 11
• Figure 2 compares the approach with nine other methods on the WorldView-II dataset.Source: Figure 2 · p. 11
• Figure 3 compares the approach with nine other methods on the WorldView-III dataset.Source: Figure 3 · p. 11
• Figure 4 compares the approach with four other methods on the full-resolution WV2 dataset.Source: Figure 4 · p. 12

• Ablations are trained and reported exclusively on WorldView-II with consistent configurations for fair comparison.Source: §4.4 · p. 12
• The second ablation part removes or substitutes three proposed modules to assess their individual functionality.Source: §4.4 · p. 12
• The first ablation part compares the state space model with prevalent low-level vision operators.Substitutions include convolution, transposed attention, window attention, and self-attention.Source: §4.4 · p. 12

• The state space model uses the fewest parameters and least computational resources among the compared core operators, while achieving superior WV2 performance.Source: Effectiveness of the State Space Model · p. 13
• Removing the Mamba block causes a decline in model performance, supporting its role in modeling long-range feature dependencies.Source: Effectiveness of the Mamba Block · p. 13
• Removing channel swapping and its associated Mamba block causes a noticeable performance decline despite the module’s low computational cost.The module is designed for shadow feature fusion and channel-feature diversity.Source: Effectiveness of the channel swapping… · p. 13

• Removing the cross-modal Mamba block causes a substantial decline in fusion results.The ablation retains only Channel Swapping Mamba for feature fusion.Source: Effectiveness of the cross mmodal… · p. 13
• Table 4 compares methods using FLOPs and parameter numbers.Source: Table 4 · p. 13
• Figure 5 compares performance and efficiency among operators and the proposed model.Source: Figure 5 · p. 14

• The efficiency analysis compares computational complexity between Pan-Mamba, ablation operators, and benchmark methods.Source: §4.5 · p. 14
• Pan-Mamba achieves the best FLOPs metric among self-attention, transposed attention, window attention, and convolution across input resolutions from 128 × 128 to 1024 × 1024.GPU memory usage and inference time are comparable to transposed attention and lower than window attention and self-attention.Source: §4.5 · p. 14
• Pan-Mamba has computational complexity comparable to benchmark methods while achieving the best results.The comparison uses similar parameter and computational complexity.Source: §4.5 · p. 14

• Pan-Mamba incorporates Mamba, channel swapping Mamba, and cross-modal Mamba blocks.Source: §5 · p. 14
• The network achieves efficient global feature extraction and cross-modal information exchange with linear complexity.Source: §5 · p. 14
• On publicly available remote-sensing datasets, Pan-Mamba outperforms state-of-the-art methods with a lightweight model.The conclusion highlights robust spectral accuracy and preservation of texture information.Source: §5 · p. 14