Source-linked AI summary
Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation
Zifu Wan, Pingping Zhang, Yuhao Wang, Silong Yong, Simon Stepputtis, Katia Sycara, Yaqi Xie
TL;DR
Multi-modal semantic segmentation must combine complementary modalities while handling difficult conditions and the computational limits of existing CNN and Transformer approaches. Sigma uses a Siamese Mamba encoder, Mamba-based fusion, and channel-aware decoding, and reports superior accuracy and efficiency across RGB-Thermal and RGB-Depth benchmarks. The authors identify limited exploration beyond two modalities as a remaining limitation.
Problem
Multi-modal semantic segmentation must align and fuse complementary modalities under challenging conditions, while existing CNN and Transformer approaches have receptive-field or computational limitations.
Method
Sigma combines a Siamese Mamba encoder, Mamba-based fusion mechanism, and channel-aware Mamba decoder for multi-modal semantic segmentation.
Results
Sigma outperforms state-of-the-art models in accuracy and efficiency across RGB-Thermal and RGB-Depth semantic segmentation benchmarks.
Takeaways & Limitations
The work presents Mamba as a successful state space model approach for multi-modal semantic segmentation with global modeling and linear complexity.
Takeaways & Limitations
The current exploration focuses primarily on two modalities, leaving Mamba’s capacity for longer multi-modal sequences underutilized.
Abstract
from arXiv · showhide
Multi-modal semantic segmentation significantly enhances AI agents' perception and scene understanding, especially under adverse conditions like low-light or overexposed environments. Leveraging additional modalities (X-modality) like thermal and depth alongside traditional RGB provides complementary information, enabling more robust and reliable prediction. In this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic segmentation utilizing the advanced Mamba. Unlike conventional methods that rely on CNNs, with their limited local receptive fields, or Vision Transformers (ViTs), which offer global receptive fields at the cost of quadratic complexity, our model achieves global receptive fields with linear complexity. By employing a Siamese encoder and innovating a Mamba-based fusion mechanism, we effectively select essential information from different modalities. A decoder is then developed to enhance the channel-wise modeling ability of the model. Our proposed method is rigorously evaluated on both RGB-Thermal and RGB-Depth semantic segmentation tasks, demonstrating its superiority and marking the first successful application of State Space Models (SSMs) in multi-modal perception tasks. Code is available at https://github.com/zifuwan/Sigma.
1. Introduction
Sigma addresses the difficulty of robust multi-modal semantic segmentation in challenging visual conditions by introducing a Siamese Mamba architecture with modality fusion and channel-aware decoding. Experiments on RGB-Thermal and RGB-Depth benchmarks report superior accuracy and efficiency.
- Multi-modal segmentation adds thermal or depth information to RGB but also creates alignment and fusion challenges.
- CNNs provide linear complexity but limited receptive fields, whereas ViTs provide global receptive fields with higher computational cost.
- The method combines a Siamese encoder, Mamba-based fusion mechanism, and channel-aware decoder to extract and integrate information across modalities.The encoder extracts modality-specific features, while fusion and decoding enhance cross-modal and channel-wise modeling.
- Sigma outperforms state-of-the-art models in accuracy and efficiency across RGB-Thermal and RGB-Depth semantic segmentation benchmarks.The reported evaluations cover MFNet, PST900, NYU Depth V2, and SUN RGB-D.
- Sigma is presented as the first successful application of Mamba-based state space models to multi-modal semantic segmentation.
2. Related Work
Prior multi-modal segmentation methods primarily use CNNs, attention mechanisms, or Transformers, while recent state space models remain underexplored for multi-modal fusion. Sigma addresses this gap with a Mamba-based mechanism designed to select and integrate cross-modal information efficiently.
- Existing RGB-Thermal and RGB-Depth methods use encoder-decoder designs, attention or cross-attention fusion, and self-supervised pre-training.
- Transformer-based RGB-X methods face quadratic self-attention scaling that limits long input sequences, whereas Sigma is designed to process long sequences with less computation.
- Sigma introduces a Mamba-based fusion mechanism for cross-modal information selection, targeting enhanced accuracy with low complexity.
- The related work identifies limited exploration of state space models in multi-modal tasks and motivates task-specific SSM fusion design.
3. Sigma: Siamese Mamba Network
Sigma combines state-space modeling with a Siamese encoder, cross-modal fusion, and a channel-aware decoder for multi-modal semantic segmentation. Its design uses selective scanning to model long-range dependencies and integrate RGB with X-modality features.
- State Space Models: Mamba uses input-dependent matrices B, C, and ∆, enabling context-aware modeling of complex interactions in long sequences.This selective mechanism addresses the invariant-parameter limitation of conventional state-space models.
- Architecture: Sigma comprises a Siamese Mamba encoder, a fusion module, and a channel-aware Mamba decoder for multi-modal semantic segmentation.The encoder extracts modality-specific features, fusion modules aggregate them, and the decoder produces the final segmentation prediction.
- Siamese Mamba Encoder: The encoder uses four cascaded Visual State Space Blocks with downsampling to extract multi-level features from two weight-sharing modality branches.Each branch processes RGB or X-modality input, while shared weights reduce computational complexity.
- Siamese Mamba Encoder: Each VSSB applies linear projection, depth-wise convolution, and four-directional selective scanning with residual connections to model long-range spatial information.The four scanned sequences are reversed to a common direction and summed.
- Fusion Module: At each encoder level, CroMB exchanges information between modalities and ConMB selectively scans their concatenated features to form the fused representation.CroMB uses cross-modal interaction, while ConMB integrates the enhanced outputs and preserves the original feature dimensions.
4. Experiments
Sigma is evaluated across RGB-Thermal and RGB-Depth benchmarks using mIoU, efficiency comparisons, qualitative analyses, and component ablations. Results show improved accuracy-efficiency trade-offs, stronger multimodal segmentation, and benefits from the proposed fusion and decoder designs.
- RGB-T Semantic Segmentation: More than 2% improvement over other methods on PST900 demonstrates Sigma’s strong RGB-Thermal segmentation performance.
- RGB-T Semantic Segmentation: Qualitative results show more comprehensive and accurate MFNet segmentations, including improved recognition of tactile paving and bollards.
- RGB-D Semantic Segmentation: Sigma’s small RGB-D model surpasses CMNeXt with 69.8M parameters, using 49.8M fewer parameters.
- RGB-D Semantic Segmentation: Depth fusion produces more coherent NYU Depth V2 segmentation, correctly identifying a round chair as one entity despite shadows.
- Qualitative Analysis of Dual-Modality Learning: Combining RGB and thermal information improves segmentation completeness and boundary precision because RGB supports color distinction while thermal supports texture differentiation.
- Effectiveness of Key Components: Removing CroMB, ConMB, or both lowers performance by 0.6%, 0.8%, and 2.1%, while the proposed decoder exceeds MLP, Swin-Transformer, and UperNet decoders by 1.1%, 0.9%, and 1.5%.
5. Conclusion
Sigma applies Siamese Mamba encoders, multimodal selective-scan fusion, and a channel-aware decoder to semantic segmentation. Experiments on RGB-T and RGB-D benchmarks report superior accuracy and efficiency.
- Sigma combines Siamese Mamba feature extraction, cross-selective and concat-selective fusion, and a channel-aware Mamba decoder for multimodal predictions.
- Experiments on RGB-T and RGB-D semantic segmentation benchmarks show superiority in both accuracy and efficiency.
A. Experimental Details
The experiments use four RGB-T and RGB-D datasets with standardized training settings and three pretrained VMamba backbone sizes. Additional evaluation compares daytime and nighttime MFNet performance.
- RGB-D Settings: NYU Depth V2 and SUN RGB-D experiments use raw depth images at 640 × 480 resolution rather than HHA depth inputs.
- Daytime and Nighttime Evaluation: Sigma achieves better MFNet results on both daytime and nighttime test images.
C. Ablation Studies
Ablations identify effective state-space and decoder configurations on MFNet. The best result uses state size 4 and decoder layers [4].
- State size 4 with decoder layers [4] produces the optimal MFNet result.
D. Complexity Comparison of CroMB and Self-Attention
Sigma’s ConM fusion mechanism provides linear-complexity scaling and substantially lower computation than self-attention-based fusion as feature dimensions or sequence length increase.
- Mamba-based fusion reduces complexity by an order of magnitude compared with Transformer-based fusion methods.The comparison attributes this advantage to Mamba’s linear scalability versus Transformer methods’ quadratic complexity.
- ConM uses far less computation than constructing the State Space Model with self-attention as input sequence length grows.Figure D2 qualitatively illustrates the FLOPs growth comparison between Concat Self-Attention and Concat Mamba.
- ConMB maintains low FLOPs across all encoding stages, while self-attention FLOPs increase substantially with feature height and width.Table D4 compares the computation of ConMB and Concat Self-Attention across four encoding stages.
E. Limitations and Future Work
Sigma’s current scope leaves Mamba’s long-sequence capacity underused for two-modal fusion, while its four-direction encoder scanning increases memory use and complicates lightweight deployment.
- Sigma primarily studies two modalities, leaving Mamba’s long-sequence capability underutilized for fusion involving more than two modalities.Future work targets datasets with more varied modalities, including RGB, depth, thermal, and LiDAR.
- The four-direction Mamba encoder quadruples memory usage, creating a deployment challenge for lightweight edge devices.Future work proposes alternative ways to incorporate positional information.
F. More Qualitative Results
The paper provides additional qualitative comparisons of Sigma against other methods on RGB-Thermal MFNet and RGB-Depth NYU Depth V2 data.
- Additional qualitative results compare Sigma with other methods on the MFNet RGB-Thermal dataset.
- Additional qualitative results compare Sigma with other methods on the NYU Depth V2 RGB-Depth dataset.HHA images are used to visualize the depth modality.