Source-linked AI summary
A Survey on Visual Mamba
Hanwei Zhang, Ying Zhu, Dan Wang, Lijun Zhang, Tianxiang Chen, Zi Ye
TL;DR
Transformers face computational demands that grow quadratically with image resolution, motivating the adaptation of Mamba to computer vision. This survey reviews Mamba’s foundations, architectural extensions, and applications across vision tasks, while identifying current challenges and future directions. It provides a first comprehensive computer-vision-focused review and organizes task-specific advances through a taxonomy.
Problem
Transformer self-attention has computational demands that increase quadratically with image resolution, motivating exploration of Mamba for computer vision.
Method
The paper synthesizes Mamba research by covering its foundations, combinations with convolution, recurrence, and attention, and applications across general, medical, and remote-sensing vision.
Results
The survey provides the first comprehensive review of Mamba techniques explicitly focused on computer vision and organizes task-specific advances into a taxonomy.
Takeaways & Limitations
The survey aims to guide researchers, encourage vision-community interest in Mamba, and support efforts to address its current limitations.
Takeaways & Limitations
Future work must address large-dataset pre-training, interpretability, adversarial robustness, real-time medical deployment, and distributed deployment challenges.
Abstract
from arXiv · showhide
State space models (SSMs) with selection mechanisms and hardware-aware architectures, namely Mamba, have recently demonstrated significant promise in long-sequence modeling. Since the self-attention mechanism in transformers has quadratic complexity with image size and increasing computational demands, the researchers are now exploring how to adapt Mamba for computer vision tasks. This paper is the first comprehensive survey aiming to provide an in-depth analysis of Mamba models in the field of computer vision. It begins by exploring the foundational concepts contributing to Mamba's success, including the state space model framework, selection mechanisms, and hardware-aware design. Next, we review these vision mamba models by categorizing them into foundational ones and enhancing them with techniques such as convolution, recurrence, and attention to improve their sophistication. We further delve into the widespread applications of Mamba in vision tasks, which include their use as a backbone in various levels of vision processing. This encompasses general visual tasks, Medical visual tasks (e.g., 2D / 3D segmentation, classification, and image registration, etc.), and Remote Sensing visual tasks. We specially introduce general visual tasks from two levels: High/Mid-level vision (e.g., Object detection, Segmentation, Video classification, etc.) and Low-level vision (e.g., Image super-resolution, Image restoration, Visual generation, etc.). We hope this endeavor will spark additional interest within the community to address current challenges and further apply Mamba models in computer vision.
1 Introduction
Transformers provide strong long-distance representations but face quadratic self-attention costs as image resolution increases. This survey reviews Mamba for computer vision, organizing its foundations, architectural enhancements, applications, and task-specific advances.
- Motivation: Transformer self-attention increases computational demands quadratically with image resolution, motivating architectures that reduce costs while retaining long-range modeling.The survey identifies SSMs as a central focus among these developments.
- Mamba foundations: Mamba improves SSMs through input-dependent parameter selection and a hardware-aware algorithm with linear sequence-length processing.These mechanisms dynamically filter information and improve computational speed on modern systems.
- Survey scope: The survey presents a comprehensive review of Mamba models and their use in the vision domain.It is described as the first survey explicitly focused on Mamba techniques in computer vision.
- Architectural developments: It examines how naive Mamba visual frameworks can be combined with other architectures, including convolution, recurrence, and attention, to enhance capabilities.The survey’s contribution includes investigating these combinations for improved performance.
- Organization and analysis: The literature is organized by application tasks to establish a taxonomy, identify task-specific advances, and provide insights for overcoming challenges.The survey structure also includes foundational concepts, competing methodologies, and applications across computer vision.
2 Formulation of Mamba
Mamba extends state space models with input-dependent selection and hardware-aware selective scanning, retaining linear processing while improving context filtering and computational efficiency.
- 2.1 State Space Models (SSMs): SSMs map one-dimensional input sequences to outputs through hidden states governed by evolution and projection parameters.The formulation uses sequence x, hidden state h, and parameters A, B, and C.
- 2.1 State Space Models (SSMs): SSMs are discretized under a Zero-Order Hold assumption, transforming continuous-time parameters with a timescale parameter ∆.
- 2.1 State Space Models (SSMs): The iterative state update can be synthesized as a global convolution with kernel K, enabling efficient sequence processing.The kernel K has length L, matching the input sequence length.
- 2.1 State Space Models (SSMs): Foundational SSM architectures include linear attention, H3, Hyena, and RetNet, which combine recurrence, gating, convolution, or simpler SSM layers in different ways.
- 2.1 State Space Models (SSMs): Traditional SSMs have linear time complexity but limited sequence-context representation because their parameterization is time-invariant.
- 2.1 State Space Models (SSMs): Selective SSMs address this limitation through input-dependent parameters that dynamically adjust state interactions and information filtering.The parameter tensors B, C, and ∆ depend on the input sequence.
- 2.2 Other Key Concepts in Mamba: The selection mechanism is connected to recurrent gating, with Softplus used for the timescale transformation in the described formulation.The paper identifies classical RNN gating as an instance of SSM selection under a simplified parameter setting.
- 2.2 Other Key Concepts in Mamba: Hardware-aware selective scan uses kernel fusion, recomputation, and parallel scan to reduce memory I/O while materializing states efficiently on GPUs.
3 Mamba for Vision
Vision Mamba adapts one-dimensional Mamba blocks to multidimensional visual data through new blocks and scanning mechanisms, then extends them with hierarchical designs, local or global processing, and other architectural components.
- 3 Mamba for Vision: Adapting Mamba to images, videos, and 3D representations requires changes to its scanning mechanism and block architecture.
- 3 Mamba for Vision: Vision Mamba and VMamba provide foundational visual designs through the ViM and VSS blocks, respectively.
- 3.1 Visual Mamba Block.: ViM flattens image patches, adds position embeddings, and uses bidirectional state space processing with one-dimensional convolutions and SiLU gating.
- 3.1 Visual Mamba Block.: VSS applies depth-wise convolution, SiLU activation, state space processing, normalization, and cross-scan traversal to address direction sensitivity.
- 3.2 Pure Mamba: LocalMamba uses windowed scans and searches directions across layers to capture local detail alongside global context.It provides plain and hierarchical variants.
- 3.2 Pure Mamba: PlainMamba improves two-dimensional spatial continuity with continuous scanning and direction-aware updating, while removing special tokens.
- 3.2 Pure Mamba: EfficientVMamba uses atrous Efficient 2D Scanning to skip patches while preserving patch count, combining a local convolution branch with a global SSM branch.
4 Visual Mamba in Application Fields
Mamba-based modules are applied across general, medical, and remote-sensing vision tasks, with linear-complexity sequence modeling supporting high-resolution and long-range processing. The survey covers classification, detection, segmentation, video understanding, restoration, generation, super-resolution, deraining, deblurring, and point-cloud analysis.
- High/Mid-level vision: Mamba backbones show decent performance on classification, object detection, and segmentation.
- High/Mid-level vision: ReMamber fuses visual and textual features for referring image segmentation through Mamba Twister blocks and a flexible decoder.Experiments on multiple referring-image-segmentation datasets achieve state-of-the-art results.
- High/Mid-level vision: The Video Mamba Suite evaluates Mamba across 12 video-understanding tasks using 14 models or modules.It assesses roles including the Decomposed Bidirectionally Mamba block and compares Mamba with Transformer-based models.
- Low-level vision: Mamba-based methods address low-level tasks including super-resolution, restoration, defogging, deraining, and deblurring.Examples include MMA, MambaIR, UVM-Net, FreqMamba, and ALGNet.
- Low-level vision: FreqMamba combines spatial Mamba, frequency-band Mamba, and Fourier global modeling for image deraining.The method processes spatial details, decomposes spectral bands with wavelet packet transforms, and captures global degradation patterns with Fourier transforms.
- High/Mid-level vision: SSPoint Mamba applies a state-space model to point-cloud analysis, addressing irregular and sparse data while avoiding Transformer complexity growth with input length.
2D Medical Image
Mamba-based 2D medical-image research is dominated by segmentation architectures, especially U-Net extensions, while also covering classification and other segmentation designs. These models combine state-space modeling with hierarchical, convolutional, semi-supervised, or task-specific mechanisms.
- 2D Medical Image: Mamba has shown potential for modeling global structure information in 2D medical segmentation.
- 2D Medical Image: Most 2D medical-segmentation architectures build on U-Net, including U-Mamba, H-vmunet, UltraLight VM-UNet, and Mamba-UNet.U-Mamba combines CNN and SSM components to model long-range dependencies, while UltraLight VM-UNet uses 0.049M parameters and 0.060 GFLOPs.
- 2D Medical Image: VM-UNet provides a foundational purely SSM-based medical-segmentation model with an encoder, decoder, and skip connections.VM-UNET-V2 enhances low-level and high-level feature fusion with a Visual State Space block.
- 2D Medical Image: Later designs refine medical segmentation through multi-scale hierarchy, bottleneck changes, semi-supervised learning, weak supervision, and dual-branch processing.Examples include Mamba-HUNet, TM-UNet, Semi-Mamba-UNet, Weak-Mamba-UNet, and P-Mamba.
- 2D Medical Image: Vision Mamba, also called MedMamba, extends Mamba to medical-image classification with a Conv-SSM module.Conv-SSM combines convolutional local-feature extraction with long-range dependency modeling.
3D Medical Image
3D medical-image research applies Mamba to segmentation, classification, registration, multimodal reconstruction, and endoscopy-related motion prediction. The survey also places these medical methods alongside remote-sensing applications and identifies deployment and robustness challenges.
- 3D Medical Image: nnMamba integrates local and global relationship modeling for 3D medical imaging through the MICCSS module.It is evaluated on six datasets across segmentation, classification, and registration tasks.
- 3D Medical Image: SegMamba is presented as the first Mamba method specifically designed for accurate 3D medical-image segmentation.Its tri-orientated Mamba module models 3D features from three directions.
- 3D Medical Image: LightM-UNet, LMa-UNet, and T-Mamba extend 3D segmentation design through lightweight residual layers, large windows, and adaptive feature selection.
- 3D Medical Image: MambaMorph and VMambaMorph apply Mamba to multimodality deformable registration and efficient processing of 3D volumetric features.VMambaMorph incorporates a redesigned 2D cross-scan module for volumetric processing.
- 3D Medical Image: CMViM uses Vision Mamba in a masked autoencoder for 3D multimodal reconstruction in Alzheimer’s disease classification, while MMH combines visual tracking with historical motion data.
- Challenge: Future medical-imaging work must address large-dataset pre-training, interpretability, adversarial robustness, real-time efficiency, and distributed deployment.
5 Conclusion
Mamba is gaining prominence in computer vision for managing long-range dependencies and offering computational efficiency relative to Transformers. This survey reviews SSMs, Mamba architectures, and their applications across general, medical, and remote-sensing vision.
- Mamba is gaining prominence in computer vision because of its long-range dependency modeling and computational efficiency relative to Transformers.
- The survey analyzes foundational SSM and Mamba concepts alongside competing methods across a broad spectrum of computer vision applications.
- Covered application areas include backbone architectures, high/mid-level vision, low-level vision, medical imaging, and remote sensing.
- The survey aims to encourage further investigation of Mamba’s possibilities and solutions to its current limitations in computer vision.