PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Mamba: A Comprehensive Survey and Taxonomy

Xiao Liu, Chenxu Zhang, Lei Zhang

arXiv:2405.04404v1cs.CVcs.AIcs.CLcs.LG

TL;DR

The paper addresses the need to understand how Mamba extends state-space modeling from language to visual tasks. It provides a comprehensive survey and taxonomy, reporting that selective mechanisms and hardware-aware state expansion support linear complexity, long-range modeling, and memory efficiency across visual applications.

  • Problem

    The survey addresses the limited organization of rapidly expanding Mamba research across general vision, multimodal, remote sensing, and medical image analysis.

  • Method

    The paper develops a comprehensive survey and taxonomy of Mamba variants, organizing methods by visual task and application domain while explaining their principles and technical details.

  • Results

    Selective mechanisms and hardware-aware state expansion give Mamba linear computational complexity, long-range dependency modeling, and memory efficiency.

  • Takeaways & Limitations

    Mamba has broad potential as an alternative or component of architectures for general vision and vertical-domain visual tasks.

  • Takeaways & Limitations

    Mamba can insufficiently model token interactions because it lacks an attention mechanism, and combining it with Transformer components remains challenging.

Abstract

from arXiv · show

State Space Model (SSM) is a mathematical model used to describe and analyze the behavior of dynamic systems. This model has witnessed numerous applications in several fields, including control theory, signal processing, economics and machine learning. In the field of deep learning, state space models are used to process sequence data, such as time series analysis, natural language processing (NLP) and video understanding. By mapping sequence data to state space, long-term dependencies in the data can be better captured. In particular, modern SSMs have shown strong representational capabilities in NLP, especially in long sequence modeling, while maintaining linear time complexity. Notably, based on the latest state-space models, Mamba merges time-varying parameters into SSMs and formulates a hardware-aware algorithm for efficient training and inference. Given its impressive efficiency and strong long-range dependency modeling capability, Mamba is expected to become a new AI architecture that may outperform Transformer. Recently, a number of works have attempted to study the potential of Mamba in various fields, such as general vision, multi-modal, medical image analysis and remote sensing image analysis, by extending Mamba from natural language domain to visual domain. To fully understand Mamba in the visual domain, we conduct a comprehensive survey and present a taxonomy study. This survey focuses on Mamba's application to a variety of visual tasks and data types, and discusses its predecessors, recent advances and far-reaching impact on a wide range of domains. Since Mamba is now on an upward trend, please actively notice us if you have new findings, and new progress on Mamba will be included in this survey in a timely manner and updated on the Mamba project at https://github.com/lx6c78/Vision-Mamba-A-Comprehensive-Survey-and-Taxonomy.

1 INTRODUCTION

The survey positions Mamba as an efficient sequence-modeling architecture extended from language to visual tasks, and organizes its applications through a targeted taxonomy. It covers general vision, multimodal learning, remote sensing, and medical image analysis.

  • CNNs struggle with long-range dependencies, whereas Transformer-based models incur substantial computational burdens from self-attention.
  • Mamba combines selective scanning with linear sequence-length scaling and is reported to be more than five times faster than Transformer.
  • Vision extensions address image-sequence challenges through pure SSM backbones, cross-scan strategies, and multidimensional processing.
  • The survey contributes a targeted taxonomy, explanations of technical principles, and examples with experimental results to improve researchers’ understanding and navigation.
  • The taxonomy groups vision-oriented Mamba methods into general vision, multimodal, remote sensing, and medical image analysis, with general vision further divided by task and data type.
  • Mamba is applied across general vision, multimodal learning, remote sensing, and medical image analysis, where high resolution and variable spatial-temporal resolution create efficiency challenges.

2 FORMULATION OF MAMBA

The formulation progresses from continuous and discretized SSMs to Mamba’s selective, input-dependent state transitions. Mamba combines this selectivity with hardware-aware computation to retain long-range modeling, linear complexity, and memory efficiency.

  • 2 FORMULATION OF MAMBA: SSMs map sequences through an intermediate hidden state, while S4 uses structured parameterization to reduce computation and memory requirements.
  • 2.1 State Space Models: Continuous SSMs transform input x(t) into output y(t) through hidden state h(t) and parameters (A, B, C, D).
  • 2.1 State Space Models: Parameter A governs historical-state evolution, B controls input influence on the state, C projects the state to output, and D provides a direct input-to-output path.
  • 2.1 State Space Models: Discretization converts continuous SSMs into sequence-to-sequence mappings x_k → y_k for discrete inputs such as images and text.
  • 2.1 State Space Models: SSMs use convolution for parallel training and recurrent computation for efficient autoregressive inference, although recurrent processing can forget information over time.
  • 2.1 State Space Models: HiPPO compresses observed input history into polynomial coefficients, while S4 uses Normal Plus Low-Rank decomposition to simplify computation through stable diagonalization and a Cauchy kernel.
  • 2.2 Selective State Space Models (S6): Mamba makes SSM parameters B, C, and ∆ input-dependent, turning a time-invariant model into a selective time-varying model with token-specific control.
  • 2.2 Selective State Space Models (S6): Because selectivity prevents fixed-kernel convolution, Mamba uses hardware-aware segmented selective scanning to achieve linear complexity, long-range modeling, and memory efficiency.

3 MAMBA IN GENERAL VISION TASKS

This section surveys Mamba variants for general vision, spanning high- and mid-level vision, low-level vision, and 3D vision. It also emphasizes selective scanning as a central design issue for visual tasks.

  • General-vision Mamba research covers high-level and mid-level vision, low-level vision, and 3D vision.
  • The surveyed variants are redesigned for the requirements of their respective vision tasks.
  • Because selective scanning is important for vision, the section summarizes classical 2D scanning mechanisms in Fig. 3.

3.1 Mamba for High-level/Mid-level Vision

Vision Mamba adapts state-space models to visual backbones by addressing image-specific spatial and directional challenges. The survey covers scanning strategies, architectural designs, and reported performance across vision tasks.

  • Vision backbone designs: Vim adapts Mamba to vision with bidirectional state-space processing and positional embeddings to address unidirectionality and limited location awareness.It patchifies images, adds positional information and a class token, then processes tokens in forward and backward directions.
  • Vision backbone designs: VMamba addresses direction-sensitive image processing with a four-way cross-scan strategy that integrates spatial information from multiple directions while retaining linear complexity.Its cross-scan module traverses rows and columns from four directional corners before combining the resulting sequences.
  • Reported performance: VMamba outperforms ResNet, DeiT, Swin, ConvNeXt, and Vim across reported classification, detection, and semantic-segmentation comparisons.The passage states that all VMamba model-size variants outperform the competing pairs in the cited evaluations.
  • Scanning strategies: Vision Mamba variants modify selective scanning through efficient sparse sampling, local windows, continuous two-dimensional traversal, direction-aware updates, or multidirectional scans.EfficientVMamba reduces scanning complexity with ES2D, LocalMamba uses windowed scans, PlainMamba preserves spatial adjacency, and Mamba-ND scans multidimensional axes.
  • Broader applications: Mamba-based vision research also explores stability, transfer learning, domain generalization, anomaly detection, industrial defect recognition, and video understanding.Examples include SiMBA for scaling stability, DGMamba for unseen domains, MambaAD for anomaly detection, MemoryMamba for industrial defects, and video-oriented applications.

3.2 Mamba for Low-level Vision

Mamba-based methods are being explored for low-level vision tasks in which images are both inputs and outputs. These approaches combine Mamba with convolutional, attention, or U-shaped designs for image restoration, denoising, enhancement, and super-resolution.

  • Image Restoration: U-shaped Mamba architectures target image restoration by combining convolutional processing with long-range sequence modeling.One design uses two convolutional blocks followed by feature flattening and transposition, then applies two Mamba branches across sequence and channel dimensions.
  • Image Restoration: MambaIR adds residual state-space blocks and channel attention to VMamba for super-resolution and real-image denoising.The residual blocks add convolutional structure for local spatial information, while channel attention enhances interactions between channels.
  • Image Restoration: Competitive or superior performance with relatively low memory and computational overheads is reported for Mamba-based single-image super-resolution.The method combines Vim with a MetaFormer-style block and complementary attention on several benchmark datasets.
  • Image Restoration: U-shaped variants also use spatial and channel state-space blocks for global context, channel features, and low-light enhancement.CU-Mamba uses sequential spatial and channel SSM blocks, while Retinexmamba uses IFSSM with SS2D and cross-attention for illumination fusion.

3.3 Mamba for 3D Vision

Mamba is applied to 3D and spectral visual data, especially where irregular structure or large spatial distances complicate sequence modeling. Point-cloud methods address ordering and causality, while hyperspectral and multimodal approaches emphasize efficient feature processing.

  • Point Cloud Analysis: Point-cloud analysis requires modifying Mamba because point clouds are unordered, irregular, and subject to Mamba’s causality requirements.Point clouds contain 3D coordinates and may include attributes such as color and intensity.
  • Point Cloud Analysis: PointMamba and related methods impose geometric or octree-based orderings to convert point clouds into sequences for Mamba.PointMamba connects tokens along x, y, and z axes, whereas Point Mamba uses octree sorting to form a z-order sequence.
  • Point Cloud Analysis: 93.4% accuracy and 75.7 mIOU are reported for Point Mamba on ModelNet40 classification and ScanNet semantic segmentation, respectively.The comparison is stated against Transformer-based methods.
  • Hyperspectral Imaging: Hyperspectral imaging requires joint modeling of spatial and spectral information, motivating efficient Mamba frameworks for large-scale hyperspectral data.Spectral signatures support applications including material identification, classification, and quantitative analysis.
  • Multimodal Vision: Mamba-FETrack uses modality-specific Vim backbones and FusionMamba to combine RGB-frame and event-stream features.It reports comparable performance on FELT and FE108 while offering advantages over ViT-based methods in FLOPs and parameters.

3.4 Mamba for Visual Generation

Mamba is integrated into diffusion, reconstruction, and video-generation systems to process long visual or temporal sequences efficiently. The surveyed designs combine Mamba with complementary attention, Gaussian splatting, or hierarchical spatiotemporal blocks.

  • Visual Generation: ZigMa incorporates Mamba into diffusion models and distributes eight scanning modes across layers to manage scanning complexity.The assignment follows the rule Ω_i = S_i%8.
  • Motion Generation: Motion Mamba uses hierarchical temporal and bidirectional spatial Mamba blocks within a denoising U-Net for motion generation.Its sub-SSMs have descending complexity across the encoder-decoder architecture, and the method reports up to 50% improvement in FID on HumanML3D and KIT-ML.
  • 3D Reconstruction: Gamba combines Gaussian splatting with Mamba for single-view 3D reconstruction and uses cross-attention for context-dependent reasoning.DINO v2 provides image tokens, camera embeddings, and learnable 3D Gaussian Splatting embeddings.
  • Video Generation: Matten combines spatiotemporal attention within frames with bidirectional Mamba between consecutive frames for video generation.The reported design targets both global and local information while retaining computational efficiency.

3.5 Discussion and Summary

Mamba-based visual methods use varied scanning mechanisms and combine with other architectures across a broad range of tasks. The survey spans low-level to high-level vision, 2D to 3D data, and discriminative to generative applications.

  • Discussion and Summary: Scanning mechanisms help Mamba-based approaches capture visual features across diverse visual tasks and data types.The survey describes applications ranging from low-level to high-level vision and from 2D to 3D inputs.
  • Discussion and Summary: Mamba’s extensibility allows it to combine with other architectures and methodologies for discriminative and generative vision tasks.The surveyed scope includes both low-level and high-level applications.

4 MAMBA IN MULTI-MODAL LEARNING TASKS

Mamba is being extended to multimodal learning, where its sequence modeling supports interactions among visual, textual, audio, and other modalities. These studies also expose challenges in cross-modal alignment, inter-channel learning, and memory consumption.

  • Mamba supports multimodal tasks by modeling interactions among image, text, audio, and other data types.Its scalable sequence modeling is presented as suitable for cross-modal interaction and fusion.
  • Mamba Twister combines visual state-space layers with a twisting layer to fuse textual and visual features across channel and spatial dimensions.The method addresses Mamba’s difficulty capturing textual information and cross-modal interactions.
  • VL-Mamba aligns non-causal visual sequences with causal text sequences through a Multi-Modal Connector using 2D vision selective scanning.The connector output is combined with a tokenized text query to generate a response.
  • MambaTalk uses audio and text sequences to control four specialized Mamba models for gesture synthesis, while SpikeMba combines spiking networks and SSMs for video-text grounding.Spiking mechanisms identify salient objects efficiently, suppress noise, and reduce unwanted effects.
  • Sigma exchanges SSM C-matrices between modalities and adds channel attention for multimodal semantic segmentation.Its four-direction scanning improves multimodal modeling but incurs high memory consumption, while the method uses only part of Mamba’s long-sequence capability.

5 MAMBA IN VERTICAL-DOMAIN TASKS

The survey organizes Mamba applications in vertical-domain vision, especially remote sensing and medical imaging. These applications use Mamba for global context and long-range modeling while targeting efficiency across diverse data and tasks.

  • Vertical-domain applications include remote sensing and medical image processing, analysis, and understanding.The survey highlights Mamba’s linear complexity and higher inference throughput as motivations for these applications.
  • Remote Sensing Image Analysis: Pan-Mamba uses Mamba blocks for feature extraction, channel-swapping interactions, and gated cross-modality learning in pan-sharpening.The architecture separates long-range modeling, lightweight modality interaction, and complementary-feature learning.
  • Remote Sensing Image Analysis: RSMamba extends Mamba to 2D non-causal remote-sensing data with position-sensitive dynamic multi-path activation, while SpectralMamba uses piece-wise scanning for hyperspectral classification.SpectralMamba is designed as a lightweight state-space model that leverages reflectance characteristics of ground objects.
  • Remote Sensing Image Change Detection: ChangeMamba develops separate frameworks for binary change detection, semantic change detection, and damage assessment using global spatial and spatio-temporal modeling.Its encoder uses Visual Mamba, while its decoder explores sequential, cross, and parallel mechanisms.
  • Remote Sensing Image Segmentation: Samba targets high-resolution remote-sensing segmentation, while RS3Mamba and RS-Mamba add auxiliary global modeling or omnidirectional scanning for dense prediction.Samba is reported to outperform state-of-the-art CNN- and Transformer-based methods across public datasets.
  • Remote Sensing Image Fusion: FusionMamba and LE-Mamba apply Mamba to image fusion by separately modeling spatial and spectral information with hierarchical or multi-scale designs.FusionMamba uses two U-shaped networks and a fusion module for progressive spatial-information injection.

5.2 Mamba for Medical Image Modeling

Medical-image research applies Mamba to 2D and multidimensional segmentation, classification, pathology, registration, reconstruction, and other modeling tasks. The surveyed methods commonly combine Mamba’s global modeling with convolutional, U-shaped, attention, or task-specific components.

  • Medical image segmentation studies are grouped into 2D segmentation and multidimensional medical-data segmentation.The survey presents a dedicated taxonomy for these Mamba models.
  • Preliminary explorations of U-shaped Mamba: U-Mamba combines CNN local-feature extraction with SSM global modeling, while VM-UNet and related U-shaped models explore pure or hybrid Mamba encoders and decoders.Reported studies examine pretraining, parameter efficiency, computational cost, and performance across multiple medical datasets.
  • Improvements to the U-shaped Mamba: LightM-UNet reduces resource demands, whereas LMa-UNet and VM-UNetV2 develop larger-window, hierarchical, bidirectional, or contextual Mamba blocks for spatial modeling.LightM-UNet is reported to outperform competing architectures with fewer parameters and lower computational overheads on two datasets.
  • U-shaped Mamba with other methodologies: Semi-Mamba-UNet combines visual Mamba and conventional UNet models in a semi-supervised framework that generates pseudo-labels and cross-supervises the two networks.The design addresses computational burdens and limited expert annotations in medical segmentation.
  • Multi-Dimensional Medical Data Segmentation: SegMamba is a general 3D segmentation framework using multiscale U-shaped modeling, gated spatial convolution, and triorientated Mamba modules.The cited multidimensional methods combine CNN and SSM strengths to model local and global features.
  • Pathological Diagnosis: MedMamba uses Conv-SSM for medical classification, while MamMIL and MambaMIL model whole-slide or instance-level relationships with bidirectional, content-aware, or reordered sequences.MamMIL is reported to use less GPU memory than Transformer-based methods on Camelyon16 and BRACS.
  • Other Medical Image Modeling: Other medical applications include Mamba-based deformable registration, endoscopic correction, image reconstruction, uncertainty estimation, radiation-dose prediction, and object tracking.These methods pair Mamba with convolutional, frequency-domain, GAN, diffusion, or recursive-registration components for task-specific processing.

5.3 Discussion and Summary

The survey concludes that Mamba’s linear complexity and global modeling capabilities suit high-resolution remote-sensing and medical images. It associates these properties with computational efficiency, lower GPU memory consumption, and faster inference in practical deployment.

  • Mamba’s linear complexity and global modeling capabilities make it suitable for high-resolution remote-sensing and medical images.The survey links these properties to computational efficiency, GPU memory consumption, and inference speed.
  • These efficiency advantages are presented as promoting deployment and implementation of deep-learning models in practical applications.

6 CONCLUSION, CHALLENGE AND OUTLOOK

Mamba has become a broadly studied alternative architecture for computer vision and vertical visual domains, but remains at an early stage relative to mature Transformer-based models. Key open directions include improving token interaction, scaling model capacity, and integrating Mamba with other methodologies.

  • Mamba variants have been deployed across general vision, visual generation, vision-language learning, remote sensing, and medical intelligence.
  • Visual Mamba must address the mismatch between Mamba’s causal sequential processing and visual data’s non-causal structure and spatial relationships.
  • Insufficient token interactions caused by the lack of attention can limit comprehensive and detailed information capture, motivating hybrid architectures.
  • Large-scale Mamba models have not yet appeared, leaving model capacity improvement as an important challenge under scaling-law considerations.
  • How Mamba can work with multi-modal processing, diffusion models, domain generalization, and visual-language models remains insufficiently explored.
Loading 2405.04404v1…