PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model

Yuheng Shi, Minjing Dong, Chang Xu

arXiv:2405.14174v1cs.CV

TL;DR

Vision SSMs face redundant multi-scan computation and long-range forgetting in parameter-limited models, while ViTs have quadratic input scaling. MSVMamba combines full- and downsampled-resolution scanning with ConvFFN; it reports gains over VMamba-Tiny and competitive results across vision tasks, with scalability still to be explored.

  • Problem

    Multi-scan vision SSMs incur redundant computation, and parameter limits constrain long-range modeling; ViTs have quadratic scaling with input size.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Method

    MSVMamba uses MS2D scanning across original- and downsampled-resolution feature maps and adds ConvFFN for channel-wise information exchange.

    Source: §1 · p. 2

  • Results

    MSVMamba-Tiny improves Top-1 accuracy over VMamba-Tiny by 0.6% while reducing GFLOPs by 17%, and MSVMamba outperforms popular architectures across classification and downstream tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The multi-scale design targets long-range forgetting and computational redundancy in parameter-limited vision models.

    Source: §1 · p. 2

    Scalability remains to be explored; as model size increases, the multi-scale design may provide only marginal improvement.

    Source: §5 · p. 9

Abstract

from arXiv · show

Despite the significant achievements of Vision Transformers (ViTs) in various vision tasks, they are constrained by the quadratic complexity. Recently, State Space Models (SSMs) have garnered widespread attention due to their global receptive field and linear complexity with respect to the input length, demonstrating substantial potential across fields including natural language processing and computer vision. To improve the performance of SSMs in vision tasks, a multi-scan strategy is widely adopted, which leads to significant redundancy of SSMs. For a better trade-off between efficiency and performance, we analyze the underlying reasons behind the success of the multi-scan strategy, where long-range dependency plays an important role. Based on the analysis, we introduce Multi-Scale Vision Mamba (MSVMamba) to preserve the superiority of SSMs in vision tasks with limited parameters. It employs a multi-scale 2D scanning technique on both original and downsampled feature maps, which not only benefits long-range dependency learning but also reduces computational costs. Additionally, we integrate a Convolutional Feed-Forward Network (ConvFFN) to address the lack of channel mixing. Our experiments demonstrate that MSVMamba is highly competitive, with the MSVMamba-Tiny model achieving 82.8% top-1 accuracy on ImageNet, 46.9% box mAP, and 42.2% instance mAP with the Mask R-CNN framework, 1x training schedule on COCO, and 47.6% mIoU with single-scale testing on ADE20K.Code is available at \url{https://github.com/YuHengsss/MSVMamba}.

1 Introduction

Vision SSMs offer global receptive fields with linear scaling, but multi-scan routes add redundancy and parameter-limited models struggle with long-range dependencies. MSVMamba combines original- and downsampled-resolution scans with ConvFFN and reports improved accuracy and computational efficiency across vision tasks.

1 Introduction

  • CNNs capture local patterns with linear scaling but can miss global context, whereas ViTs capture long-range dependencies at quadratic input-size complexity.The paper presents SSMs as combining a global receptive field with linear scaling.Source: §1 · p. 1
  • Multi-scan routes let long-range decay occur in different image directions, but increase computation and redundancy in parameter-constrained vision models.The paper calls the resulting difficulty integrating distant information the long-range forgetting problem.Source: §1 · p. 2
  • On ImageNet, MSVMamba-Tiny improves Top-1 accuracy over VMamba-Tiny by 0.6% while reducing GFLOPs by 17%.The comparison is shown in Figure 1.Source: Figure 1 · p. 2, §1 · p. 2
  • MSVMamba combines original-resolution and downsampled scanning directions to learn long-range dependencies with less redundancy, while ConvFFN supports channel exchange and local feature capture.The downsampled branches are upsampled after S6 processing.Source: §1 · p. 2

2 Related work

The related work traces visual backbones from CNNs to ViTs and surveys the development of SSMs, including their adoption in vision models.

2.1 Generic Vision Backbone

  • CNNs capture visual patterns with convolution, while ViTs process image patches with self-attention to model global dependencies, at quadratic cost with input sequence length.The review also describes subsequent work on more efficient training, hierarchical structures, spatial attention, and convolution-based designs.Source: §2.1 · pp. 2–3

2.2 State Space Models

  • SSM research improves efficiency through structured models and parallel scanning; vision approaches adapt S6 with multiple scan directions and extend it across backbone, medical-imaging, and video tasks.The review identifies S4ND, U-Mamba, ViM, and VMamba among these developments.Source: §2.2 · p. 3

3 Method

The method analyzes why multi-scan vision SSMs help, then introduces hierarchical multi-scale scanning and an MS3 block to address long-range forgetting and computational redundancy.

3.1 Preliminaries

  • Classical SSMs map an input sequence through a latent representation to an output; discretization adapts this formulation for deep learning, while S6 makes B, C, and ∆ input-dependent.The method also describes implementing the discretized process as a global convolution.Source: §3.1 · pp. 3–4

3.2 Analysis of Multi-Scan Strategy

  • Multi-scan routes address images’ non-causal structure and reduce effective token distances, but route-specific decay dominance makes some scans computationally redundant.The analysis attributes long-range forgetting to decaying contributions from distant tokens; increasing model depth or parameters can mitigate it at additional computational cost.Source: §3.2 · pp. 4–5

3.3 Multi-Scale 2D Scanning

  • MS2D combines full-resolution and downsampled feature maps across scanning routes, shortening token distances while retaining full-resolution scans to avoid losing fine-grained features.Downsampling by a factor of s in each dimension reduces the total sequence length across four scans from 4L to (1+3/s^2)L; the downsampling rate is set to 2, and downsampled routes share an S6 block.Source: §3.3 · pp. 5–6

3.4 Overall Model Architecture

  • MS3 replaces VMamba’s SS2D with MS2D within an MSVSS block, then pairs it with a ConvFFN channel mixer to support information flow across channels.The ConvFFN uses a depth-wise convolution and two fully connected layers, with channel expansion ratio 2; Nano, Micro, and Tiny variants use 6.9M, 11.9M, and 33.0M parameters, respectively.Source: §3.4 · pp. 6–7

4 Experimental Validation

MSVMamba is evaluated on ImageNet classification, COCO detection and instance segmentation, and ADE20K semantic segmentation, with ablations testing its scanning and channel-mixing components. The results report improvements over competing models and examine accuracy–compute trade-offs.

4.1 ImageNet Classification

  • 82.8% top-1 accuracy: MSVMamba-T reaches this on ImageNet-1K with 33M parameters and 4.6G FLOPs, exceeding VMamba-T by 0.6%.MSVMamba-N reaches 77.3% with 7M parameters and 0.9G FLOPs; MSVMamba-M reaches 79.8% with 12M parameters and 1.5G FLOPs, surpassing ViM-T by 3.7%.Source: §4.1 · p. 7

4.2 Object Detection

  • +4.2 box AP and +2.9 mask AP: MSVMamba-T outperforms Swin-T under the 1× COCO schedule.Improvements in both box AP and mask AP are also reported under the 3× schedule.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 Semantic Segmentation

  • +2.2, +1.6, and +0.3 mIoU: the MSVMamba tiny variant exceeds Swin, ConNeXt, and VMamba, respectively, on ADE20K.The results include single-scale and multi-scale testing.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 Ablation Study

  • 71.9% versus 69.6% accuracy: replacing VMamba’s SS2D with MS2D raises accuracy at roughly equivalent FLOPs.Putting all scans on the half-resolution branch substantially reduces accuracy; adding two or three full-resolution scans improves accuracy by 0.1% or 0.6% at approximately 12% or 25% additional compute, while four scans add 34% compute for a 0.4% gain. One full-resolution scan is selected as the default trade-off, and an SE block adds 0.5% accuracy with minimal additional compute.Source: §4.4 · pp. 8–9, Table 5 · p. 9
  • 2.0% additional accuracy: adding ConvFFN improves the model’s channel interaction; setting N = 1 and adding a block yields a further 0.7% improvement at roughly constant computational cost.The number of blocks is adjusted to keep compute roughly equivalent when adding ConvFFN.Source: §4.4 · p. 9, Table 4 · p. 9

5 Limitations

MSVMamba’s scalability remains unexplored; as model sizes increase, its multi-scale design may provide only marginal improvement.

  • MSVMamba’s scalability remains unexplored, and its multi-scale design may offer only marginal improvement as model sizes increase.Source: §5 · p. 9

6 Conclusion

The conclusion presents MSVMamba as an SSM-based vision backbone using MS2D to reduce computational redundancy and alleviate long-range forgetting in parameter-limited models. It also incorporates ConvFFN to improve channel information exchange and reports consistent outperformance across classification and downstream tasks.

  • MSVMamba combines MS2D scanning to reduce computational redundancy and alleviate long-range forgetting with ConvFFN to enhance channel information exchange.Source: §6 · p. 9

Throughput Comparison

On ImageNet-1K, MSVMamba-Tiny nearly doubles VMamba-Tiny throughput under similar GFLOPs.

  • Nearly doubles throughput: MSVMamba-Tiny processes images at almost twice the rate of baseline VMamba-Tiny under similar GFLOPs.Throughput is measured in images per second on an NVIDIA GeForce RTX 2080 Ti, with 224×224 images and batch size 128.Source: Table 6 · p. 14, Throughput Comparison · p. 14

Qualitative Analysis

Visualizations compare MS2D and SS2D attention maps, showing that MS2D’s full-resolution scan captures finer details while half-resolution scans focus on broader architectural features. The hierarchical pattern supports combining features across abstraction levels.

Qualitative Analysis

  • MS2D’s full-resolution scan captures more detailed features than SS2D, while its half-resolution scans focus on broader architectural details.The maps are generated by converting the S6 layer into an attention format, following VMamba [31].Source: Qualitative Analysis · p. 14
  • The hierarchical scanning pattern facilitates the current layer’s ability to discern and amalgamate features across abstraction levels.Source: Qualitative Analysis · p. 14
  • Full-resolution scanning captures fine-grained features, whereas half-resolution scans capture coarse-grained features.Figure 6 shows maps from four distinct scanning directions in the last layer of the second stage.Source: Figure 6 · p. 14

Network Architecture

The MSVMamba series includes Nano, Micro, and Tiny variants with different channel counts and block numbers. Table 7 presents their architectural overview.

Network Architecture

Loading 2405.14174v1…