PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba

Xiaohuan Pei, Tao Huang, Chang Xu

arXiv:2403.09977v1cs.CVcs.AI

TL;DR

Lightweight vision models face a trade-off between local CNN extraction, globally capable but costly Transformers, and accuracy. EfficientVMamba addresses this with atrous selective scanning, skip sampling, and fused global-local blocks, achieving competitive results across vision tasks while reducing complexity; its EfficientVMamba-S reaches 5.6% higher ImageNet accuracy than Vim-Ti with 1.3G versus 1.5G FLOPs.

  • Problem

    Lightweight vision models must balance accuracy and efficiency because CNNs emphasize local features while Transformers provide global reach with O(N^2) computation.

  • Method

    EfficientVMamba combines atrous-based selective scanning with skip sampling, a convolution branch, channel attention, and stage-specific SSM/CNN block allocation.

  • Results

    EfficientVMamba delivers competitive performance across image classification, object detection, and semantic segmentation while reducing computational complexity.

  • Takeaways & Limitations

    The architecture provides a lightweight visual state-space framework that combines global receptive-field coverage with local feature extraction.

  • Takeaways & Limitations

    Visual SSMs remain computationally sophisticated, complicating efficient parallel processing and leaving efficiency and scalability open for further investigation.

Abstract

from arXiv · show

Prior efforts in light-weight model development mainly centered on CNN and Transformer-based designs yet faced persistent challenges. CNNs adept at local feature extraction compromise resolution while Transformers offer global reach but escalate computational demands $\mathcal{O}(N^2)$. This ongoing trade-off between accuracy and efficiency remains a significant hurdle. Recently, state space models (SSMs), such as Mamba, have shown outstanding performance and competitiveness in various tasks such as language modeling and computer vision, while reducing the time complexity of global information extraction to $\mathcal{O}(N)$. Inspired by this, this work proposes to explore the potential of visual state space models in light-weight model design and introduce a novel efficient model variant dubbed EfficientVMamba. Concretely, our EfficientVMamba integrates a atrous-based selective scan approach by efficient skip sampling, constituting building blocks designed to harness both global and local representational features. Additionally, we investigate the integration between SSM blocks and convolutions, and introduce an efficient visual state space block combined with an additional convolution branch, which further elevate the model performance. Experimental results show that, EfficientVMamba scales down the computational complexity while yields competitive results across a variety of vision tasks. For example, our EfficientVMamba-S with $1.3$G FLOPs improves Vim-Ti with $1.5$G FLOPs by a large margin of $5.6\%$ accuracy on ImageNet. Code is available at: \url{https://github.com/TerryPei/EfficientVMamba}.

1 Introduction

EfficientVMamba targets lightweight vision models by combining efficient global state-space scanning with local convolutional extraction. Its design reduces computation while maintaining competitive performance across vision tasks.

  • Lightweight vision models must balance competitive performance with resource consumption as architectures increasingly scale model sizes.
  • EfficientVMamba uses skip sampling to scan fewer spatial tokens while preserving a global receptive field.The method is illustrated as an efficient 2D scan that reduces computational demands from 4N to N.
  • The network places global SSM blocks in shallow, high-resolution layers and efficient convolution blocks in deeper layers.
  • Experiments cover image classification, object detection, and semantic segmentation, reporting reduced FLOPs with significant improvements over existing lightweight models.
  • Its dual-pathway design combines a global state-space branch with a convolution branch for local feature extraction, fused through channel attention.

2 Related Work

Related work spans CNNs, Transformers, and emerging state-space models for vision. SSM-based approaches model long dependencies, while visual variants adapt sequence processing to two-dimensional image structure.

  • CNN and Transformer architectures have achieved strong results across image classification, detection, and segmentation tasks.
  • SSMs transform sequences while modeling long dependencies, but their high computational and memory usage makes training challenging.
  • Mamba improves SSM efficiency through input-specific parameterization and a hardware-optimized algorithm for long-sequence processing.
  • Visual SSM methods extend the framework to one-, two-, and three-dimensional visual data and address the mismatch between 1D sequences and multidimensional images.

3 Preliminaries

SSMs map input sequences through a latent state to output sequences, with continuous dynamics discretized for sampled computation. Selective and two-dimensional scans adapt this framework to input-dependent and visual processing.

  • An SSM transforms an input D-dimensional sequence into an output sequence through a learnable latent state h(t) that is not directly observable.
  • The continuous state-space equations update h(t) from the input x(t) and produce y(t) through matrices A, B, and C.
  • Discretization: Discretization converts continuous differential equations into discrete functions aligned with the input sampling frequency for efficient computation.
  • Discretization: Repeated discrete updates can be performed simultaneously through a global convolution using the resulting SSM kernel.
  • Selective SSMs: Mamba makes SSM parameters input-dependent, enabling selective information processing across sequences.
  • Visual SSMs: VMamba’s SS2D scans four directed feature sequences independently and combines them into a comprehensive 2D feature map.

4 Method

EfficientVMamba reduces selective-scan cost through atrous skip sampling, combines global and local feature extraction in EVSS, and places local modules in later stages.

  • Efficient 2D Scanning (ES2D): EfficientVMamba introduces ES2D, an atrous-based selective scanning strategy using skip sampling and spatial regrouping to extract global dependencies efficiently.The method decomposes scanning into sparse local and global forms before regrouping patches to reconstruct global spatial structure.
  • Efficient 2D Scanning (ES2D): O(N) → O(N/p^2) reduces ES2D scan complexity with step p while preserving a global receptive field among tokens.Skip sampling processes patches at intervals of p and reduces the number of scanned tokens from N to N/p^2.
  • Efficient Visual State Space Block (EVSS): EVSS combines modified ES2D for global information capture with a convolutional branch for local feature extraction.Each pathway uses Squeeze-Excitation, and their outputs are combined by element-wise summation.
  • Efficient Visual State Space Block (EVSS): Squeeze-Excitation dynamically re-balances global and local features to emphasize salient information in the fused EVSS representation.The two pathway outputs are fused after separate SE operations.
  • Inverted Insertion of EfficientNet Blocks: Inverted insertion places EVSS blocks in the first two stages and Inverted Residual blocks in subsequent stages for local feature extraction.This reverses the conventional ordering of efficient local modules and later global blocks; the placement was selected based on empirical observations in Table 6.
  • Model Variants: EfficientVMamba-T, -S, and -B provide 6M, 11M, and 33M parameters, with corresponding computational loads of 0.8G, 1.3G, and 4.0G FLOPs.The variants represent increasing model complexity and feature size.

5 Experiments

Experiments evaluate EfficientVMamba across image classification, object detection, semantic segmentation, and ablations of its scanning, fusion, and stage designs. The results report competitive accuracy with reduced computational or parameter costs.

  • ImageNet Classification: EfficientVMamba-T achieves 76.5% Top-1 accuracy with 0.8 GFLOPs, surpassing PVTv2-B0 by 6% and MobileViT-XS by 1.7%.
  • ImageNet Classification: EfficientVMamba-S reaches 78.7% Top-1 accuracy with 1.3 GFLOPs, exceeding DeiT-Ti’s 72.2% and MobileViT-S’s 78.4%.
  • ImageNet Classification: EfficientVMamba-B achieves 81.8% Top-1 accuracy with 33 M parameters and 4.0 GFLOPs, surpassing DeiT-S and Vim-S while using fewer FLOPs than VMamba-T.
  • Object Detection: EfficientVMamba variants achieve competitive RetinaNet detection results, including EfficientVMamba-T’s 37.5% AP with 13M parameters and EfficientVMamba-S’s 39.1% AP with 19M parameters.
  • Semantic Segmentation: EfficientVMamba-B reaches 46.5% single-scale and 47.3% multi-scale mIoU, while EfficientVMamba-S reaches 41.5% and 42.1%, respectively, on semantic segmentation.
  • Ablation Study: Ablations show ES2D reduces computation while preserving or improving accuracy, and SSM-Conv fusion raises tiny-model accuracy from 73.6% to 75.1% and base-model accuracy from 80.9% to 81.2%.

6 Conclusion

EfficientVMamba combines atrous selective scanning and convolutional processing to balance global and local feature extraction. Experiments report reduced computational complexity and competitive performance across vision tasks.

  • EfficientVMamba combines atrous-based selective scanning with skip sampling to cover global receptive fields while minimizing computational load.
  • A convolutional branch and Squeeze-and-Excitation module rebalance global and local features, while inverted residual insertion refines multi-layer stages.
  • Experiments report O(N) computational complexity and competitive performance across image classification, object detection, and semantic segmentation.

Appendix

The appendix evaluates EfficientVMamba as a lightweight backbone under Mask R-CNN schedules, reporting AP improvements over several comparison backbones at lower computational or parameter budgets.

  • Mask R-CNN: Under the Mask R-CNN 1× schedule, EfficientVMamba-T achieves 35.6% AP with 11M parameters and 60G FLOPs, 1.6% above ResNet-18.
  • Mask R-CNN: EfficientVMamba-S reaches 39.3% AP with 31M parameters and 197G FLOPs, 0.5% above ResNet-50’s result.
  • Mask R-CNN: EfficientVMamba-B achieves 43.7% AP with 53M parameters and 252G FLOPs, outperforming VMamba-T by 2.8%.
  • Mask R-CNN: Under the Mask R-CNN 3× multi-scale schedule, EfficientVMamba-T, S, and B achieve 38.3%, 41.5%, and 45.0% AP, respectively.

Comparisons with MobileNetV2 Backbone

The study compares EVSS and Inverted Residual blocks across network stages. The reported results indicate that combining EVSS early with InRes later improves the architecture’s accuracy-efficiency balance.

  • Stage-wise Architecture: The proposed fusion uses EVSS in initial stages and InRes in later stages, improving accuracy relative to the compared stage configurations.
  • Stage-wise Architecture: Inverted Residual blocks used across all stages achieve 81.4% accuracy in the base variant.
  • Stage-wise Architecture: Applying EVSS across all stages causes a slight accuracy decrease in both tiny and base variants.

Limitations

Visual state space models provide linear-time processing and strong high-resolution performance, but their computational sophistication complicates efficient parallel processing and leaves efficiency and scalability open for further optimization.

  • Visual state space models achieve linear-time complexity O(N) relative to sequence length and marked enhancements in high-resolution downstream tasks.
  • Their computational design is more sophisticated than convolutional and self-attention mechanisms, complicating efficient parallel processing.
  • Optimizing the computational efficiency and scalability of visual state space models remains a promising direction for future investigation.
Loading 2403.09977v1…