Source-linked AI summary
Mamba YOLO: A Simple Baseline for Object Detection with State Space Model
Zeyu Wang, Chen Li, Huiying Xu, Xinzhong Zhu, Hongbo Li
TL;DR
Transformer-based detectors improve receptive-field modeling but incur quadratic self-attention costs. Mamba YOLO introduces a linear-complexity SSM-based detector with an RG Block for local dependencies, and reports competitive MSCOCO performance across model scales.
Problem
Transformer self-attention improves global modeling but introduces quadratic complexity, creating a performance-speed challenge for real-time object detection.
Method
Mamba YOLO uses an SSM-based ODMamba backbone with ODSSBlock and an RG Block combining gated aggregation, convolution, and residual connectivity.
Results
Mamba YOLO achieves significant performance improvements over existing approaches on MSCOCO across Tiny, Base, and Large variants.
Takeaways & Limitations
The paper establishes an SSM-based YOLO baseline that does not require pretraining on large-scale datasets.
Abstract
from arXiv · showhide
Driven by the rapid development of deep learning technology, the YOLO series has set a new benchmark for real-time object detectors. Additionally, transformer-based structures have emerged as the most powerful solution in the field, greatly extending the model's receptive field and achieving significant performance improvements. However, this improvement comes at a cost as the quadratic complexity of the self-attentive mechanism increases the computational burden of the model. To address this problem, we introduce a simple yet effective baseline approach called Mamba YOLO. Our contributions are as follows: 1) We propose that the ODMamba backbone introduce a \textbf{S}tate \textbf{S}pace \textbf{M}odel (\textbf{SSM}) with linear complexity to address the quadratic complexity of self-attention. Unlike the other Transformer-base and SSM-base method, ODMamba is simple to train without pretraining. 2) For real-time requirement, we designed the macro structure of ODMamba, determined the optimal stage ratio and scaling size. 3) We design the RG Block that employs a multi-branch structure to model the channel dimensions, which addresses the possible limitations of SSM in sequence modeling, such as insufficient receptive fields and weak image localization. This design captures localized image dependencies more accurately and significantly. Extensive experiments on the publicly available COCO benchmark dataset show that Mamba YOLO achieves state-of-the-art performance compared to previous methods. Specifically, a tiny version of Mamba YOLO achieves a \textbf{7.5}\% improvement in mAP on a single 4090 GPU with an inference time of \textbf{1.5} ms. The pytorch code is available at: \url{https://github.com/HZAI-ZJNU/Mamba-YOLO}
Introduction
Mamba YOLO addresses the tension between global modeling and real-time efficiency by applying linear-complexity SSMs to object detection. Its ODSSBlock and RG Block target SSM limitations, while scaled variants are evaluated on MSCOCO.
- Motivation: Transformer self-attention provides global modeling but has quadratic complexity, motivating lower-complexity detector designs.
- Approach: Mamba YOLO applies SSMs in an ODSSBlock-based detector backbone and trains without pretraining on large-scale datasets.
- Approach: The RG Block combines gated aggregation, effective convolution, and residual connectivity to capture local dependencies and enhance robustness.
- Results: Mamba YOLO provides Tiny, Base, and Large variants, and experiments on MSCOCO report significant performance improvements over existing approaches.
Related Work
Prior object detectors use convolutional, Transformer, and hybrid designs, while recent visual SSMs seek efficient long-range modeling. Mamba YOLO extends this direction into a detector architecture using selective SSM, multilevel feature fusion, and a decoupled head.
- Real-Time Object Detectors: YOLO backbones evolved from Dark-Net and E-ELAN toward lightweight CSPDarknet53-based designs that balance accuracy with deployment flexibility.
- End-to-End Object Detectors: DETR introduced Transformer encoder-decoder detection without handcrafted anchor generation and non-maximum suppression, while Deformable DETR uses sparse keypoint attention for high-resolution maps.
- Vision State Space Models: Mamba and related visual SSMs provide linear complexity for long-sequence modeling, with Vision Mamba and VMamba introducing selective SSMs and cross-scan processing to vision.
- Mamba YOLO Architecture: Mamba YOLO uses an ODSSBlock with selective SSM in its backbone, fuses C3, C4, and C5 through PAFPN, and sends P3, P4, and P5 to a decoupled head.
Method
Mamba YOLO builds its detector around an SSM-based ODMamba backbone and ODSSBlock modules, combining global spatial processing with gated local and global channel modeling. Its architecture uses visual-clue-preserving downsampling and a PAFPN-style neck to support efficient detection.
- State-space modeling: SSM maps input sequences through latent states, with Mamba using discretization for deep-learning sequence processing and convolutional training alongside recursive inference.The state-space formulation uses transition, input, and observation matrices; zero-order hold is one discretization method.
- Overall architecture: Mamba YOLO applies structured state-space modeling to object detection through an ODMamba backbone and ODSSBlock modules.The detector separates the ODMamba backbone from a PAFPN-style neck that also uses ODSSBlock modules.
- Vision Clue Merge: Vision Clue Merge preserves selected feature maps during downsampling by splitting dimensions, appending excess maps, and using a 4× compressed pointwise convolution.Unlike a stride-2 3 × 3 convolution, this approach preserves the feature map selected by SS2D from the previous layer.
- SS2D: SS2D expands image features along four directions, processes the resulting sequences with S6, and merges them into global features.The directional scans cover top-down, bottom-up, left-right, and right-to-left paths.
- RG Block: RG Block uses two branches and gating to model local and global channel information, capturing richer dependencies with only a slight computational increase.Its design incorporates depthwise convolution, residual concatenation, and nonlinear GELU activation.
Experiments
Experiments show that Mamba YOLO provides favorable accuracy–efficiency trade-offs across model scales, while ablations identify effective ODSSBlock, RG Block, and architectural configurations.
- State-of-the-art comparison: Mamba YOLO-T improves AP by 1.1%/1.5% over PPYOLOE-S/YOLO-MS-XS while reducing GPU inference latency by 0.9ms/0.2ms.
- State-of-the-art comparison: Against YOLOv8-S, Mamba YOLO-T reduces parameters by 48%, FLOPs by 53%, and GPU inference latency by 0.4ms at similar accuracy.
- State-of-the-art comparison: Mamba YOLO-B gains 3.7% AP over similarly sized Gold-YOLO-M, while Mamba YOLO-L gains 0.3% AP over Gold-YOLO-L with 0.9% fewer parameters.
- Resolution and efficiency: Mamba YOLO-L maintains better precision and speed than DINO-R50 at increased resolutions, with linear growth in memory efficiency and FLOPs.
- RG Block ablations: RG Block ablations indicate that convolution improves complex-image performance when combined with gated aggregation and residual connectivity.
- Architecture ablations: The selected [3, 6, 6, 3] backbone configuration balances accuracy and complexity, whereas alternative placements add overhead or duplicate ODSSBlock excessively.
- Qualitative evaluation: Visualization results show accurate detection under overlapping, occluded, and complex-background conditions.
Conclusion
The paper proposes a YOLO detector based on SSM, emphasizing simple scratch training and an RG Block that propagates important features through hierarchical representations.
- Mamba YOLO uses an SSM-based YOLO detector that does not require pre-training on extensive datasets.
- The RG Block combines gating and deep convolutional residual connectivity to propagate important features in the hierarchical structure.