Source-linked AI summary
Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding
Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang, Kunchang Li, Tong Lu, Limin Wang
TL;DR
Video understanding needs architectures that capture complex spatial-temporal dynamics efficiently, but Mamba’s effectiveness across video tasks had not been comprehensively established. The paper studies Mamba in four modeling roles through a 14-model/module suite evaluated across 12 tasks and reports strong performance and promising efficiency-performance trade-offs, while identifying a spatial-temporal scanning limitation.
Problem
Mamba’s effectiveness as a transformer alternative for diverse video understanding tasks remains unclear because comprehensive evidence across roles and tasks is limited.
Method
The paper evaluates Mamba in four video-modeling roles through a Video Mamba Suite comprising 14 models/modules across 12 tasks and 13 datasets.
Results
Across 12 video understanding tasks, Mamba demonstrates strong performance and promising efficiency-performance trade-offs compared with traditional transformers.
Takeaways & Limitations
The findings support Mamba as a viable alternative for video understanding and provide data points for future SSM-based video research.
Takeaways & Limitations
Scanning-based spatial-temporal modeling may damage the spatial feature distribution produced by pretrained space-attention blocks.
Abstract
from arXiv · showhide
Understanding videos is one of the fundamental directions in computer vision research, with extensive efforts dedicated to exploring various architectures such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state space model, e.g., Mamba, shows promising traits to extend its success in long sequence modeling to video modeling. To assess whether Mamba can be a viable alternative to Transformers in the video understanding domain, in this work, we conduct a comprehensive set of studies, probing different roles Mamba can play in modeling videos, while investigating diverse tasks where Mamba could exhibit superiority. We categorize Mamba into four roles for modeling videos, deriving a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12 video understanding tasks. Our extensive experiments reveal the strong potential of Mamba on both video-only and video-language tasks while showing promising efficiency-performance trade-offs. We hope this work could provide valuable data points and insights for future research on video understanding. Code is public: https://github.com/OpenGVLab/video-mamba-suite.
1 Introduction
Video understanding requires modeling spatial-temporal dynamics, while existing architectures trade off joint modeling, global context, and efficiency. This paper investigates whether Mamba can serve as a transformer alternative through four modeling roles and a broad suite of models, tasks, and datasets.
- Video understanding requires capturing spatial-temporal dynamics to localize activities or infer their evolution.
- RNN-based approaches separate frame encoding from temporal modeling, while 3D CNNs jointly model spatial and temporal correlations.
- Video transformers provide unified temporal or spatial-temporal modeling but face limited computational efficiency on long videos.
- Mamba combines strong long-sequence representation with linear-time complexity and hardware-aware efficient training and inference, motivating its study for video understanding.
- The study categorizes Mamba into temporal model, temporal module, multi-modal interaction network, and spatial-temporal model roles.
- The resulting Video Mamba Suite comprises 14 models/modules for 12 video understanding tasks and is intended as a resource for future SSM-based research.
2 Related Work
Prior video modeling progressed from convolutional and recurrent approaches toward transformers and linear-complexity architectures. Related SSM work established efficient long-range sequence modeling, but video applications remained limited in scope.
- Video convolutional methods integrate temporal modules or expand kernels, but static local operators constrain representation capacity.
- Video transformers, especially those using joint spatial-temporal attention, improve modeling capabilities but motivate research into alternative efficient architectures.
- RetNet, RWKV, and SSMs pursue linear-complexity modeling, while Mamba adds data-dependent inference and efficient implementations.
- S4 introduced linearly scaling long-range dependency modeling, followed by S5, H3, gated SSMs, and Mamba improvements in efficiency or expressivity.
- The paper extends Mamba’s language-modeling success to video understanding by developing, validating, and analyzing its performance.
- Existing SSM video studies primarily address long-term video classification, whereas this work covers broader SSM usages and video-related tasks.
3 Preliminaries
The preliminaries describe SSMs as state-space sequence processors and introduce Mamba-family blocks used in the suite. These formulations update hidden states over discrete steps or scan sequences in multiple directions.
- An SSM transforms an input sequence x(t) into outputs y(t) through a hidden state h(t), with matrices A, B, and C governing state evolution and projections.
- S4 and Mamba discretize continuous dynamics using a timescale parameter ∆ and zero-order hold before applying discrete state updates.
- At each discrete step, the hidden state updates from the previous state and current input, then produces the output through a projection.
- The sequence output can also be computed by applying a structured convolutional kernel K across the input sequence.
- Mamba-family blocks: The vanilla Mamba block expands model dimension D by factor E and concentrates most parameters in linear projections rather than the inner SSM.
- Mamba-family blocks: ViM adds a backward selective-scanning branch, shares linear projection and gating layers across directions, and averages gated forward and backward features.
- Mamba-family blocks: DBM separates forward and backward input projections, shares SSM parameters across directions, and concatenates separately gated outputs.
4 Video Mamba Suite
Video Mamba Suite evaluates Mamba across four modeling roles and diverse video-only and video-language tasks, comparing Mamba-based modules with transformer counterparts. Results indicate strong performance across temporal, cross-modal, long-form, and retrieval settings, with some task- and arrangement-specific differences.
- Suite design: The suite organizes Mamba into temporal models, temporal modules, multi-modal interaction models, and space-time sequence models for diverse video understanding tasks.The evaluation includes temporal localization, segmentation, captioning, anticipation, temporal grounding, long-form video QA, retrieval, and action recognition.
- Temporal tasks: Mamba-based models outperform transformer counterparts across the evaluated temporal tasks, including localization, segmentation, dense captioning, paragraph captioning, and causal anticipation.The comparisons replace transformer blocks with vanilla Mamba, ViM, or DBM blocks, with causal Mamba used for action anticipation.
- Temporal tasks: 44.56 average mAP is achieved by ActionMamba with DBM on HACS Segment, exceeding its transformer counterpart by 1.22 points.The corresponding transformer result is 43.34, while the ViM version reaches 44.26 average mAP.
- Cross-modal interaction: 44.74 average mAP is achieved by Mamba on QvHighlight, compared with 38.48 for the transformer, while Charade-STA performance remains comparable.Fusion experiments find the best results when textual conditions are placed on the left side of visual features, with Charade-STA more sensitive to text position.
- Retrieval and recognition: ViM-based temporal modules outperform attention-based modules in zero-shot retrieval, while space-time ViM unexpectedly decreases performance relative to temporal ViM.ViViM-S also improves over ViT-S by +2.1 mAP@Avg, and TimeMamba surpasses TimeSformer by 2.8 points for verb recognition after fine-tuning.
- Long-form video understanding: TimeMamba and TimeSformer improve on zero-shot long-form video QA as testing frames increase, with TimeMamba generally benefiting more beyond 32 frames.A significant improvement is observed at 8192 frames despite both models being pretrained on four frames.
5 Efficiency Analysis
The study evaluates inference speed as frame counts grow, using fixed spatial token counts and matched precision and attention settings. Mamba shows increasing speed advantages over transformer models particularly for long videos.
- Inference speed is measured from 4 to 8192 frames with 196 spatial tokens on one A100 GPU at half precision.All attention blocks use Flash-attention for fair comparison.
- Mamba offers speed advantages over transformer-series models when the number of frames is substantial.ViViM-T is also compared with ViT both with and without Flash-attention.
6 Conclusion
The Video Mamba Suite evaluates Mamba across 14 models or modules and 12 video understanding tasks. The study reports efficient handling of spatial-temporal dynamics, strong performance, and promising efficiency-performance trade-offs.
- Mamba demonstrates efficient handling of complex spatial-temporal dynamics with superior performance and promising efficiency-performance trade-offs.The supplementary material provides additional details and experimental results.
A7 More Experimental Results
Supplementary experiments extend the evaluation across temporal localization, segmentation, dense video captioning, and action recognition. Mamba-based models generally perform strongly, while results vary by dataset, structure, and comparison model.
- Temporal action localization: ActionMamba outperforms ActionFormer on temporal action localization across THUMOS-14, ActivityNet, and FineAction.DBM also outperforms the ViM block in these comparisons.
- Temporal action segmentation: Mamba-based methods achieve significantly stronger performance on Breakfast, while 50Salads results favor ASFormer on several metrics but ASMamba on F1@25.ASMamba outperforms encoder-only ASFormer, whereas encoder-decoder structure can be better in some scenarios.
- Dense video captioning: Mamba-based temporal encoders improve localization metrics in dense video captioning, consistent with their temporal action localization results.The reported localization metrics are Recall and Precision.
- Action recognition: 77.4% and 80.1% Top-1 accuracy are achieved by ViViM-T and ViViM-S on Kinetics-400, respectively.ViViM-T reaches competitive performance with 7M parameters compared with the 28M VideoSwin-T, while a gap remains versus dedicated module designs such as UniFormer.
A8 ViM and DBM
This section describes ViM and DBM blocks and compares their parameterization, scanning context, and computational cost. DBM separates directional features, trades dynamic-modeling capacity for compatibility in some settings, and reduces time cost relative to ViM.
- ViM block: ViM expands an input sequence into scanning and gating hidden states, then applies bidirectional selective scanning and gating.The input sequence has length N and feature dimension d, and expansion uses factor E.
- DBM block: DBM separates forward and backward features along the channel dimension into four hidden states before scanning, gating, concatenating, and projecting them.
- Analysis: DBM provides static direction separation and reduces dynamic-modeling capacity compared with ViM, making it more compatible with certain downstream datasets.
- Analysis: DBM reduces training and inference time cost by half relative to ViM.The reduction is consistent with the vanilla Mamba block.
A9 Hyperparameter Sensitivity
Mamba-series models were generally insensitive to training hyperparameters, but video temporal grounding benefited from targeted adjustments to learning rate and video-text alignment loss weight.
- Most experiments found Mamba-series training hyperparameters to be insensitive.
- For most tasks, the Transformer block was simply replaced with a Mamba-based block.
- In video temporal grounding, a larger learning rate yielded better optimization results.
- Increasing the video-text alignment loss weight facilitated model convergence.