Source-linked AI summary
VideoMamba: State Space Model for Efficient Video Understanding
Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, Yu Qiao
TL;DR
Video understanding must address local spatiotemporal redundancy and long-range dependencies, which existing 3D CNNs and video transformers do not simultaneously handle. VideoMamba adapts a purely SSM-based model with linear-complexity spatiotemporal context modeling, and experiments demonstrate scalability, short-term sensitivity, long-term superiority, and multimodal compatibility.
Problem
Video understanding requires jointly modeling local spatiotemporal redundancy and complex dependencies across long contexts, while existing 3D CNNs and video transformers address these challenges separately.
Method
VideoMamba is a purely SSM-based model that uses linear-complexity dynamic spatiotemporal context modeling for efficient video understanding.
Results
Experiments demonstrate scalability without large-scale pretraining, sensitivity to fine-grained short-term actions, superiority in long-term understanding, and compatibility with other modalities; it operates 6× faster than TimeSformer and uses 40× less GPU memory for 64-frame videos.
Takeaways & Limitations
VideoMamba shows promising potential as an efficient approach for long-video comprehension across short-term, long-term, and multimodal settings.
Takeaways & Limitations
The study does not fully validate larger VideoMamba models, additional modalities such as audio, or integration with large language models for hour-level video understanding.
Abstract
from arXiv · showhide
Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain. The proposed VideoMamba overcomes the limitations of existing 3D convolution neural networks and video transformers. Its linear-complexity operator enables efficient long-term modeling, which is crucial for high-resolution long video understanding. Extensive evaluations reveal VideoMamba's four core abilities: (1) Scalability in the visual domain without extensive dataset pretraining, thanks to a novel self-distillation technique; (2) Sensitivity for recognizing short-term actions even with fine-grained motion differences; (3) Superiority in long-term video understanding, showcasing significant advancements over traditional feature-based models; and (4) Compatibility with other modalities, demonstrating robustness in multi-modal contexts. Through these distinct advantages, VideoMamba sets a new benchmark for video understanding, offering a scalable and efficient solution for comprehensive video understanding. All the code and models are available at https://github.com/OpenGVLab/VideoMamba.
1 Introduction
Video understanding must address both short-clip redundancy and long-context dependencies, challenges that existing convolutional and attention-based approaches do not simultaneously resolve. VideoMamba applies a purely SSM-based, linear-complexity design and demonstrates scalability, short-term sensitivity, long-term effectiveness, and multimodal compatibility.
- Motivation: Video understanding faces large spatiotemporal redundancy in short clips and complex dependencies across long contexts.Existing 3D CNNs and video transformers address these challenges separately, while UniFormer struggles with long videos.
- Approach: VideoMamba is a purely SSM-based model using linear-complexity dynamic spatiotemporal context modeling for high-resolution long videos.Its design merges convolutional and attention strengths in a vanilla ViT-style architecture.
- Scalability: Self-distillation lets VideoMamba scale model and input sizes without large-scale dataset pretraining.The pure Mamba model tends to overfit as it scales without this strategy.
- Short-term understanding: VideoMamba distinguishes short-term actions with fine-grained motion differences and outperforms existing attention-based models.The model is also suitable for masked modeling, which further enhances temporal sensitivity.
- Long-term understanding: 6× faster and 40× less GPU memory than TimeSformer for 64-frame videos, VideoMamba supports efficient long-term understanding.It also shows superiority over conventional feature-based methods through end-to-end training.
- Evaluation scope: Experiments report VideoMamba’s potential across short-term and long-term video contents, alongside compatibility with other modalities.The evaluated short-term datasets are K400 and SthSthV2, while long-term datasets include Breakfast, COIN, and LVU.
2 Related Works
State space models provide a linear-complexity route to modeling long-range sequence dependencies, and recent vision adaptations have demonstrated efficiency on visual tasks. VideoMamba extends this direction to unified short- and long-term video understanding.
- State Space Models: S4 introduced structured state-space sequence modeling for long-range dependencies with linear complexity, followed by models including S5, H3, GSS, and Mamba.Mamba adds a data-dependent SSM layer.
- SSMs in vision: Vision research applies SSMs to image classification, movie-clip classification, object detection, and semantic segmentation with improved GPU efficiency over Transformers.These applications motivate extending Mamba to video.
- Video datasets: Video datasets support benchmarking action recognition and related tasks, including UCF101 and Kinetics.Additional datasets provide annotated activity videos for action localization and related evaluation settings.
- Video architectures: Video understanding research evolved from 3D CNNs and temporal-stream methods toward attention-based models such as TimeSformer and ViViT.These architectures progressively target spatial, temporal, and spatiotemporal information in videos.
3 Method
VideoMamba adapts bidirectional selective state-space modeling to 3D video tokens, combining convolutional patch projection, position-aware token sequences, and efficient spatiotemporal scans. Its design uses linear-complexity sequence modeling, self-distillation, and masking strategies to support scalable and temporally sensitive video understanding.
- 3.1 Preliminaries: SSMs map 1D inputs to outputs through hidden states, while Mamba discretizes the continuous system and uses input-dependent selective scanning for contextual sensitivity.The continuous formulation uses evolution and projection matrices; Mamba derives B, C, and Δ from the input and adaptively modulates the sequence computation.
- 3.1 Preliminaries: VideoMamba adapts the bidirectional Mamba block from 2D vision to 3D video sequences within a vanilla ViT-style architecture.The framework uses stacked bidirectional Mamba blocks and classifies from the final normalized [CLS] representation.
- 3.2 VideoMamba: 3D convolution projects videos into non-overlapping spatiotemporal patches, which are combined with a learnable [CLS] token and spatial and temporal position embeddings.The spatial and temporal embeddings retain spatiotemporal position information because SSM modeling is sensitive to token position.
- 3.2 VideoMamba: VideoMamba evaluates Spatial-First, Temporal-First, and hybrid Spatiotemporal bidirectional scans, with experiments identifying Spatial-First as the most effective simple choice.The Spatiotemporal variant includes alternatives with different computation, including a full version requiring 2× computation.
- 3.2 VideoMamba: Linear Mamba complexity enables efficient processing of long, high-resolution videos, while row masking preserves continuous tokens favored by the preceding 1D convolution.Clip-row masking removes an entire video clip, whereas frame-row masking masks each frame individually; attention masking is also explored to preserve token adjacency.
- 3.4 Masked Modeling: Larger VideoMamba models can overfit, so the method investigates self-distillation, early stopping, and masked modeling to improve scalability and temporal sensitivity.The reported experiments found early stopping unhelpful, while self-distillation was presented as a viable scalability strategy and masked alignment was adapted to VideoMamba’s SSM architecture.
4 Experiments
Experiments evaluate VideoMamba across image classification, short- and long-term video understanding, and multimodal retrieval, including scaling, scan design, masking, and computational efficiency.
- 4.1 Scaling Up: VideoMamba-M improves ImageNet-1K accuracy by +0.8% over ConvNeXt-B and +2.0% over DeiT-B while using fewer parameters.Increasing resolution reaches 84.0% top-1 accuracy with 74M parameters.
- 4.1 Scaling Up: Self-distillation enables VideoMamba to scale model and input sizes without large-scale dataset pretraining.The strategy addresses overfitting observed when scaling the pure Mamba model.
- 4.2 Short-term Video Understanding: VideoMamba-M outperforms ViViT-L by +2.0% on Kinetics-400 and +3.0% on Something-Something V2 with lower computational demands and less pretraining data.It performs on par with UniFormer and masked pretraining surpasses VideoMAE.
- 4.2 Short-term Video Understanding: Spatial-first scanning performs best because it leverages 2D pretrained knowledge by scanning frames sequentially.Temporal-first scanning performs worst among the evaluated scan strategies.
- 4.3 Long-term Video Understanding: VideoMamba-Ti improves over ViS4mer by +6.1% and Turbo by +3.0% on long-term video tasks while achieving outstanding or comparable LVU results.Scaling the model and frame count positively affects long-term tasks.
- 4.4 Multi-modality Video Understanding: VideoMamba achieves superior zero-shot video retrieval to UMT under the same pretraining corpus and similar training strategies.Improvements are significant on longer-video and more complex benchmarks, including ActivityNet, DiDeMo, and LSMDC.
5 Conclusion
The paper concludes that VideoMamba provides an efficient SSM-based approach with scalability, short-term sensitivity, long-term understanding, and multimodal compatibility.
- 5 Conclusion: Experiments demonstrate VideoMamba’s scalability, short-term action sensitivity, long-term video understanding, and compatibility with other modalities.The authors identify these as the model’s four demonstrated abilities.
- 5 Conclusion: Resource constraints leave larger models, additional modalities such as audio, and hour-level video understanding with large language models insufficiently validated.The authors plan further exploration of these capabilities.
A More Results
Additional Kinetics-400 results show consistent gains as VideoMamba’s input resolution and frame count increase.
- A More Results: VideoMamba outperforms previous attention-based methods on Kinetics-400, with performance improving as resolution and frame count increase.The passage reports the trend but does not provide specific accuracy values.
B.1 Training Details
The appendix documents sparse frame sampling and the training details used for masked pretraining, unmasked multimodal pretraining, and dataset-specific fine-tuning.
- B.1 Training Details: Frames are sparsely sampled from raw videos following TSN for all datasets.Tables II–VI provide masked-pretraining, multimodal-pretraining, and fine-tuning details.
B.2 Dataset Descriptions
This section identifies the paper’s dataset-statistics and training-setting tables, covering both multi-modality and single-modality datasets.
- Table I provides additional scene-related Kinetics-400 results and defines “iso.” as an isotropic architecture without downsampling layers.
- Tables II–V document masked pre-training and training settings for ImageNet-1K, Kinetics-400, and SthSthV2.The notation “†” denotes masked pretraining in the Kinetics-400 and SthSthV2 settings.
- Table VI lists training settings for Breakfast, COIN, and LVU, including sparse frame sampling directly from raw videos.The table also marks masked pretraining with “†”.
- Table VII reports statistics for multi-modality datasets.
- Table VIII reports statistics for single-modality datasets.