Source-linked AI summary
MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Yuhuan Yang, Chaofan Ma, Zhenjie Mao, Jiangchao Yao, Ya Zhang, Yanfeng Wang
TL;DR
Video understanding needs spatial-temporal representations, but adapting image foundation models with PEFT often separates spatial and temporal processing while full attention is costly. MoMa integrates lightweight Mamba and SeqMod into a Divide-and-Modulate architecture, and experiments report improved performance and efficiency across multiple video benchmarks. The method preserves pre-trained IFM parameters while adding spatial-temporal modeling through trainable SSM layers.
Problem
PEFT-based adaptation of image foundation models often adds temporal modules separately, while full spatial-temporal attention has quadratic cost for long video sequences.
Method
MoMa combines window-based local attention with SeqMod and lightweight Mamba to inject spatial-temporal information into pre-trained IFMs.
Results
MoMa improves performance and computational efficiency compared with existing methods across multiple video understanding benchmarks.
Takeaways & Limitations
MoMa provides a parameter-efficient way to capture full spatial-temporal dynamics while retaining pre-trained IFM features.
Takeaways & Limitations
The approach assumes CLIP lacks inherent temporal modeling and therefore requires additional temporal modules for video adaptation.
Abstract
from arXiv · showhide
Video understanding is a complex challenge that requires effective modeling of spatial-temporal dynamics. With the success of image foundation models (IFMs) in image understanding, recent approaches have explored parameter-efficient fine-tuning (PEFT) to adapt IFMs for video. However, most of these methods tend to process spatial and temporal information separately, which may fail to capture the full intricacy of video dynamics. In this paper, we propose MoMa, an efficient adapter framework that achieves full spatial-temporal modeling by integrating Mamba's selective state space modeling into IFMs. We propose a novel SeqMod operation to inject spatial-temporal information into pre-trained IFMs, without disrupting their original features. By incorporating SeqMod into a Divide-and-Modulate architecture, MoMa enhances video understanding while maintaining computational efficiency. Extensive experiments on multiple video benchmarks demonstrate the effectiveness of MoMa, achieving superior performance with reduced computational cost.
1. Introduction
MoMa adapts image foundation models to video by combining parameter-efficient spatial-temporal modulation with windowed attention. It targets the efficiency and modeling limitations of prior approaches and reports improved performance and computational efficiency across video benchmarks.
- 1. Introduction: PEFT methods must add temporal modules because IFMs provide strong image representations but do not explicitly model temporal dynamics.Full attention over long spatial-temporal sequences is costly, motivating linear-complexity Mamba-based modeling.
- 1. Introduction: The Divide-and-Modulate design addresses efficiency by narrowing attention locally while using modulation to capture global spatial-temporal relationships.This separates short-term spatial dependency modeling from broader temporal-dynamic modeling within the IFM.
- 1. Introduction: Mamba is introduced as an efficient adapter for IFMs that captures full spatial-temporal dynamics without interfering with pre-trained parameters.The approach keeps most IFM parameters frozen while using SeqMod to incorporate information from lightweight Mamba.
- 1. Introduction: MoMa combines window-based spatial local attention with SeqMod to inject full spatial-temporal information into each IFM layer.The framework uses a Divide stage for local attention and a Modulate stage for spatial-temporal modulation.
- 1. Introduction: MoMa’s experiments and ablations on multiple video benchmarks show improved performance and computational efficiency compared with existing methods.The paper presents this as the overall empirical outcome of the proposed framework.
2. Related Works
Related work spans video temporal modeling, image foundation model adaptation, and state space models. MoMa builds on these directions by combining pre-trained IFM representations with Mamba’s linear-complexity sequence modeling.
- Video Understanding: Video understanding methods model temporal patterns using 3D convolutions or Transformer global attention to capture dependencies across frames.Transformer-based approaches provide long-range temporal modeling, while 3D convolutions have more limited receptive fields.
- Image Foundation Models Adaptation: Image foundation models learn powerful visual representations from large-scale self-supervised, multimodal, and web-scale training resources.These representations motivate adapting IFMs rather than training video models entirely from scratch.
- State Space Model: State space models process long sequences with linear complexity, and Mamba adds data-dependent selection and hardware-efficient algorithms.These properties make Mamba relevant for efficient long-term dynamic modeling.
- State Space Model: MoMa differs from Mamba architectures trained from scratch by building a hybrid model on pre-trained IFMs.This design aims to retain Mamba’s linear-complexity advantages while leveraging representations learned by IFMs.
Modulate Stage…
MoMa’s Modulate stage uses lightweight Mamba and SeqMod to inject spatial-temporal information into IFM features while preserving the original attention and feed-forward components. This complements hybrid Mamba-attention architectures that combine layers or modules as alternatives to direct feature modulation.
- Modulate Stage: The Modulate stage forwards sequences through an SSM to produce scale and bias sequences for SeqMod-based spatial-temporal feature injection.The modulated output is then passed to CLIP’s feed-forward layer, while only SSM layers are trainable.
- Modulate Stage: MoMa uses SeqMod to adapt pre-trained IFM features rather than directly modifying them with a newly inserted Mamba module.This design responds to the integration challenge posed by frozen Transformer-based IFMs.
- Mamba and Attention Hybrid Architectures: Prior hybrid architectures interleave attention, MLP, and Mamba components or combine them with convolution for vision tasks.These approaches generally construct hybrid networks rather than applying Mamba as a modulation adapter within pre-trained IFMs.
3. Method
MoMa adapts image foundation models for video through a Divide-and-Modulate architecture that combines local spatial attention with Mamba-based full spatial-temporal modeling. Its SeqMod operation injects Mamba features as learnable scale and bias sequences while preserving the original IFM pathway and computational efficiency.
- Divide Stage: MoMa applies window-based local attention in a Divide stage, reducing attention overhead while retaining the video feature shape for subsequent processing.Each flattened window is processed independently with the pre-trained CLIP attention layer, then the outputs are concatenated.
- Modulate Stage: The Modulate stage uses Mamba’s SSM to capture full spatial-temporal interactions with linear-time complexity.The SSM layer models long-range dependencies and uses bidirectional scans across spatial and temporal dimensions.
- Sequence Modulation (SeqMod): SeqMod injects Mamba output into pre-trained Transformer features through learnable scale and bias sequences rather than directly replacing IFM features.This sequence-to-sequence modulation extends scalar adaptive-normalization parameters into tensors matching the input sequence and retains a skip connection.
- SSM Forwarding Layer: The SSM forwarding layer doubles its output projection channels and splits them into two sequences that provide modulation parameters for the attention output.The two sequences are learned from each Divide-stage output and are used in the subsequent SeqMod operation.
- Sequence Modulation (SeqMod): MoMa avoids the suboptimal fusion of weighted averaging, max pooling, or channel concatenation by minimally disrupting frozen Transformer features while integrating Mamba information.The design balances IFM pre-training benefits with Mamba’s linear-complexity processing, whereas direct Mamba insertion may confuse a frozen Transformer.
4. Experiments
MoMa is evaluated across standard, long-video, zero-shot, efficiency, and ablation settings, with results showing strong recognition performance alongside reduced computational demands. The experiments also identify SeqMod, windowed Divide, and alternating Transformer–Mamba layers as important design choices.
- 4.1. Standard Video Recognition Benchmarks: MoMa achieves the best performance on K400 and SSv2 while using substantially fewer FLOPs than AIM and DiST.The method uses 25.6% fewer FLOPs than AIM and 8.5% fewer than DiST, while requiring fewer trainable parameters than most approaches.
- 4.2. Long-term Video Recognition Benchmark: MoMa surpasses non-end-to-end methods on Breakfast and COIN, and PEFT on CLIP also outperforms VideoMamba for long-video recognition.These results support the method’s effectiveness for long-term video understanding under the reported evaluation setting.
- 4.3. Zero-shot Transfer: MoMa surpasses DiST on both HMDB51 and UCF101 in zero-shot transfer evaluation.The models are evaluated using the zero-shot setting based on K400-trained models.
- 4.4. Ablation Studies: SeqMod outperforms Skip, Add, Max, and Concat, while direct fusion methods can produce feature confusion or interfere with pre-trained representations.Raw-AdaN’s soft gating motivates SeqMod by making sequence changes orthogonal to the original CLIP feature outputs.
- 4.4. Ablation Studies: Uniformly alternating Transformer and Mamba layers performs best, while adapting only later layers or using [TTM]6 is sub-optimal.The ablation is conducted on Kinetics-400 with a CLIP ViT-B/16 backbone and 16 input frames unless otherwise specified.
- 4.5. Speed Comparison: As input frames increase, MoMa maintains stable memory growth and gradual speed decline, while UMT eventually runs out of memory at 32 frames.The SSM module is also described as more parameter-efficient than attention, supporting the overall efficiency comparison.
5. Conclusion
MoMa adapts image foundation models for video by injecting spatial-temporal information without disrupting pre-trained features. Its Divide-and-Modulate design combines local attention with modulation, improving performance and efficiency across multiple video benchmarks.
- SeqMod injects spatial-temporal information into IFM features without interfering with their pre-trained parameters.
- The Divide-and-Modulate strategy applies local attention within video frames before modulation captures full spatial-temporal dynamics.
- MoMa improves performance and efficiency over existing methods across multiple video understanding benchmarks.
Impact Statement
The method is intended to improve spatial-temporal modeling efficiency for pre-trained image foundation models in video understanding. Potential applications include video analytics, while privacy and ethical considerations remain relevant.
- The work targets improved spatial-temporal modeling efficiency for pre-trained image foundation models.
- Potential application areas include video analytics in healthcare, security, and autonomous systems.
- The statement identifies privacy concerns and ethical considerations as broad societal implications of video-based AI.