Source-linked AI summary
STM: SpatioTemporal and Motion Encoding for Action Recognition
Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, Junjie Yan
TL;DR
Action recognition requires complementary spatiotemporal and motion features, while existing 3D CNN and flow-stream approaches can be computationally expensive. STM encodes both features with CSTM and CMM in lightweight 2D ResNet blocks, and experiments show state-of-the-art performance across temporal- and scene-related datasets.
Problem
Existing approaches capture complementary spatiotemporal and motion features using costly 3D CNN and optical-flow streams.
Method
STM combines Channel-wise SpatioTemporal and Channel-wise Motion Modules in replaceable ResNet blocks within a unified 2D CNN framework.
Results
STM outperforms state-of-the-art methods on temporal-related and scene-related action-recognition benchmarks.
Takeaways & Limitations
STM encodes spatiotemporal and motion features together without 3D convolution or pre-calculated optical flow, with 1.2% more FLOPs than the TSN baseline.
Abstract
from arXiv · showhide
Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose an STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in the ResNet architecture with STM blcoks to form a simple yet effective STM network by introducing very limited extra computation cost. Extensive experiments demonstrate that the proposed STM network outperforms the state-of-the-art methods on both temporal-related datasets (i.e., Something-Something v1 & v2 and Jester) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51) with the help of encoding spatiotemporal and motion features together.
1. Introduction
Action recognition depends on both spatiotemporal and motion features, but existing 3D CNN and two-stream approaches incur substantial computational costs. STM integrates both feature types in a unified 2D framework through lightweight STM blocks and achieves state-of-the-art results across multiple benchmarks.
- Spatiotemporal and motion features are identified as the most important information for video action recognition.
- Two-stream methods use RGB and optical-flow streams, but optical flow captures only neighboring-frame motion and lacks long-range temporal modeling.
- 3D CNNs capture long-range spatiotemporal relationships, yet combining them with flow streams increases computation by an order of magnitude.
- STM combines a Channel-wise SpatioTemporal Module and a Channel-wise Motion Module in a unified 2D CNN framework without 3D convolution or optical-flow pre-calculation.
- STM blocks replace residual blocks in existing ResNet architectures while introducing very limited extra computation cost.
- STM outperforms state-of-the-art methods on Something-Something, Kinetics, Jester, UCF101, and HMDB-51 benchmarks.
2. Related Works
Related work includes 2D CNNs, 3D CNNs, and efficiency-oriented spatiotemporal convolution methods. These approaches improve temporal modeling or reduce computation, but commonly retain flow streams to capture motion features and remain costly for deployment.
- Frame-aggregation methods based on 2D CNNs model appearance independently and ignore dynamics between frames.
- 3D CNN methods learn spatiotemporal features directly from RGB frames, with I3D additionally modeling motion through a flow stream.
- Despite strong benchmark performance, 3D CNN methods remain computationally heavy for real-world deployment.
- Methods such as P3D, R(2+1)D, and TSM reduce 3D convolution costs through decomposition or temporal channel shifting.
- Efficiency-oriented spatiotemporal methods still use flow streams to incorporate motion features and achieve their best performance.
3. Approach
STM combines channel-wise spatiotemporal fusion and feature-level motion extraction in a lightweight block that can replace ResNet residual blocks. Its 2D architecture avoids 3D convolution and optical-flow pre-computation while jointly encoding both feature types.
- CSTM: CSTM reshapes feature maps for channel-wise temporal fusion, then applies 3×3 spatial convolution to model local spatiotemporal information.Independent kernels are learned for each channel, reducing computation relative to ordinary convolution.
- CMM: CMM estimates motion between adjacent feature maps by applying channel-wise convolution to the later map and subtracting it from the earlier map.A 1×1 convolution first reduces channels by a factor of 16 in the reported experiments.
- CMM: CMM produces T −1 motion representations, pads the final time step with zeros, concatenates them temporally, and restores channel count with a 1×1 convolution.The module is applied to each adjacent pair of feature maps.
- STM block: The STM block reduces channels, processes the compressed maps through CSTM and CMM, and combines their spatiotemporal and motion features.This design encodes complementary feature types within one residual-compatible block.
- STM network: STM replaces ResNet residual blocks in a 2D ResNet-50 network, using very limited extra computation while avoiding 3D convolution and optical-flow pre-computation.The network samples one frame from each of N equally divided video segments and performs no temporal dimension reduction before final score fusion.
4. Experiments
Experiments evaluate STM across temporal-related and scene-related action-recognition datasets, using segment-based training, multi-crop inference, and comparisons with competing methods. STM improves over baselines and achieves strong results across both dataset categories while reducing computation relative to several alternatives.
- Datasets: The evaluation covers temporal-related datasets Something-Something v1/v2 and Jester, plus scene-related datasets Kinetics-400, UCF-101, and HMDB-51.Temporal-related datasets emphasize motion interactions, whereas scene-related datasets often benefit more from background information.
- Implementation: Training divides each video into T equal-duration segments and randomly samples one frame per segment to model long-range temporal structure.Frames use a fixed 256-pixel short side with corner cropping and scale-jittering for augmentation.
- Temporal-related results: 29.5% and 30.8% top-1 accuracy improvements over TSN are reported on Something-Something v1 with 8 and 16 input frames, respectively.On Something-Something v2, STM gains 34.5% and 34.2% over TSN with the same frame counts.
- Temporal-related results: STM achieves state-of-the-art performance on Something-Something v1 and v2 with 8 RGB frames, and the best validation performance with 16 frames.The 16-frame model uses only RGB frames while competing methods may use more frames, optical flow, or 3D CNN backbones.
- Scene-related results: On Kinetics-400, STM is 0.5% below two-stream I3D in top-1 accuracy, outperforms major 3D and 2D CNN methods, and achieves the best top-5 accuracy.STM also performs competitively on UCF-101 and HMDB-51; only two optical-flow-based methods perform slightly better, while STM surpasses I3D's RGB stream on UCF-101.
- Ablation studies: Element-wise summation of CSTM and CMM outperforms concatenation by 7.4% top-1 and 6.1% top-5 accuracy.The ablation also reports that replacing only one residual block with STM already improves significantly over TSN.
- Runtime analysis: STM uses approximately 10x and 2x fewer FLOPs than I3D and ECO while achieving 5.9% and 6.1% higher accuracy, respectively.Compared with TSM16F, STM8F has 2.7% higher accuracy, 1.5x faster speed, and half the FLOPs.
5. Conclusion
The STM network jointly encodes spatiotemporal and motion features in a unified 2D CNN, achieving state-of-the-art results with only 1.2% more FLOPs than the TSN baseline.
- STM replaces ResNet residual blocks with STM blocks containing CSTM and CMM modules for joint spatiotemporal and motion representation.The network avoids 3D convolution and pre-calculated optical flow.
- 1.2% more FLOPs than the TSN baseline accompanies state-of-the-art results on temporal-related and scene-related action-recognition datasets.