Source-linked AI summary
MotionSqueeze: Neural Motion Feature Learning for Video Understanding
Heeseung Kwon, Manjin Kim, Suha Kwak, Minsu Cho
TL;DR
Video recognition needs motion information, but external optical-flow extraction creates a major computational bottleneck. MotionSqueeze learns frame correspondences internally through a lightweight trainable module, and the resulting system improves benchmark performance with small overhead, including state-of-the-art results on Something-Something-V1&V2.
Problem
External dense optical-flow extraction is computationally expensive, and separate motion streams can substantially increase video-model cost.
Method
MotionSqueeze is an end-to-end trainable module that learns correspondences across adjacent frames and converts them into motion features within a video network.
Results
The method outperforms existing state-of-the-art methods on Something-Something-V1&V2 for video classification with only a small amount of additional cost.
Takeaways & Limitations
Motion features can be learned on the fly through a lightweight module that can be inserted into existing video architectures.
Takeaways & Limitations
Estimated displacement maps are often inaccurate around regions of occlusion or severe deformation.
Abstract
from arXiv · showhide
Motion plays a crucial role in understanding videos and most state-of-the-art neural models for video classification incorporate motion information typically using optical flows extracted by a separate off-the-shelf method. As the frame-by-frame optical flows require heavy computation, incorporating motion information has remained a major computational bottleneck for video understanding. In this work, we replace external and heavy computation of optical flows with internal and light-weight learning of motion features. We propose a trainable neural module, dubbed MotionSqueeze, for effective motion feature extraction. Inserted in the middle of any neural network, it learns to establish correspondences across frames and convert them into motion features, which are readily fed to the next downstream layer for better prediction. We demonstrate that the proposed method provides a significant gain on four standard benchmarks for action recognition with only a small amount of additional cost, outperforming the state of the art on Something-Something-V1&V2 datasets.
1 Introduction
Video understanding requires motion modeling, but conventional optical-flow pipelines impose substantial computational and architectural costs. MotionSqueeze addresses this bottleneck with a trainable module that learns frame correspondences and offers a favorable accuracy–efficiency trade-off.
- 1 Introduction: Motion is essential for video understanding, yet spatio-temporal convolution alone does not adequately model relative object movement.CNNs capture appearance and translation-equivariant patterns, but motion patterns require analyzing relative movement across frames.
- 1 Introduction: On Something-Something-V1, MSNet achieves the best reported trade-off among accuracy, computational cost, and model size against several state-of-the-art methods.The comparison includes TSM, TRN, ECO, I3D, NL-I3D, and GCN.
- 1 Introduction: Dense optical flows remain common but are costly because frame-by-frame extraction is slow and often requires a separate motion stream.Optical-flow extraction is typically an order of magnitude slower than neural-network feed-forwarding, while a separate stream increases parameters and computation.
- 1 Introduction: MotionSqueeze is an end-to-end trainable module that establishes correspondences across adjacent frames and converts them into motion features for downstream prediction.The module is designed to be inserted within neural networks for video understanding.
2 Related work
Prior video-recognition methods commonly rely on precomputed optical flow or internal correspondence modeling, each with efficiency or performance limitations. This work contributes a lightweight, model-agnostic motion module and evaluates it across four benchmark datasets.
- Video classification architectures: Video architectures have used 3D convolutions or separate optical-flow streams to learn spatio-temporal and motion features.Two-stream 3D CNNs combine spatio-temporal processing with motion representations from precomputed optical flows.
- Learning motions in a video: Optical-flow-based approaches improve motion modeling but reduce processing efficiency, while some alternatives still require flow during training or impose substantial architectural cost.Prior methods include multitask flow estimation, motion distillation, and internal motion modeling.
- Learning visual correspondences: Neural correspondence methods construct correlations between feature maps and estimate displacements, but many require dense flow supervision or target related correspondence problems.Correlation computation and displacement estimation provide the conceptual basis for learning visual correspondences.
- Learning visual correspondences: Existing correspondence-based video methods either have model sizes comparable to two-stream networks or focus on spatio-temporal relationships rather than the same motion-feature objective.These differences motivate a lightweight alternative for video understanding.
- Contributions: The paper proposes an end-to-end trainable, model-agnostic, lightweight motion extractor and an efficient recognition architecture, evaluated on four benchmarks.The authors report state-of-the-art results on Something-Something-V1 and V2.
3 Proposed approach
MotionSqueeze is a lightweight, trainable module that extracts motion from adjacent-frame appearance features and integrates it into video networks. It computes correlations, estimates displacements and confidence, transforms them into motion features, and fuses them with appearance features for prediction.
- 3 Proposed approach: MotionSqueeze extracts motion features from adjacent-frame appearance features through correlation computation, displacement estimation, and feature transformation.The module is inserted into a neural network and produces motion features for downstream layers.
- 3 Proposed approach: The module computes local correlations between adjacent feature maps, restricting candidate displacements to a neighborhood with P = 2k + 1 positions per dimension.The correlation tensor has size H × W × P^2, and its computation costs THWCP^2 FLOPs per video.
- 3 Proposed approach: Kernel-soft-argmax converts the correlation tensor into a differentiable displacement estimate while reducing sensitivity to noisy correlation outliers.A Gaussian kernel emphasizes closer neighbors during displacement estimation.
- 3 Proposed approach: The module augments displacement information with a pooled correlation confidence map before transforming both into motion features.The displacement and confidence maps form a 3-channel tensor D(t), and the confidence map can help identify displacement outliers.
- 3 Proposed approach: Depth-wise separable convolutions transform D(t) into a motion feature with the same channel count as the input, enabling lower-cost processing.The transformation uses one 1 × 7 × 7 layer followed by three 1 × 3 × 3 layers.
- 3 Proposed approach: MSNet inserts MotionSqueeze into a ResNet-based video architecture and fuses motion with appearance by element-wise addition, jointly learning both at modest overhead.The default design increases FLOPs by 2.5% and parameters by 1.2%.
4 Experiments
Experiments show that MotionSqueeze improves action-recognition accuracy across datasets and backbones while adding limited computational cost. Ablations identify effective displacement estimation, module placement, and feature fusion choices.
- Comparison with other motion representation methods: The proposed 16-frame model outperforms conventional CNNs and motion-representation methods while remaining competitive with the R(2+1)D two-stream model.It is also reported as highly efficient in FLOPs, clips, and number of frames.
- Ablation studies: Placement at res3 performs best because earlier features are weak for matching and later features have insufficient spatial resolution, while multiple modules add only 0.2 points.The computational cost increases quadratically with spatial resolution because of feature-transformation convolutions.
- Ablation studies: Additive fusion achieves the best ablation accuracy at 45.5% top-1, while the kernel-soft-argmax displacement estimator outperforms soft-argmax.The authors attribute additive fusion's effectiveness to amplifying appearance features of moving objects, and report performance saturation after P = 15.
- Ablation studies: The MS module improves ResNet-18, MobileNet-V2, I3D, and TSM ResNet-18 by 21.3%, 19.2%, 2.4%, and 4.0% top-1 points, respectively.The gains on 3D CNNs indicate that its explicit adjacent-frame motion features are complementary to spatio-temporal features.
- Comparison with other motion representation methods: 0.8% higher top-1 accuracy and about two times fewer FLOPs than the fair two-stream comparison demonstrate improved efficiency without optical flow.The standard two-stream variant with stacked optical flows is more accurate but requires substantially more computation and external flow extraction.
5 Conclusion
The MS module generates motion features on the fly, can be inserted into existing video architectures, and improves video classification with small additional cost.
- The MS module outperforms existing state-of-the-art methods on Something-Something-V1&V2 for video classification with only a small amount of additional cost.
- The module is trainable by backpropagation and readily insertable into existing video architectures.
- Ablation studies demonstrate effectiveness in accuracy, computational cost, and model size.
Supplementary Material of “MotionSqueeze: Neural Motion Feature Learning for Video Understanding”
The supplementary material provides omitted results and details and states that the project code and data are released online.
- The supplementary material provides additional results and details omitted from the main paper because of space limitations.
- The authors state that all code and data are released online at the project page.
1 Effects of depth-wise separable (DWS) convolutions
The feature-transformation layers use depth-wise separable convolutions to reduce computational cost while supporting deeper and wider layers, achieving the best accuracy-FLOPs tradeoff.
- DWS convolutions achieve the best accuracy-FLOPs tradeoff among the evaluated feature-transformation layers.
- Accuracy increases when the feature-transformation layers become deeper and have wider receptive fields.
- DWS convolutions enable deeper and wider feature-transformation layers while saving computational cost.
2 Comparison with the CP module [22]
The supplementary comparison distinguishes MS from CP in temporal scope and differentiability, while reporting a more efficient MS result under comparison settings.
- MS extracts motion features across consecutive frames, whereas CP captures long-term spatio-temporal relationships across the whole video.
- CP selects top-k correspondences with a non-differentiable operation, while the MS module is designed for effective motion-feature extraction.
- The supplementary material includes a performance-comparison table between the CP and MS modules.
- The MS module achieves higher accuracy than the CP module while consuming 26% less FLOPs.
3 Backbone architectures in experiments
The experiments evaluate MotionSqueeze with ResNet, TSM ResNet, MobileNet-V2, and I3D backbones, inserting one MS module after res3 by default for ResNet-based architectures.
- Experiments cover ResNet, TSM ResNet, MobileNet-V2, and I3D backbone architectures.
- ResNet & TSM ResNet: For ResNet and TSM ResNet, one MS module is inserted immediately after res3 by default.
4 Additional examples of visualization
Additional visualizations show displacement and confidence maps for varied motions across Something-Something V1 and Kinetics-400, while failure cases occur around occlusions or severe deformation.
- Visualizations cover horizontal and vertical movements, rotations, scale changes, and deformations across Something-Something V1 and Kinetics-400.
- The figures display RGB frames, color-coded displacement maps, and confidence maps from the top of each figure.
- Estimated displacement maps are often inaccurate around regions of occlusion or severe deformation.
- The MobileNet-V2 backbone uses bottleneck modules, while the supplementary figures identify the I3D backbone architecture.