Source-linked AI summary
Real-time Action Recognition with Enhanced Motion Vector CNNs
Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, Hanli Wang
TL;DR
Two-stream action recognition is accurate but not real-time because optical-flow computation is expensive, while motion vectors are cheaper but coarse and noisy. The paper replaces optical flow with motion vectors and transfers optical-flow CNN knowledge during training, achieving comparable performance with substantially higher processing speed.
Problem
Optical-flow computation makes accurate two-stream CNN action recognition too slow for real-time use, while motion vectors introduce coarse, noisy, and inaccurate motion information.
Method
The method uses motion vectors as the temporal CNN input and transfers knowledge from a frozen optical-flow CNN teacher to a motion-vector CNN during training.
Results
391 fps on UCF101 and 403 fps on THUMOS14 are achieved with high performance, comparable to the state-of-the-art two-stream approach.
Takeaways & Limitations
Optical flow is needed only during training, while testing uses motion vectors solely, so the proposed transfer strategy does not affect system speed at test time.
Takeaways & Limitations
Motion vectors remain noisy and inaccurate because compression algorithms balance encoding speed and compression rate.
Abstract
from arXiv · showhide
The deep two-stream architecture exhibited excellent performance on video based action recognition. The most computationally expensive step in this approach comes from the calculation of optical flow which prevents it to be real-time. This paper accelerates this architecture by replacing optical flow with motion vector which can be obtained directly from compressed videos without extra calculation. However, motion vector lacks fine structures, and contains noisy and inaccurate motion patterns, leading to the evident degradation of recognition performance. Our key insight for relieving this problem is that optical flow and motion vector are inherent correlated. Transferring the knowledge learned with optical flow CNN to motion vector CNN can significantly boost the performance of the latter. Specifically, we introduce three strategies for this, initialization transfer, supervision transfer and their combination. Experimental results show that our method achieves comparable recognition performance to the state-of-the-art, while our method can process 390.7 frames per second, which is 27 times faster than the original two-stream method.
1. Introduction
Two-stream CNNs achieve strong action-recognition performance but are not real-time because optical-flow computation is expensive. The paper replaces optical flow with motion vectors and transfers optical-flow knowledge to recover recognition performance.
- Two-stream CNNs use RGB and optical-flow CNNs and achieve state-of-the-art performance on several large action datasets.
- 60 ms per frame is required to calculate optical flow on a K40 GPU, preventing real-time processing.
- Motion vectors can be decoded directly from compressed videos with very low computational cost, avoiding extra optical-flow calculation.
- Motion vectors are coarser, noisier, and less precise than optical flow, so directly substituting them severely degrades CNN recognition performance.
- The method transfers fine features learned by an optical-flow CNN teacher to a motion-vector CNN student, using optical flow only during training.
- The paper introduces initialization transfer, supervision transfer, and their combination to improve motion-vector CNN recognition.
2. Related Work
Action recognition research progressed from hand-crafted spatio-temporal descriptors and Bag-of-Visual-Words representations toward deep neural networks and CNN-based video classification.
- Early action-recognition methods extracted local spatio-temporal descriptors and encoded them with Bag of Visual Words or related representations.
- Subsequent approaches modeled temporal relationships using spatio-temporal interest points and trajectories.
- Deep neural networks learned powerful features from large-scale image datasets, reducing reliance on hand-crafted feature design.
- CNNs were extended from image classification to video classification tasks, motivating deep action-recognition architectures.
3. Motion Vector for Deep Action Recognition
The paper uses motion vectors as a low-cost temporal input for a two-stream-style CNN, addressing their coarse, noisy, and incomplete motion information through training and architectural choices.
- 360 ms with an efficient implementation and 60 ms with GPU acceleration illustrate that optical-flow computation is the main two-stream bottleneck.
- Motion vectors describe local block offsets and can be obtained directly during video decoding without extra calculation.
- Motion vectors have low resolution and imprecise fine-motion descriptions, which can harm CNN recognition when networks are trained directly on them.
- 3.1. Motion Vector: Unlike pixel-level optical flow, motion vectors provide block-level motion and lose fine details such as bow structure or a baseball hold.
- 3.1. Motion Vector: Compression-oriented motion estimation introduces noisy and inaccurate movement patterns that hamper temporal-CNN performance.
- 3.1. Motion Vector: I-frames contain no movement information, so the method replaces each empty I-frame with motion vectors from the previous frame.
- 3.2. Real-time Action Recognition Frameworks: The framework decodes RGB images and motion vectors, then feeds them to spatial and temporal CNNs modeled on the two-stream architecture.
4. Enhanced Motion Vector CNNs
The paper enhances motion vector CNNs by transferring optical-flow CNN knowledge through teacher initialization, supervision transfer, or their combination. These strategies retain motion vectors’ speed advantage while improving recognition performance.
- Motivation: Motion vectors directly replace optical flow but cause around 7% recognition-performance degradation because they lack fine details and contain noisy, inaccurate motion patterns.Motion vectors are designed for compression rather than precise motion representation.
- Knowledge Transfer: Optical-flow and motion-vector CNNs are treated as teacher and student networks because their inputs both describe local motion information.The teacher is used only during training, so optical flow is unnecessary during testing.
- Teacher Initialization: Teacher initialization copies optical-flow CNN parameters into the motion-vector CNN before fine-tuning on motion-vector samples.The paper assumes both CNNs have the same network structure, although the techniques can generalize to different structures.
- Supervision Transfer: Supervision transfer trains the motion-vector CNN with both teacher supervision and ground-truth supervision, while freezing the teacher weights.The teacher supervision uses the optical-flow CNN’s FC-layer representation and a weighted combination with ground-truth loss.
- Combination: The combined strategy initializes the motion-vector CNN with teacher parameters and then trains it to mimic teacher predictions alongside ground-truth labels.This combines inherited parameters with prediction-based supervision during fine-tuning.
5. Experiment
Experiments evaluate the proposed CNNs on UCF101 and THUMOS14, comparing training strategies, accuracy, speed, and competing methods. Knowledge transfer improves motion-vector recognition, while the resulting system achieves high throughput with performance comparable to two-stream CNNs.
- Datasets and evaluation: Experiments use UCF101 with average accuracy over three splits and THUMOS14 with testing mAP on untrimmed videos.THUMOS14 testing samples CNNs every 20 frames because its videos are untrimmed and contain many frames.
- Training strategies: 74.4% and 29.8% are achieved by MV-CNN trained from scratch on UCF101 Split1 and THUMOS14, respectively, representing 7% and 25% degradation compared with OF-CNN.The larger THUMOS14 degradation is attributed to untrimmed videos, incorrect labels for many frames, and shot shifts.
- Training strategies: 4.9% and 12% performance improvements are obtained by EMV-CNN over MV-CNN on UCF101 Split1 and THUMOS14, respectively.The results support the effectiveness of the proposed EMV-CNN training method and its ability to improve MV-CNN generalization.
- Training strategies: 3.8% improvement comes from teacher initialization, while supervision transfer provides 3.1% improvement on UCF101 Split1.Combining the strategies further boosts performance; soft targets from OF-CNN provide richer training information than hard labels.
- Comparison with the state of the art: 1 crop is four times faster than 5 crops+mirror with only slight performance degradation, and the method is 27 times faster than two-stream CNNs while achieving comparable results.On UCF101, the method also achieves higher accuracy than C3D (1 net) and C3D (3 net).
- Speed evaluation: 390.7 fps and 403.2 fps are achieved on UCF101 and THUMOS14, respectively, exceeding the 25 fps real-time requirement by one order of magnitude.Motion-vector extraction is also substantially faster than optical-flow computation, which is identified as the main two-stream bottleneck.
6. Conclusions
The paper proposes motion vector CNNs to accelerate deep-learning action recognition and uses knowledge transfer to address their less precise motion information. It reports high-performance processing at 391 fps on UCF101 and 403 fps on THUMOS14.
- Motion vectors can be extracted during video decoding without extra computation, enabling faster action recognition.
- Three knowledge transfer techniques adapt optical flow CNN models to motion vector CNNs and significantly improve recognition performance.
- 391 fps on UCF101 and 403 fps on THUMOS14 are achieved with high performance.