Source-linked AI summary
Compressed Video Action Recognition
Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, Philipp Krähenbühl
TL;DR
Deep video learning is hindered by raw-video redundancy and weak temporal information in RGB frames. This paper trains CNNs directly on compressed video, using I-frames, motion vectors, residuals, and back-tracing to model the representation. On action-recognition benchmarks, it outperforms RGB-based methods while being substantially faster than Res3D and ResNet-152.
Problem
Raw videos are enormous and temporally redundant, while RGB-only models have difficulty learning temporal structure for robust video representation.
Method
The model operates directly on compressed video, applying CNNs to I-frames, motion vectors, and residuals, with back-tracing to reduce P-frame dependency chains.
Results
The approach significantly outperforms traditional RGB methods on UCF-101, HMDB-51, and Charades, running 4.6 times faster than Res3D and 2.7 times faster than ResNet-152.
Takeaways & Limitations
Compressed video provides a denser representation with free motion information, and the proposed techniques make it effective for accurate, efficient action recognition.
Takeaways & Limitations
The study focuses on videos encoded using only backward references, namely I- and P-frames; arbitrary decoding order is beyond scope.
Abstract
from arXiv · showhide
Training robust deep video representations has proven to be much more challenging than learning deep image representations. This is in part due to the enormous size of raw video streams and the high temporal redundancy; the true and interesting signal is often drowned in too much irrelevant data. Motivated by that the superfluous information can be reduced by up to two orders of magnitude by video compression (using H.264, HEVC, etc.), we propose to train a deep network directly on the compressed video. This representation has a higher information density, and we found the training to be easier. In addition, the signals in a compressed video provide free, albeit noisy, motion information. We propose novel techniques to use them effectively. Our approach is about 4.6 times faster than Res3D and 2.7 times faster than ResNet-152. On the task of action recognition, our approach outperforms all the other methods on the UCF-101, HMDB-51, and Charades dataset.
1. Introduction
Video understanding remains difficult because raw videos contain massive redundancy and RGB frames provide limited temporal information. The paper therefore uses compressed video directly, exploiting its denser signals and motion information for faster, stronger action recognition.
- Video carries rich temporal and spatial information, but little of this information is currently exploited.
- Deep video models often process frames independently with image-designed CNNs, achieving results no better than hand-crafted techniques.
- Raw video has low information density: 1h of 720p footage can shrink from 222GB uncompressed to 1GB compressed.
- RGB-only learning struggles with temporal structure, while precomputed optical flow almost always improves performance.
- The proposed model operates directly on compressed video, using CNNs for motion vectors, residuals, and a small number of complete images.
- Compressed representations remove up to two orders of magnitude of superfluous information and provide motion signals that can simplify training and improve generalization.
- On UCF-101, HMDB-51, and Charades, the approach outperforms traditional RGB methods and runs 4.6 times faster than Res3D and 2.7 times faster than ResNet-152.
2. Background
Video compression stores complete reference frames and compact changes between frames, creating motion and residual signals that can support recognition. The paper builds on this structure while addressing noisy motion estimates and frame dependencies.
- Traditional action recognition uses hand-crafted features such as HOG, HOF, and dense trajectories, with iDT remaining competitive.
- Deep video improvements have largely come from image representations, while temporal modeling commonly relies on frame subsampling, pooling, RNNs, or temporal CNNs.
- Optical flow often boosts performance, highlighting both the importance of input representation and the limitations of RGB frames.
- Video Compression: Modern codecs exploit frame similarity by storing I-frames completely and encoding P-frames through motion vectors and residual differences.
- Video Compression: Motion vectors describe block movements, while residuals encode differences remaining after motion compensation; both are transformed and entropy-encoded.
- Video Compression: Accumulating motion vectors and residuals over longer temporal differences produces clearer, smoother patterns than the original signals.
- Video Compression: The paper focuses on I- and P-frames with backward references and leaves arbitrary decoding order involving B-frames beyond scope.
- Features from Compressed Data: Prior compressed-video work used non-deep features, whereas MV-CNN still requires decompressed RGB video and optical-flow supervision.
3. Modeling Compressed Representations
CoViAR models compressed-video components directly, using back-traced motion and residual signals to decouple P-frames from one another. Separate CNN streams process I-frames, motion vectors, and residuals, with lightweight P-frame updates improving computational efficiency.
- Compressed-video representation: CoViAR operates directly on stored compressed video rather than raw RGB frames.Its inputs include I-frames, motion vectors, and residuals.
- Decoupled model: Back-tracing traces motion vectors to the reference I-frame and accumulates residuals so each P-frame depends only on that I-frame.This removes the long dependency chain among consecutive P-frames.
- Decoupled model: The accumulated motion and residual signals can be computed in linear time while decoding the video.The method uses a simple feed-forward accumulation algorithm.
- Decoupled model: With P-frames depending only on the I-frame, they can be processed in parallel and the corresponding networks can be trained independently.Models are shared across P-frames in the decoupled model.
- Network design: The network uses CNNs for each input source, with I-frame features retained and P-frame motion and residual features fused with RGB information.Simple score summation performed best among the tested fusion strategies.
- Implementation: Computation concentrates on I-frames while smaller models capture P-frame updates, yielding savings because modern codecs contain mostly P-frames.The implementation uses ResNet-152 for I-frames and ResNet-18 for motion vectors and residuals.
4. Experiments
The experiments evaluate CoViAR for action recognition across three datasets and test its accuracy against RGB-based representations. Compressed-video training improves performance on UCF-101 and HMDB-51, while the evaluation protocol also covers the longer, multilabel Charades dataset.
- Experimental goals: The experiments validate compressed video as a representation for action recognition by measuring accuracy and speed.The study separately examines representation benefits, accuracy, and computational speed.
- Datasets and protocol: CoViAR is evaluated on UCF-101, HMDB-51, and Charades using established action-recognition protocols.UCF-101 and HMDB-51 contain short trimmed videos, whereas Charades contains longer untrimmed videos with one or more labels and temporal intervals.
- Datasets and protocol: The testing procedure samples 25 frames with flips and five crops, then averages scores for final prediction.Temporal-segment averaging is used for UCF-101 and HMDB-51; Charades is evaluated with mAP and wAP.
- Ablation study: 5.8% absolute improvement occurs on HMDB-51 and 2.7% on UCF-101 when training on compressed videos instead of RGB frames.Residuals alone achieve good accuracy, while motion vectors provide complementary information despite lacking spatial details.
- Ablation study: Using only I-frames as full images, CoViAR still achieves good performance while exploiting compressed representations.The reported analysis focuses on UCF-101 and HMDB-51.
I M R I+M I+R I+M+R (gain)
CoViAR combines I-frame RGB images, motion vectors, and residuals from compressed video, achieving strong action-recognition accuracy with substantially lower computational cost than RGB-based alternatives.
- I M R I+M I+R I+M+R (gain): 5.6% improvement is obtained on the motion-vector stream with accumulated motion vectors, while the full model improves by 0.4% and the residual stream by 0.9%.The full-model gain corresponds to a 4.2% error reduction, and the residual-stream gain to a 4.3% error reduction.
- Speed and accuracy: 4.6 times greater efficiency than Res3D and 2.7 times greater efficiency than ResNet-152 are reported while CoViAR remains significantly more accurate.The method shares I-frame computation across multiple frames and uses cheaper P-frame computation.
- Speed and accuracy: Optical-flow computation is the bottleneck for two-stream networks, whereas CoViAR’s CPU-only preprocessing is much faster.CoViAR preprocesses accumulated motion vectors and residuals instead of computing optical flow.
- Speed and accuracy: Up to 100 times faster than traditional methods, CoViAR reaches 1,300 frames per second with multi-thread preprocessing while requiring much less input data.Figure 6 summarizes the trade-off between efficiency, accuracy, and input data size.
- Accuracy: CoViAR outperforms RGB-based and optical-flow-based baselines on UCF-101, HMDB-51, and Charades, including when paired with a temporal-stream network.On Charades, it achieves the best performance without the additional annotations used by one comparison method.
5. Conclusion
The paper trains deep networks directly on compressed videos, treating compression as an advantage rather than a drawback. The resulting method is faster, more accurate, and simpler than previous approaches.
- The method trains deep networks directly on compressed videos instead of decompressed RGB frames.
- Compression removes irrelevant information, making compressed data more robust and allowing simpler networks for motion vectors and residuals.
- The method is faster, more accurate, and simpler to implement than previous works.
- In t-SNE space, two Jumping Jack videos overlap in motion-vector and residual representations but remain separated in RGB space.
Appendix A. RNN-Based Models
The appendix evaluates recurrent models for compressed video by replacing CoViAR’s accumulated representations and average pooling with original dependencies and Conv-LSTM aggregation. CoViAR achieves the best performance among the compared models.
- A recurrent compressed-video model uses a Conv-LSTM to model the recurrent definition of P-frames.
- The Conv-LSTM variant uses original T and Δ representations instead of accumulated D and R to preserve the original dependency.
- The Conv-LSTM aggregates CNN features instead of using average pooling.
- RGB features are reduced from 2048 to 512 dimensions so their dimensionality matches the P-frame features.
- One every two P-frames is subsampled to reduce sequence length because of memory constraints.
- CoViAR achieves the best performance, while the Conv-LSTM model still outperforms traditional RGB-based methods.
Appendix B. Feature Fusion
The appendix compares several ways to combine P-frame and I-frame features. Late fusion performs best for CoViAR and enables a decoupled model that is easier to train.
- The experiments compare maximum, mean, multiplicative, concatenation, and late fusion of P-frame and I-frame features.
- For maximum, mean, and multiplicative fusion, a 1 × 1 convolution matches I-frame feature-map dimensionality to P-frame features.
- Late fusion works best for CoViAR.
- Late fusion allows training a decoupled model, whereas the other fusion methods require jointly training multiple CNNs.
Appendix C. CoViAR without Temporal Segments
CoViAR remains effective without temporal segments, significantly outperforming RGB-only models including ResNet-152 and Res3D.
- CoViAR significantly outperforms RGB-only models even without temporal segments.
- 83.4% is reported for ResNet-152 in ST-Mult, compared with 84.7% using the authors’ implementation.
- 85.8% is reported for Res3D in the comparison.
I M R I+M I+R I+M+R
This section reports CoViAR’s accuracy without temporal segments on UCF-101 split 1.
- Table 10 reports CoViAR’s accuracy without temporal segments on UCF-101 split 1.
Appendix D. Confusion Matrix
The confusion matrices compare CoViAR with an RGB-only model on UCF-101. CoViAR corrects many RGB-model mistakes, including confusion between Cricket Bowling and Cricket Shot.
- CoViAR and the RGB-only model are compared using confusion matrices on UCF-101.
- CoViAR corrects many mistakes made by the RGB-based model, shown as purple off-diagonal blocks in the prediction-difference figure.
- CoViAR better distinguishes Cricket Bowling from Cricket Shot than the RGB-based model.