Source-linked AI summary
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
Shuyang Sun, Zhanghui Kuang, Wanli Ouyang, Lu Sheng, Wei Zhang
TL;DR
Video action recognition needs effective temporal representations, but optical flow is computationally expensive and alternatives may be less effective. The paper introduces OFF, a differentiable optical-flow-guided representation built from CNN feature gradients and frame differences. OFF achieves competitive RGB-only recognition with substantially higher efficiency and also improves a two-stream framework across UCF-101 and HMDB-51.
Problem
Optical flow is useful for action recognition but inefficient to extract, while other motion representations can be less effective or fail to match two-stream performance.
Method
OFF uses differentiable spatial and temporal gradients of CNN feature maps, including differences between frames, to provide an optical-flow-guided motion representation.
Results
93.3% accuracy on UCF-101 is achieved with RGB-only input, while integrating OFF into a two-stream framework yields 96.0% on UCF-101 and 74.2% on HMDB-51.
Takeaways & Limitations
OFF provides a fast, robust motion representation that captures spatial and temporal information and complements optical flow in CNN-based action recognition.
Takeaways & Limitations
The framework costs more memory than the original TSN framework, so it samples fewer frames during training than testing to reduce computational and memory costs.
Abstract
from arXiv · showhide
Motion representation plays a vital role in human action recognition in videos. In this study, we introduce a novel compact motion representation for video action recognition, named Optical Flow guided Feature (OFF), which enables the network to distill temporal information through a fast and robust approach. The OFF is derived from the definition of optical flow and is orthogonal to the optical flow. The derivation also provides theoretical support for using the difference between two frames. By directly calculating pixel-wise spatiotemporal gradients of the deep feature maps, the OFF could be embedded in any existing CNN based video action recognition framework with only a slight additional cost. It enables the CNN to extract spatiotemporal information, especially the temporal information between frames simultaneously. This simple but powerful idea is validated by experimental results. The network with OFF fed only by RGB inputs achieves a competitive accuracy of 93.3% on UCF-101, which is comparable with the result obtained by two streams (RGB and optical flow), but is 15 times faster in speed. Experimental results also show that OFF is complementary to other motion modalities such as optical flow. When the proposed method is plugged into the state-of-the-art video action recognition framework, it has 96:0% and 74:2% accuracy on UCF-101 and HMDB-51 respectively. The code for this project is available at https://github.com/kevin-ssy/Optical-Flow-Guided-Feature.
1. Introduction
Video action recognition depends on temporal information, but dense optical flow is costly and alternative motion representations can be less effective. The paper proposes OFF, a fast, robust, end-to-end feature representation that captures spatial and temporal information from CNN features.
- 1. Introduction: Dense optical flow costs over 90% of runtime in two-stream pipelines, while RGB-based 3D CNNs do not match two-stream performance.
- 1. Introduction: OFF addresses the need for economical, motion-guided representation by comprehensively representing motion dynamics in video clips.
- 1. Introduction: OFF combines horizontal and vertical spatial gradients with temporal gradients from feature differences across frames.
- 1. Introduction: Because its pixel-wise operations are differentiable, OFF can be trained end-to-end and enables joint learning of spatial and motion representations without precomputed motion modalities.
- 1. Introduction: 93.3% accuracy on UCF-101 was achieved using only RGB input, while the OFF-equipped network runs at over 200 frames per second.
- 1. Introduction: 96.0% on UCF-101 and 74.2% on HMDB-51 were obtained when OFF was plugged into a two-stream framework.
2. Related Work
Prior action-recognition methods use hand-crafted or learned motion representations, but many depend on precomputed optical flow or struggle to match its effectiveness. OFF instead derives a fast, optical-flow-guided representation and remains complementary to other temporal methods.
- 2. Related Work: Traditional approaches encoded hand-crafted motion features such as 3DHOG, MBH, improved Dense Trajectory, and motion vectors for classification.
- 2. Related Work: Two-stream frameworks improved accuracy with optical flow but remained constrained by the need to precompute motion inputs.
- 2. Related Work: Methods using training-only optical flow or motion vectors reduced computation but produced degraded results and did not match traditional optical-flow approaches.
- 2. Related Work: Unlike 3D CNNs that learn motion filters, OFF is mathematically derived from optical flow and enables RGB performance on par with two-stream methods.
- 2. Related Work: OFF can complement optical flow and LSTM-based temporal modeling rather than replacing those motion or sequence representations.
- 2. Related Work: Dynamic images summarize a series of frames, whereas OFF is designed to capture motion information related to optical flow.
3. Optical Flow Guided Feature
OFF generalizes optical-flow constraints from pixels to deep features, combining spatial and temporal gradients into a representation guided by feature-level motion.
- Definition and derivation: The feature-level brightness constraint assumes feature values remain unchanged as a moving point shifts across space and time.This generalizes the optical-flow brightness-constant constraint to any differentiable feature-extraction mapping.
- Definition and derivation: OFF comprises horizontal and vertical spatial gradients plus a temporal gradient formed from feature differences across frames.The temporal gradient is explicitly identified with the difference between feature maps from different times.
- Definition and derivation: OFF extends the optical-flow representation from pixel intensity I(p) to differentiable feature maps f(I; w).The feature-level velocity is called feature flow, and the resulting OFF is orthogonal and complementary to it.
- Network overview: Figure 3 processes two segments through separate feature-generation branches before OFF units and residual blocks refine the resulting representations.The residual blocks between resolution levels form a ResNet-20 when considered together, with independent supervision for subnetwork scores.
4. Using Optical Flow Guided Feature in Convolutional Neural Network
The CNN implementation generates OFF from multi-level deep features using channel reduction, spatial Sobel gradients, temporal subtraction, residual refinement, and independently supervised classification outputs.
- Network architecture: The architecture has feature-generation, OFF, and classification subnetworks that produce basic features, refined OFF features, and classification scores.The feature-generation backbone is replaceable; the described experiment uses BN-Inception.
- OFF units: Each OFF unit reduces feature channels with 1×1 convolutions, computes spatial and temporal gradients, and concatenates OFF with lower-level features.Spatial gradients use Sobel operators, while temporal gradients use element-wise subtraction.
- OFF units: OFF units operate at multiple CNN levels, using previous semantic-level outputs to refine OFF representations at the current resolution.This multi-level design receives basic features from two segments and, when available, the preceding OFF-unit feature.
- Training and classification: Classification scores from sampled segments are averaged over time, while feature-level subnetworks can receive independent intermediate supervision.The loss is standard categorical cross-entropy, with a separate loss for each supervised feature level.
- Training and classification: Training uses a two-stage strategy: first train feature generation, then train OFF and classification subnetworks with feature-generation weights frozen.Intermediate supervision is added at each OFF level, and the network may subsequently be fine-tuned end-to-end.
- Training and classification: The framework reduces training computation and memory by sampling fewer frames during training than testing, while preserving satisfactory results.This introduces a possible mismatch in the segment time interval Δt between training and testing, which can invalidate the derivation and affect performance.
5. Experiments and Evaluations
Experiments evaluate OFF across datasets, modalities, efficiency settings, and comparisons with alternative architectures and state-of-the-art methods. OFF achieves competitive accuracy with high processing speed, improves fused modalities, and outperforms the hypercolumn comparison.
- Datasets and implementation: Experiments use UCF-101 and HMDB-51, following three official training/testing splits and averaging accuracy across splits.The evaluation includes modality, efficiency, robustness, hypercolumn, and state-of-the-art comparisons.
- Experimental investigations on OFF: 4.5% improvement is obtained when OFF(RGB) is fused with RGB on UCF-101 Split1.Table 2 compares different input modalities and reports speed as network-forward time.
- Efficiency evaluation: OFF(RGB) reaches 93.3% accuracy on UCF-101 over three splits while running at over 200 fps with RGB inputs.The result is comparable with some two-stream methods, whereas the compared alternatives are described as inefficient or less effective.
- Hypercolumn comparison: OFF(RGB) performs better than the hypercolumn network on UCF-101 Split1 despite the hypercolumn network’s slight 0.5% improvement.The comparison keeps the architecture and training strategies the same except for removing the OFF unit from the hypercolumn network.
- Comparison with the state-of-the-art: 96.0% accuracy on UCF-101 and 74.2% on HMDB-51 are obtained by fusing RGB, OFF(RGB), optical flow, corresponding OFF(optical flow), and OFF(RGB Diff).Adding OFF(RGB Diff) provides a further 0.3% gain; the final version prepares only RGB and optical flow in advance.
- Comparison with the state-of-the-art: The final method improves over baseline Two-Stream TSN by 2.0% on UCF-101 and 5.7% on HMDB-51.The compared final-version TSN uses three modalities: RGB, optical flow, and improved Dense Trajectories.
6. Conclusion
OFF is a fast, robust motion representation that achieves competitive RGB-only recognition while remaining efficient and complementary to optical flow. The resulting architecture outperforms many state-of-the-art methods on UCF-101 and HMDB-51.
- OFF enables RGB-only recognition comparable to Two-Stream approaches while operating at over 200 frames per second.The OFF-plugged network remains efficient while matching the reported Two-Stream comparison on UCF-101.
- OFF remains complementary to other motion representations, including optical flow.
- The proposed CNN architecture outperforms many state-of-the-art video action recognition methods on UCF-101 and HMDB-51.
- The authors identify validating OFF on other video tasks and datasets as future work.