Source-linked AI summary
On the Integration of Optical Flow and Action Recognition
Laura Sevilla-Lara, Yiyi Liao, Fatma Guney, Varun Jampani, Andreas Geiger, Michael J. Black
TL;DR
The paper examines why optical flow helps action recognition and whether conventional flow accuracy predicts recognition performance. It compares flow methods and transformations, then fine-tunes neural flow networks for classification, finding that task-optimized motion improves recognition and differs from traditional flow around the human body and its boundaries.
Problem
Optical flow is commonly used as a black-box input to action recognition, but the reasons for its usefulness and the relevance of conventional EPE accuracy remain unclear.
Method
The authors evaluate multiple optical flow methods and input transformations with an action recognition system, and fine-tune FlowNet and SpyNet end-to-end using classification accuracy instead of EPE.
Results
Optical flow is useful because it is appearance-invariant; EPE is weakly correlated with recognition, while boundary and small-displacement accuracy is more relevant, and classification-trained flow improves recognition.
Takeaways & Limitations
The findings support integrating optical flow estimation with action recognition rather than optimizing the two modules independently.
Takeaways & Limitations
The experiments cannot determine whether the lack of correlation for large motions reflects poor flow estimates or the limited importance of large motions for human action recognition.
Abstract
from arXiv · showhide
Most of the top performing action recognition methods use optical flow as a "black box" input. Here we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better. In particular, we investigate the impact of different flow algorithms and input transformations to better understand how these affect a state-of-the-art action recognition method. Furthermore, we fine tune two neural-network flow methods end-to-end on the most widely used action recognition dataset (UCF101). Based on these experiments, we make the following five observations: 1) optical flow is useful for action recognition because it is invariant to appearance, 2) optical flow methods are optimized to minimize end-point-error (EPE), but the EPE of current methods is not well correlated with action recognition performance, 3) for the flow methods tested, accuracy at boundaries and at small displacements is most correlated with action recognition performance, 4) training optical flow to minimize classification error instead of minimizing EPE improves recognition performance, and 5) optical flow learned for the task of action recognition differs from traditional optical flow especially inside the human body and at the boundary of the body. These observations may encourage optical flow researchers to look beyond EPE as a goal and guide action recognition researchers to seek better motion cues, leading to a tighter integration of the optical flow and action recognition communities.
NVIDIA
Michael J. Black is affiliated with the MPI for Intelligent Systems.
- Michael J. Black is listed with the MPI for Intelligent Systems.
1. Introduction
The paper examines whether optical flow is necessary and how its properties affect action recognition, challenging the conventional modular combination of flow estimation and classification. Experiments show that appearance invariance, boundary and small-displacement accuracy, and task-specific training matter more than standard end-point error alone.
- 1. Introduction: Standard flow evaluation uses end-point-error, but current methods’ EPE is not well correlated with action-recognition accuracy.The paper tests several flow methods within a baseline recognition system and compares their classification accuracy with standard optical-flow accuracy.
- 1. Introduction: Optical flow is investigated as a video-classification feature even though many UCF101 categories can often be identified from a single frame.The paper frames this as a test of whether motion is necessary for recognizing actions involving human-object interactions.
- 1. Introduction: Shuffling flow fields in time causes only a small accuracy decrease, while shuffling frames before computing flow causes a larger decrease.These experiments suggest that motion trajectories are not the main information source, although adjacent-frame flow retains temporal information unless the frames are shuffled first.
- 1. Introduction: Altering appearance reduces optical-flow accuracy by 1% but raw-image accuracy by 50%, supporting appearance invariance as a key reason flow helps action recognition.The paper therefore distinguishes flow’s appearance invariance from motion trajectories as the source of its usefulness.
- 1. Introduction: End-to-end integration of optical-flow estimation and action recognition shows that minimizing classification error improves recognition and produces flow differing on the human body and near its boundaries.These findings challenge isolated, classical flow optimization and support a more integrated vision approach.
- 1. Introduction: For the tested methods, flow accuracy at boundaries and for small displacements matters most for action recognition.This identifies properties of flow estimation that are more relevant to recognition than optimizing EPE alone.
2. Related Work
Prior work treats optical flow and action recognition as separate modules, typically optimizing flow numerically before using it for classification. This paper instead focuses on their interaction and on training flow for recognition.
- Optical flow estimation: Optical flow research commonly minimizes end-point-error (EPE), the average Euclidean distance between estimated and ground-truth flow.EPE is treated as a standard numerical measure of flow accuracy.
- Learning flow for recognition: The authors retrain FlowNet and SpyNet using recognition loss and find motion representations that are better for action recognition.The improvement is consistent for both neural-network flow methods.
- Learning action recognition: Two-stream action recognition uses images in a spatial stream and optical flow fields in a temporal stream.The streams are processed by parallel networks and their predictions are combined for video classification.
- Learning action recognition: The paper uses Temporal Segment Networks as its baseline because they separately expose image and flow modalities.TSN samples snippets throughout each video and aggregates their predictions.
- Relation to prior work: Unlike related ActionFlowNet work, this paper studies why flow helps, which flow properties matter, and how recognition fine-tuning changes flow.Its experiments vary temporal structure and appearance, compare six flow methods, and analyze fine-tuned flow fields.
3. Why Use Optical Flow as Input for Video Classification?
The experiments test whether optical flow helps because of temporal trajectories or because it provides appearance-invariant information. Results favor appearance invariance: disrupting temporal structure reduces accuracy, but appearance changes strongly hurt image-based recognition while barely affecting flow-based recognition.
- Temporal structure: 78.64% accuracy remains after temporally shuffling flow fields, indicating coherent trajectories are not the most important feature in this setup.The result is reported for the TSN temporal stream on UCF101.
- Interpretation: The study challenges the assumption that explicit motion trajectories are the main source of optical flow’s recognition value.Flow remains informative even when temporal coherence is substantially disrupted.
- Temporal structure: 59.5% accuracy remains after shuffling RGB frames before computing flow, despite chance-level accuracy of approximately 1%.The resulting flow no longer generally represents physical motion but still captures moving-object shapes.
- Conclusion: The results identify appearance invariance, rather than temporal coherence alone, as key to optical flow’s usefulness in many action-recognition applications.This conclusion is supported by both temporal-shuffling and appearance-alteration experiments.
4. Is Optical Flow Accuracy Correlated with Action Recognition Accuracy?
The paper finds that standard optical-flow accuracy, measured by EPE, is weakly related to action-recognition accuracy. Region-specific analysis points instead to boundaries and small displacements, while end-to-end recognition training improves both tested neural flow networks.
- Region-specific analysis: Flow accuracy near motion boundaries and at small displacements has a higher correlation with recognition accuracy.The analysis groups pixels by distance from boundaries and by motion speed.
- Overall correlation: EPE does not correlate strongly with action-recognition accuracy across the tested flow methods.LDOF has relatively large EPE on Sintel and Real-World Scenes but produces the best action-recognition performance.
- Region-specific analysis: Large-motion flow accuracy is uncorrelated with action-recognition performance in the experiments.The study cannot determine whether this reflects poor estimation of large motions or limited importance of large motions for recognition.
- Implications: The findings motivate seeking task-specific metrics beyond whole-image EPE when selecting or training flow methods for action recognition.The paper identifies region-dependent flow information as more relevant than aggregate error alone.
5. Are There Better Motion Representations for Action Recognition than Optical Flow?
The paper fine-tunes SpyNet and FlowNet end-to-end with action-recognition loss, finding improved recognition despite slightly increased Sintel EPE. The largest learned flow changes occur on human regions and motion boundaries.
- 5. Are There Better Motion Representations for Action Recognition than Optical Flow?: Fine-tuning SpyNet and FlowNet through TSN with action-recognition loss improves action-recognition accuracy over EPE-trained flow.The training keeps the TSN recognition model fixed while observing how the optical-flow modules change.
- 5. Are There Better Motion Representations for Action Recognition than Optical Flow?: 3% improvement for FlowNet and 0.5% for SpyNet occur under TSN evaluation with 25 snippets and test-time data augmentation.With three snippets, improvements reach 4% for FlowNet and 2% for SpyNet.
- 5. Are There Better Motion Representations for Action Recognition than Optical Flow?: Slightly increased Sintel EPE after action-recognition fine-tuning is consistent with weak correlation between EPE and recognition accuracy.The task-specific models improve recognition without improving the conventional Sintel EPE measure.
- 5. Are There Better Motion Representations for Action Recognition than Optical Flow?: Action-recognition training changes flow most around motion boundaries and human regions in both SpyNet and FlowNet.Examples include fingers, a person’s boundary during pushups, and other human-centered motion regions.
- 5. Are There Better Motion Representations for Action Recognition than Optical Flow?: Flow changes inside human regions are one order of magnitude larger than changes outside them for both networks.Human regions are estimated with Mask-RCNN, while boundaries are measured using a 20-pixel dilated mask edge.
6. Conclusion
The paper analyzes the interaction between optical flow and action recognition, showing that appearance invariance makes flow useful even when its vectors are inaccurate. It further finds that task-specific flow training improves recognition, with changes concentrated on human bodies and their boundaries.
- 6. Conclusion: Optical flow is useful for action recognition because it is invariant to appearance, even when its flow vectors are inaccurate.The conclusion also reports weak correlation between traditional EPE and recognition accuracy.
- 6. Conclusion: EPE at boundaries and on small displacements is more relevant to recognition than the traditional overall EPE metric.This conclusion follows the paper’s comparison of optical-flow quality measures with action-recognition accuracy.
- 6. Conclusion: Training optical flow to minimize action-recognition error produces numerical improvements in action-recognition accuracy.The task-specific objective changes the motion representation rather than merely selecting among fixed flow algorithms.
- 6. Conclusion: The recognition-driven improvements arise from flow changes on the human body and near the body boundary.The authors present this as a connection relevant to both optical-flow and action-recognition researchers.