Source-linked AI summary
Representation Flow for Action Recognition
AJ Piergiovanni, Michael S. Ryoo
TL;DR
Action recognition benefits from motion information, but traditional optical flow is computationally expensive and often requires separate CNN streams. The paper introduces a differentiable representation-flow layer that learns motion within CNN features and reports improved speed and accuracy over existing methods, while also extending the idea to “flow of flow” representations.
Problem
Traditional optical flow is expensive to compute, requires parallel RGB and flow CNN streams, and prior flow-free motion representations underperformed two-stream models.
Method
The paper unrolls TV-L1-inspired iterative flow optimization into a differentiable CNN layer whose parameters are learned end-to-end on intermediate feature maps.
Results
The model outperformed existing methods in both speed and accuracy on standard datasets and showed performance benefits from stacked “flow of flow” layers.
Takeaways & Limitations
Representation flow provides a single-stream, RGB-based way to model motion while avoiding optical-flow preextraction and extending motion modeling to longer-term representations.
Takeaways & Limitations
The method assumes feature-value consistency as objects move, reflecting the spatial invariance of CNN features.
Abstract
from arXiv · showhide
In this paper, we propose a convolutional layer inspired by optical flow algorithms to learn motion representations. Our representation flow layer is a fully-differentiable layer designed to capture the `flow' of any representation channel within a convolutional neural network for action recognition. Its parameters for iterative flow optimization are learned in an end-to-end fashion together with the other CNN model parameters, maximizing the action recognition performance. Furthermore, we newly introduce the concept of learning `flow of flow' representations by stacking multiple representation flow layers. We conducted extensive experimental evaluations, confirming its advantages over previous recognition models using traditional optical flows in both computational speed and performance. Code/models available here: https://piergiaj.github.io/rep-flow-site/
1. Introduction
Two-stream CNNs benefit from explicit optical flow but incur substantial computation and parameter costs. The paper proposes a learnable representation-flow layer that captures motion within CNN features without optical-flow preextraction or multiple streams.
- Two-stream CNNs combine RGB appearance with optical flow and provide state-of-the-art action-recognition results.
- Optical flow requires hundreds of optimization iterations per frame, separate RGB and flow streams, and substantial inference computation.
- Earlier motion-representation methods reduced parameters and computation but performed worse than two-stream models on Kinetics and HMDB.
- The representation-flow layer learns motion from any CNN representation channel through differentiable, end-to-end learned flow optimization.
- Stacking representation-flow layers enables learning “flow of flow” representations without training multiple network streams.
- The paper evaluates flow placement, hyperparameters, learning parameters, and fusion techniques for action classification.
2. Related Works
Prior work captures temporal information through hand-crafted motion, optical flow, temporal CNN operations, attention, or learned motion representations. The proposed representation-flow model uses RGB alone and is reported to improve speed and accuracy relative to prior motion-representation methods.
- Earlier approaches captured motion using dense trajectories, optical flow, frame ordering, temporal pooling, temporal convolutions, or attention.
- Two-stream networks separately process RGB frames for appearance and optical-flow frames for motion before averaging predictions.
- Activity-recognition-optimized flow can differ from true optical flow, motivating end-to-end learning of motion representations.
- The proposed RGB-only model learns fewer parameters, represents motion through iterative optimization, and is reported to outperform TVNet and OFF in speed and accuracy.
3. Approach
The approach unrolls optical-flow optimization into a differentiable CNN layer that computes motion flow over CNN feature maps. Learned parameters, reduced-scale computation, and stacked layers adapt the representation flow to action recognition.
- Representation Flow Layer: The representation flow layer extends optical-flow algorithms to compute learned flow over arbitrary CNN feature maps rather than only RGB images.It learns θ, λ, τ, and divergence weights end-to-end, together with the rest of the CNN.
- Review of Optical Flow Methods: Standard variational optical flow estimates a two-dimensional flow field from sequential images through iterative optimization of a variational energy.The flow field stores x- and y-direction motion at each image location, and coarse-to-fine methods use multiple scales and warpings.
- Representation Flow Layer: The layer replaces costly optical-flow operations with a single-scale, unwarped computation on smaller CNN tensors.It computes gradients on sequential feature maps, initializes u and p to zero, and repeatedly applies tensor operations for a fixed number of iterations.
- Representation Flow Layer: Feature-value consistency extends the brightness-consistency assumption from images to CNN representations as objects move spatially.The layer therefore receives sequential feature-map channels and computes their representation flow using Sobel-filter gradients.
- Representation Flow Layer: Unrolled iterations behave like parameter-shared convolutional layers, making the flow computation fully differentiable and trainable for action recognition.The flow-layer parameters are optimized jointly with the classification CNN under the final recognition objective.
- Flow-of-Flow and CNN Integration: Stacking representation flow layers enables flow-of-flow representations that capture longer temporal intervals and motion-consistent locations.Intermediate CNN feature maps feed the flow layer, whose outputs are then used for prediction; channel reduction limits the cost of computing flow across many feature channels.
4. Experiments
Experiments examine where and how representation flow should be computed, how flow-of-flow should be constructed, and how the method compares with motion-representation and optical-flow baselines. The results support learned iterative flow on intermediate features, with an intermediate convolution improving stacked flow representations and RGB-only inference improving efficiency.
- Flow location: Computing flow after CNN feature extraction improves performance over applying it directly to the input, while computing it too late causes degradation from abstracted spatial information.The authors use the layer after the third residual block for subsequent experiments.
- Learned parameters: Learning divergence and τ, λ, θ is beneficial, whereas learning Sobel kernel values reduces performance because of noisy gradients with limited batch sizes.The comparison evaluates different learnable components with flow computed after Block 3.
- Iterations: Learned parameters provide better performance with fewer iterations, while iterative feature computation remains important.The remaining experiments use 10 or 20 iterations because they provide good performance.
- Flow-of-flow: Flow-conv-flow performs best among the flow-of-flow variants because the intermediate convolution smooths flow and creates a better input for the second flow layer.A third flow layer reduces performance as the motion representation becomes unreliable.
- Temporal CNNs: The learned flow layer improves performance even when 3D and (2+1)D CNNs already capture temporal information, unlike OFF in the reported comparison.These experiments evaluate flow on features from temporally modeling CNNs.
- Efficiency and comparison: Compared with optical-flow-based and other motion-representation methods, the model is faster and performs similarly or better, using about half as many parameters as two-stream competitors.The reported 2D CNN example compares 21M parameters with 42M, and optical-flow models were more than 10 times slower in the cited setting.
5. Conclusion
The paper introduces a learnable representation flow layer and extends it to flow-of-flow representations. Experiments report improved speed and accuracy over existing methods while using flow-of-flow to represent longer-term motion.
- Conclusion: The learnable representation flow layer and flow-of-flow representations improve both speed and accuracy over existing methods on standard datasets.The conclusion also states that iterative optimization and learnable parameters are important.
A. Training and Implementation Details
Implementation choices reduce the cost of representation flow and define the temporal input used for evaluation. Training uses standard momentum SGD across datasets and CNN architectures, with separate testing protocols for the full-resolution model.
- Representation-flow layer: A 1x1 convolution reduces feature channels to 32 before flow computation, producing 64 output channels for x and y flows.The reduction is chosen as a trade-off between performance and speed.
- Temporal aggregation: Sequential-image representation flows are averaged across 16 frames because stacking representation flows performed poorly.This differs from two-stream networks that stack 10 optical-flow frames.
- Optimization: Training uses stochastic gradient descent with momentum 0.9, with dataset- and architecture-specific learning rates, decay schedules, epochs, batch sizes, and GPU counts.The reported training duration is 200 epochs for the main Kinetics and Tiny-Kinetics setting.
- Testing: Only Table 9 uses 25 random crops and the full 32 × 224 × 224 input, while Tables 1–8 use smaller spatially and/or temporally sized models without random cropping.Random cropping is used to increase performance slightly.