PaperlayerView exact revision on arXiv

Source-linked AI summary

ActionFlowNet: Learning Motion Representation for Action Recognition

Joe Yue-Hei Ng, Jonghyun Choi, Jan Neumann, Larry S. Davis

arXiv:1612.03052v3cs.CV

TL;DR

Action recognition systems rely on hand-crafted motion features, while learning motion effectively from raw video remains challenging. ActionFlowNet jointly estimates optical flow and recognizes actions in a single network, and the authors report improved recognition over feature-learning methods without external large-scale data or additional flow input.

  • Problem

    Learning motion from raw video is challenging, and classification supervision may not learn motion effectively; video annotation is also costly, while two-stream systems require separate optical-flow input.

    Source: §1 · p. 1

  • Method

    ActionFlowNet is a single-stream multitask model that uses raw video frames to jointly learn optical flow and action classes.

    Source: §1 · p. 2, §3 · p. 2

  • Results

    ActionFlowNet significantly outperforms prior feature-learning methods trained without external large-scale data or additional optical-flow input; on UCF101, it exceeds Sports-1M-pretrained C3D by 1.6%.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The model captures motion information for action recognition without requiring explicit optical-flow computation as input.

    Source: §5 · p. 8

    The reported comparisons use only UCF101 and HMDB51, so results are not directly comparable with systems using substantially more labeled data.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

Even with the recent advances in convolutional neural networks (CNN) in various visual recognition tasks, the state-of-the-art action recognition system still relies on hand crafted motion feature such as optical flow to achieve the best performance. We propose a multitask learning model ActionFlowNet to train a single stream network directly from raw pixels to jointly estimate optical flow while recognizing actions with convolutional neural networks, capturing both appearance and motion in a single model. We additionally provide insights to how the quality of the learned optical flow affects the action recognition. Our model significantly improves action recognition accuracy by a large margin 31% compared to state-of-the-art CNN-based action recognition models trained without external large scale data and additional optical flow input. Without pretraining on large external labeled datasets, our model, by well exploiting the motion information, achieves competitive recognition accuracy to the models trained with large labeled datasets such as ImageNet and Sport-1M.

1. Introduction

ActionFlowNet addresses the challenge of learning motion from raw video by jointly estimating optical flow and recognizing actions in a single network. The introduction reports improved action recognition without external large-scale training data or additional optical-flow input.

1. Introduction

  • Frame-by-frame video annotation and temporal trimming are costly, limiting applications in domains such as medical imaging where data collection is difficult.The introduction contrasts video annotation with image labeling.Source: §1 · p. 1
  • Classification loss may be insufficient for learning motion features: even models trained with extensive labeled video data benefit from an additional optical-flow input stream.The authors use this as motivation to explore alternative motion supervision.Source: §1 · p. 1
  • Two-stream networks separately process images and optical flow, missing motion–appearance interaction and adding flow computation to the classification pipeline.The authors instead seek to infer motion from raw intensities within the model.Source: §1 · pp. 1–2
  • ActionFlowNet jointly learns optical flow and action recognition from raw pixels in a single network, using flow supervision to learn motion representations with less human annotation.After motion modeling, action annotations supervise learning of action-class-specific details.Source: §1 · p. 2
  • 1.6%: ActionFlowNet outperforms Sports-1M-pretrained C3D on UCF101 while using no external training data.The authors also report gains over state-of-the-art representation-learning methods trained without external labeled data.Source: §1 · p. 2

2. Related Work

Prior action-recognition work uses optical flow, large-scale video pretraining, and video-based self-supervision in different ways. ActionFlowNet builds on these directions while emphasizing motion representation from raw pixels without large labeled-dataset pretraining.

2. Related Work

  • Two-stream networks process appearance and optical flow separately, while ActionFlowNet focuses on learning from raw pixels rather than hand-crafted representations.Follow-up two-stream methods include.Source: §2 · p. 2
  • FlowNet and video optical-flow estimation with 3D convolutions motivate ActionFlowNet’s use of motion estimation, while its emphasis is action recognition.Source: §2 · p. 2
  • Large-scale video pretraining supports 3D CNN approaches such as C3D and I3D, whereas ActionFlowNet targets learning with small amounts of labeled data.The related-work discussion notes the computational expense of training on large-scale datasets and cites competitive performance as the motivation for its focus.Source: §2 · p. 2
  • Video self-supervision has used patch similarity, temporal ordering, future-trajectory prediction, and out-of-order sequence detection.Unlike those approaches, ActionFlowNet learns motion representations across video frames rather than representations for a single image, and does not focus on video generation.Source: §2 · p. 2
  • Diba et al. also combine flow estimation and action recognition [3], but use a C3D-based two-stream model rather than ActionFlowNet’s single stream.The authors distinguish their work by its focus on motion representation without large labeled-dataset pretraining and its analysis of flow representations for recognition.Source: §2 · p. 2

3. Approach

ActionFlowNet jointly learns action classes and optical flow from video frames in a single end-to-end model. The approach extends this idea to multi-frame and two-frame architectures, including a stacked classifier that uses FlowNet output.

3. Approach

  • ActionFlowNet learns action classes and optical flow jointly from video frames in a single end-to-end model.Unlike methods that use externally computed flow as input, it simultaneously learns flow and class labels.Source: §3 · p. 2

3.1. Multi-frame Optical Flow with 3D-ResNet

  • 3D-ResNet-18 extends ResNet with temporal convolutions and decoder skip connections to estimate optical flow across multiple frames.It uses highest-resolution loss, avoids temporal max pooling, and predicts flow between the final input frame and an unseen next frame.Source: §3.1 · p. 3

3.2. ActionFlowNet

  • ActionFlowNet transfers motion features by initializing its classifier from an optical-flow-trained network, then jointly trains flow estimation and action classification.The multitask framework is intended to prevent catastrophic forgetting; at inference, the decoder can be removed, leaving the encoder and classifier.Source: §3.2 · pp. 3–4

3.3. Two-Frame Based Models

  • Two-frame models take consecutive frames as input, enabling more efficient training and evaluation of motion modeling and external flow-data effects.The model concatenates the frames and uses a weighted sum of endpoint errors across decoder outputs.Source: §3.3 · p. 4

3.3.1 Stacked Model

  • The Stacked Model places a CNN classifier on FlowNet output, but cannot model appearance and motion simultaneously.It removes the classifier’s first two layers because flow output has four times lower resolution; the model has about twice the parameters and inference cost of the original ResNet.Source: §3.3.1 · p. 4

3.3.2 ActionFlowNet-2F

  • ActionFlowNet-2F adds a classifier to the two-frame FlowNet and trains classification and optical-flow losses together for each frame.Classification uses average pooling of the encoder’s last convolutional layer followed by a linear classifier.Source: §3.3.2 · pp. 4–5

4. Experiments

The experiments evaluate ActionFlowNet on UCF101 and HMDB51 and compare action recognition with prior models, including models trained with large labeled datasets. They examine learned motion representations, classwise performance, and how optical-flow quality relates to recognition accuracy.

4.1. Datasets

  • UCF101 (13,320 videos, 101 classes) and HMDB51 (6,766 videos, 51 classes) were used for action recognition; EpicFlow supplied pseudo-ground-truth flow because neither dataset has flow annotations.FlyingChairs provides ground-truth flow for training, and Sintel is used to validate flow quality.Source: §4.1 · p. 5

4.2. Experimental Setup

  • The experiments use task-specific data augmentation, Adam optimization for 40,000 iterations, and averaged predictions over 25 sampled video segments with 10 crops each.For UCF101 flow estimation, translation and rotation augmentation are omitted to preserve the original flow distribution.Source: §4.2 · p. 5

4.3. Improving Action Recognition

  • ActionFlowNet-2F scored 71.0% versus 80.7% for ImageNet-pretrained ResNet on UCF101, and 42.6% versus 47.1% on HMDB51.The authors attribute the larger UCF101 gap to its rich background context; HMDB51 backgrounds are less discriminative.Source: §4.3 · p. 6
  • 83.9% action recognition accuracy on UCF101 was 23.6% higher than the best previous work; 16-frame ActionFlowNet scored 83.9% versus 70.0% for its two-frame model.The 16-frame model also significantly improved on two-frame models by taking more input frames.Source: Table 1 · p. 5, Table 2 · p. 6, §4.3 · pp. 6–7

4.3.1 Learning Motions for Discriminative Regions

  • ActionFlowNet-2F focuses more on motion regions than ImageNet-pretrained ResNet, but can also use discriminative appearance such as writing on a board.The comparison suggests the model can use appearance cues when they are informative, unlike flow-only models.Source: §4.3.1 · p. 7

4.3.2 Optical Flow and Future Prediction

  • The multi-frame model produces reasonable future optical-flow predictions despite lacking accurate optical-flow ground truth for training.The authors interpret these predictions as reflecting semantic understanding beyond matching input frames.Source: §4.3.2 · p. 7

4.3.3 Classes Improved By Learning Motions

  • ActionFlowNet better recognizes simple-motion classes such as WallPushups and ApplyEyeMakeup, while C3D and ImageNet models do better on complex-appearance classes such as MoppingFloor and BaseballPitch.The authors note that object and contextual appearances are more discriminative for some action classes.Source: §4.3.3 · p. 8

4.4. Recognition and Optical Flow Quality

  • The Stacked model pretrained on FlyingChairs achieved 51.7% accuracy despite lower EPE, versus 69.6% for UCF101 pretraining; high-quality flow input reached 77.7%.The authors suggest FlyingChairs-trained flow misses small displacements; EPE may not capture shape preservation, which they say matters for action recognition.Source: §4.4 · p. 8

5. Conclusion

ActionFlowNet uses optical-flow supervision in a multitask framework to capture motion without requiring explicit flow as input. It outperforms prior feature-learning methods trained without external large-scale data or additional optical-flow input.

  • ActionFlowNet is a multitask framework for learning action recognition with motion flow.Source: §5 · p. 8
  • Optical flow supervises classification, allowing the model to capture motion without requiring explicit optical-flow computation as input.Source: §5 · p. 8
  • The model significantly outperforms prior feature-learning methods trained without external large-scale data or additional optical-flow input.Source: §5 · p. 8
Loading 1612.03052v3…