Source-linked AI summary
DynamoNet: Dynamic Action and Motion Network
Ali Diba, Vivek Sharma, Luc Van Gool, Rainer Stiefelhagen
TL;DR
Action-recognition ConvNets still lack effective motion-cue modeling. DynamoNet learns video-specific motion representations by predicting future frames while training for classification, and the authors report better performance than state-of-the-art methods on three benchmarks.
Problem
Action-recognition ConvNets still lack effective motion-cue modeling, despite the importance of motion cues for recognizing actions in videos.
Source: §1 · p. 1Method
DynamoNet jointly trains a ConvNet for action classification and future-frame prediction, using input-conditioned dynamic filters to learn video-specific motion representations.
Source: §1 · pp. 1–2, §3.1 · p. 3, §3.2 · p. 5Results
The authors report better performance than state-of-the-art methods on UCF101, HMDB51, and Kinetics, with 97.8% accuracy on UCF101.
Source: §5 · p. 8, §4.7 · p. 8Takeaways & Limitations
The learned motion representations are described as robust and compact, retaining global motion information for action classification.
Source: §5 · p. 8The paper focuses on action classification, leaving other possible uses of its motion information for future exploration.
Source: §5 · p. 8
Abstract
from arXiv · showhide
In this paper, we are interested in self-supervised learning the motion cues in videos using dynamic motion filters for a better motion representation to finally boost human action recognition in particular. Thus far, the vision community has focused on spatio-temporal approaches using standard filters, rather we here propose dynamic filters that adaptively learn the video-specific internal motion representation by predicting the short-term future frames. We name this new motion representation, as dynamic motion representation (DMR) and is embedded inside of 3D convolutional network as a new layer, which captures the visual appearance and motion dynamics throughout entire video clip via end-to-end network learning. Simultaneously, we utilize these motion representation to enrich video classification. We have designed the frame prediction task as an auxiliary task to empower the classification problem. With these overall objectives, to this end, we introduce a novel unified spatio-temporal 3D-CNN architecture (DynamoNet) that jointly optimizes the video classification and learning motion representation by predicting future frames as a multi-task learning problem. We conduct experiments on challenging human action datasets: Kinetics 400, UCF101, HMDB51. The experiments using the proposed DynamoNet show promising results on all the datasets.
1. Introduction
Action-recognition ConvNets still struggle to model motion cues effectively. DynamoNet jointly learns video-specific motion representations through future-frame prediction and uses them for action classification.
1. Introduction
- Action-recognition ConvNets still lack effective motion-cue modeling, while camera motion and changing viewpoints further complicate recognition.The paper also notes that computer-vision systems perform below humans in practice.Source: §1 · p. 1
- DynamoNet jointly optimizes video action classification and future-frame prediction, learning action-specific motion representations with video-dependent dynamic filters.The prediction task is intended to selectively use motion features that improve classification.Source: §1 · pp. 1–2
- Given T input frames, the network generates T dynamic motion kernels to predict consecutive next frames, then combines motion features with 3D-ConvNet features for classification.The filters vary between video samples and extract motion representations conditioned on each input.Source: §1 · p. 2
- The authors report 97.8% accuracy on UCF101, 76.8% on HMDB51, and 77.9% on Kinetics 400 for their method.They describe these results as state of the art.Source: §1 · p. 2
2. Background and Related Work
Prior work uses fixed spatiotemporal filters, encoding methods, and self-supervised video tasks. DynamoNet instead applies input-conditioned dynamic filters to learn motion representations for action recognition.
2. Background and Related Work
- Prior 3D video architectures use fixed temporal convolution-kernel depths, while DynamoNet proposes video-conditioned dynamic motion filters for adaptive motion-feature extraction.The cited related work also includes variable temporal kernel depths in T3D and channel correlations in [6].Source: §2 · p. 2
- Self-supervised action-recognition methods use frame shuffling [12] [28], clip ordering, or optical-flow estimation; DynamoNet uses future-frame prediction to learn motion representations.The paper says its goal is motion representation learning, rather than future-frame prediction itself.Source: §2 · pp. 2–3
- Earlier filter-generating networks learn steerable, weather-prediction, or enhancement filters, whereas DynamoNet applies dynamic filters to video motion representation and action classification.The authors distinguish their technical approach and application scope from those works.Source: §2 · p. 3
3. Proposed Method
DynamoNet learns video-specific motion representations by predicting future frames, then combines them with spatiotemporal action features for classification. The method also supports an unsupervised frame-prediction setup whose pretraining is reported to improve initialization stability and reduce the need for large labeled video datasets.
3. Proposed Method
- DynamoNet’s action-recognition architecture jointly learns motion representations through future-frame prediction and action classification.A separate architecture learns dynamic motion filters through short-term future-frame prediction in an end-to-end self-supervised fashion.Source: §3 · p. 3
3.1. Dynamic Motion Filters
- Each video clip generates sample-specific dynamic motion filters that predict consecutive frames and form a video-specific motion representation.For a 16-frame clip, the predictions include 15 reconstructed input frames and one new future frame; a 5×5 filter performed best in the reported setup.Source: §3.1 · pp. 3–4
3.2. Action Recognition
- The action-recognition network trains motion-filter and 3D-STCnet feature branches together, flattening the filters into a d-dimensional representation concatenated with action features for classification.The total objective combines frame-prediction and classification losses.Source: §3.2 · pp. 4–5
3.3. Unsupervised Training
- In the unsupervised setup, removing the classification branch leaves frame prediction, and pretraining on unlabeled videos is reported to stabilize initialization and reduce the need for large labeled datasets.The reported investigation trained the network from scratch on unlabeled videos.Source: §3.3 · p. 5
4. Experiments
DynamoNet is evaluated on Kinetics, UCF101, and HMDB51 using a 3D-CNN with classification and frame-prediction branches. Experiments examine unsupervised pre-training, prediction loss, future-frame prediction, and action-recognition performance.
4.1. Network Design
- The network uses a 3D-CNN backbone with classification and frame-prediction branches, concatenating action-recognition and dynamic-motion features for classification.The prediction branch generates dynamic filters; the experiments use STCnet or 3D-ResNet/ResNext backbones.Source: §4.1 · p. 5
4.2. Datasets
- Experiments use HMDB51, UCF101, and Kinetics, following their provided splits and protocols; Kinetics evaluation uses trimmed videos.Accuracy is averaged across three splits for HMDB51 and UCF101, and measured on the Kinetics validation set.Source: §4.2 · p. 5
4.3. Implementation Details
- DynamoNet’s frame-prediction-only unsupervised pre-training uses 500K unlabeled YouTube8M clips; afterward, both action-recognition and frame-prediction branches are trained on Kinetics for up to 200 epochs.Source: §4.3 · p. 6, §4.4 · p. 6
4.4. Unsupervised Pre-Training
- DynamoNet’s self-supervised pre-training outperforms prior self-supervised methods on UCF101 and HMDB51; with half of Kinetics, its performance remains comparable to models trained on the full dataset.The UCF101/HMDB51 comparison uses models fine-tuned after self-supervised training; the Kinetics comparison is on its validation set.Source: §4.4 · p. 6, Table 1 · p. 6, Table 2 · p. 6
4.5. Frame Prediction Learning Impact
- 50.2% versus 43.2% on UCF101: removing the prediction loss lowers DynamoNet’s performance, and the filters lack meaningful information without that objective.The comparison evaluates DynamoNet with and without the prediction loss.Source: §4.5 · p. 6
4.6. Frame Prediction
- On 378 UCF101 test videos, the paper compares frame-prediction performance with Mathieu et al. [27] and reports qualitative predictions that capture motion information.Figure 3 shows ground-truth frames, predicted frames, their difference images, and optical flow extracted from predictions.Source: Table 3 · p. 6, §4.6 · p. 7, Figure 3 · p. 7
4.7. Action Recognition
- On Kinetics-400, DynamoNet (STCnet101) with 64-frame input outperforms STC-ResNext101 with 32-frame input and I3D with 64-frame input.Table 4 compares performance (%) on Kinetics-400.Source: §4.7 · p. 8, Table 4 · p. 8
- 97.8% UCF101 and 76.8% HMDB51 accuracy; DynamoNet outperforms STCnet, 3D-ResNet, RGB-I3D, and C3D on both datasets.The RGB-only comparison covers all three splits of UCF101 and HMDB51.Source: §4.7 · p. 8, Table 5 · p. 8
5. Conclusion
DynamoNet learns motion representations by predicting future frames and uses them for action classification. The paper reports better-than-state-of-the-art performance, while its experiments focus on action classification.
5. Conclusion
- The proposed method yields better performance than state-of-the-art methods on UCF101, HMDB51, and Kinetics.Source: §5 · p. 8
- Dynamic motion representations are described as robust and compact, retaining global motion in a more expressive way.Source: §5 · p. 8
- The study focuses on action classification; video understanding and retrieval are proposed as possible complementary uses of the motion information.Source: §5 · p. 8