Source-linked AI summary
Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition
Kensho Hara, Hirokatsu Kataoka, Yutaka Satoh
TL;DR
3D CNNs offer direct spatio-temporal modeling but have been relatively shallow and difficult to train because of their large parameter counts. This paper extends ResNets to 3D CNNs and evaluates them on ActivityNet and Kinetics. Kinetics-trained 3D ResNets performed well without overfitting despite their many parameters.
Problem
Very deep 3D CNNs have been insufficiently explored because their large parameter counts make training difficult, despite the success of deep architectures in image recognition.
Method
The paper extends 2D ResNets to 3D convolutional networks and trains them on ActivityNet and Kinetics.
Results
Kinetics-trained 3D ResNets performed well without overfitting despite the model’s large number of parameters.
Takeaways & Limitations
The experiments support the effectiveness of ResNets with 3D convolutional kernels for action recognition.
Takeaways & Limitations
The paper notes that further experiments with deeper models such as ResNet-50 and ResNet-101, and architectures such as DenseNet-201, remain to be conducted.
Abstract
from arXiv · showhide
Convolutional neural networks with spatio-temporal 3D kernels (3D CNNs) have an ability to directly extract spatio-temporal features from videos for action recognition. Although the 3D kernels tend to overfit because of a large number of their parameters, the 3D CNNs are greatly improved by using recent huge video databases. However, the architecture of 3D CNNs is relatively shallow against to the success of very deep neural networks in 2D-based CNNs, such as residual networks (ResNets). In this paper, we propose a 3D CNNs based on ResNets toward a better action representation. We describe the training procedure of our 3D ResNets in details. We experimentally evaluate the 3D ResNets on the ActivityNet and Kinetics datasets. The 3D ResNets trained on the Kinetics did not suffer from overfitting despite the large number of parameters of the model, and achieved better performance than relatively shallow networks, such as C3D. Our code and pretrained models (e.g. Kinetics and ActivityNet) are publicly available at https://github.com/kenshohara/3D-ResNets.
1. Introduction
3D CNNs directly capture spatio-temporal video information but have faced training difficulty and overfitting risks. The paper explores extending ResNets to 3D CNNs and evaluates them on ActivityNet and Kinetics.
- Human-action recognition supports applications including surveillance, video indexing, and human-computer interaction.
- 2D CNNs commonly use separate RGB and optical-flow streams to represent video appearance and motion information.
- 3D convolutional kernels directly capture spatio-temporal information but contain many parameters and historically underperformed 2D CNNs on smaller datasets.
- Very deep 3D CNNs remain underexplored because their large parameter counts make training difficult.
- The paper extends 2D ResNets to 3D and evaluates the resulting networks on ActivityNet and Kinetics.
- The main contribution is exploring the effectiveness of ResNets with 3D convolutional kernels for action recognition.
2. Related Work
Prior action-recognition research spans two-stream 2D CNNs and 3D CNNs, with dataset scale, annotation quality, and architecture depth shaping performance. This paper positions 3D ResNets as a deep alternative to earlier 3D architectures.
- Small action-recognition datasets such as UCF101 and HMDB51 make training models without overfitting difficult.
- Large video datasets provide more training data but may contain noisy video-level labels and unrelated frames.
- Two-stream CNNs use RGB frames and stacked optical flow as appearance and motion inputs, respectively.
- 3D CNN studies found that temporal input length and optical-flow inputs can improve recognition, with combined RGB and flow performing best in the cited work.
- C3D established 3 × 3 × 3 convolutional kernels as a strong configuration in its experiments.
- Residual blocks use shortcut connections that bypass signals and sum them at the block tail.
- The inception architecture brought a 22-layer deep design to 3D CNNs, while this paper introduces ResNets to the same setting.
3. 3D Residual Networks
The proposed network replaces 2D convolutions and pooling with 3D operations in a ResNet architecture, using 16-frame RGB clips and residual shortcuts. Training combines stochastic optimization with spatial and temporal augmentation, and inference averages clip probabilities.
- 3.1. Network Architecture: 3D ResNets use 3D convolution and pooling with 3 × 3 × 3 kernels on 16-frame RGB clips sized 3 × 16 × 112 × 112.
- 3.1. Network Architecture: Down-sampling occurs in conv3_1, conv4_1, and conv5_1 using stride 2.
- 3.1. Network Architecture: When feature-map dimensions increase, identity shortcuts with zero-padding avoid increasing the number of parameters.
- 3.1. Network Architecture: Each convolutional layer is followed by batch normalization and ReLU, and the final fully connected layer is configured for Kinetics’ 400 categories.
- 3.2.1 Training: Training uses SGD with momentum and generates augmented samples through temporal sampling, spatial cropping, scale selection, flipping, and mean subtraction.
- 3.2.1 Training: Large learning rate and batch size were important for good recognition performance in preliminary ActivityNet experiments.
- 3.2.2 Testing: Inference splits videos into non-overlapping 16-frame clips and averages clip-level class probabilities across each video.
4. Experiments
Experiments on ActivityNet and Kinetics examined whether 3D ResNets train effectively at different dataset scales. The 3D ResNets overfit on smaller ActivityNet but trained well on Kinetics and outperformed shallow C3D models.
- 4.1. Dataset: ActivityNet contains 20,000 videos, while Kinetics contains 300,000 or more videos and Sports-1M contains 1,000,000.The ActivityNet experiments used 200 action classes, whereas Kinetics used 400 classes and about 240,000 training videos.
- 4.1. Dataset: The 18-layer 3D ResNet overfitted on ActivityNet, with validation accuracy significantly below training accuracy.Accuracies were calculated from 16-frame clips rather than entire videos.
- 4.2. Results: The 34-layer 3D ResNet did not overfit on Kinetics and achieved good performance, while the Sports-1M-pretrained C3D underfitted.Both training and validation accuracies were calculated on 16-frame clips.
- 4.2. Results: The 3D ResNet was competitive with C3D without Sports-1M pretraining, indicating that C3D was too shallow for Kinetics-scale training.The authors trained the deeper 34-layer model because Kinetics contained significantly more activity instances than ActivityNet.
- 4.2. Results: ResNet-34 achieved higher Kinetics accuracies than Sports-1M-pretrained C3D and batch-normalized C3D trained from scratch, while RGB-I3D achieved the best performance.The comparison is reported in Table 2; the models used different GPU counts and input clip sizes.
- 4.2. Results: The ResNet-34 input was limited to 3 × 16 × 112 × 112, whereas RGB-I3D used 3 × 64 × 224 × 224 inputs.The authors suggest that larger batch sizes, higher spatial resolutions, and longer temporal durations might further improve 3D ResNets.
5. Conclusion
3D ResNets trained on Kinetics performed well without overfitting despite their large parameter count. Evaluation focused mainly on ResNets-34 because training required three weeks, leaving deeper models for future experiments.
- 3D ResNets trained on Kinetics performed well without overfitting despite the model’s large number of parameters.
- Figure 4 presents three correctly recognized Kinetics examples and one wrongly recognized example from 3D ResNets-34.Each row contains center-cropped frames showing part of the original video.
- Three weeks of training time led the authors to focus mainly on ResNets-34.
- Experiments with deeper models, including ResNets-50 and ResNets-101, were left for future work.The authors also mention DenseNets-201 as another future architecture.