PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?

Kensho Hara, Hirokatsu Kataoka, Yutaka Satoh

arXiv:1711.09577v2cs.CV

TL;DR

The paper asks whether current video datasets provide enough data to train very deep 3D CNNs, addressing the limited evidence from mostly shallow architectures and small action-recognition datasets. It evaluates residual 3D CNNs from shallow to very deep depths across video datasets and finds that Kinetics supports training through ResNet-152, while Kinetics-pretrained 3D models transfer effectively to smaller benchmarks.

  • Problem

    Small video datasets and prior shallow-model studies leave limited evidence about whether very deep 3D CNNs can be trained effectively from scratch.

  • Method

    The study trains and fine-tunes shallow-to-very-deep residual 3D CNNs on Kinetics, UCF-101, HMDB-51, and ActivityNet.

  • Results

    Kinetics supports 3D ResNet-152 training, and Kinetics-pretrained simple 3D architectures outperform complex 2D architectures on UCF-101 and HMDB-51.

  • Takeaways & Limitations

    Deep 3D CNNs together with Kinetics have the potential to contribute to progress in action recognition and related video tasks.

  • Takeaways & Limitations

    Sports-1M and YouTube-8M are excluded because noisy annotations, unrelated frames, and scales exceeding 10 TB may hinder training and practical use.

Abstract

from arXiv · show

The purpose of this study is to determine whether current video datasets have sufficient data for training very deep convolutional neural networks (CNNs) with spatio-temporal three-dimensional (3D) kernels. Recently, the performance levels of 3D CNNs in the field of action recognition have improved significantly. However, to date, conventional research has only explored relatively shallow 3D architectures. We examine the architectures of various 3D CNNs from relatively shallow to very deep ones on current video datasets. Based on the results of those experiments, the following conclusions could be obtained: (i) ResNet-18 training resulted in significant overfitting for UCF-101, HMDB-51, and ActivityNet but not for Kinetics. (ii) The Kinetics dataset has sufficient data for training of deep 3D CNNs, and enables training of up to 152 ResNets layers, interestingly similar to 2D ResNets on ImageNet. ResNeXt-101 achieved 78.4% average accuracy on the Kinetics test set. (iii) Kinetics pretrained simple 3D architectures outperforms complex 2D architectures, and the pretrained ResNeXt-101 achieved 94.5% and 70.2% on UCF-101 and HMDB-51, respectively. The use of 2D CNNs trained on ImageNet has produced significant progress in various tasks in image. We believe that using deep 3D CNNs together with Kinetics will retrace the successful history of 2D CNNs and ImageNet, and stimulate advances in computer vision for videos. The codes and pretrained models used in this study are publicly available. https://github.com/kenshohara/3D-ResNets-PyTorch

1. Introduction

The study asks whether increasingly large video datasets can support very deep 3D CNNs, paralleling the progress enabled by deep 2D CNNs and ImageNet. Experiments across video datasets show that Kinetics supports deep 3D training through ResNet-152.

  • 1. Introduction: Video datasets such as UCF-101 and HMDB-51 are too small to optimize deep CNN representations from scratch, while Kinetics exceeds 300,000 videos.ActivityNet is larger but still contains a limited number of action instances.
  • 1. Introduction: 3D CNNs directly model spatio-temporal information, but their larger parameter counts and reliance on video-only pretraining have limited their comparison with ImageNet-pretrained 2D CNNs.Earlier 3D CNNs failed to overcome two-stream 2D CNNs combining RGB and stacked optical flow.
  • 1. Introduction: The study evaluates shallow-to-very-deep residual 3D CNNs on Kinetics, UCF-101, HMDB-51, and ActivityNet using training from scratch and fine-tuning.The tested architectures are based on ResNets and extended residual networks.
  • 1. Introduction: 152 layers: Kinetics trains 3D ResNet-152 from scratch to a level similar to 2D ResNets trained on ImageNet.Accuracy improvements continued as depth increased until ResNet-152, while ResNet-200 was nearly unchanged.
  • 1. Introduction: Deeper 3D CNNs are reported as more effective, supporting further progress in computer vision for videos.The paper presents this work as the first study focused on training very deep 3D CNNs from scratch for action recognition.

2. Related Work

Related work contrasts small action-recognition benchmarks and noisy large-scale datasets with approaches using two-stream 2D CNNs or increasingly effective 3D CNNs. The study focuses on whether large, labeled video data can support deeper 3D architectures.

  • 2. Related Work: UCF-101 and HMDB-51 remain popular benchmarks, but consensus indicates they are too small for training deep CNNs from scratch.ActivityNet contains 849 hours and 28,000 action instances, while Kinetics contains more than 300,000 trimmed videos across 400 categories.
  • 2. Related Work: Sports-1M and YouTube-8M are larger than Kinetics but have noisy video-level labels, unrelated frames, and file sizes exceeding 10 TB.The authors therefore refrain from discussing them further.
  • 2. Related Work: Two-stream 2D CNNs combine RGB frames with stacked optical flow to represent appearance and motion, improving action-recognition accuracy.Numerous subsequent methods built on this approach.
  • 2. Related Work: 3D CNNs extract spatio-temporal features directly from raw videos and recently outperform 2D CNNs when trained with large-scale video datasets.Earlier 3D CNNs trained on UCF-101 and HMDB-51 underperformed ImageNet-pretrained 2D CNNs, while Kinetics-trained models were comparable.
  • 2. Related Work: Prior 3D ResNet studies examined only relatively shallow architectures, leaving the trainability of very deep 3D CNNs unresolved.This study addresses that gap using current video datasets.

3. Experimental configuration

The study tests whether current video datasets support training deep 3D CNNs by evaluating ResNet-based architectures across datasets, depths, and transfer settings. It combines architecture comparisons with training-from-scratch, fine-tuning, and standard clip-based recognition procedures.

  • Experiments: ResNet-18 is trained from scratch on UCF-101, HMDB-51, ActivityNet, and Kinetics to assess whether each dataset can support deeper 3D CNN training.The study treats overfitting of ResNet-18 as evidence that a dataset is too small for training deep 3D CNNs from scratch.
  • Experiments: 3D ResNets with depths from 18 to 200 are trained on Kinetics to test how deeply the dataset can train 3D CNNs.The experiment specifically examines whether Kinetics can train models such as ResNet-152, comparable to the successful 2D ImageNet setting.
  • Experiments: Kinetics-pretrained 3D CNNs are fine-tuned on UCF-101 and HMDB-51 to evaluate transfer of visual representations across video domains.The fine-tuning experiment targets the use of large-scale pretraining for performance on relatively small datasets.
  • Network architectures: The evaluated models include ResNet, pre-activation ResNet, wide ResNet, ResNeXt, and DenseNet, spanning deeper, wider, grouped-convolution, and densely connected designs.The architectures are summarized in Figure 3 and Table 1; shortcut connections use summation except in DenseNet, which uses concatenation.
  • Network architectures: ResNet basic blocks use two convolutional layers, whereas bottleneck blocks use three layers with 1 × 1 × 1, 3 × 3 × 3, and 1 × 1 × 1 kernels.ResNet-18 and ResNet-34 use basic blocks; ResNet-50, 101, 152, and 200 use bottleneck blocks.
  • Training and recognition: Training uses stochastic gradient descent with momentum, randomly sampled 16-frame clips, spatial cropping and resizing to 112 × 112 pixels, and horizontal flipping.Scratch training starts at learning rate 0.1, while fine-tuning starts at 0.001; recognition averages class scores across non-overlapping 16-frame clips.

4. Results and discussion

Experiments show that dataset scale determines whether deep 3D CNNs can be trained effectively: smaller datasets overfit, whereas Kinetics supports very deep models and useful transfer.

  • Training on each dataset: 40.1%, 16.2%, and 26.8% validation accuracy was obtained for UCF-101, HMDB-51, and ActivityNet, respectively, alongside clear overfitting.Their validation losses quickly converged high and exceeded training losses; accuracies are per-clip rather than per-video.
  • Training on each dataset: Kinetics training of ResNet-18 did not overfit, indicating that the dataset can support training deep 3D CNNs.Validation losses were only slightly higher than training losses.
  • Deeper networks: 152 layers marked the point through which Kinetics ResNet accuracy improved with depth, while ResNet-200 was nearly unchanged and began to overfit.This depth pattern was similar to 2D ResNets trained on ImageNet.
  • Deeper networks: ResNeXt-101 achieved the best validation accuracies among the tested Kinetics architectures, while DenseNet-121 and DenseNet-201 were slightly lower.The authors relate this to successful cardinality and the absence of a need for parameter-efficiency techniques on Kinetics.
  • Deeper networks: ResNeXt-101 achieved higher Kinetics test accuracies than C3D with batch normalization, CNN+LSTM, and two-stream CNN, while RGB-I3D performed better.The comparison used average accuracy over Top-1 and Top-5 on the Kinetics test set.
  • Fine-tuning: Kinetics-pretrained ResNeXt-101 achieved the highest performance on both UCF-101 and HMDB-51 among the compared networks.Performance generally improved with depth, and the difference between architectures was smaller than on Kinetics.
  • Fine-tuning: Kinetics-pretrained simple 3D architectures outperformed complex 2D architectures, although two-stream I3D achieved the best accuracies in the state-of-the-art comparison.ResNeXt-101 (64f) slightly outperformed ST Multiplier Net and TSN, while two-stream I3D ranked highest.

5. Conclusion

The study finds that Kinetics supports training very deep 3D CNNs and that Kinetics-pretrained models perform strongly on smaller action-recognition datasets. The authors suggest that deep 3D CNNs and Kinetics could parallel the progress enabled by 2D CNNs and ImageNet.

  • ResNet-18 significantly overfits on UCF-101, HMDB-51, and ActivityNet but not on Kinetics.
  • Kinetics provides sufficient data to train 3D ResNets up to 152 layers, similarly to 2D ResNets trained on ImageNet.
  • 94.5% and 70.2% are achieved by pretrained ResNeXt-101 on UCF-101 and HMDB-51, respectively.
  • Deep 3D CNNs together with Kinetics could contribute to progress in video recognition and related tasks, following advances associated with 2D CNNs and ImageNet.
Loading 1711.09577v2…