Source-linked AI summary
Towards Good Practices for Very Deep Two-Stream ConvNets
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao
TL;DR
Action-recognition ConvNets had limited improvement because existing two-stream models were shallow and action datasets were much smaller than ImageNet. The report adapts very deep architectures with training practices designed for small datasets, achieving 91.4% recognition accuracy on UCF101.
Problem
Action recognition lacked the strong deep-ConvNet improvement seen in image classification because existing two-stream models were relatively shallow and action datasets were extremely small.
Method
The report adapts GoogLeNet and VGG-16 to two-stream video ConvNets and combines pre-training, smaller learning rates, data augmentation, high dropout, and multi-GPU training.
Results
91.4% recognition accuracy is achieved on the UCF101 dataset.
Takeaways & Limitations
Very deep two-stream ConvNets outperform the original two-stream ConvNets and prior compared methods under the proposed training practices.
Abstract
from arXiv · showhide
Deep convolutional networks have achieved great success for object recognition in still images. However, for action recognition in videos, the improvement of deep convolutional networks is not so evident. We argue that there are two reasons that could probably explain this result. First the current network architectures (e.g. Two-stream ConvNets) are relatively shallow compared with those very deep models in image domain (e.g. VGGNet, GoogLeNet), and therefore their modeling capacity is constrained by their depth. Second, probably more importantly, the training dataset of action recognition is extremely small compared with the ImageNet dataset, and thus it will be easy to over-fit on the training dataset. To address these issues, this report presents very deep two-stream ConvNets for action recognition, by adapting recent very deep architectures into video domain. However, this extension is not easy as the size of action recognition is quite small. We design several good practices for the training of very deep two-stream ConvNets, namely (i) pre-training for both spatial and temporal nets, (ii) smaller learning rates, (iii) more data augmentation techniques, (iv) high drop out ratio. Meanwhile, we extend the Caffe toolbox into Multi-GPU implementation with high computational efficiency and low memory consumption. We verify the performance of very deep two-stream ConvNets on the dataset of UCF101 and it achieves the recognition accuracy of $91.4\%$.
1. Introduction
Action recognition remains challenging because videos exhibit substantial variation and complexity, while deep ConvNets have not clearly surpassed traditional methods. The report addresses shallow architectures and small datasets by proposing very deep two-stream ConvNets with specialized training practices.
- Action recognition is difficult because videos contain large intra-class variation, low resolution, and high-dimensional data.
- Deep action-recognition methods include handcrafted feature pipelines and end-to-end ConvNets trained from RGB images or optical flow.Two-stream ConvNets are identified as the most competitive deep model.
- Unlike image classification, deep ConvNets had not yielded significant improvement over traditional action-recognition methods.
- Current two-stream ConvNets are relatively shallow, while UCF101 contains only 13,320 clips, creating architectural and data constraints.
- The report presents very deep two-stream ConvNets and good practices intended to stabilize training and reduce over-fitting on small action datasets.It also extends Caffe to multi-GPU implementation with high efficiency and low memory consumption.
2. Very Deep Two-stream ConvNets
The method adapts GoogLeNet and VGG-16 architectures to spatial and temporal video streams, then uses pre-training, smaller learning rates, augmentation, dropout, and multi-GPU training to address over-fitting and computational cost.
- Network architectures: GoogLeNet uses stacked multi-filter Inception modules in a 22-layer architecture, while VGGNet emphasizes small filters, small strides, and deeper structures up to 19 layers.
- Network architectures: Very deep two-stream ConvNets adapt GoogLeNet and VGG-16 architectures for video action recognition.
- Network architectures: The spatial net processes a 224 × 224 × 3 frame, whereas the temporal net processes 10 stacked optical-flow frames with input size 224 × 224 × 20.
- Training practices: ImageNet models initialize spatial nets and also work for temporal nets despite their optical-flow input modality.
- Training practices: Multi-GPU data-parallel training addresses long video-model training times and high activation memory consumption.With 4 GPUs, training is 3.7x faster for VGGNet-16 and 4.0x faster for GoogLeNet, using 4x less memory per GPU.
- Testing strategy: Testing averages predictions across 25 sampled frames or flow fields and 10 crops per selected input.
3. Experiments
Experiments on UCF101 show that deeper architectures and the proposed training practices improve two-stream recognition, with very deep models reaching 91.4% accuracy and outperforming prior methods.
- Dataset and evaluation: The UCF101 evaluation uses 13,320 clips across 101 action classes and three training/testing splits.
- Architecture results: VGGNet-16 outperforms shallower architectures by around 5% for spatial nets and around 4% for temporal nets.
- Architecture results: Very deep two-stream ConvNets outperform the original two-stream ConvNets by 3.4%.
- Training-practice analysis: The authors conjecture that ImageNet pre-training for temporal nets and additional data augmentation reduce over-fitting across datasets.This interpretation follows different performance between UCF101 experiments and earlier THUMOS15 training without the proposed practices.
- Comparison with prior work: The proposed results outperform Fisher-vector representations, DeepNets, and recurrent two-stream networks, exceeding the best prior result by 2.8%.
4. Conclusions
The work evaluates very deep two-stream ConvNets for action recognition and proposes training practices suited to extremely small action-recognition datasets. With carefully designed strategies, the models achieve 91.4% recognition accuracy on UCF101, alongside a high-efficiency, low-memory Multi-GPU Caffe implementation.
- 91.4% recognition accuracy is achieved on the UCF101 dataset with carefully designed training strategies.
- The Caffe toolbox is extended into a Multi-GPU implementation with high efficiency and low memory consumption.