Source-linked AI summary
Learning Spatiotemporal Features with 3D Convolutional Networks
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri
TL;DR
The paper addresses the need for effective spatiotemporal video features by training deep 3D ConvNets on large-scale supervised video data. It develops C3D with homogeneous small kernels and evaluates it across video benchmarks, where the features outperform or approach current best methods while remaining compact, efficient, and simple to use.
Problem
Large-scale video analysis needs a generic descriptor that represents diverse videos while remaining compact, efficient to compute, and simple to use.
Method
The paper learns spatiotemporal features with deep 3D ConvNets trained on large-scale video datasets, using a homogeneous architecture with 3 × 3 × 3 kernels.
Results
C3D features with a simple linear classifier outperform or approach current best methods on 4 different tasks and 6 different benchmarks.
Takeaways & Limitations
C3D features jointly capture appearance and motion and are compact, efficient to compute, and simple to use across video analysis tasks.
Takeaways & Limitations
C3D needs to be combined with iDT to outperform two-stream networks, other iDT-based methods, and a method focused on long-term modeling.
Abstract
from arXiv · showhide
We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised video dataset. Our findings are three-fold: 1) 3D ConvNets are more suitable for spatiotemporal feature learning compared to 2D ConvNets; 2) A homogeneous architecture with small 3x3x3 convolution kernels in all layers is among the best performing architectures for 3D ConvNets; and 3) Our learned features, namely C3D (Convolutional 3D), with a simple linear classifier outperform state-of-the-art methods on 4 different benchmarks and are comparable with current best methods on the other 2 benchmarks. In addition, the features are compact: achieving 52.8% accuracy on UCF101 dataset with only 10 dimensions and also very efficient to compute due to the fast inference of ConvNets. Finally, they are conceptually very simple and easy to train and use.
1. Introduction
The paper seeks a generic video descriptor for large-scale analysis and proposes supervised 3D ConvNet features that jointly model spatiotemporal information. C3D is designed to be broadly useful, compact, efficient, and simple to use.
- A generic video descriptor is needed to support large-scale tasks such as search, recommendation, and ranking across diverse video types.
- An effective descriptor should be generic, compact, efficient to compute, and simple enough to work with a linear classifier.
- 3D ConvNet features model appearance and motion simultaneously, unlike image-based deep features that lack motion modeling.
- 3 × 3 × 3 convolution kernels in all layers performed best among the explored 3D ConvNet architectures.
- C3D features with a simple linear model outperform or approach current best methods on 4 different tasks and 6 different benchmarks.
2. Related Work
Video representation research spans handcrafted spatiotemporal features and increasingly deep ConvNet-based approaches. Related 3D ConvNet work differs in preprocessing and input scope, while two-stream networks reported strong action-recognition results.
- Video representation research includes STIPs, SIFT-3D, and HOG3D, which extend image descriptors or detectors into the spatiotemporal domain.
- ConvNets have been applied to visual recognition, human pose estimation, and image feature learning as computational resources and training data expanded.
- Two-stream networks achieved best results on action recognition in the cited related work.
- A closely related 3D ConvNet method segments human subjects using detection and head tracking before classifying actions with a three-convolution-layer network.
- In contrast, the proposed method uses full video frames without preprocessing, supporting easier scaling to large datasets.
3. Learning Features with 3D ConvNets
The paper develops 3D ConvNets for learning video features that preserve and model temporal information, then selects a homogeneous 3 × 3 × 3 architecture and trains C3D for large-scale feature learning.
- 3.1. 3D convolution and pooling: 3D ConvNets perform convolution and pooling across spatial and temporal dimensions, preserving temporal information that 2D ConvNets collapse.Applying 3D convolution to a video volume produces another volume, whereas 2D convolution produces an image.
- 3.2. Exploring kernel temporal depth: The architecture search varies convolutional kernel temporal depth while fixing the spatial receptive field and other network settings.The study compares homogeneous temporal depths, where all convolution layers share one temporal depth, with varying-depth architectures.
- 3.2. Exploring kernel temporal depth: Depth-3 performs best among homogeneous networks on UCF101 test split-1, and 3D ConvNets consistently outperform 2D ConvNets in the reported experiments.The paper also reports similar behavior with larger spatial receptive fields, full input resolution, and a large-scale internal dataset.
- 3.3. Spatiotemporal feature learning: C3D uses 8 convolution layers, 5 pooling layers, 2 fully connected layers, and a softmax output, with 3 × 3 × 3 convolution kernels throughout.Its first pooling layer uses 1 × 2 × 2 kernels to preserve temporal information early; later pooling layers use 2 × 2 × 2 kernels.
- 3.3. Spatiotemporal feature learning: C3D is trained on Sports-1M, containing 1.1 million videos across 487 sports categories, using randomly extracted 2-second clips and 16-frame crops.The networks are trained with SGD on augmented clips resized to 128 × 171 frames.
- 3.3. Spatiotemporal feature learning: 85.5% video top-5 accuracy is achieved by C3D fine-tuned from I380K, outperforming DeepVideo while remaining below convolution pooling using much longer clips.The comparison notes that convolution pooling uses 120-frame clips, so its results are not directly comparable to the short-clip C3D and DeepVideo settings.
4. Action recognition
On UCF101 action recognition, C3D features are evaluated with linear SVMs against hand-crafted and image-based baselines, showing strong accuracy, compactness, and motion-sensitive representations.
- Feature complementarity: C3D captures appearance and motion information, making Imagenet combination yield only 0.6% improvement while iDT provides a larger complementary gain.The paper characterizes iDT as low-level optical-flow and gradient features, whereas C3D captures high-level abstract or semantic information.
- Learned representations: Deconvolution visualizations show C3D initially attends to appearance and subsequently tracks salient motion.The observed behavior includes tracking a person and pole-vault motion across later frames.
- Action recognition results: 85.2% accuracy is achieved by C3D with three nets, compared with 9% and 16.4% gains over iDT and Imagenet baselines, respectively.A single 4,096-dimensional net reaches 82.3%, while combining C3D with iDT reaches 90.4%.
- Comparisons: C3D outperforms several RGB-only and recurrent baselines, including LRCN and an LSTM composite model, using only RGB inputs.The reported margins over LRCN and the LSTM composite model are 14.1% and 9.4%, respectively.
- Compactness: At 10 dimensions, C3D reaches 52.8% UCF101 accuracy, about 20% above Imagenet and iDT at approximately 32%.At 500 dimensions, C3D reaches 79.4%, exceeding iDT by 6% and Imagenet by 11%.
- Feature embedding: C3D features are qualitatively more semantically separable than Imagenet features in t-SNE embeddings without fine-tuning.The visualization uses 100K UCF101 clips and compares fc6 features from Imagenet and C3D.
5. Action Similarity Labeling
On ASLAN action-similarity labeling, C3D averages clip features and classifies video-pair distances with a linear SVM, outperforming the reported state-of-the-art method.
- ROC analysis: 86.5% AUC is achieved by C3D on ASLAN, according to the ROC-curve result.The ROC comparison includes current methods and human performance, reported as 98.9%.
- Results: C3D significantly outperforms the state-of-the-art method by 9.6% in accuracy and 11.1% in area under ROC curve.The task predicts whether video pairs depict the same or different actions, including never-seen-before actions in testing.
- Method: C3D uses averaged clip features and a linear SVM, whereas many competing methods use multiple hand-crafted features, strong encodings, and complex learning models.The video representation averages features from overlapping 16-frame clips before pairwise-distance classification.
6. Scene and Object Recognition
C3D is evaluated on dynamic scene and object recognition benchmarks using clip features with a linear SVM. It outperforms current methods on scene recognition and object recognition while remaining competitive with an ImageNet baseline.
- Evaluation setting: The evaluations cover two scene-recognition benchmarks and one egocentric object-recognition dataset.YUPENN contains 420 videos across 14 scene categories, Maryland contains 130 videos across 13 categories, and the object dataset covers 42 everyday-object types.
- Scene recognition: C3D outperforms the state-of-the-art method by 10% on Maryland and 1.9% on YUPENN scene classification.C3D uses simple clip-feature averaging and a linear SVM, whereas the second-best method uses multiple complex feature encodings.
- Object recognition: 22.3% accuracy on object recognition exceeds the comparison method by 10.3% using only a linear SVM.The comparison method uses an RBF kernel with strong SIFT-RANSAC feature matching.
- Comparison with ImageNet: C3D matches the ImageNet baseline on Maryland and is 1.4% higher on YUPENN.On object recognition, C3D is 3.4% below the ImageNet baseline, partly because it uses 128 × 128 rather than 256 × 256 input resolution.
7. Runtime Analysis
The runtime analysis compares C3D with improved dense trajectories and Brox optical flow implementations for extracting features from UCF101. C3D is substantially faster, exceeding real-time processing speed.
- Runtime comparison: 313 fps: C3D processes UCF101 features much faster than the other methods, which operate below 4 fps.The comparison includes feature extraction and I/O using a single CPU or a single K40 Tesla GPU.
- Measurement conditions: Brox’s GPU implementation takes 0.85–0.9s per image pair including I/O, while the cited competing timing excludes I/O.The comparison with improved dense trajectories is not fully fair because that method uses only CPU and lacks an available GPU implementation.
8. Conclusions
The paper concludes that C3D learns appearance and motion jointly and performs strongly across video-analysis benchmarks. It emphasizes that the features are efficient, compact, and simple to use.
- Conclusions: C3D models appearance and motion information simultaneously and outperforms 2D ConvNet features on various video-analysis tasks.The conclusion frames this as the paper’s approach to learning spatiotemporal video features with 3D ConvNets.
- Conclusions: C3D with a linear classifier outperforms or approaches current best methods on different video-analysis benchmarks.The paper presents this result as applying across multiple benchmarks rather than a single task.
- Conclusions: The proposed C3D features are efficient, compact, and extremely simple to use.The paper also provides source code and a pre-trained model for C3D.
Appendix A: Effects of Input Resolution
The appendix studies how input resolution affects 3D ConvNet performance while holding the convolutional architecture fixed. It compares 64 × 64, 128 × 128, and 256 × 256 inputs.
- Appendix A: Effects of Input Resolution: The input-resolution study fixes all convolution kernels at 3 × 3 × 3 and varies only the input resolution.The experiments use the common network setting described in the paper.
- Appendix A: Effects of Input Resolution: Three networks are evaluated: net-64, net-128, and net-256, corresponding to 64 × 64, 128 × 128, and 256 × 256 inputs.Net-128 is equivalent to the depth-3 network described in section 3.2.
Appendix B: Visualization of C3D Learned Features
The appendix visualizes what C3D feature maps respond to by projecting strongly activating video clips and their gradients back into image space. It examines representations across convolutional layers, including comparisons with optical flow.
- Purpose: The appendix provides additional visualizations to improve understanding of C3D’s internally learned features.The visualizations are designed to inspect feature-map responses rather than to fine-tune the model.
- Visualization Method: 20K randomly selected UCF101 clips are grouped by shared strong activation at a chosen convolutional layer, then projected into image space using deconvolution.The visualization combines activation-causing gradients with corresponding cropped image sequences, without fine-tuning C3D on UCF101.
- Layer Coverage: The visualizations cover C3D feature maps from conv2a, conv3b, and conv5b, grouping views of the same feature map together.Figures 9 and 10 show conv2a and conv3b, while Figures 11–14 each show one conv5b feature map.
- Layer Coverage: Several conv5b feature maps are compared with optical flows to examine their learned motion representations.Figure 15 presents this comparison across multiple C3D conv5b feature maps.