Source-linked AI summary
Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Amir Hossein Karami, Mohammad Mahdi Arzani, Rahman Yousefzadeh, Luc Van Gool
TL;DR
Video ConvNets have difficulty capturing long-range temporal information and often require large labeled datasets or costly optical flow. The paper introduces T3D with TTL and transfers supervision from a pretrained 2D CNN to a 3D CNN; it reports strong results across HMDB51, UCF101, and Kinetics while reducing reliance on training from scratch.
Problem
Existing 3D ConvNets use fixed temporal kernel depths, while training them from scratch requires large labeled datasets and substantial computational effort.
Method
T3D extends DenseNet with 3D filters and uses TTL to combine temporal features from diverse depths, while transferring supervision from a pretrained 2D CNN to a 3D CNN.
Results
T3D achieves state-of-the-art performance among 3D ConvNets on HMDB51 and UCF101, competitive results on Kinetics, and outperforms Sports-1M-trained generic 3D ConvNets after fine-tuning.
Takeaways & Limitations
Variable temporal-depth modeling and 2D-to-3D supervision transfer provide the paper’s supported route to expressive video features and stable 3D initialization without training from scratch.
Takeaways & Limitations
T3D has 1.3 times more model parameters than DenseNet3D, despite its reported performance advantage.
Abstract
from arXiv · showhide
The work in this paper is driven by the question how to exploit the temporal cues available in videos for their accurate classification, and for human action recognition in particular? Thus far, the vision community has focused on spatio-temporal approaches with fixed temporal convolution kernel depths. We introduce a new temporal layer that models variable temporal convolution kernel depths. We embed this new temporal layer in our proposed 3D CNN. We extend the DenseNet architecture - which normally is 2D - with 3D filters and pooling kernels. We name our proposed video convolutional network `Temporal 3D ConvNet'~(T3D) and its new temporal layer `Temporal Transition Layer'~(TTL). Our experiments show that T3D outperforms the current state-of-the-art methods on the HMDB51, UCF101 and Kinetics datasets. The other issue in training 3D ConvNets is about training them from scratch with a huge labeled dataset to get a reasonable performance. So the knowledge learned in 2D ConvNets is completely ignored. Another contribution in this work is a simple and effective technique to transfer knowledge from a pre-trained 2D CNN to a randomly initialized 3D CNN for a stable weight initialization. This allows us to significantly reduce the number of training samples for 3D CNNs. Thus, by finetuning this network, we beat the performance of generic and recent methods in 3D CNNs, which were trained on large video datasets, e.g. Sports-1M, and finetuned on the target datasets, e.g. HMDB51/UCF101. The T3D codes will be released
1. Introduction
The paper targets limitations in video ConvNets, especially fixed temporal receptive fields, high parameter and data demands, and costly optical-flow inputs. It proposes T3D with TTL and 2D-to-3D supervision transfer, then evaluates the approach on three action-recognition datasets.
- Existing 3D ConvNets use fixed temporal kernels and fail to exploit long-range temporal information.
- TTL concatenates temporal feature maps from different depth ranges within a DenseNet3D-based architecture called T3D.The architecture models short-, mid-, and long-range temporal information.
- T3D is evaluated on HMDB51, UCF101, and Kinetics, achieving state-of-the-art performance on HMDB51 and UCF101 among other 3D ConvNets and competitive Kinetics results.
- A pretrained 2D CNN transfers supervision to a randomly initialized 3D CNN for stable weight initialization, avoiding training 3D ConvNets from scratch.The approach is intended to reduce excessive training time and computational workload.
2. Related Work
Prior video-recognition methods include hand-engineered spatio-temporal descriptors and ConvNets using fixed temporal kernel depths. Related work also demonstrates transfer learning across datasets, architectures, and modalities.
- Hand-engineered descriptors capture appearance and motion but can be computationally expensive and lack scalability for semantic concepts.The passage identifies improved dense trajectories as the strongest among the listed hand-engineered methods, with these trade-offs.
- Temporal ConvNets: Temporal ConvNets process RGB clips or stacked optical-flow frames with 3D spatial-temporal filters and pooling kernels.
- Temporal ConvNets: Earlier 3D architectures use fixed temporal depths, whereas the proposed architecture integrates variable temporal-depth information over shorter and longer ranges.
- Transfer Learning: Transfer learning commonly specializes representations learned on another dataset and has been applied within and across modalities.Examples include RGB-to-RGB, RGB-to-depth, RGB-to-optical-flow, RGB-to-sound, and near-infrared-to-RGB transfer.
3. Proposed Method
The proposed T3D combines a 3D DenseNet backbone with a Temporal Transition Layer that aggregates temporal features across multiple kernel depths. It also transfers supervision from a frozen, pretrained 2D ConvNet to stabilize 3D ConvNet initialization and avoid training from scratch.
- Temporal 3D ConvNets: T3D extends DenseNet with 3D filters and pooling kernels, replacing the standard transition layer with the Temporal Transition Layer (TTL).The same DenseNet building blocks and architecture choices are retained while the kernels and transition mechanism are adapted for video.
- Temporal 3D ConvNets: 3D dense connectivity feeds each layer’s feature maps directly to all subsequent layers, with preceding maps concatenated as input to later composite functions.This propagates 3D feature representations throughout each Dense block.
- Temporal 3D ConvNets: TTL uses variable temporal-depth 3D convolution kernels to capture short-, mid-, and long-term dynamics instead of fixed homogeneous depths.Its outputs are concatenated and passed through a 3D pooling layer, producing a dense-aggregated feature representation.
- Temporal 3D ConvNets: T3D has 1.3 times more model parameters than DenseNet3D, although the authors report significantly better performance.The TTL can also be employed in other architectures such as Res3D or I3D.
- Supervision or Knowledge Transfer: A frozen ImageNet-pretrained 2D ConvNet supervises a randomly initialized 3D ConvNet through image-video correspondence at matching timestamps.Only the 3D model parameters are updated, using positive same-video timestamp pairs and negative pairs sampled from different videos.
- Supervision or Knowledge Transfer: The authors report that T3D is state-of-the-art on HMDB51 and UCF101 and competitive on Kinetics, while the transfer procedure supports training directly on smaller datasets.The transfer method is presented as avoiding scratch training and enabling faster adaptation during fine-tuning.
4. Experiments
The experiments search T3D architectures and input configurations, then evaluate T3D and transfer learning across action-recognition benchmarks. Results report strong RGB-only performance, while transfer learning reduces reliance on large labeled video datasets.
- Architecture and input search: The study searches DenseNet3D and T3D architectures, network sizes, temporal depths, frame resolutions, and sampling rates.The experiments include architecture and input-data studies before benchmark comparisons.
- Architecture and input search: TTL adds variable temporal convolution kernel sizes between 3D DenseBlocks, with features densely propagated through subsequent layers.The layer is intended to improve spatial-temporal connections and extract more informative features.
- Architecture and input search: 224×224 input frames yield better performance than 112×112 in the frame-resolution study, so the final T3D uses 224×224 inputs on Kinetics.
- Architecture and input search: Sampling rate 2 gives the best result among the evaluated temporal strides and is used for T3D and ResNet3D experiments.
- Transfer learning: The 2D→3D supervision transfer initializes a 3D CNN from an ImageNet-pretrained 2D CNN and allows training with a fraction of a large dataset.The paper reports that this reduces the need for large labeled datasets while retaining good fine-tuning performance on UCF101.
- Benchmark comparisons: T3D achieves state-of-the-art performance among 3D ConvNets on HMDB51 and UCF101 and competitive results on Kinetics.On Kinetics, RGB-I3D performs better, while the comparison notes longer clips, larger minibatches, and 64 GPUs.
- Benchmark comparisons: 93.2% on UCF101 and 63.5% on HMDB51 are reported for T3D and DenseNet3D, outperforming Res3D, Inception3D, and C3D on both datasets.T3D performs almost 4% better than Inception3D on UCF101; RGB-I3D reports higher results, with differences attributed to longer clips and larger minibatches.
- Benchmark comparisons: T3D achieves comparable performance to state-of-the-art methods without dense optical-flow maps.The paper presents this as avoiding the computational cost and scalability difficulty of obtaining optical-flow information.
5. Conclusion
The paper introduces TTL within T3D to model variable temporal depths and reports strong action-recognition results. It also presents cross-architecture supervision transfer from 2D to 3D ConvNets for stable initialization and reduced computational costs.
- TTL models temporal convolution at variable depths, capturing feature interactions over shorter and longer time ranges.
- T3D extends DenseNet with 3D convolutions, densely propagates TTL feature-maps, and learns them end-to-end.
- T3D achieves state-of-the-art performance on HMDB51 and UCF101 and comparable results on Kinetics against other temporal deep neural networks.
- Supervision transfer from 2D to 3D ConvNets provides stable weight initialization instead of training 3D models from scratch.The approach also avoids associated computational costs and can be adopted across modalities.