Source-linked AI summary
Spatio-Temporal Channel Correlation Networks for Action Classification
Ali Diba, Mohsen Fayyaz, Vivek Sharma, M. Mahdi Arzani, Rahman Yousefzadeh, Juergen Gall, Luc Van Gool
TL;DR
The paper addresses limited channel-correlation modeling in 3D CNNs and the costly requirement for large labeled datasets and training from scratch. It introduces STC blocks and transfers knowledge from pretrained 2D CNNs to 3D CNNs, improving Kinetics accuracy by 2-3% while reducing training demands.
Problem
3D CNNs neglect hidden spatial and temporal correlations between channels, while training them from scratch requires large labeled datasets and substantial computation.
Method
The paper adds STC residual blocks to 3D CNNs and uses a pretrained 2D CNN as a teacher for stable initialization of a randomly initialized 3D CNN.
Results
2-3% accuracy improvements on Kinetics were obtained by equipping 3D-ResNet and 3D-ResNext with STC blocks, with evaluations spanning HMDB51, UCF101, and Kinetics.
Takeaways & Limitations
Knowledge transfer enables training with a fraction of a large dataset while still achieving good fine-tuning performance on UCF101.
Takeaways & Limitations
Comparison with some state-of-the-art CNNs is constrained because their experiments require expensive optical-flow inputs and high computation.
Abstract
from arXiv · showhide
The work in this paper is driven by the question if spatio-temporal correlations are enough for 3D convolutional neural networks (CNN)? Most of the traditional 3D networks use local spatio-temporal features. We introduce a new block that models correlations between channels of a 3D CNN with respect to temporal and spatial features. This new block can be added as a residual unit to different parts of 3D CNNs. We name our novel block 'Spatio-Temporal Channel Correlation' (STC). By embedding this block to the current state-of-the-art architectures such as ResNext and ResNet, we improved the performance by 2-3\% on Kinetics dataset. Our experiments show that adding STC blocks to current state-of-the-art architectures outperforms the state-of-the-art methods on the HMDB51, UCF101 and Kinetics datasets. The other issue in training 3D CNNs is about training them from scratch with a huge labeled dataset to get a reasonable performance. So the knowledge learned in 2D CNNs is completely ignored. Another contribution in this work is a simple and effective technique to transfer knowledge from a pre-trained 2D CNN to a randomly initialized 3D CNN for a stable weight initialization. This allows us to significantly reduce the number of training samples for 3D CNNs. Thus, by fine-tuning this network, we beat the performance of generic and recent methods in 3D CNNs, which were trained on large video datasets, e.g. Sports-1M, and fine-tuned on the target datasets, e.g. HMDB51/UCF101.
1 Introduction
The paper identifies two limitations of 3D CNNs: neglected inter-channel spatio-temporal correlations and costly training from scratch. It proposes an STC block and supervision transfer from pretrained 2D CNNs to address them.
- 3D CNNs learn local correlations along input channels but neglect hidden correlations between channels across space and time.
- The proposed STC block captures spatial-channel and temporal-channel correlation information throughout network layers.
- A pretrained 2D CNN transfers knowledge to a randomly initialized 3D CNN for stable weight initialization without training the 3D model from scratch.
- The paper evaluates STC-equipped 3D architectures on HMDB51, UCF101, and Kinetics datasets.
2 Related Work
Prior video-classification methods evolved from hand-engineered spatio-temporal descriptors to deep architectures using 2D or 3D information. Related work also includes transfer learning across datasets and modalities.
- Video Classification with and without CNNs: Hand-engineered descriptors such as HOG3D, SIFT3D, HOF, MBH, and iDTs were developed for spatio-temporal video representation.
- Video Classification with and without CNNs: Deep-learning methods learn spatio-temporal representations end to end using frame-level 2D models or video-level 3D models.
- Video Classification with and without CNNs: 3D convolutional architectures use filters and pooling kernels spanning spatial dimensions and time, with RGB clips or optical-flow frames as input.
- Transfer Learning: Transfer learning commonly specializes representations learned on another dataset, and prior work has transferred knowledge across modalities including RGB, depth, optical flow, and sound.
3 Proposed Method
The proposed method combines a dual-branch STC block with cross-architecture supervision transfer to enrich 3D CNN representations and avoid training them from scratch.
- Spatio-Temporal Channel Correlation (STC) Block: STC captures correlations between channels across spatial and temporal dimensions in 3D CNN feature maps.The block extracts relations between channels at different network layers.
- Spatio-Temporal Channel Correlation (STC) Block: STC blocks are inserted after convolutional blocks in 3D ResNet and ResNext architectures and added to their residual layers.The two branch outputs are averaged before being combined with the residual layer.
- Spatio-Temporal Channel Correlation (STC) Block: The STC block uses separate spatial and temporal correlation branches to model complementary channel information.Its dual-path structure includes channel or filter information embedding and dependency capture.
- Spatio-Temporal Channel Correlation (STC) Block: The temporal branch applies global spatio-temporal pooling, bottleneck fully connected layers, sigmoid activation, and channel-wise rescaling.The first fully connected layer reduces the feature dimension to C/r, and the second restores it to C.
- Spatio-Temporal Channel Correlation (STC) Block: The spatial branch performs global spatial pooling after shrinking channel-wise information over time, then models temporal-channel relations.Its pooled descriptor has size T × C, followed by fully connected layers of sizes (T × C)/r and C.
- Knowledge Transfer: The knowledge-transfer method uses a frozen pretrained 2D CNN as a teacher for a randomly initialized 3D CNN through image-video correspondence.Frames and video clips from the same timestamp form positive pairs, while samples from different videos form negative pairs; only the 3D CNN is updated.
4 Experiments
Experiments evaluate STC-Nets across HMDB51, UCF101, and Kinetics, including architecture, temporal-input, branch, sampling, transfer-learning, and state-of-the-art comparisons. Results show that STC blocks improve compact 3D models, longer clips and combined branches help, and 2D-to-3D supervision transfer provides stable initialization for small-dataset training.
- Evaluation setup: The study evaluates STC-Nets on HMDB51, UCF101, and Kinetics using standard dataset protocols and reports validation or test performance as appropriate.HMDB51 and UCF101 results are averaged over three splits, while Kinetics results are reported on validation and test sets.
- Input and branch ablations: Longer input clips yield better performance, and combining spatial and temporal correlation branches outperforms either branch alone.These findings come from temporal-depth and TCB-versus-SCB ablations.
- Input and branch ablations: A frame sampling rate of 2 produces the best STC-ResNet101 results and is used for the other reported 3D CNN experiments.The sampling-rate study evaluates temporal strides of 1, 2, 4, and 16 on UCF101 split 1.
- Knowledge transfer: 2D-to-3D supervision transfer from ImageNet-pretrained models enables stable initialization and provides a starting model for training on small datasets such as UCF101.The transfer experiments use 2D ResNet and ResNext teachers with randomly initialized 3D-ResNet and STC-ResNet targets, and approximately 500K unlabeled YouTube8M videos.
- Comparison with the state-of-the-art: STC-ResNext101 outperforms 3D-ResNet, Inception3D, and C3D on UCF101 and HMDB51, while STC-ResNet achieves higher Kinetics accuracy than several listed 3D baselines.The reported STC-ResNext101 accuracies are 95.8% on UCF101 and 72.6% on HMDB51; RGB-I3D remains competitive under different experimental conditions.
5 Conclusion
The paper introduces STC-Nets, which add spatio-temporal channel-correlation blocks to 3D CNNs and achieve strong action-recognition performance. It also demonstrates supervision transfer from 2D to 3D CNNs.
- Contributions: STC blocks model correlations between channels in 3D CNNs and can be inserted as residual units.They are learned end-to-end within other network architectures.
- Results: 3D-ResNet and 3D-ResNext equipped with STC blocks improve accuracy by 2-3% on Kinetics.
- Results: STC-Nets achieve state-of-the-art performance on HMDB51 and UCF101, with comparable results on Kinetics.
- Contributions: The STC architecture models feature interactions more expressively and efficiently without undesired information loss through the network.
- Contributions: The paper demonstrates supervision transfer between 2D and 3D CNN architectures.