Source-linked AI summary
Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks
Lin Sun, Kui Jia, Dit-Yan Yeung, Bertram E. Shi
TL;DR
Training 3D convolutional kernels for human action recognition is difficult because of high kernel complexity and insufficient training videos. FSTCN factorizes learning into sequential spatial and temporal convolutions, and experiments show improved recognition performance on benchmark datasets without additional training videos.
Problem
High kernel complexity and insufficient training videos make learning effective 3D convolutional kernels for action recognition difficult.
Method
FSTCN sequentially learns 2D spatial kernels in lower SCL layers and 1D temporal kernels in upper TCL layers, with a transformation and permutation operator enabling factorization.
Results
FSTCN experiments on benchmark action-recognition datasets show superiority without additional training videos; concatenating SCL and TCL features yields about 10% gain.
Takeaways & Limitations
The factorized architecture provides effective spatio-temporal features, while SCI-based score fusion and video-clip sampling further improve recognition performance.
Takeaways & Limitations
The factorization is restricted to 3D kernels expressible in factorized form, and non-low-rank kernels may lose representation power.
Abstract
from arXiv · showhide
Human actions in video sequences are three-dimensional (3D) spatio-temporal signals characterizing both the visual appearance and motion dynamics of the involved humans and objects. Inspired by the success of convolutional neural networks (CNN) for image classification, recent attempts have been made to learn 3D CNNs for recognizing human actions in videos. However, partly due to the high complexity of training 3D convolution kernels and the need for large quantities of training videos, only limited success has been reported. This has triggered us to investigate in this paper a new deep architecture which can handle 3D signals more effectively. Specifically, we propose factorized spatio-temporal convolutional networks (FstCN) that factorize the original 3D convolution kernel learning as a sequential process of learning 2D spatial kernels in the lower layers (called spatial convolutional layers), followed by learning 1D temporal kernels in the upper layers (called temporal convolutional layers). We introduce a novel transformation and permutation operator to make factorization in FstCN possible. Moreover, to address the issue of sequence alignment, we propose an effective training and inference strategy based on sampling multiple video clips from a given action video sequence. We have tested FstCN on two commonly used benchmark datasets (UCF-101 and HMDB-51). Without using auxiliary training videos to boost the performance, FstCN outperforms existing CNN based methods and achieves comparable performance with a recent method that benefits from using auxiliary training videos.
1. Introduction
Human action recognition must model both spatial appearance and temporal motion in complex 3D video signals. FSTCN addresses this challenge by factorizing spatio-temporal learning and adding operators, sampling, and score fusion strategies.
- Motivation: Complex human actions challenge methods that must represent spatial appearance and temporal motion in 3D video signals.Performance can degrade significantly for realistic actions with large intra-category variation.
- Motivation: Image-based deep architectures learn spatially discriminative features, but action recognition also requires exploiting the temporal domain.Directly applying image-based CNNs has shown promising performance because learned features characterize visual appearance spatially.
- FSTCN architecture: FSTCN factorizes 3D convolution learning into sequential 2D spatial kernels in lower layers and 1D temporal kernels in upper layers.The factorized scheme reduces the number of parameters, mitigating high kernel complexity and insufficient training video data.
- FSTCN architecture: The T-P operator forms an intermediate FSTCN layer that facilitates learning temporal convolution kernels in subsequent layers.It is introduced specifically to make the factorized architecture possible.
- Training and inference: Multiple sampled video clips address sequence alignment by improving robustness to variations caused by misaligned action sequences.Clips are temporally sampled with a stride and spatially cropped from the same location in the source sequence.
- Additional contribution: SCI-based score fusion gives greater weight to class-probability score vectors with higher sparsity and consistently improves over existing fusion schemes.The complete FSTCN stacks spatial and temporal convolutional components and is trained globally using back-propagation.
2. The proposed deep architecture
FSTCN approximates 3D spatio-temporal convolution through sequential spatial and temporal processing, reducing kernel complexity while preserving separate appearance and motion modeling. Its architecture combines SCLs, a T-P operator, TCLs, and a parallel appearance branch, with sampled clips and SCI fusion supporting robustness and inference.
- Factorized convolution: 3D convolution kernels are challenging to learn because of their high complexity and insufficient training videos.The paper contrasts this with learning 2D spatial kernels and motivates lower-complexity approximations.
- Factorized convolution: FSTCN restricts 3D kernels to a Kronecker factorization of a 2D spatial kernel and a 1D temporal kernel.Although general equivalence does not hold, this factorization enables sequential spatial-then-temporal convolution.
- Factorized convolution: Sequentially convolving each frame spatially and each resulting temporal vector temporally reduces kernel complexity from nxnynt to nxny + nt.The factorized scheme can also use existing image datasets to support learning spatial kernels.
- Network architecture: FSTCN stacks spatial convolutional layers, inserts a transformation and permutation operator, and places a temporal convolutional layer above them to model evolving motion patterns.The T-P operator reorganizes feature channels so subsequent temporal convolution has better local support across feature-channel and temporal directions.
- Network architecture: A parallel spatial convolutional layer learns abstract appearance features, and fully connected and classifier layers combine the temporal and spatial representations.The additional spatial branch is intended to improve spatial invariance by extracting salient appearance features.
- Clip sampling and fusion: The method samples multiple spatially cropped, temporally strided clips, including corresponding difference clips, to improve robustness to sequence misalignment.SCI score fusion consistently improves over commonly used averaging during inference.
3. Experiments
Experiments on UCF-101 and HMDB-51 show that FSTCN’s factorized architecture, clip sampling, SCI fusion, and combined spatial-temporal features improve recognition, matching or surpassing CNN-based comparisons without auxiliary videos.
- Datasets: FSTCN was evaluated on UCF-101 and HMDB-51, reporting mean classification accuracy over the datasets’ three splits.UCF-101 contains 101 categories and more than 13K videos; HMDB-51 contains 51 categories and 6766 videos.
- Experimental setup: Five temporally sampled video-clip pairs used dt = 9 and st = 5 to represent long-range motion dynamics.The experiments used sampled clip pairs as inputs to FSTCN.
- Controlled comparisons: Using two TCL kernels outperformed either single-kernel choice, and the larger kernel outperformed the smaller one.These results were obtained without the proposed score-fusion scheme.
- Controlled comparisons: 48.4% TCL-path accuracy exceeded the 46.6% optical-flow CNN-stream result on HMDB-51 split 1.The comparison used only split 1 training videos for the reported results, while the competing method also used auxiliary training videos elsewhere in the table.
- Ablation studies: About 10% gain resulted when SCL and TCL features were concatenated and trained globally, while clip sampling and SCI fusion also improved recognition.The authors characterize the SCL and TCL features as complementary.
- State-of-the-art comparison: FSTCN outperformed the state-of-the-art CNN-based method by about 1% on both datasets with averaging fusion and achieved better or comparable performance overall.The compared results used mean accuracy over three dataset splits; the competing methods used auxiliary training videos, unlike these FSTCN results.
4. Visualization
Visualization analyses indicate that FSTCN learns discriminative action features and focuses on representative image regions, while closely related facial actions remain difficult to separate.
- Saliency visualization: Back-propagated saliency maps suggest that FSTCN captures representative regions for several action categories.The maps were generated by back-propagating an action-category classifier neuron to the input image domain.
- Feature visualization: t-SNE visualizations examine learned features for seven HMDB-51 facial-action categories involving mouth movements.The categories include smile, laugh, chew, talk, eat, smoke, and drink.
- Feature visualization: The visualized facial-action categories cannot be easily distinguished because their actions are mainly concerned with similar face motions.The passage specifically highlights mouth movements as the shared visual characteristic.
5. Conclusion
FSTCN is a cascaded architecture for learning spatio-temporal action features, with factorization addressing training difficulty and auxiliary components improving representation. Visualizations and experiments support the learned features’ relevance to action recognition.
- FSTCN learns effective spatio-temporal features through a cascaded architecture trained with standard back-propagation.
- Saliency maps visualize representative regions for smile, clap, pull-up, and climbing action categories.
- The factorization design mitigates the compound difficulty of high kernel complexity and insufficient training videos.
- The T-P operator provides a novel feature and temporal representation for actions, while two parallel TCL kernels learn more representative temporal features.
- An additional SCL extracts more abstract spatial appearance, compensating for TCL’s deficiency as shown in the experimental results.