Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

Zuxuan Wu; Xi Wang; Yu-Gang Jiang; Hao Ye; Xiangyang Xue

Source-linked AI summary

Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

Zuxuan Wu, Xi Wang, Yu-Gang Jiang, Hao Ye, Xiangyang Xue

arXiv:1504.01561v1Apr 7, 2015cs.CVcs.MM

TL;DR

Video classification needs representations that capture spatial appearance, short-term motion, and longer-term temporal structure. The paper proposes a hybrid framework combining separate CNN streams, LSTM temporal modeling, and regularized feature fusion, and reports best performance on UCF-101 and CCV. Its components are complementary, while temporal modeling is less effective for long, noisy videos or classes without clear temporal order.

Problem
Existing approaches do not model long-term temporal clues because short-term optical-flow order can be discarded, limiting recognition of complex sequential content.
Method
The framework extracts spatial and short-term motion features with separate CNNs, models each feature sequence with LSTMs, and combines video-level features through regularized fusion.
Results
The hybrid framework achieves the best reported performance on UCF-101 and CCV, with its LSTM classifiers complementary to traditional order-agnostic classifiers.
Takeaways & Limitations
Combining long-term sequence modeling with spatial-motion fusion is particularly effective for long videos containing rich temporal clues.
Takeaways & Limitations
On CCV, LSTM models perform worse than CNN models for long, noisy videos and classes lacking clear temporal order.

Abstract

from arXiv · show

Classifying videos according to content semantics is an important problem with a wide range of applications. In this paper, we propose a hybrid deep learning framework for video classification, which is able to model static spatial information, short-term motion, as well as long-term temporal clues in the videos. Specifically, the spatial and the short-term motion features are extracted separately by two Convolutional Neural Networks (CNN). These two types of CNN-based features are then combined in a regularized feature fusion network for classification, which is able to learn and utilize feature relationships for improved performance. In addition, Long Short Term Memory (LSTM) networks are applied on top of the two features to further model longer-term temporal clues. The main contribution of this work is the hybrid learning framework that can model several important aspects of the video data. We also show that (1) combining the spatial and the short-term motion features in the regularized fusion network is better than direct classification and fusion using the CNN with a softmax layer, and (2) the sequence-based LSTM is highly complementary to the traditional classification strategy without considering the temporal frame orders. Extensive experiments are conducted on two popular and challenging benchmarks, the UCF-101 Human Actions and the Columbia Consumer Videos (CCV). On both benchmarks, our framework achieves to-date the best reported performance: $91.3\%$ on the UCF-101 and $83.5\%$ on the CCV.

1. INTRODUCTION

Video classification must capture static spatial content, short-term motion, and long-term temporal structure. The proposed hybrid framework combines CNN features, LSTM temporal modeling, and regularized fusion for video-level prediction.

Video classification of human actions and complex events remains challenging despite extensive study.
Existing CNN-based approaches capture spatial information and short-term optical-flow motion but discard the order of short-term actions.
The framework uses separate CNNs for spatial frames and stacked optical flows, then applies LSTMs to model long-term temporal clues.
A regularized feature-fusion network learns correlations between spatial and motion features for video-level fusion and classification.
The sequence-based LSTM classifiers complement traditional classifiers that ignore temporal frame order.
Extensive experiments report that the framework outperforms alternative methods and achieves the best performance on UCF-101 and CCV.

2. RELATED WORKS

Related work spans handcrafted and CNN-based video representations, temporal modeling with graphical and recurrent models, and early or late feature fusion. The paper identifies limitations in conventional fusion and short-term temporal modeling that motivate its approach.

Video representations have evolved from handcrafted spatial-temporal features toward learned CNN-based representations from raw data.
Graphical models such as HMMs, Bayesian networks, and CRFs have been used to represent long-term temporal information in videos.
Early fusion combines features before classification, whereas late fusion combines outputs from independently trained classifiers.
The paper argues that conventional early and late fusion do not explore shared feature correlations and introduces regularized neural-network fusion to learn dimension-wise correlations.
RNN and LSTM approaches have been explored for video temporal modeling, including video description, motion prediction, and classification.

3. METHODOLOGY

The framework combines two-stream CNN features for static spatial and short-term motion information with LSTMs for long-term temporal modeling, then fuses predictions for video classification. A regularized fusion network explores correlations between spatial and motion features, while separate training preserves flexibility and avoids the limited gains reported for joint training.

Spatial and Motion CNN Features: Two CNN streams separately extract static spatial features from individual frames and short-term motion features from stacked optical flows.The spatial stream captures scene backgrounds and basic objects, while the motion stream uses displacement fields computed between adjacent frames.
Temporal Modeling with LSTM: The CNN training discards frame order, so two LSTM models process spatial and motion feature sequences to model long-term temporal clues.LSTM hidden states recursively represent sequence information, and the final time-step outputs provide video-level prediction scores.
Temporal Modeling with LSTM: LSTM maps feature sequences through controllable memory units and gates that regulate information retention, updating, and output over time.The paper identifies the input, forget, memory-cell, and output activations and describes the current input and previous hidden state as recurrent inputs.
Regularized Feature Fusion Network: A regularized fusion network combines spatial and motion representations using an ℓ21 norm to exploit shared discriminative information while preserving feature-specific characteristics.The structural regularizer encourages row-sparse fusion weights, producing similar zero/nonzero patterns across feature columns.
Discussion: The framework models spatial, short-term motion, and long-term temporal clues, while separate component training supports replacing CNN or LSTM modules without retraining the entire system.The authors note that joint training is more complex and report only a limited improvement in a cited prior benchmark example, from 70.5% to 71.1%.

4.1 Experimental Setup

The experiments evaluate the framework on UCF-101 and CCV using established dataset protocols, CNN feature extractors, LSTM temporal models, a regularized fusion network, and multiple comparison methods.

Datasets: UCF-101 contains 13,320 clips from 101 human-action classes, evaluated using three train/test splits with classification accuracy.
Datasets: CCV contains 9,317 videos across 20 classes, using 4,659 training and 4,658 test videos and mean average precision as the overall measure.
Feature extraction: Spatial and motion features are extracted with VGG 19 and CNN M, respectively, with the spatial CNN pretrained on ImageNet before video fine-tuning.
Temporal modeling: The temporal models use two-layer LSTMs for both spatial and motion features, with 1,024 bottom-layer and 512 upper-layer hidden units.
Feature fusion: The regularized fusion network separately abstracts spatial and motion features, fuses them with structural norms, and applies logistic regression for classification.
Compared approaches: Evaluation compares the proposed components and fusion network with two-stream CNN, separate LSTMs, SVM fusion methods, and neural-network fusion alternatives.

4.2 Results and Discussions

Experiments show that long-term temporal modeling and regularized fusion provide complementary gains across UCF-101 and CCV, yielding the strongest results when combined. The hybrid framework achieves high performance while retaining computational efficiency, though LSTM performance can decline on long, noisy videos.

Temporal Modeling: LSTM models underperform CNN models on both CCV streams because videos are longer and noisier, and some classes lack clear temporal order.CCV videos average 80 seconds, around ten times longer than UCF-101 videos.
Temporal Modeling: 90.1% on UCF-101 and 81.7% on CCV are achieved by fusing all CNN and LSTM models, the highest performance in Table 1.The gains include approximately 1% and 4% on UCF-101, and 12% on CCV's motion stream, relative to the best single models.
Feature Fusion: The regularized feature fusion network consistently outperforms alternative neural-network fusion methods, while slightly surpassing RDNN at much lower complexity.The comparison includes M-DBM and the two-stream CNN approach.
Feature Fusion: Adding regularization to the same fusion network improves performance by 1.4% on UCF-101 and 0.8% on CCV.The baseline uses the same network structure without regularization.
The Hybrid Framework: 91.3% on UCF-101 and 83.5% on CCV are obtained by the complete hybrid framework, improving over the regularized fusion network by 2.9% and 7.3%.The framework combines temporal LSTM outputs with the regularized fusion network using cross-validated linear weights.
The Hybrid Framework: On CCV, combining LSTM outputs with regularized video-level predictions significantly improves results for almost all classes, including object-related classes such as cat and dog.Cat videos often contain a single cat running around on the floor, providing a temporal clue for the LSTM.
Efficiency: The framework's UCF-101 testing pipeline takes 3.9 seconds for frame and flow extraction, 9 seconds for CNN features, and 2.8 seconds for prediction on one video.These timings are reported for an 8-second UCF-101 video on a single NV system.
Comparison with State of the Arts: The hybrid framework produces the highest reported performance on both UCF-101 and CCV, although LSTM comparisons with prior work are not directly comparable because input features differ.The approach relies fully on deep learning techniques for UCF-101 and outperforms recent CCV methods using multiple features.

5. CONCLUSIONS

The proposed hybrid framework combines CNN-based spatial and motion features with LSTM temporal modeling and regularized fusion, achieving strong benchmark performance. The authors identify improved temporal modeling as a priority for future work and note that audio features could be incorporated.

The framework models static visual features, short-term motion patterns, and long-term temporal clues using CNNs, LSTMs, and regularized feature fusion.Spatial and motion features are extracted separately, passed to LSTMs for temporal modeling, and combined through video-level fusion.
Combining frame-level temporal modeling with video-level fusion and classification produces highly complementary information and a substantial performance gain.
The authors state that effective architectures for sequential video data remain urgently needed, despite deep learning's success on other problems.
Further improving temporal modeling is identified as a high-priority direction, while audio features are presented as an easily incorporable extension.