Source-linked AI summary
Two Stream LSTM: A Deep Fusion Framework for Human Action Recognition
Harshala Gammulle, Simon Denman, Sridha Sridharan, Clinton Fookes
TL;DR
The paper addresses human action recognition from video by combining spatial CNN features with temporal LSTM modeling. It evaluates four CNN-LSTM fusion methods, finding that hierarchical two-stream fusion improves recognition and reaches or exceeds state-of-the-art results across three datasets.
Problem
Human action recognition requires exploiting both spatial and temporal information, while limited action datasets make training CNNs from scratch challenging.
Method
The paper evaluates four CNN-to-LSTM designs, including separate streams for final convolutional and first fully connected features with hierarchical fusion and joint back propagation.
Results
The proposed methods outperform state-of-the-art models on UCF11 and UCF Sports, while fu-2 matches state-of-the-art results on jHMDB.
Takeaways & Limitations
Fully connected features act as an attention mechanism directing the LSTM toward relevant portions of convolutional feature sequences.
Takeaways & Limitations
Training from scratch is constrained by limited dataset sizes, including around 150 UCF11 videos per action class.
Abstract
from arXiv · showhide
In this paper we address the problem of human action recognition from video sequences. Inspired by the exemplary results obtained via automatic feature learning and deep learning approaches in computer vision, we focus our attention towards learning salient spatial features via a convolutional neural network (CNN) and then map their temporal relationship with the aid of Long-Short-Term-Memory (LSTM) networks. Our contribution in this paper is a deep fusion framework that more effectively exploits spatial features from CNNs with temporal features from LSTM models. We also extensively evaluate their strengths and weaknesses. We find that by combining both the sets of features, the fully connected features effectively act as an attention mechanism to direct the LSTM to interesting parts of the convolutional feature sequence. The significance of our fusion method is its simplicity and effectiveness compared to other state-of-the-art methods. The evaluation results demonstrate that this hierarchical multi stream fusion method has higher performance compared to single stream mapping methods allowing it to achieve high accuracy outperforming current state-of-the-art methods in three widely used databases: UCF11, UCFSports, jHMDB.
1. Introduction
Human action recognition remains challenging in realistic videos because background clutter and occlusions complicate recognition. The paper targets the complementary spatial and temporal information needed to distinguish visually similar actions.
- Real-world action recognition is difficult when videos contain background clutter and occlusions.
- Combining spatial and temporal information is widely regarded as a strong opportunity for recognizing actions from video sequences.
- Spatial cues such as green backgrounds and golf equipment can distinguish golf swing from ride bike but confuse it with croquet swing.
- The proposed approach combines convolutional and fully connected CNN activations with a two-stream LSTM to improve action recognition.
- The paper evaluates four databases and experimental methods, with sections covering related work, the model, experiments, and conclusions.
2. Relation to previous efforts
Prior work includes static, handcrafted, CNN-LSTM, and attention-based approaches, but the paper argues that combining both convolutional and fully connected CNN features in multiple streams is underexplored.
- Video-based approaches represent both spatial and temporal aspects of human actions, unlike static approaches that rely on background or body-part information.
- Handcrafted features commonly use engineered descriptors and classifiers, but they capture information only at an abstract level.
- Some methods combine deep CNN features with recurrent models, including LSTMs that learn sequential relationships from framewise CNN features.
- CNN feature-selection approaches range from hyper-columns using several layers to methods extracting only dense or convolutional representations.
- LSTM attention models learn where to focus in each frame, but incorrect attention can lead to misclassification and learning attention weights is exhaustive.
- This paper differs from single-dense-layer CNN-LSTM methods by jointly using the final convolutional and first fully connected layers in multiple streams.
3. Action Recognition Model
The action-recognition model extracts framewise CNN features and models their temporal relationships with LSTMs through four fusion designs. It uses pretrained VGG-16 representations and hierarchical two-stream fusion to combine complementary feature types.
- Action Recognition Model: The framework extracts features from each video frame with a CNN and feeds them sequentially into an LSTM for sequence classification.
- Convolutional Neural Network (CNN): A pretrained CNN is used because available action-recognition datasets, such as UCF11 with around 150 videos per class, are too limited for training from scratch.
- Convolutional Neural Network (CNN): VGG-16 contains 13 convolutional layers followed by 3 fully connected layers, with features extracted from the last convolutional and first fully connected layers.
- Convolutional Neural Network (CNN): Fine-tuning produces salient CNN activations around the athlete’s body, stage floor, and equipment, which are combined and compressed in deeper layers.
- Combining CNNs and LSTMs for Action Recognition: The four models comprise single-stream conv-L and fc-L systems plus fused fu-1 and fu-2 systems that enable joint back propagation between streams.
- Conv-L + fc-L fusion method 1 (fu-1): The third model merges the outputs of Equations 2 and 5 and applies a soft-max layer for final classification.
- Conv-L + fc-L fusion method 2 (fu-2): The fu-2 architecture represents each video as hidden-state sequences before a second LSTM produces one video-level hidden unit, enabling hierarchical information capture.
Xi,T
The model uses both sequence-to-sequence and sequence-to-one LSTM configurations to process video representations and produce final predictions.
- Figure 5 distinguishes sequence-to-one LSTM and sequence-to-sequence LSTM* configurations used in the final model.
- The resultant prediction sequence is passed to a final sequence-to-one LSTM that produces a single output for the video.
4. Experiments
Experiments evaluate four CNN–LSTM fusion models across UCF11, UCF Sports, and jHMDB, using dataset-specific sampling and validation protocols. The results favor deep two-stream fusion, with fu-2 outperforming or matching state-of-the-art baselines while combining complementary spatial and discriminative features.
- Datasets and evaluation: The approach was evaluated on UCF11, UCF Sports, and jHMDB using comparisons against state-of-the-art methods.UCF11 contains 1600 videos across 11 classes, UCF Sports contains 150 sequences across 10 classes, and jHMDB contains 923 videos across 21 classes.
- Datasets and evaluation: Videos were sampled at evenly spaced positions, using T equal to the shortest sequence length within each database.For a video with N frames, the method selects a T-frame subsequence using rounded frame intervals.
- Datasets and evaluation: The experiments used leave-one-out cross-validation for UCF11 and UCF Sports, while jHMDB used its three prescribed splits and averaged their performance.The CNN was fine-tuned and the LSTM trained for each database; all LSTMs used 100 hidden units and dropout ratio 0.25.
- Overall results: fc-L, fu-1, and fu-2 outperformed state-of-the-art models on UCF11 and UCF Sports, while fu-2 matched state-of-the-art results on jHMDB.These comparisons are reported from Table 1 across all three datasets.
- Overall results: fu-2 achieved the best accuracy among the four proposed models, with its deep layer-wise fusion improving accuracy over fu-1 across all three databases.Unlike fu-1, fu-2 couples the convolutional and fully connected streams through a merged LSTM layer and joint backpropagation.
- Model behavior: Fully connected features were more discriminative but sparse, whereas convolutional outputs retained spatial information useful for distinguishing objects and sub-actions.Direct conv-L mapping had the lowest accuracy, while fc-L was more prone to confusing target actions with multiple classes.
- Dataset-specific results: On UCF11, fu-2 reached 94.6 accuracy, 5.1 percent higher than the previous highest result reported in [26].Residual confusions involved classes with similar backgrounds and motions, including volleyball spiking, basketball shooting, cycling, and walking.
- Dataset-specific results: On UCF Sports, automatic CNN and LSTM feature learning learned salient regions and substantially improved discrimination of confusable sports actions.The reported confusion included kicking, skating, golf swing, and skate boarding in the comparison with.
5. Conclusion
The proposed approach trains LSTMs on convolutional outputs and evaluates four fusion methods, finding that multi-stream fusion improves recognition across three challenging datasets.
- The approach uses convolutional layer outputs to train an LSTM for human action recognition.
- Four fusion methods combine convolutional neural network outputs with LSTM networks and outperform state-of-the-art approaches on three challenging datasets.
- Two LSTM streams using final convolutional and first fully connected layer outputs recognize actions better than either stream alone.
- A third LSTM layer further improves performance by combining the two streams, while fully connected outputs direct attention to important convolutional feature regions.