Source-linked AI summary
Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks
Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, Xiaohui Xie
TL;DR
Skeleton-based action recognition needs representations that capture both joint configurations and temporal dynamics. The paper proposes a fully connected deep LSTM with co-occurrence regularization and in-depth dropout, and reports state-of-the-art performance across several datasets.
Problem
Skeleton-based action recognition requires robust intra-frame features and models of interframe temporal dependencies, while prior handcrafted subnet designs ignore inherent joint co-occurrences.
Method
An end-to-end fully connected deep LSTM learns joint co-occurrence features through regularization and applies in-depth dropout to LSTM gates, cells, and outputs.
Results
The model consistently outperforms other state-of-the-art algorithms for skeleton-based human action recognition on three public datasets.
Takeaways & Limitations
The proposed regularization enables automatic co-occurrence learning from skeleton joints while the in-depth dropout supports effective deep LSTM learning.
Abstract
from arXiv · showhide
Skeleton based action recognition distinguishes human actions using the trajectories of skeleton joints, which provide a very good representation for describing actions. Considering that recurrent neural networks (RNNs) with Long Short-Term Memory (LSTM) can learn feature representations and model long-term temporal dependencies automatically, we propose an end-to-end fully connected deep LSTM network for skeleton based action recognition. Inspired by the observation that the co-occurrences of the joints intrinsically characterize human actions, we take the skeleton as the input at each time slot and introduce a novel regularization scheme to learn the co-occurrence features of skeleton joints. To train the deep LSTM network effectively, we propose a new dropout algorithm which simultaneously operates on the gates, cells, and output responses of the LSTM neurons. Experimental results on three human action recognition datasets consistently demonstrate the effectiveness of the proposed model.
1 Introduction
Skeleton-based action recognition uses skeletal trajectories to represent actions, while the proposed deep LSTM learns features and motion dynamics with regularization for joint co-occurrences.
- 1 Introduction: Skeleton-based representations provide an attractive option for recognizing human actions from their temporal joint evolution.Kinect v2 can generate accurate skeletons in real time, supporting skeleton-based action recognition.
- 1 Introduction: Skeleton-based recognition requires both discriminative intra-frame features and temporal modeling of interframe action dynamics.Prior approaches use models including hierarchical maximum entropy Markov models, hidden Markov models, and Conditional Random Fields for temporal dependencies.
- 1 Introduction: The paper proposes an end-to-end fully connected deep LSTM network for automatic feature learning and motion modeling.Alternating LSTM and feedforward layers capture motion information and model relationships among joints.
- 1 Introduction: Regularization drives fully connected layers to learn joint co-occurrence features and applies new dropout to LSTM neurons for complex motion dynamics.The regularization is designed to improve feature and dynamics learning while mitigating over-fitting.
- 1 Introduction: The proposed model consistently outperforms other state-of-the-art algorithms on three public skeleton-based action-recognition datasets.The result is reported as consistent across the evaluated datasets.
2 Related Work
Related work contrasts learned neural representations with handcrafted or limited temporal models and motivates automatic mining of action-relevant joint co-occurrences. It also motivates dropout that operates within LSTM neurons rather than only on their outputs.
- 2 Related Work: Earlier neural approaches either classify individual frames with MLPs or use shallow bidirectional LSTMs, limiting temporal or architectural depth.A deep recurrent model with handcrafted subnets still ignores inherent joint co-occurrences.
- 2 Related Work: Actions are characterized by interactions among subsets of joints, motivating automatic co-occurrence mining instead of pre-specifying joint groups.Examples include hand, arm, and head interactions during making a telephone call and interactions between two people’s joints.
- 2 Related Work: Prior recurrent-network dropout commonly avoids recurrent connections to preserve sequence modeling and has considered only LSTM output responses.The paper identifies the internal cell and gate units as additional parts for dropout design.
- 2 Related Work: The proposed in-depth dropout addresses LSTM’s internal structure by operating on its gates, cells, and output responses.This extends the dropout design beyond output-only regularization.
3 Deep LSTM with Co-occurrence Exploration and In-depth Dropout
The paper combines a fully connected deep bidirectional LSTM with co-occurrence regularization and in-depth LSTM dropout for skeleton-based action recognition. The model learns joint-feature connections and regularizes gates, cells, and outputs while modeling temporal dependencies.
- Architecture: The network alternates three bidirectional LSTM layers with two feedforward layers and a softmax prediction layer, using 3D skeleton-joint positions as inputs at each time step.
- Co-occurrence Exploration: The method is motivated by action-specific joint interactions, because different actions can involve different correlated subsets of joints, including joints from separate body parts.Walking and drinking are presented as examples with distinct active joint sets.
- Co-occurrence Exploration: Co-occurrence exploration uses fully connected layers so neurons can automatically learn discriminative joint or feature connections rather than fixed part-based subnetworks.Neurons are divided into groups so different groups can explore different conjunction patterns.
- Co-occurrence Exploration: The co-occurrence regularization combines an ℓ1 term with a structural ℓ21 term to select discriminative inputs and encourage group-wise column-sparse conjunctions.The structural norm is applied to grouped connection matrices, while different groups learn different co-occurrence patterns.
- In-depth Dropout: The proposed LSTM dropout can drop internal gates, the cell, and output responses, while dropout is applied along layers rather than recurrent time connections.At testing, all neurons are used and the affected units are multiplied by 1−p; output-only dropout is a special case.
- Action Recognition: The learned network converts a sequence into class probabilities and selects the action class with the highest probability.The class probabilities are computed from the output responses of the last bidirectional LSTM layer.
4 Experiments
Experiments evaluate the proposed schemes on SBU, HDM05, and CMU skeleton-action datasets using cross-validation protocols. The combined co-occurrence regularization and in-depth dropout scheme achieves the strongest reported performance, while learned weights reveal both within-part and cross-part joint interactions.
- SBU Kinect Interaction Dataset: On SBU, co-occurrence regularization and in-depth dropout improve average recognition accuracy over basic Deep LSTM by 3.4% and 4.1%, respectively.In-depth dropout outperforms simple dropout, and the combined regularizations achieve the best performance.
- SBU Kinect Interaction Dataset: Compared with hierarchical RNN on SBU, the deep LSTM network achieves a 5.6% performance improvement.The authors attribute this comparison to fully connecting joints and features rather than imposing predefined subnet constraints.
- HDM05 and CMU Datasets: The full model outperforms the manually designed hierarchical RNN approach on HDM05 and achieves significant improvement on the challenging CMU dataset.The CMU dataset includes strongly varying sequence lengths, substantial within-class diversity, and complex actions such as dance and yoga.
- Discussions: Visualization shows that co-occurrence regularization learns strong within-part and cross-part joint connections, while different neuron groups prefer different joint conjunctions.The visualization uses absolute first-layer input-gate weights for 30 joints and groups of LSTM neurons on SBU.
5 Conclusion
The paper presents an end-to-end fully connected deep LSTM for skeleton-based action recognition, learning joint co-occurrences through regularization and improving LSTM training with specialized dropout. Experiments demonstrate state-of-the-art performance across several datasets.
- The model automatically learns feature co-occurrences from skeleton joints through designed regularization.
- An in-depth dropout algorithm regularizes internal gates, cells, and output responses of LSTM neurons.
- Experimental results demonstrate state-of-the-art performance on several human action recognition datasets.