Source-linked AI summary
Differential Recurrent Neural Networks for Action Recognition
Vivek Veeriah, Naifan Zhuang, Guo-Jun Qi
TL;DR
Conventional LSTMs do not explicitly account for salient spatio-temporal motion when gating information across video frames. This paper proposes dRNNs, which use different-order Derivative of States to control LSTM gates, and reports state-of-the-art performance on 2D and 3D action-recognition datasets while outperforming conventional LSTMs.
Problem
Conventional LSTMs gate information without explicitly considering whether video frames contain salient motion patterns, limiting their sensitivity to dynamic action evolution.
Method
dRNNs extend LSTMs by using different-order Derivative of States to make gate units sensitive to changes in spatio-temporal action dynamics.
Results
dRNNs achieve state-of-the-art performance on 2D and 3D action-recognition datasets and outperform conventional LSTMs with the same feature or input sequence.
Takeaways & Limitations
The results demonstrate the potential of high-order state derivatives for learning complex time-series representations of human actions.
Takeaways & Limitations
Fair model comparisons require the same input feature, and higher-order DoS may increase computational cost without compensating performance gains.
Abstract
from arXiv · showhide
The long short-term memory (LSTM) neural network is capable of processing complex sequential information since it utilizes special gating schemes for learning representations from long input sequences. It has the potential to model any sequential time-series data, where the current hidden state has to be considered in the context of the past hidden states. This property makes LSTM an ideal choice to learn the complex dynamics of various actions. Unfortunately, the conventional LSTMs do not consider the impact of spatio-temporal dynamics corresponding to the given salient motion patterns, when they gate the information that ought to be memorized through time. To address this problem, we propose a differential gating scheme for the LSTM neural network, which emphasizes on the change in information gain caused by the salient motions between the successive frames. This change in information gain is quantified by Derivative of States (DoS), and thus the proposed LSTM model is termed as differential Recurrent Neural Network (dRNN). We demonstrate the effectiveness of the proposed model by automatically recognizing actions from the real-world 2D and 3D human action datasets. Our study is one of the first works towards demonstrating the potential of learning complex time-series representations via high-order derivatives of states.
1. Introduction
Conventional LSTMs integrate information from every frame without explicitly distinguishing salient motion, limiting their sensitivity to dynamic action patterns. The proposed dRNN addresses this with derivative-based gating and achieves strong performance on 2D and 3D action-recognition datasets.
- Motivation: LSTMs model sequential dynamics with memory cells and input, output, and forget gates controlling information over time.These gates determine how dynamic information enters and leaves the memory cell.
- Problem: Many action-video frames contain non-salient motions that are irrelevant to the performed action.Conventional LSTMs integrate available frame information without explicitly identifying salient spatio-temporal motion.
- Proposed approach: The dRNN learns salient spatio-temporal action representations by making gate units sensitive to the dynamic evolution of hidden states.It extends the LSTM model to address the conventional gates’ insensitivity to salient dynamic patterns.
- Proposed approach: Different-order Derivative of States (DoS) values guide dRNN gates in selecting information needed to model action dynamics.Training uses truncated back propagation to limit exploding or diminishing errors through time.
- Results: dRNNs achieve state-of-the-art performance on both 2D and 3D action-recognition datasets, outperforming existing LSTMs with the same input sequences.They remain competitive with algorithms designed around special spatio-temporal assumptions.
2. Related Work
Action recognition seeks to understand human activities despite substantial variation across actors, speeds, environments, and imaging conditions. Prior work uses engineered spatio-temporal descriptors and deep models, including 3DCNNs and LSTMs, but descriptors often target specific structures.
- Problem: Action recognition aims to automatically understand activities performed by people interacting with their environment and one another.The task is difficult because actions vary across actors, speeds, camera angles, lighting, and cluttered backgrounds.
- Engineered representations: Handcrafted descriptors represent specific motion structures using 3D gradients, scale-invariant video-volume features, or tracked-joint positions.Examples include HOG3D, 3D-SIFT, and actionlet ensemble.
- Deep approaches: These descriptors have achieved success but are usually engineered ad hoc for particular spatio-temporal structures.Deep neural networks such as 3DCNNs and RNNs were subsequently applied to action recognition.
- Deep approaches: An LSTM combining Harris3D and 3DCNN descriptors achieved state-of-the-art performance on the KTH dataset.This illustrates the use of recurrent models with spatio-temporal descriptors.
3. Background
RNNs model sequences through evolving hidden states, while LSTMs add memory cells and gates to preserve long-range dependencies. Their outputs are converted into class probabilities for action recognition.
- Recurrent neural networks: A classical RNN updates each hidden state by combining the current frame input with the previous hidden state.The hidden state sequence models the spatio-temporal structure of the input frames.
- Recurrent neural networks: The hidden state is mapped to output vectors whose entries represent confidence scores for action classes.A softmax transformation converts these outputs into class probabilities for each frame.
- Long short-term memory: Classical RNNs struggle with long-term video representation because retained context decays exponentially over time.LSTM was designed to learn long-range dependencies between output labels and input frames.
- Long short-term memory: LSTMs use memory cells with internal states and three gate types to control information entering and leaving over time.The gates are activated by nonlinear functions of input/output sequences and internal states.
- Long short-term memory: The input gate controls how incoming information influences the current cell state, while the forget gate controls the previous state’s contribution.The output gate controls information emitted from the memory cell toward future states.
4. Differential Recurrent Neural Networks
The dRNN extends LSTM action modeling by using derivatives of memory-cell states to detect salient spatio-temporal changes and control information flow. It computes discretized derivatives, updates gated memory states, and trains with backpropagation procedures adapted to recurrent dynamics.
- Differential Recurrent Neural Networks: The model addresses conventional LSTM gates that do not explicitly account for salient motion patterns in video frames.Many frames contain non-salient motions irrelevant to the performed action, motivating dynamic saliency modeling.
- Differential Recurrent Neural Networks: dRNN uses Derivative of States (DoS) to detect salient spatio-temporal dynamics and control information entering and leaving the memory cell.Large DoS magnitudes indicate informative action-state changes, while small magnitudes gate incoming information out.
- Discretized Model: Higher-order DoS captures higher-order action dynamics, with first-order DoS corresponding to velocity and second-order DoS to acceleration.The paper considers the first two DoS orders, while higher orders can be derived similarly.
- Learning Algorithm: The model uses discretized DoS up to order N and truncated backpropagation to limit exponentially decaying or exploding errors through time.Truncated derivatives prevent back-propagated errors from re-entering memory cells or gates after leaving them.
- Recurrent Computation: At each time step, dRNN computes input and forget gates, updates the state, computes the output gate, and produces a gated output.A softmax can optionally convert the output into an action label.
- Learning Algorithm: Training minimizes loss against either the final video-level label or cumulative frame-level labels using backpropagation through time.For a video-level label, the loss is minimized at time T; frame-level supervision uses cumulative sequence loss.
5. Experiments and Results
Experiments evaluate dRNN on 2D KTH and 3D MSR Action3D datasets, using shared architectures and dataset-specific frame features. The model converges during training, outperforms conventional LSTM with the same features, and remains competitive with specialized approaches.
- The study compares dRNN with state-of-the-art LSTM and non-LSTM methods on 2D and 3D human action datasets.
- KTH experiments use six action classes across KTH-1 and KTH-2, while MSR Action3D contains 20 depth-based action classes.KTH videos use densely sampled HOG3D features; MSR Action3D uses concatenated position, angle, and joint-based features.
- The same dRNN architecture is used for both DoS orders and across the two datasets, with fewer memory-cell units than input units.The paper interprets the smaller state space as sufficient for action trajectories embedded in a low-dimensional manifold.
- The objective loss continuously decreases over 50 epochs, with training usually beginning to converge after 40 epochs.The reported learning rate for truncated back propagation through time is 0.0001.
- With the same HOG3D feature, both dRNN orders outperform conventional LSTM on KTH, and 2-order dRNN exceeds its 1-order counterpart.The paper reports 92.12% accuracy for 2-order dRNN with HOG3D on KTH-2, compared with 87.78% for LSTM with Harris3D.
- On MSR Action3D, both dRNN orders outperform conventional LSTM with the same feature and perform competitively against other algorithms.The comparison follows an evaluation setting in which half of the actors train and the remainder test across all 20 action classes.
6. Conclusion and Future Work
The paper presents dRNNs as an LSTM extension that models evolving state dynamics through derivatives, improving salient spatio-temporal action representation. Experiments on 2D and 3D human-action datasets show performance exceeding conventional LSTMs and remaining competitive with specialized approaches.
- dRNNs extend LSTM by modeling the dynamics of states evolving over time.
- Their gates are controlled by different orders of state derivatives to learn salient spatio-temporal action structure.
- Experiments on 2D and 3D human-action datasets show that dRNNs outperform conventional LSTMs.
- The general-purpose dRNN model remains competitive with approaches built on strong assumptions about motion structure.
- Future work will combine dRNN with more sophisticated input feature sequences to explore specific action-motion structures.