Source-linked AI summary
Lattice Long Short-Term Memory for Human Action Recognition
Lin Sun, Kui Jia, Kevin Chen, Dit Yan Yeung, Bertram E. Shi, Silvio Savarese
TL;DR
Video action recognition requires modeling both appearance and long-term motion, but conventional CNNs and convolutional recurrent models are limited by temporal or spatial assumptions. The paper proposes L2STM with spatially independent memory transitions, multimodal gate training, and long-short-term sampling, and reports superior benchmark performance against comparable LSTM- and CNN-based methods.
Problem
Existing CNN and recurrent architectures struggle to represent long-term, complex, and spatially non-stationary motion dynamics in videos.
Method
L2STM uses independent memory-cell transitions across spatial locations, jointly trains RGB-and-flow input and forget gates, and samples clips to capture long- and short-term temporal information.
Results
On UCF-101 and HMDB-51, the proposed methods outperform existing LSTM- and CNN-based methods with similar model complexities.
Takeaways & Limitations
L2STM provides a motion-modeling architecture that enhances, encodes, and selects motion dynamics for human action recognition.
Takeaways & Limitations
More complex deep architectures achieve about 1% to 2% better performance than the authors’ baseline.
Abstract
from arXiv · showhide
Human actions captured in video sequences are three-dimensional signals characterizing visual appearance and motion dynamics. To learn action patterns, existing methods adopt Convolutional and/or Recurrent Neural Networks (CNNs and RNNs). CNN based methods are effective in learning spatial appearances, but are limited in modeling long-term motion dynamics. RNNs, especially Long Short-Term Memory (LSTM), are able to learn temporal motion dynamics. However, naively applying RNNs to video sequences in a convolutional manner implicitly assumes that motions in videos are stationary across different spatial locations. This assumption is valid for short-term motions but invalid when the duration of the motion is long. In this work, we propose Lattice-LSTM (L2STM), which extends LSTM by learning independent hidden state transitions of memory cells for individual spatial locations. This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity. Additionally, we introduce a novel multi-modal training procedure for training our network. Unlike traditional two-stream architectures which use RGB and optical flow information as input, our two-stream model leverages both modalities to jointly train both input gates and both forget gates in the network rather than treating the two streams as separate entities with no information about the other. We apply this end-to-end system to benchmark datasets (UCF-101 and HMDB-51) of human action recognition. Experiments show that on both datasets, our proposed method outperforms all existing ones that are based on LSTM and/or CNNs of similar model complexities.
1. Introduction
L2STM addresses the difficulty of modeling long-term, complex, and spatially non-stationary motion in videos by combining spatially varying memory transitions with multimodal gate training. Experiments on UCF-101 and HMDB-51 report better performance than comparable LSTM- and CNN-based methods.
- CNNs capture frame-level appearance, but they do not model interactions across the temporal domain.
- Existing convolutional recurrent models work well for stationary motions but struggle with long-term and complex motions.
- L2STM learns independent memory-cell transitions for individual spatial locations to address non-stationary long-term motion dynamics without significantly increasing model complexity.
- RGB and optical flow jointly train shared input and forget gates, improving control over information entering and leaving the memory cell.
- A long-short-term sampling strategy uses randomly extracted clips to learn both long- and short-term temporal information and augment the temporal domain.
- Experiments on UCF-101 and HMDB-51 show that the proposed methods outperform existing LSTM- and CNN-based methods with similar model complexities.
2. Related Work
Related work combines CNNs, optical flow, and recurrent models to capture appearance and motion in videos. However, existing approaches remain limited by short temporal coverage, spatially shared filters, or complex architectures with limited gains.
- Frame-based CNNs obtain temporal information through pooling, but early multi-frame fusion produced only marginal gains over a single-frame baseline.
- C3D learns space-time convolution kernels but covers only a short range of the video sequence.
- Two-stream CNNs improve recognition by combining appearance and optical-flow features, even through simple probability-score fusion.
- ConvLSTM incorporates neighboring-pixel relations and learns spatial patterns over time, while LSTM-based methods model temporal dependencies.
- VideoLSTM adds attention and another recurrent network for motion modeling, but its performance depends heavily on improved dense trajectories and shows limited improvement.
3. Models and Algorithm
L2STM addresses the difficulty of modeling long-term, spatially nonstationary motion by giving memory-cell transitions location-specific capacity and jointly training RGB-flow control gates. Its end-to-end design combines lattice recurrence with temporal sampling that exposes recurrent models to both short- and long-term dynamics.
- Revisiting RNN modeling: Long-term video motion requires more capacity than shared short-range filters provide, but directly learning long-duration filters makes model size and filter count prohibitively large.The limitation arises because long-term dynamics require large temporal support, increasing learning difficulty and model complexity.
- Revisiting RNN modeling: Standard RNN transitions reuse shared weights across time, while ConvLSTM applies the same patch-level filters at different spatial locations, limiting spatially varying motion modeling.The recurrent derivation identifies shared transition parameters as the source of this restriction.
- Lattice-LSTM: L2STM applies independently initialized local filters to each memory-cell location, using local superposition on the hidden transition to model spatially nonhomogeneous long-term dynamics.Only the hidden transition of the cell memory uses local superposition; other linear combinations remain convolutional.
- Lattice-LSTM: RGB and optical flow are jointly used to train shared input and forget gates, allowing both modalities to control information entering and leaving the memory cell.The gates regulate the dynamic information stored in the updated cell memory.
- Long-Short Term Sampling: Randomized crop centers and temporal strides generate clips spanning stationary short-term motion and changing long-term motion for end-to-end recurrent training.This sampling procedure provides about a 2% gain over fixed-stride traditional sampling and is used throughout the experiments.
4. Experiments
Experiments evaluate L2STM on UCF-101 and HMDB-51 using RGB and optical-flow inputs, component analyses, visualizations, and comparisons with LSTM-like methods. Results show stronger handling of complex motion, with gains concentrated in interactive actions and performance advantages over comparable LSTM architectures.
- Evaluation setup: Experiments use UCF-101 and HMDB-51, reporting mean classification accuracy across their three train-test splits.UCF-101 contains more than 13K videos across 101 actions, while HMDB-51 contains 6766 videos across 51 categories.
- Implementation details: The system uses VGG16 features for RGB and optical flow, trains L2STM from scratch, and applies random-stride clips with eight unrolled time steps.Each training step uses eight clips of five frames, while convolutional and superposition kernels are 3×3.
- Visualization: Saliency maps show L2STM detects more dynamic regions than ConvLSTM, including small scissors movements and both leg and lifting movements.The paper attributes this visualization to local superposition and multimodal control of input and forget gates.
- Complex movements: L2STM improves over ConvLSTM on UCF-101 coarse categories, especially human-object interaction by 10.7% and human-human interaction by 6.3%.The improvement is smaller for body-motion-only actions, at 0.5%, where movements are simpler.
- Component analysis: On UCF-101 split 1, L2STM outperforms its LSTM variants, with local superposition helping temporal networks and shared gates helping spatial networks more.The comparison analyzes the contributions of local superposition and shared control gates.
- State-of-the-art comparison: L2STM performs best among the compared LSTM-like architectures despite not using 1M-sports pre-training, while more complex baseline networks are about 1%–2% better.The paper also compares against traditional and deep-learning approaches on UCF-101 and HMDB-51.
5. Conclusion
The conclusion presents L2STM as a solution for action recognition that combines spatially varying memory operations, multimodal gates, and long-short-term sampling. The paper reports large gains over LSTM-like architectures and state-of-the-art performance relative to more complex networks.
- Method: L2STM applies spatially varying superposition operations to memory cells and trains input and forget gates using RGB and optical flow.The method also introduces long-short-term sampling to improve recurrent learning.
- Conclusion: The paper reports a huge gain over other LSTM-like action-recognition architectures and state-of-the-art performance against more complex networks.The conclusion frames the result as evidence that space-variant filters can model semantic meanings in CNN feature maps across time.
- Future work: An extension proposed by the authors is to adaptively group semantic meanings based on videos rather than learning them independently at each location.This is identified as future work beyond the presented space-variant filtering approach.