Source-linked AI summary
Pose-conditioned Spatio-Temporal Attention for Human Action Recognition
Fabien Baradel, Christian Wolf, Julien Mille
TL;DR
Human action recognition must combine articulated pose with information-rich RGB video despite limited datasets and high visual complexity. The paper proposes a pose convolutional stream with topology-preserving joint ordering and a pose-conditioned RGB attention stream, followed by temporal attention. It achieves state-of-the-art results on NTU-RGB+D and SBU, and near-state-of-the-art performance on MSR Daily Activity 3D.
Problem
Human action recognition from multimodal video requires using articulated pose and RGB cues while handling high information volume and limited training data.
Method
A two-stream model processes topology-ordered pose sub-sequences convolutionally and uses pose-conditioned spatial and temporal attention over RGB features.
Results
The method achieves state-of-the-art performance on NTU-RGB+D and SBU, with performance close to state-of-the-art on MSR Daily Activity 3D.
Takeaways & Limitations
Pose can guide both where RGB evidence is sampled and how it is weighted, while adaptive temporal pooling supports multimodal activity recognition.
Takeaways & Limitations
The MSR and SBU evaluations are constrained by very small training sets of 160 and 225 videos, respectively.
Abstract
from arXiv · showhide
We address human action recognition from multi-modal video data involving articulated pose and RGB frames and propose a two-stream approach. The pose stream is processed with a convolutional model taking as input a 3D tensor holding data from a sub-sequence. A specific joint ordering, which respects the topology of the human body, ensures that different convolutional layers correspond to meaningful levels of abstraction. The raw RGB stream is handled by a spatio-temporal soft-attention mechanism conditioned on features from the pose network. An LSTM network receives input from a set of image locations at each instant. A trainable glimpse sensor extracts features on a set of predefined locations specified by the pose stream, namely the 4 hands of the two people involved in the activity. Appearance features give important cues on hand motion and on objects held in each hand. We show that it is of high interest to shift the attention to different hands at different time steps depending on the activity itself. Finally a temporal attention mechanism learns how to fuse LSTM features over time. We evaluate the method on 3 datasets. State-of-the-art results are achieved on the largest dataset for human activity recognition, namely NTU-RGB+D, as well as on the SBU Kinect Interaction dataset. Performance close to state-of-the-art is achieved on the smaller MSR Daily Activity 3D dataset.
1. Introduction
The paper addresses limited and information-heavy human activity recognition by fusing articulated pose with raw RGB video. It combines pose-based representations and attention mechanisms to focus spatially and temporally on activity-relevant evidence.
- 1. Introduction: The method fuses pose and raw RGB streams, using pose for activity cues, glimpse locations, and conditioning RGB soft attention.Pose features guide the importance assigned to RGB glimpses, while raw joints select image positions.
- 1. Introduction: A topology-preserving joint ordering encodes pose sequences as 3D tensors for convolutional processing instead of recurrent modeling.The ordering preserves neighborhood relationships between body joints.
- 1. Introduction: Spatial attention is conditioned on pose features from the full sub-sequence, allowing the RGB model to weight activity-relevant glimpse locations.The RGB stream uses an LSTM, while pose features provide temporal context for attention decisions.
- 1. Introduction: Adaptive temporal attention learns how to pool LSTM outputs over time rather than using a fixed temporal aggregation.The mechanism assigns learned importance to recurrent features at different instants.
- 1. Introduction: Knowledge transfer from NTU’s 57000 activities to MSR Daily Activity 3D’s 300 videos is experimentally demonstrated.The authors describe this as an ImageNet-style transfer setting not previously attempted for human activities, to their knowledge.
2. Related Work
Prior work uses multimodal convolutional or recurrent models and separate pose processing, while attention methods typically operate within individual modalities. This paper combines pose-guided hard location choices with learned soft weighting over RGB features.
- 2. Related Work: Existing activity-recognition methods commonly stack RGB or depth frames into 3D volumes, while available pose data is typically processed by a separate network.Prior work also augments pose with velocities, accelerations, or normalization.
- 2. Related Work: Recurrent approaches include part-aware, hierarchical, bidirectional, and multidimensional LSTMs that model temporal or anatomical structure.Multidimensional variants can traverse joints according to topology-preserving paths.
- 2. Related Work: The pose model is convolutional rather than recurrent, despite using a joint traversal similar to prior multidimensional LSTM work.The RGB network remains recurrent.
- 2. Related Work: Hard attention selects input locations stochastically, whereas soft attention dynamically weights the full input and remains compatible with gradient-based optimization.These mechanisms have been applied to images, video, translation, and captioning.
- 2. Related Work: Unlike prior attention models, this method uses articulated pose and RGB simultaneously: pose determines locations while learned soft attention weights features from them.The location choices depend on pose rather than being learned, while the feature weighting is learned and pose-conditioned.
3. Proposed Model
The proposed model fuses articulated pose and RGB streams: a topology-aware convolutional pose network conditions spatial attention over hand-centered RGB glimpses, while temporal attention adaptively pools recurrent features.
- 3.1. Convolutional pose features: The pose stream uses a convolutional network on pose subsequences, producing features that represent temporal behavior and correlations among joints.The model uses a sub-sequence rather than only an individual frame, allowing pose features to capture a larger temporal range.
- 3.1. Convolutional pose features: A topology-aware joint ordering preserves body-neighborhood relationships in a 3D pose tensor, enabling convolutional layers to learn increasingly broader joint correlations.The tensor stacks time, ordered joint coordinates, and feature channels for coordinates, velocities, and accelerations; two people are stacked along the joint dimension.
- 3.2. Spatial Attention on RGB videos: The RGB stream extracts convolutional features from crops centered at predefined hand joints, then integrates the hand glimpses as input to a recurrent LSTM.The attention points are the four hands of the two interacting people, and the glimpse sensor is trainable and differentiable.
- 3.2. Spatial Attention on RGB videos: Spatial attention weights hand features using both the LSTM state and pose features, allowing attention to shift across hands as motion provides contextual cues.The pose-conditioned distribution dynamically weights the glimpse outputs rather than selecting a single learned location.
- 3.3. Temporal Attention: Temporal attention adaptively pools recurrent feature vectors over the subsequence instead of relying on fixed sum or average pooling.The temporal attention distribution is predicted from stacked temporal features and spatial-attention information, then used to weight the feature matrix.
4. Network architectures and Training
The model combines a convolutional pose stream with an RGB stream using trainable spatial and temporal attention. The RGB pathway uses pose-guided glimpses and recurrent processing to focus on relevant image regions over time.
- Architectures: The pose network uses three convolutional layers, while the RGB pathway uses an Inception V3 glimpse sensor, a 1024-unit LSTM, and learned spatial and temporal attention networks.The full model without the glimpse sensor has 38 million trainable parameters.
- Training: The glimpse sensor is pretrained on ILSVRC 2012, while the RGB stream is trained with pose and glimpse parameters frozen.End-to-end training did not improve performance.
- Attention: The attention example shows focus shifting to the active hands during an interaction.The figure illustrates spatial and temporal attention changing over time when something is given to another person.
5. Experiments
The method is evaluated on NTU RGB+D, MSR Daily Activity3D, and SBU Kinect Interaction using standard dataset protocols and transfer from NTU for smaller datasets. It achieves state-of-the-art performance on NTU and SBU, while remaining close to state of the art on MSR.
- Datasets: The experiments use three datasets, with NTU as the large benchmark and MSR and SBU as smaller transfer targets.MSR contains 320 videos and SBU contains 282 sequences, while the smaller datasets provide only 160 and 225 training videos, respectively.
- Results: The full pose-plus-RGB model reaches 94.1% on NTU, compared with 90.5% for pose only and 72.0% for RGB only.The NTU evaluation follows cross-subject and cross-view protocols.
- Results: The method achieves state-of-the-art performance on NTU and SBU, and performs close to state of the art on MSR.On MSR, it outperforms all methods in the hand-crafted-method group; NTU and MSR results include knowledge transfer from NTU.
- Ablations: Topology-respecting joint ordering improves NTU performance by more than one percentage point over random ordering.Retaining redundant double joint entries also improves performance, despite increasing trainable parameters.
- Ablations: Conditioning RGB attention on pose adds one point in the multimodal setting and more than 12 points in the RGB-only setting.The comparison is reported in the NTU pose-conditioned attention ablation.
- Runtime: A full prediction from features takes 1.4 ms on a Titan-X GPU, excluding RGB preprocessing, which takes about 1 second.The authors report that classification can therefore be performed close to real time.
6. Conclusion
The paper presents a pose-and-RGB method that combines convolutional pose processing, hand-centered soft attention, and temporal pooling. It reports state-of-the-art results on several benchmarks and introduces pose-conditioned attention and knowledge transfer for human action recognition.
- Conclusion: The method combines organized pose convolutions, hand-joint soft attention for RGB, and adaptive temporal pooling.The attention extracts hand-shape and manipulated-object information while the pose stream provides complementary activity cues.
- Conclusion: The method achieves state-of-the-art results on several benchmarks and is presented as the first approach to combine pose-RGB attention with knowledge transfer in human action recognition.