Source-linked AI summary
Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks
Pichao Wang, Zhaoyang Li, Yonghong Hou, Wanqing Li
TL;DR
The paper addresses how to use ConvNets for skeleton-sequence action recognition while retaining spatio-temporal information. It converts joint trajectories into Joint Trajectory Maps and reports effective encoding across three public benchmarks, with the method outperforming previous ones.
Problem
The central problem is how to model and use spatio-temporal information for action recognition without insufficient temporal modeling causing overfitting.
Method
The method constructs three Joint Trajectory Maps from skeleton sequences in orthogonal planes, feeds them to three ConvNets, and late-fuses their classifications.
Results
The method was evaluated on MSRC-12, G3D, and UTD-MHAD, and the proposed encoding techniques effectively captured spatio-temporal information for ConvNet action recognition.
Takeaways & Limitations
Encoding skeleton trajectories as images supports real-time human action recognition and allows image-recognition methods to be leveraged.
Takeaways & Limitations
Temporal-axis normalization makes jog and walk produce very similar JTMs, limiting their discrimination on UTD-MHAD.
Abstract
from arXiv · showhide
Recently, Convolutional Neural Networks (ConvNets) have shown promising performances in many computer vision tasks, especially image-based recognition. How to effectively use ConvNets for video-based recognition is still an open problem. In this paper, we propose a compact, effective yet simple method to encode spatio-temporal information carried in $3D$ skeleton sequences into multiple $2D$ images, referred to as Joint Trajectory Maps (JTM), and ConvNets are adopted to exploit the discriminative features for real-time human action recognition. The proposed method has been evaluated on three public benchmarks, i.e., MSRC-12 Kinect gesture dataset (MSRC-12), G3D dataset and UTD multimodal human action dataset (UTD-MHAD) and achieved the state-of-the-art results.
1 Introduction
The paper targets the difficulty of using ConvNets for video action recognition when temporal information is insufficiently modeled. It encodes joint trajectories into texture images, allowing ConvNets to learn discriminative features.
- ConvNet-based video recognition can lose temporal information, especially when training data is insufficient, leading to overfitting.
- Joint Trajectory Maps encode joint trajectories into texture images that serve as ConvNet inputs for action recognition.
- Action recognition must model spatio-temporal information without letting spatial or temporal cues dominate disproportionately.
- The encoding approach lets ConvNets learn discriminative features while retaining spatio-temporal information in image form.
2 The Proposed Method
The method constructs three Joint Trajectory Maps from 3D skeleton sequences projected onto orthogonal planes, encoding spatial-temporal motion with HSV colors for ConvNet-based recognition. Multiple colormaps distinguish body parts, while saturation and brightness represent motion magnitude.
- Joint Trajectory Maps: Three ConvNets process JTMs constructed from skeleton trajectories projected onto three orthogonal Cartesian planes, with late fusion producing the final classification.The framework uses one JTM and ConvNet for each plane.
- Joint Trajectory Maps: JTM construction accumulates a frame-wise function f(i) that encodes spatial-temporal information over an action.The accumulation requires f(i) to preserve useful spatial-temporal information with little adverse impact over time.
- Encoding Joint Motion Direction: Hue represents trajectory motion direction and assigns temporal positions to a common-length jet colormap.The encoding preserves direction through color variation even when actions contain different numbers of cycles.
- Encoding Body Parts: Three colormaps distinguish left, right, and middle body parts, using a jet map, its reversal, and a light-gray-to-black map, respectively.This separates trajectories from body regions with different motion patterns.
- Encoding Motion Magnitude: Saturation and brightness encode joint motion magnitude, enriching trajectory texture and highlighting faster motion for ConvNets.Saturation is linked to joint speed, while brightness is modulated according to speed within a specified range.
- Implementation: The ConvNets use the layer configuration from, ImageNet-pretrained initialization, mini-batch stochastic gradient descent, and 256 × 256 JTM inputs.Training uses momentum 0.9, weight decay 0.0005, and mini-batches of 256 shuffled training JTMs.
3 Experimental Results
Experiments evaluated the JTM encoding schemes and the proposed ConvNet method on three public skeleton-action benchmarks. The results indicate that the encoding captures spatio-temporal information effectively, while confusion matrices expose action-specific weaknesses.
- Evaluation setup: The evaluation covered MSRC-12, G3D, and UTD-MHAD, with comparisons against previously reported methods on the same datasets.The experiments also assessed individual encoding schemes, including front-plane JTM representations on G3D.
- Encoding evaluation: The proposed encoding techniques effectively capture spatio-temporal information, and ConvNets learn discriminative features from the resulting JTMs.This conclusion was supported by the encoding-scheme evaluation on G3D using the front JTM.
- MSRC-12 Kinect Gesture Dataset: On MSRC-12, most actions were distinguished well, but “goggles” and “had enough” remained difficult to separate.The similar JTM appearance was attributed probably to the 3D-to-2D projection.
- G3D Dataset: On G3D, the proposed method outperformed LRBM in spatial information mining and distinguished several action pairs that LRBM confused.These included “tennis swing forehand” versus “bowling” and “jump” versus “walk”; “aim and fire gun” versus “wave” also benefited from motion-magnitude encoding.
- UTD-MHAD: On UTD-MHAD, some actions such as “jog” and “walk” were difficult to distinguish because temporal normalization produced very similar JTMs.The comparison method used depth and inertial sensor data rather than skeleton data alone.
4 Conclusion
The paper encodes skeleton joint trajectories as Joint Trajectory Maps and applies ConvNets to learn discriminative features for real-time human action recognition. The authors report effective encoding techniques and identify data augmentation as future work.
- 4 Conclusion: The method converts joint trajectories into JTMs where motion information is represented as texture patterns.ConvNets then learn discriminative features from these maps.
- 4 Conclusion: The approach applies ConvNets to skeleton sequences for real-time human action recognition.
- 4 Conclusion: The authors report that the encoding techniques worked effectively and propose effective data augmentation as future work.