Source-linked AI summary
SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition
Carlos Caetano, Jessica Sena, François Brémond, Jefersson A. dos Santos, William Robson Schwartz
TL;DR
Skeleton-image methods for 3D action recognition often encode spatial structure while representing temporal dynamics indirectly or naively. SkeleMotion explicitly encodes joint-motion magnitude and orientation across temporal scales, and its combination with a spatial structural representation outperforms the best reported method on NTU RGB+D 120 by up to 3.1 p.p. on cross-subject evaluation.
Problem
Existing skeleton-image methods do not explicitly model rich motion information across multiple temporal scales, limiting how temporal dynamics are represented.
Method
SkeleMotion represents skeleton sequences for CNNs by explicitly encoding joint-motion magnitude and orientation at multiple temporal scales.
Results
Up to 3.1 p.p. improvement over Body Pose Evolution Map was achieved on the cross-subject protocol when SkeleMotion was combined with Yang et al.’s representation.
Takeaways & Limitations
SkeleMotion can capture temporal dynamics explicitly while preserving spatial joint relations, supporting effective skeleton-based 3D action recognition.
Takeaways & Limitations
Future work will evaluate SkeleMotion with other architectures and on 2D action datasets using estimated skeletons.
Abstract
from arXiv · showhide
Due to the availability of large-scale skeleton datasets, 3D human action recognition has recently called the attention of computer vision community. Many works have focused on encoding skeleton data as skeleton image representations based on spatial structure of the skeleton joints, in which the temporal dynamics of the sequence is encoded as variations in columns and the spatial structure of each frame is represented as rows of a matrix. To further improve such representations, we introduce a novel skeleton image representation to be used as input of Convolutional Neural Networks (CNNs), named SkeleMotion. The proposed approach encodes the temporal dynamics by explicitly computing the magnitude and orientation values of the skeleton joints. Different temporal scales are employed to compute motion values to aggregate more temporal dynamics to the representation making it able to capture longrange joint interactions involved in actions as well as filtering noisy motion values. Experimental results demonstrate the effectiveness of the proposed representation on 3D action recognition outperforming the state-of-the-art on NTU RGB+D 120 dataset.
1. Introduction
Skeleton data offers a compact, robust basis for 3D action recognition, while existing skeleton-image methods encode temporal dynamics mainly through spatially organized matrices. SkeleMotion instead uses explicit multiscale motion information and achieves state-of-the-art performance when combined with a spatial joint representation.
- Skeleton data is computationally efficient and robust to illumination changes, background noise, and camera views.
- Existing skeleton-image representations encode temporal dynamics as matrix variations while organizing each frame’s spatial structure across the representation.These compact representations encode an entire video sequence in one image and can be processed by CNNs.
- SkeleMotion explicitly computes skeleton-joint motion magnitude and orientation at multiple temporal scales for CNN input.The scales aggregate longer-range dynamics and filter noisy motion values.
- Unlike prior approaches, SkeleMotion models explicit motion information in multiple temporal scales rather than only using consecutive-frame differences.
- SkeleMotion achieves state-of-the-art performance on NTU RGB+D 120 when combined with a spatial structural joint representation.
2. Related Work
Prior skeleton-image methods encode spatial structure and temporal dynamics through joint arrangements, trajectory maps, reference-joint configurations, or consecutive-frame differences. SkeleMotion addresses limited motion richness by explicitly encoding orientation and magnitude across temporal scales while preserving relevant joint relations.
- Du et al. represent skeleton sequences as matrices whose rows contain concatenated joint coordinates and whose columns track joint evolution over time.The matrix has size J × T × 3 before image quantization and normalization.
- Joint Trajectory Maps encode spatial configuration and joint-trajectory dynamics into three texture images using color encoding and orthogonal-plane projections.The method also rotates skeleton data to mimic multiple views and enlarge the data.
- Other methods address sparse sequence data with four reference-joint representations or encode motion through differences between consecutive frames.
- Tree Structure Skeleton Image preserves spatial relations by traversing the skeleton tree in depth-first order, prioritizing joint pairs with fewer edges.
- SkeleMotion differs from these methods by encoding motion explicitly through orientation and magnitude at different temporal scales.This design targets richer temporal information while retaining spatial relations through a structured joint organization.
3. Proposed Approach
SkeleMotion represents skeleton sequences through explicit joint-motion magnitude and orientation, organized as CNN-ready skeleton images. Multiple temporal scales aggregate dynamics, filter noisy motion, and expose longer-range joint interactions.
- SkeleMotion: SkeleMotion is a skeleton image representation based on magnitude and orientation motion information.It is designed as input to convolutional neural networks for 3D action recognition.
- SkeleMotion: The method orders joints with a depth-first traversal to preserve spatial relations, then arranges their coordinates across video frames.The resulting matrix S has dimensions C × T × 3, where C is the chain length and T is the frame count.
- SkeleMotion: Motion structure D computes each joint’s temporal displacement between frames separated by distance d.This produces a C × (T − d) × 3 matrix and supports motion representations at different temporal scales.
- SkeleMotion: The motion structure yields separate magnitude and orientation representations, with M sized J × (T − d) × 1 and θ sized J × (T − d) × 3.The three orientation channels correspond to the coordinate axes.
- SkeleMotion: Orientation values are filtered using a magnitude threshold m to suppress noisy estimates from joints without meaningful movement.The generated matrices are normalized to [0, 1] and resized to a fixed C × 100 representation.
- Temporal Scale Aggregation (TSA): Temporal-scale aggregation stacks representations computed with different d values, explicitly exposing more temporal dynamics to the network.The authors state that this allows the network to learn important movements and capture long-range joint interactions.
4. Experimental Results
Experiments evaluate SkeleMotion on NTU RGB+D 60 and NTU RGB+D 120 under established protocols, using a common implementation and network setup. Multi-scale motion encoding performs competitively on NTU RGB+D 60, improves through fusion with spatial representations, and reaches state-of-the-art results on NTU RGB+D 120.
- Experimental setup: The experiments compare skeleton image representations on NTU RGB+D 60 and state-of-the-art methods on NTU RGB+D 120.All compared representations use the same datasets, network architecture, data splits, evaluation protocols, and metrics to isolate representation effects.
- Temporal-scale optimization: Three temporal scales perform best for magnitude scales (5, 10, 15) and orientation scales (1, 10, 20), while four scales saturate or reduce performance.The optimization uses 20 frames in total on a cross-view subset of NTU RGB+D 60.
- NTU RGB+D 60: 69.6% accuracy is achieved by SkeleMotion Magnitude (TSA) on cross-subject NTU RGB+D 60, 12.8 p.p. above the Li et al. baseline.The Ke et al. Reference Joints representation remains higher at 70.8% under cross-subject evaluation.
- NTU RGB+D 60: 80.1% accuracy is achieved by SkeleMotion Magnitude (TSA) on cross-view NTU RGB+D 60, improving 4.5 p.p. over Yang et al.'s TSSI baseline.The improvement over Li et al.'s motion-encoding baseline is 18.8 p.p.
- Fusion: Late fusion of SkeleMotion with Yang et al.'s spatial representation reaches 76.5% cross-subject and 84.7% cross-view accuracy on NTU RGB+D 60.These results exceed the corresponding early-fusion results of 73.5% and 82.4%.
- NTU RGB+D 120: On NTU RGB+D 120, combining SkeleMotion with Yang et al. achieves state-of-the-art cross-subject performance, up to 3.1 p.p. above Body Pose Evolution Map.The combined method achieves competitive cross-setup results and uses late fusion.
5. Conclusions and Future Works
SkeleMotion is a CNN-oriented skeleton image representation that explicitly encodes joint motion through magnitude and orientation, with temporal scales that filter noise and aggregate dynamics. Experiments on two public datasets found strong performance, including state-of-the-art results on NTU RGB+D 120 when combined with existing methods.
- SkeleMotion encodes skeleton-joint temporal dynamics using motion magnitude and orientation in a CNN-compatible image representation.
- A temporal-scale variation of the magnitude representation filters noisy motion values while aggregating more temporal dynamics.
- Experiments on two publicly available datasets demonstrated excellent performance for the proposed approach.
- Combining SkeleMotion with methods from the literature improved 3D action recognition and outperformed the state of the art on NTU RGB+D 120.
- Future Works: Future work will evaluate SkeleMotion with distinct architectures and on 2D action datasets using skeletons estimated by methods from the literature.