Source-linked AI summary
Make Skeleton-based Action Recognition Model Smaller, Faster and Better
Fan Yang, Sakriani Sakti, Yang Wu, Satoshi Nakamura
TL;DR
Skeleton-based action recognition often uses models that are too large or slow for efficient deployment. This paper proposes DD-Net, which combines compact invariant and two-scale motion features with a lightweight architecture, achieving strong performance on SHREC and JHMDB while running at 3,500 FPS on one GPU or 2,000 FPS on one CPU.
Problem
Existing skeleton-based action-recognition methods may suffer from large model sizes and slow execution speeds.
Source: §I · p. 1Method
DD-Net combines a location-viewpoint invariant JCD feature, a two-scale global motion feature, and a lightweight 1D-CNN network.
Source: §I · p. 1, B. Modeling Global Scale-invariant… · p. 3Results
DD-Net achieves superior results on SHREC and JHMDB while reaching around 3,500 FPS on one GPU and 2,000 FPS on one CPU.
Source: D. Result Analysis and Discussion · pp. 4–5Takeaways & Limitations
The results support DD-Net as an efficient model that generalizes across a wide range of skeleton-based action-recognition scenarios.
Source: D. Result Analysis and Discussion · pp. 4–5, §V · p. 5
Abstract
from arXiv · showhide
Although skeleton-based action recognition has achieved great success in recent years, most of the existing methods may suffer from a large model size and slow execution speed. To alleviate this issue, we analyze skeleton sequence properties to propose a Double-feature Double-motion Network (DD-Net) for skeleton-based action recognition. By using a lightweight network structure (i.e., 0.15 million parameters), DD-Net can reach a super fast speed, as 3,500 FPS on one GPU, or, 2,000 FPS on one CPU. By employing robust features, DD-Net achieves the state-of-the-art performance on our experimental datasets: SHREC (i.e., hand actions) and JHMDB (i.e., body actions). Our code will be released with this paper later.
I. INTRODUCTION
DD-Net addresses the large size and slow execution of existing skeleton-based action-recognition methods with a lightweight network built around complementary invariant and motion features. The authors report higher accuracy, efficient computation, and generalization across hand and body-action settings.
- Existing skeleton-based action-recognition methods may have large model sizes and slow execution speeds.Source: §I · p. 1
- DD-Net combines a location-viewpoint invariant JCD feature with a two-scale global motion feature.The design targets varied skeleton settings, including hand/body actions, 2D/3D skeletons, and actions with or without global trajectories.Source: §I · p. 1
- The JCD feature is compact and invariant, while the two-scale motion feature improves robustness to motion-scale variation.Source: §I · p. 1
- An embedding process lets DD-Net learn joint correlations that are difficult to predefine from joint indices.Source: §I · p. 2
- DD-Net provides higher action-recognition accuracy and computational efficiency than methods using more complicated model structures.Source: §I · p. 2
III. METHODOLOGY
The methodology represents each skeleton frame with Joint Collection Distances and processes the resulting sequence through a configurable lightweight 1D-CNN architecture. JCD preserves location-viewpoint invariance while reducing feature redundancy.
- DD-Net uses a 1D ConvNet architecture whose model size can be changed by modifying the filter count.The architecture includes convolutional layers, max pooling, global average pooling, and fully connected layers.Source: Figure 2 · p. 2, §III · p. 2
- Cartesian coordinates vary with skeleton location and viewpoint, whereas geometric features such as distances are location-viewpoint invariant.Source: A. Modeling Location-viewpoint… · p. 2
- The method is motivated by the need for a compact geometric representation that avoids dataset-specific redesign and redundant elements.Source: A. Modeling Location-viewpoint… · p. 2
- JCD computes Euclidean distances between pairs of collective joints and uses the symmetric matrix’s lower triangle without its diagonal.This removes redundant entries from the distance representation.Source: A. Modeling Location-viewpoint… · p. 2
- The flattened JCD feature is used as a one-dimensional model input.Source: A. Modeling Location-viewpoint… · p. 3
B. Modeling Global Scale-invariant Motions by a Two-scale Motion Feature
Because JCD omits global motion, DD-Net supplements it with location-invariant temporal differences computed at two time scales. The resulting feature captures both slower and faster motion patterns and is resized to match the JCD sequence length.
- JCD is location-viewpoint invariant but lacks global motion information, making it insufficient alone for actions tied to global trajectories.Source: B. Modeling Global Scale-invariant… · p. 3
- DD-Net computes temporal differences of Cartesian coordinates to obtain location-invariant global motion features.Source: B. Modeling Global Scale-invariant… · p. 3
- The method models both fast and slow global motions because the same action can occur at different motion scales.Source: B. Modeling Global Scale-invariant… · p. 3
- The two-scale motion sequences are reshaped into one-dimensional inputs and linearly interpolated to match the JCD feature’s frame count.Source: B. Modeling Global Scale-invariant… · p. 3
- The resulting two-scale global motion feature can be generated using only Cartesian-coordinate input.Source: B. Modeling Global Scale-invariant… · p. 3
C. Modeling Joint Correlations by an Embedding
DD-Net addresses the lack of local correlation among joint indices by embedding joint-distance and two-scale motion features before temporal modeling. This embedding automatically learns joint relationships, reduces skeleton-noise effects, and enables 1D ConvNet processing independent of joint ordering.
- Joint indices are not locally correlated, and their relationships can change across actions, making manual joint ordering difficult.Source: C. Modeling Joint Correlations by an… · p. 3
- DD-Net embeds JCD and two-scale motion features into latent vectors so joint correlations are learned automatically rather than predefined.The embedding also reduces the effect of skeleton noise.Source: C. Modeling Joint Correlations by an… · p. 3
- Embed1 and Embed2 use stacked 1D convolutions, with Embed2 additionally applying max pooling because the slow features have twice the temporal length of the fast features.Source: C. Modeling Joint Correlations by an… · p. 3
- The embedded features are concatenated into a representation εk using ⊕ as the concatenation operation.Source: C. Modeling Joint Correlations by an… · p. 3
- After embedding, joint indices no longer affect subsequent processing, allowing a 1D ConvNet to learn temporal information.Source: C. Modeling Joint Correlations by an… · p. 3
A. Experimental Datasets
The experiments use SHREC and JHMDB to assess DD-Net across different skeleton modalities and action properties. SHREC provides 3D hand-action skeletons, whereas JHMDB provides 2D body-action skeletons interpreted from RGB videos.
- DD-Net is evaluated on the SHREC and JHMDB skeleton-based action-recognition datasets.Source: A. Experimental Datasets · p. 3
- SHREC provides 3D skeletons derived from RGB-D data, while JHMDB provides 2D skeletons interpreted from RGB videos.Source: A. Experimental Datasets · pp. 3–4
- The dataset-properties table organizes the experimental datasets used for evaluation.Source: Table I · p. 4
- SHREC actions are strongly correlated with subjects’ global trajectories, whereas JHMDB actions may have a weaker connection with global trajectories.Source: A. Experimental Datasets · p. 4
B. Evaluation Setup
Evaluation covers multiple SHREC gesture settings and three JHMDB training/testing splits, with ablations examining component contributions and model-size effects.
- SHREC is evaluated with 14 gestures and 28 gestures, while JHMDB results are averaged across three training/testing splits.Source: B. Evaluation Setup · p. 4
- Ablation studies remove one DD-Net component at a time while keeping the others unchanged to assess component contributions.Source: B. Evaluation Setup · p. 4
- Model-size effects are examined by adjusting the number of filters in the network.Source: B. Evaluation Setup · p. 4
C. Implementation Details
DD-Net uses compact, single-GPU training with Adam, annealed learning rates, and temporal augmentation. The implementation avoids ensembles and pretrained weights and uses Keras with TensorFlow despite its slower execution speed.
- DD-Net is small enough to place all training sets in one batch on a single GTX 1080Ti GPU.Source: C. Implementation Details · p. 4
- Training uses Adam with β1 = 0.9 and β2 = 0.999, alongside an annealing learning rate that drops from 1−3 to 1−5.Source: C. Implementation Details · p. 4
- Temporal augmentation randomly selects 0.9 of the entire frames during training.Source: C. Implementation Details · p. 4
- The experiments use neither ensemble strategies nor pretrained weights to boost performance.Source: C. Implementation Details · p. 4
- DD-Net is implemented with a Keras backend in TensorFlow, while other neural-network frameworks may execute it faster.Source: C. Implementation Details · p. 4
D. Result Analysis and Discussion
DD-Net achieves strong recognition results across SHREC and JHMDB while using a compact model and fast inference. Its confusion matrices and ablations support robustness across action classes, datasets, motion scales, and trajectory properties.
- The SHREC evaluations cover 14 and 28 hand actions, with results reported alongside confusion matrices.Source: D. Result Analysis and Discussion · p. 4, Table II · p. 4, Figure 4 · p. 4
- DD-Net achieves superior results on both SHREC and JHMDB despite using fewer parameters.Source: D. Result Analysis and Discussion · p. 4
- DD-Net demonstrates robustness across individual action classes and generalization despite differences between the evaluated datasets.Source: D. Result Analysis and Discussion · pp. 4–5
- 0.15 million parameters are sufficient for DD-Net to generate comparable results on SHREC and JHMDB.Source: D. Result Analysis and Discussion · p. 5
- 3,500 FPS on one GPU and 2,000 FPS on one CPU demonstrate DD-Net's fast inference speed.Source: D. Result Analysis and Discussion · p. 5
V. CONCLUSION
The paper proposes DD-Net, a compact approach for efficient skeleton-based action recognition. It reports state-of-the-art performance on the experimental datasets and identifies extensions for broader studies.
- The paper proposes two features and DD-Net by analyzing basic skeleton-sequence properties for efficient action recognition.Source: §V · p. 5
- DD-Net achieves state-of-the-art performance on the experimental datasets despite containing few parameters.Source: §V · p. 5
- The paper identifies possible extensions including online recognition, RGB or depth integration, and temporal action detection.Source: §V · p. 5