PaperlayerView exact revision on arXiv

Source-linked AI summary

Make Skeleton-based Action Recognition Model Smaller, Faster and Better

Fan Yang, Sakriani Sakti, Yang Wu, Satoshi Nakamura

arXiv:1907.09658v8cs.CV

TL;DR

Skeleton-based action recognition often uses models that are too large or slow for efficient deployment. This paper proposes DD-Net, which combines compact invariant and two-scale motion features with a lightweight architecture, achieving strong performance on SHREC and JHMDB while running at 3,500 FPS on one GPU or 2,000 FPS on one CPU.

  • Problem

    Existing skeleton-based action-recognition methods may suffer from large model sizes and slow execution speeds.

  • Method

    DD-Net combines a location-viewpoint invariant JCD feature, a two-scale global motion feature, and a lightweight 1D-CNN network.

  • Results

    DD-Net achieves superior results on SHREC and JHMDB while reaching around 3,500 FPS on one GPU and 2,000 FPS on one CPU.

  • Takeaways & Limitations

    The results support DD-Net as an efficient model that generalizes across a wide range of skeleton-based action-recognition scenarios.

Abstract

from arXiv · show

Although skeleton-based action recognition has achieved great success in recent years, most of the existing methods may suffer from a large model size and slow execution speed. To alleviate this issue, we analyze skeleton sequence properties to propose a Double-feature Double-motion Network (DD-Net) for skeleton-based action recognition. By using a lightweight network structure (i.e., 0.15 million parameters), DD-Net can reach a super fast speed, as 3,500 FPS on one GPU, or, 2,000 FPS on one CPU. By employing robust features, DD-Net achieves the state-of-the-art performance on our experimental datasets: SHREC (i.e., hand actions) and JHMDB (i.e., body actions). Our code will be released with this paper later.

I. INTRODUCTION

DD-Net addresses the large size and slow execution of existing skeleton-based action-recognition methods with a lightweight network built around complementary invariant and motion features. The authors report higher accuracy, efficient computation, and generalization across hand and body-action settings.

  • Existing skeleton-based action-recognition methods may have large model sizes and slow execution speeds.
  • DD-Net combines a location-viewpoint invariant JCD feature with a two-scale global motion feature.The design targets varied skeleton settings, including hand/body actions, 2D/3D skeletons, and actions with or without global trajectories.
  • The JCD feature is compact and invariant, while the two-scale motion feature improves robustness to motion-scale variation.
  • An embedding process lets DD-Net learn joint correlations that are difficult to predefine from joint indices.
  • DD-Net provides higher action-recognition accuracy and computational efficiency than methods using more complicated model structures.

III. METHODOLOGY

The methodology represents each skeleton frame with Joint Collection Distances and processes the resulting sequence through a configurable lightweight 1D-CNN architecture. JCD preserves location-viewpoint invariance while reducing feature redundancy.

  • DD-Net uses a 1D ConvNet architecture whose model size can be changed by modifying the filter count.The architecture includes convolutional layers, max pooling, global average pooling, and fully connected layers.
  • Cartesian coordinates vary with skeleton location and viewpoint, whereas geometric features such as distances are location-viewpoint invariant.
  • The method is motivated by the need for a compact geometric representation that avoids dataset-specific redesign and redundant elements.
  • JCD computes Euclidean distances between pairs of collective joints and uses the symmetric matrix’s lower triangle without its diagonal.This removes redundant entries from the distance representation.
  • The flattened JCD feature is used as a one-dimensional model input.

B. Modeling Global Scale-invariant Motions by a Two-scale Motion Feature

Because JCD omits global motion, DD-Net supplements it with location-invariant temporal differences computed at two time scales. The resulting feature captures both slower and faster motion patterns and is resized to match the JCD sequence length.

  • JCD is location-viewpoint invariant but lacks global motion information, making it insufficient alone for actions tied to global trajectories.
  • DD-Net computes temporal differences of Cartesian coordinates to obtain location-invariant global motion features.
  • The method models both fast and slow global motions because the same action can occur at different motion scales.
  • The two-scale motion sequences are reshaped into one-dimensional inputs and linearly interpolated to match the JCD feature’s frame count.
  • The resulting two-scale global motion feature can be generated using only Cartesian-coordinate input.

C. Modeling Joint Correlations by an Embedding

DD-Net addresses the lack of local correlation among joint indices by embedding joint-distance and two-scale motion features before temporal modeling. This embedding automatically learns joint relationships, reduces skeleton-noise effects, and enables 1D ConvNet processing independent of joint ordering.

  • Joint indices are not locally correlated, and their relationships can change across actions, making manual joint ordering difficult.
  • DD-Net embeds JCD and two-scale motion features into latent vectors so joint correlations are learned automatically rather than predefined.The embedding also reduces the effect of skeleton noise.
  • Embed1 and Embed2 use stacked 1D convolutions, with Embed2 additionally applying max pooling because the slow features have twice the temporal length of the fast features.
  • The embedded features are concatenated into a representation εk using ⊕ as the concatenation operation.
  • After embedding, joint indices no longer affect subsequent processing, allowing a 1D ConvNet to learn temporal information.

A. Experimental Datasets

The experiments use SHREC and JHMDB to assess DD-Net across different skeleton modalities and action properties. SHREC provides 3D hand-action skeletons, whereas JHMDB provides 2D body-action skeletons interpreted from RGB videos.

  • DD-Net is evaluated on the SHREC and JHMDB skeleton-based action-recognition datasets.
  • SHREC provides 3D skeletons derived from RGB-D data, while JHMDB provides 2D skeletons interpreted from RGB videos.
  • The dataset-properties table organizes the experimental datasets used for evaluation.
  • SHREC actions are strongly correlated with subjects’ global trajectories, whereas JHMDB actions may have a weaker connection with global trajectories.

B. Evaluation Setup

Evaluation covers multiple SHREC gesture settings and three JHMDB training/testing splits, with ablations examining component contributions and model-size effects.

  • SHREC is evaluated with 14 gestures and 28 gestures, while JHMDB results are averaged across three training/testing splits.
  • Ablation studies remove one DD-Net component at a time while keeping the others unchanged to assess component contributions.
  • Model-size effects are examined by adjusting the number of filters in the network.

C. Implementation Details

DD-Net uses compact, single-GPU training with Adam, annealed learning rates, and temporal augmentation. The implementation avoids ensembles and pretrained weights and uses Keras with TensorFlow despite its slower execution speed.

  • DD-Net is small enough to place all training sets in one batch on a single GTX 1080Ti GPU.
  • Training uses Adam with β1 = 0.9 and β2 = 0.999, alongside an annealing learning rate that drops from 1−3 to 1−5.
  • Temporal augmentation randomly selects 0.9 of the entire frames during training.
  • The experiments use neither ensemble strategies nor pretrained weights to boost performance.
  • DD-Net is implemented with a Keras backend in TensorFlow, while other neural-network frameworks may execute it faster.

D. Result Analysis and Discussion

DD-Net achieves strong recognition results across SHREC and JHMDB while using a compact model and fast inference. Its confusion matrices and ablations support robustness across action classes, datasets, motion scales, and trajectory properties.

  • The SHREC evaluations cover 14 and 28 hand actions, with results reported alongside confusion matrices.
  • DD-Net achieves superior results on both SHREC and JHMDB despite using fewer parameters.
  • DD-Net demonstrates robustness across individual action classes and generalization despite differences between the evaluated datasets.
  • 0.15 million parameters are sufficient for DD-Net to generate comparable results on SHREC and JHMDB.
  • 3,500 FPS on one GPU and 2,000 FPS on one CPU demonstrate DD-Net's fast inference speed.

V. CONCLUSION

The paper proposes DD-Net, a compact approach for efficient skeleton-based action recognition. It reports state-of-the-art performance on the experimental datasets and identifies extensions for broader studies.

  • The paper proposes two features and DD-Net by analyzing basic skeleton-sequence properties for efficient action recognition.
  • DD-Net achieves state-of-the-art performance on the experimental datasets despite containing few parameters.
  • The paper identifies possible extensions including online recognition, RGB or depth integration, and temporal action detection.
Loading 1907.09658v8…