PaperlayerView exact revision on arXiv

Source-linked AI summary

Skeleton Image Representation for 3D Action Recognition based on Tree Structure and Reference Joints

Carlos Caetano, François Brémond, William Robson Schwartz

arXiv:1909.05704v1cs.CVcs.LG

TL;DR

3D action recognition needs representations that capture both temporal dynamics and spatial relations among skeleton joints. The paper introduces TSRJI, which combines reference-joint relationships with depth-first tree ordering for CNN input, and reports accurate results on two datasets, including state-of-the-art performance on NTU RGB+D 120.

  • Problem

    Existing 3D action-recognition approaches either use RNN-LSTMs that lack efficient spatial-relation learning or skeleton images whose joint organization can be improved.

  • Method

    TSRJI combines reference joints with depth-first tree traversal to construct CNN-ready skeleton image representations.

  • Results

    TSRJI recognizes actions accurately on NTU RGB+D 60 and NTU RGB+D 120 and achieves state-of-the-art performance on NTU RGB+D 120.

  • Takeaways & Limitations

    Combining TSRJI with an explicit temporal structural joint representation achieves state-of-the-art 3D action-recognition performance on NTU RGB+D 120.

Abstract

from arXiv · show

In the last years, the computer vision research community has studied on how to model temporal dynamics in videos to employ 3D human action recognition. To that end, two main baseline approaches have been researched: (i) Recurrent Neural Networks (RNNs) with Long-Short Term Memory (LSTM); and (ii) skeleton image representations used as input to a Convolutional Neural Network (CNN). Although RNN approaches present excellent results, such methods lack the ability to efficiently learn the spatial relations between the skeleton joints. On the other hand, the representations used to feed CNN approaches present the advantage of having the natural ability of learning structural information from 2D arrays (i.e., they learn spatial relations from the skeleton joints). To further improve such representations, we introduce the Tree Structure Reference Joints Image (TSRJI), a novel skeleton image representation to be used as input to CNNs. The proposed representation has the advantage of combining the use of reference joints and a tree structure skeleton. While the former incorporates different spatial relationships between the joints, the latter preserves important spatial relations by traversing a skeleton tree with a depth-first order algorithm. Experimental results demonstrate the effectiveness of the proposed representation for 3D action recognition on two datasets achieving state-of-the-art results on the recent NTU RGB+D~120 dataset.

I. INTRODUCTION

The paper targets 3D action recognition by improving CNN-ready skeleton image representations, combining reference joints with tree-structured joint ordering. TSRJI is reported to perform accurately on two large-scale datasets, with state-of-the-art performance on NTU RGB+D 120.

  • Motivation: CNN skeleton images learn structural information from 2D arrays, whereas RNN-LSTM approaches do not efficiently learn spatial relations between skeleton joints.Existing approaches primarily use recurrent models or skeleton images as inputs to CNNs.
  • Contribution: TSRJI combines reference joints and a tree-structure skeleton to encode temporal dynamics and preserve spatial relationships for CNN-based 3D action recognition.The representation incorporates different joint relationships while depth-first traversal preserves relevant spatial organization.
  • Implementation: The shallow network uses only three convolutional layers and two fully connected layers, making the compact representation extremely fast to train.The authors also make the TSRJI code publicly available.
  • Results: TSRJI recognizes actions accurately on NTU RGB+D 60 and NTU RGB+D 120 and achieves state-of-the-art performance on NTU RGB+D 120.Combining TSRJI with a temporal structural joint representation improves performance by up to 3.3 percentage points over the best reported skeleton-based method.

II. RELATED WORK

Prior skeleton-image methods encode temporal and spatial information through matrices, trajectories, heat maps, reference joints, motion images, or tree traversal. The paper motivates TSRJI by combining tree-based spatial organization with reference-joint relationships.

  • Matrix-based representations: Earlier methods represent skeleton sequences as matrices, encoding temporal evolution and spatial structure before CNN classification.Du et al. use a J × T × 3 matrix that is quantified into an image and processed by a CNN.
  • Trajectory and heat-map representations: Joint Trajectory Maps encode joint trajectories, motion direction, and magnitude as three color-texture images, while heat-map methods map 5D joint points into color spaces.Trajectory methods also use rotations for multi-view simulation and data enlargement.
  • Reference and motion representations: Reference-joint representations compute relative joint positions to four body reference joints, whereas motion images encode differences between consecutive frames.These approaches provide multiple spatial or temporal skeleton images for CNN classification.
  • Tree-structured representations: Tree Structure Skeleton Image methods use depth-first traversal because fixed joint concatenation can lose semantic and structural information.The traversal prioritizes joint pairs connected by fewer skeleton-tree edges.
  • Motivation for TSRJI: TSRJI combines tree-based preservation of relevant spatial relations with reference-joint modeling of different joint relationships.This combination is proposed to improve temporal-dynamics encoding in skeleton images.

III. PROPOSED APPROACH

TSRJI constructs CNN-ready skeleton images by ordering joints with depth-first tree traversal and computing chains relative to four reference joints. The resulting four matrices encode arranged joint trajectories over time and are normalized to a fixed image size.

  • Tree Structure Reference Joints Image: Depth-first traversal generates a predefined chain Ct in which neighboring image columns correspond to spatially related joints in the original skeleton.The method assumes directly linked joints in the skeleton graph are more spatially related.
  • Tree Structure Reference Joints Image: The method computes four chains using the left shoulder, right shoulder, left hip, and right hip as reference joints.Relative positions are used because the reference joints are stable in most actions and can reflect other-joint motions.
  • Tree Structure Reference Joints Image: Figure 2 illustrates how one ordered chain is transformed into generated chains using the reference-joint technique.The figure provides the visual counterpart to the four-reference-joint computation.
  • Tree Structure Reference Joints Image: Each video produces four matrices, Sa, Sb, Sc, and Sd, by concatenating the ordered chains across frames.Each matrix column represents the temporal evolution of an arranged chain joint, with size J × T × 3.
  • Tree Structure Reference Joints Image: The four matrices are normalized to [0, 1] and resized to J × 100 so variable-length skeleton sequences can enter CNNs at a fixed size.Figure 3 summarizes the construction of the proposed skeleton image representation.

B. Convolutional Neural Network Architecture Employed

The approach uses a compact CNN to learn features from the generated skeleton images, with a modified shallow architecture and stacked inputs for multi-person actions.

  • Convolutional Neural Network Architecture Employed: The CNN uses three convolutional layers and two fully connected layers to learn features from the generated skeleton image representations.The architecture is a reduced version of the network used by Li et al.
  • Convolutional Neural Network Architecture Employed: The convolutional layers use 3 × 3 kernels, with stride 2 in the third layer, alongside max pooling, ReLU, and dropout.The network is designed to be trainable from scratch without pre-training.
  • Convolutional Neural Network Architecture Employed: Skeleton images from different people are stacked as the network input for multi-person interactions such as shaking hands.This is a common strategy used to handle actions involving multiple people.

IV. EXPERIMENTAL RESULTS

The experiments evaluate whether structural organization of joints matters by comparing TSRJI with a representation built using randomly ordered joints.

  • The study compares TSRJI with a baseline that uses random joint ordering when constructing the representation.The random ordering does not account for semantic meaning between adjacent joints.
  • The comparison is designed to test whether a meaningful joint structure preserves spatial relations in skeleton data.
  • The baseline changes the chains’ order without using semantic information about adjacent joints.

1) NTU RGB+D 60 [9]:

The datasets provide large-scale RGB-D action-recognition benchmarks with skeleton data and subject-, view-, or setup-based evaluation protocols.

  • 1) NTU RGB+D 60 [9]:: NTU RGB+D 60 contains 56,880 videos across 60 action categories performed by 40 subjects.The videos were collected using three Microsoft Kinect sensors and include RGB, depth, infrared, and skeleton-joint data.
  • 1) NTU RGB+D 60 [9]:: NTU RGB+D 60 evaluates recognition using cross-subject and cross-view splits, with performance averaged across classes.
  • 2) NTU RGB+D 120 [21]:: NTU RGB+D 120 contains 114,480 RGB+D videos across 120 action categories performed by 106 subjects.It was captured with Microsoft Kinect under various environmental conditions and includes RGB, depth, infrared, and skeleton-joint data.
  • 2) NTU RGB+D 120 [21]:: NTU RGB+D 120 uses cross-subject and cross-setup protocols, with recognition averaged across all classes.The cross-setup split assigns even setup IDs to training and odd setup IDs to testing.

B. Implementation Details

The implementation controls comparisons by using the same datasets, architecture, data splits, evaluation protocols, and metrics, with fixed regularization and optimization settings.

  • All compared skeleton image representations use the same datasets, network architecture, training/testing split, evaluation protocols, and metrics.This setup isolates the contribution of the proposed representation.
  • The evaluation follows the protocols and metrics defined by the dataset creators.
  • The network uses a dropout ratio of 0.5, a learning rate of 0.001, and a batch size of 1000.

C. Evaluation

Experiments evaluate TSRJI against skeleton-image and literature baselines on NTU RGB+D 60 and 120, using stacked and late-fusion variants. TSRJI achieves the strongest reported results on several protocols, especially when fused with motion information.

  • TSRJI (Stacked) achieves 76.7% accuracy on NTU RGB+D 60 cross-view evaluation, exceeding Ke et al. by 1.2 p.p. and Yang et al. by 1.1 p.p.
  • Compared with random joint ordering, TSRJI (Stacked) improves accuracy by 1.5 p.p. on both cross-subject and cross-view protocols.
  • 80.3% accuracy on NTU RGB+D 60 cross-view with TSRJI (Late Fusion), improving 4.7 p.p. over Yang et al..
  • TSRJI (Late Fusion) reaches 73.3% accuracy on NTU RGB+D 60 cross-subject evaluation, improving 2.5 p.p. over Ke et al..
  • Up to 3.3 p.p. improvement over Body Pose Evolution Map on NTU RGB+D 120 cross-subject evaluation when TSRJI is fused with Li et al..
  • On NTU RGB+D 120 cross-subject evaluation, TSRJI exceeds the best reported LSTM method by 4.3 p.p., while fusion with Li et al. increases the margin to 6.7 p.p.

D. Discussion

TSRJI improves recognition of actions involving arm and hand movements by combining reference joints with depth-first structural ordering. However, it remains challenged by visually similar actions, especially when objects distinguish the movements.

  • TSRJI correctly classified arm- and hand-movement actions that the baselines commonly confused.Examples include standing up, writing, tearing paper, and handshaking.
  • The late-fusion TSRJI comparison targets NTU RGB+D 60 cross-view recognition against a late-fused combination of prior representations.
  • The method attributes these gains to explicit structural organization of relevant joint pairs and preserved adjacency from depth-first traversal.Reference joints can capture arm and hand motion through stable shoulder locations.
  • TSRJI confuses actions with similar movements, including clapping versus rubbing hands and reading versus writing or typing.The authors suggest explicit motion information could help distinguish these cases.

V. CONCLUSIONS AND FUTURE WORKS

The paper concludes that TSRJI is an effective CNN input representation for skeleton-based 3D action recognition. It also identifies evaluation with other architectures and 2D skeleton datasets as future work.

  • TSRJI preserves spatial relations among relevant joint pairs while incorporating multiple spatial relationships between joints.
  • Combining TSRJI with an explicit motion method improves 3D action recognition beyond the reported state of the art on NTU RGB+D 120.
  • Future work will evaluate TSRJI with other architectures and on 2D action datasets using estimated skeletons.
Loading 1909.05704v1…