PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph Edge Convolutional Neural Networks for Skeleton Based Action Recognition

Xikun Zhang, Chang Xu, Xinmei Tian, Dacheng Tao

arXiv:1805.06184v2cs.CV

TL;DR

Skeleton action-recognition methods commonly focus on joints, while the paper investigates whether bone dynamics provide a more useful representation. It introduces spatial-temporal graph edge convolution and hybrid node-edge networks, which outperform prior state-of-the-art methods across Kinetics and NTU-RGB+D evaluations. The authors note that their class-level analysis is preliminary and requires further validation.

  • Problem

    Existing skeleton-based action recognition primarily focuses on joints, although bone shapes, lengths, positions, and movements may provide more stable action cues.

  • Method

    The paper applies graph edge convolution to spatial and temporal neighboring bones and combines edge and node convolutions through hybrid networks with shared layers.

  • Results

    The graph edge model outperforms previous state-of-the-art methods across Kinetics and NTU-RGB+D settings, while hybrid models further improve performance.

  • Takeaways & Limitations

    Analyzing bone dynamics and combining bone- and joint-based graph convolutions can improve skeleton-based action recognition performance.

  • Takeaways & Limitations

    The paper's class-level model analysis is not rigorous enough for fully convincing validation, so more detailed validation is left for future work.

Abstract

from arXiv · show

This paper investigates body bones from skeleton data for skeleton based action recognition. Body joints, as the direct result of mature pose estimation technologies, are always the key concerns of traditional action recognition methods. However, instead of joints, we humans naturally identify how the human body moves according to shapes, lengths and places of bones, which are more obvious and stable for observation. Hence given graphs generated from skeleton data, we propose to develop convolutions over graph edges that correspond to bones in human skeleton. We describe an edge by integrating its spatial neighboring edges to explore the cooperation between different bones, as well as its temporal neighboring edges to address the consistency of movements in an action. A graph edge convolutional neural network is then designed for skeleton based action recognition. Considering the complementarity between graph node convolution and graph edge convolution, we additionally construct two hybrid neural networks to combine graph node convolutional neural network and graph edge convolutional neural network using shared intermediate layers. Experimental results on Kinetics and NTU-RGB+D datasets demonstrate that our graph edge convolution is effective to capture characteristic of actions and our graph edge convolutional neural network significantly outperforms existing state-of-art skeleton based action recognition methods. Additionally, more performance improvements can be achieved by the hybrid networks.

1 Introduction

Skeleton-based action recognition typically models joints, but the paper argues that bone movements can provide more informative and stable cues. It proposes graph edge convolution to model spatial and temporal relationships among bones, alongside hybrid networks combining edge and node convolutions.

  • Motivation: Skeleton data offers a high-level, viewpoint- or appearance-invariant representation for recognizing human actions.Depth-derived skeletons simplify action representation by tracking human joints over time.
  • Motivation: Bone shapes, lengths, and positions can reveal action-relevant motion even when connected joints show little coordinate change.The paper highlights walking and running as examples where hip and shoulder joints may have subtle 3D changes while bones move significantly.
  • Proposed approach: Graph edge convolution represents each bone by learning weights over spatially neighboring edges and extends this operation to temporal neighboring edges.The temporal extension addresses skeleton changes across continuous video frames.
  • Proposed approach: Two hybrid networks combine graph node and edge convolutions through shared intermediate layers to exploit their complementary perspectives.The paper describes one hybrid using a shared dense layer and another using shared convolutional layers.
  • Evaluation: The proposed models are evaluated on the Kinetics and NTU-RGB+D datasets for skeleton-based action recognition.The experimental section follows the model and hybrid-network introduction.

2 Related Work

Related work spans graph convolution, handcrafted skeleton features, recurrent models, and CNN-based action recognition. In contrast, this paper applies convolution to skeleton-graph edges, which represent bones, rather than only to nodes representing joints.

  • Graph convolutional networks: Graph convolution research includes spectral methods that apply convolution in a graph-derived frequency domain.The paper identifies spectral approaches as one major category of graph convolutional networks.
  • Skeleton-based recognition: Skeleton-based action recognition includes handcrafted features that encode joint locations, motion dynamics, and temporal dependencies.Examples include covariance descriptors for joint locations over time and other geometric representations.
  • Skeleton-based recognition: CNN- and recurrent-based methods analyze skeleton sequences using temporal, spatial-temporal, coordinate, motion, or geometric representations.The reviewed approaches include RNNs, LSTMs, two-stream CNNs, cylindrical-coordinate clips, temporal CNNs, and spatial-temporal skeleton graphs.
  • Position of this work: The proposed model differs from existing CNN-based skeleton methods by performing convolution on graph edges instead of graph nodes.In the skeleton graph, edges correspond to bones and nodes correspond to joints.

3 Graph Edge Convolutional Neural Networks

The paper defines graph edge convolution over skeleton bones by aggregating spatial and temporal neighboring edges, then combines edge and node representations through hybrid networks. These designs use complementary bone and joint dynamics for action recognition.

  • Graph Edge Convolution: An edge neighborhood contains edges within distance R of a root edge, with R = 1 selecting directly neighboring edges.The distance-based neighborhood is illustrated by paths and edge distances from a root edge.
  • Graph Edge Convolution: Neighboring edges receive weights through a labeling function that assigns an order, with normalization balancing differently sized labeling groups.The convolution computes a weighted summation of neighboring edge features, and Z_pq normalizes contributions sharing a labeling value.
  • Graph Edge Convolution: Graph edge convolution focuses on information carried by skeleton edges, while nodes indicate connections between bones.The method represents each edge by learning weights over its spatial neighboring edges and extends this to temporal changes.
  • Graph Edge Convolutional Neural Networks: For action sequences, edge neighborhoods incorporate spatial neighbors within a frame and corresponding spatial neighbors from nearby frames restricted by temporal kernel size K_t.This design processes skeleton graphs across frames concurrently to model temporal movement consistency.
  • Combining Graph Edge and Node Convolutions: The hybrid networks combine node and edge features, either after separate extraction or through a shared convolutional layer where joint and bone information flow into each other.The shared layer produces updated features for both nodes and edges, allowing each representation to be informed by the other.
  • Combining Graph Edge and Node Convolutions: Hybrid models exploit complementary dynamics: bone motion can help when joint changes are subtle, while joint dynamics can distinguish actions with similar bone movements.The resulting classification uses both joint and bone dynamics rather than relying on either representation alone.

4 Experiments

The experiments first examine temporal-kernel effects on Kinetics, then evaluate the model on Kinetics and NTU-RGB+D against prior state-of-the-art methods.

  • The study selects a temporal kernel size through ablation on the Kinetics dataset.
  • Performance is evaluated on Kinetics and NTU-RGB+D, then compared with previous state-of-the-art methods.

4.1 Datasets and Settings

The experiments use skeletal representations from Kinetics and NTU-RGB+D, with dataset-specific graph structures, bone features, splits, and accuracy metrics.

  • Kinetics: Kinetics contains 300,000 YouTube video clips across 400 action classes, with skeletal tensors shaped (3, T, 18, 2).OpenPose provides 2D joint coordinates and confidence scores; the two bodies with highest average joint confidence are retained.
  • Kinetics: The Kinetics skeleton graph uses 18 joints and 17 bones, representing each bone with center coordinates, confidence, and a 2D orientation vector.
  • Kinetics: Kinetics contains 266,440 samples, split into 246,534 training and 19,906 testing samples, with top-1 and top-5 accuracy reported.
  • NTU-RGB+D: NTU-RGB+D contains 56,880 samples across 60 classes, with each body represented by 25 joints and 3D coordinates.
  • NTU-RGB+D: The NTU-RGB+D graph contains 25 joints and 24 bones, with each bone represented by a six-dimensional location-and-orientation feature vector.
  • NTU-RGB+D: NTU-RGB+D uses cross-subject and cross-view evaluation splits, reporting top-1 accuracy.

4.2 Pipeline

The pipeline processes skeletal sequences through separate edge- and node-convolution streams, then combines their pooled representations for classification or shares convolutional layers before pooling.

  • Edge-convolution pipeline: The edge-convolution pipeline normalizes joint coordinates, computes bone centers and orientations, and applies nine edge-convolution layers with channel expansion and pooling.The first, middle, and final layer groups output 64, 128, and 256 channels; strides of 2 are used in the fourth and seventh layers.
  • Sequence-level hybrid model: The sequence-level hybrid model uses separate edge- and node-convolution streams, each followed by global pooling.
  • Sequence-level hybrid model: The two pooled 256-dimensional stream outputs are concatenated and passed to a fully connected layer producing class scores.
  • Body-part-level hybrid model: The second hybrid model removes initial global pooling, applies two shared convolutional layers to the two streams, then pools and classifies the result.

4.3 Implementation of Edge Convolution

Edge convolution processes skeletal sequences with temporal and spatial stages over tensors indexed by channels, edges, and time.

  • Edge convolution consists of temporal convolution followed by spatial convolution.
  • Each input sequence has shape (C,E,T), representing feature channels, graph edges, and sequence length.
  • The temporal stage applies a 2D convolution with kernel size 1×K_t along the temporal dimension.

4.4 Hyper parameter analysis

The ablation varies temporal kernel size to assess how many consecutive frames edge convolution should incorporate. Accuracy improves through size 9, while larger kernels add little benefit and increase computation.

  • Temporal kernel size: Temporal kernel sizes from 3 to 9 produce significant top-1 accuracy gains on Kinetics.The study compares six temporal sizes for the edge convolution model.
  • Temporal kernel size: Kernels of size 11 and 13 provide little additional improvement over size 9.The authors select temporal kernel size 9 after comparison.
  • Interpretation: Using enough consecutive frames captures action motion, whereas too many frames increase computation without promising gains.

4.5 Experiment Results

Experiments on Kinetics and NTU-RGB+D show that edge convolution improves skeleton action recognition, while hybrid models generally perform best. The analysis links these gains to bone dynamics and robustness when joints overlap, while noting that class-level interpretations remain preliminary.

  • Kinetics: On Kinetics, GECNN exceeds previous state-of-the-art models by 0.7% top-1 and 1.1% top-5 accuracy.
  • Kinetics: On Kinetics, BPLHM performs best, improving over previous state-of-the-art models by 2.7% top-1 and 3.4% top-5 accuracy.SLHM improves by 0.4% top-1 and 0.3% top-5 accuracy.
  • NTU-RGB+D: Under NTU-RGB+D cross-subject evaluation, BPLHM achieves accuracy 3.9% higher than the previous state-of-the-art model.GECNN improves by 2.5%, while SLHM and BPLHM add 0.7% and 1.4%, respectively.
  • NTU-RGB+D: Under cross-view evaluation, BPLHM achieves accuracy 2.8% higher than the previous state-of-the-art model.GECNN improves by 1.1%, while SLHM and BPLHM further improve it by 0.3% and 1.7%.
  • Class-level analysis: Bone-based recognition is more robust to joint overlap because bones extend through 3D space and are less likely to completely overlap.The paper associates this advantage with classes where edge convolution substantially outperforms node convolution.
  • Limitations: The class-level analysis is not fully rigorous, and more detailed validation is left for future work.
  • Class-level analysis: Across 60 classes, BPLHM is best in 25, SLHM in 18, GECNN in 15, and node convolution in 8, with ties preventing the counts from summing to 60.The results also show that node convolution remains best for some classes.
  • Hybrid models: Hybrid models outperform joint-only and bone-only models, with body-part fusion benefiting from earlier information exchange between connected joints and bones.The body-part hybrid outperforms the sequence-level hybrid because feature interaction occurs during feature extraction rather than only after whole-sequence extraction.

5 Conclusion

The paper proposes graph edge convolution to model relationships among graph edges representing bones rather than only graph nodes. It also combines edge and node convolution in hybrid models, which further improves performance on skeleton action recognition.

  • Graph edge convolution: Graph edge convolution learns relationships and dependencies between graph edges, complementing node-based graph convolution.The approach leverages bone dynamics for skeleton-based action recognition.
  • Hybrid models: Hybrid models combine node and edge convolution and further improve performance over the individual approaches.
Loading 1805.06184v2…