PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatio-Temporal Graph Convolution for Skeleton Based Action Recognition

Chaolong Li, Zhen Cui, Wenming Zheng, Chunyan Xu, Jian Yang

arXiv:1802.09834v1cs.CV

TL;DR

Skeleton motion is naturally represented as a dynamic sequence of irregular graphs, motivating models that capture both spatial graph structure and temporal variation. The paper proposes recursive multi-scale spatio-temporal graph convolution, proves stability and an upper-bound, and reports state-of-the-art performance across four benchmark datasets.

  • Problem

    Skeleton actions form dynamic sequences of irregular graphs, while prior dynamic-graph recurrent methods had not absorbed CNN-style local convolutional filtering.

  • Method

    STGC recursively applies multi-scale local graph filters with receptive-field matrices and signal mappings across temporal motions and spatial structures, then stacks the model into a deep architecture.

  • Results

    STGC achieves state-of-the-art performance on four benchmark datasets, including 99.07% on Florence and 86.28% and 74.85% on NTU.

  • Takeaways & Limitations

    The approach combines local convolutional filtering with recursive sequence learning for dynamic skeletal graphs and extends to a multi-layer architecture.

  • Takeaways & Limitations

    The stability analysis assumes nonnegative recursive weights and bounded normalized spectral terms.

Abstract

from arXiv · show

Variations of human body skeletons may be considered as dynamic graphs, which are generic data representation for numerous real-world applications. In this paper, we propose a spatio-temporal graph convolution (STGC) approach for assembling the successes of local convolutional filtering and sequence learning ability of autoregressive moving average. To encode dynamic graphs, the constructed multi-scale local graph convolution filters, consisting of matrices of local receptive fields and signal mappings, are recursively performed on structured graph data of temporal and spatial domain. The proposed model is generic and principled as it can be generalized into other dynamic models. We theoretically prove the stability of STGC and provide an upper-bound of the signal transformation to be learnt. Further, the proposed recursive model can be stacked into a multi-layer architecture. To evaluate our model, we conduct extensive experiments on four benchmark skeleton-based action datasets, including the large-scale challenging NTU RGB+D. The experimental results demonstrate the effectiveness of our proposed model and the improvement over the state-of-the-art.

1 Introduction

Skeleton-based action recognition models human motion as sequences of irregular body graphs rather than grid-shaped image structures. The paper proposes STGC to combine graph-local convolution with recursive temporal learning, with theoretical analysis and benchmark evaluation.

  • Human skeletons form irregular geometric structures whose motions unfold as dynamic sequences of joints and bones.
  • Existing approaches model joint trajectories with statistical, shape, geometric, HMM, or recursive methods such as LSTMs.
  • Prior dynamic-graph recurrent methods had not incorporated the convolutional neural network advances that benefited other applications.
  • STGC uses multi-scale receptive fields and signal mappings to perform local filtering across temporal motions and spatial graph structures.
  • The recursive STGC model is extended into a deep architecture, theoretically analyzed for stability and an upper-bound, and evaluated on four benchmark datasets.

2 Graph Preliminary

Graph preliminaries represent skeleton data through nodes, weighted adjacency matrices, Laplacians, and graph Fourier components. These operators provide the spectral basis for graph filtering.

  • An undirected graph G=(V,A) contains nodes V and a weighted adjacency matrix A recording which node pairs are connected.
  • The graph Laplacian is L=D−A, where D is the diagonal degree matrix with D_ii equal to the summed adjacency weights for node i.
  • The normalized Laplacian is formed as I−D^−1/2AD^−1/2 and is used unless otherwise specified.
  • L has orthonormal eigenvectors and nonnegative eigenvalues, with the normalized Laplacian bounded by λ_max=2.
  • The graph Fourier transform maps a spatial signal x to x̂=Φ^⊤x, while filtering modifies its graph-frequency coefficients through the frequency response H(λ_n).

3 The Model

The model represents skeleton sequences as dynamic graphs and combines multi-scale spatial graph filtering with recursive temporal modeling. It also establishes stability and transformation bounds, then extends the recursive model into a deep architecture.

  • Graph representation: Skeleton sequences are modeled as streams of attributed graphs whose nodes are joints, edges represent bones, and node signals contain 3D coordinates.Each graph at time slice k is G_k = (V, A_k, X_k).
  • Multi-scale graphical convolution: Multi-scale graph convolution uses polynomial receptive fields and signal mappings to summarize information from k-scale neighborhoods with homogeneous-graph-invariant responses.The receptive-field computation is based on the graph Laplacian, while V_k performs signal transformation.
  • Spatio-temporal graph convolution: The spatio-temporal model recursively regresses temporal signals with local convolutional kernels, while its outputs combine spatial graph signals and dynamic temporal signals.The formulation is inspired by autoregressive moving average models and supports dependent or independent input channels.
  • Frequency-domain interpretation: For independent signals, graph filtering is interpreted as a (K-1)-order polynomial approximation of the graph Fourier transform, with dependent signals additionally transformed across channels.The general case separates per-signal polynomial graph filtering from multi-channel mappings.
  • Stability analysis: Under stated norm, nonnegativity, and spectral conditions, the recursive model is stable, its frequency response converges, and the transformation function has an upper bound.Normalization strategies can be used in practice to satisfy the stability preconditions.
  • Deep architecture: Stacking the recursive model into multiple layers enlarges convolutional receptive fields so higher layers can abstract more global information.The output of a lower layer is used as the input to the next layer.

4 Experiments

Experiments evaluate STGC across four public skeleton-based action datasets using configurations that vary neighborhood use, signal dependency, multi-scale filtering, and depth.

  • Datasets: Experiments use Florence 3D, HDM05, Large Scale Combined, and NTU RGB+D skeleton-based action datasets.The evaluation includes the large-scale NTU RGB+D dataset.
  • Results: Table 1 summarizes performance comparisons among the different configurations of the proposed model.The supplied table passage provides the comparison scope but no numerical values.
  • Configurations: The compared configurations range from channel mapping without neighbors to one-hop filtering, multi-scale independent or dependent filtering, and corresponding deep schemes.Deep variants stack the multi-scale STGCK configurations.

4.1 Datasets and Settings

The evaluation uses four public skeleton-action datasets with varied sizes, action counts, sensors, views, and difficulty. HDM05 is also used for kernel-scale comparison under Protocol 1.

  • Florence 3D contains 215 sequences from 10 subjects, nine actions, and 15 joints captured by a stationary Kinect.Several similar actions are difficult to distinguish because the skeletons contain few joints.
  • HDM05 contains 2,337 sequences spanning 130 motion classes, with each skeleton represented by 31 joints from an optical marker-based Vicon system.Its intra-class variation and many motion classes make recognition challenging.
  • The Large Scale Combined dataset merges nine public datasets into a complex 88-action collection with heterogeneous capture conditions.The constituent datasets differ in execution manner, background, viewpoint, resolution, and sensor type.
  • NTU RGB+D contains 56,880 sequences of 60 actions from 40 subjects, captured by Kinect v2 from different views using 25-joint skeletons.Its large intra-class and viewpoint variation makes it particularly challenging.

4.2 Implementation Details

Implementation preprocesses skeleton coordinates for coordinate-system invariance, augments sequences by segmentation, and constructs graph signals from human-bone connections. Stability is enforced through normalized graph operators and constrained weights.

  • The coordinate origin is shifted to the joints’ orthocenter in each frame to reduce dependence on the user coordinate system.
  • Each action sequence is split into 12 equal-sized subsequences for data augmentation.
  • Graph edges follow human bones, with weight 1 for connected joints and 0 otherwise; node signals are their 3D coordinates.
  • The recursive outputs feed a softmax classifier, while the deep STGC implementation uses two layers with output dimensions 32 and 64.

4.3 Selection of Convolutional Kernel Size

Kernel scales control temporal and spatial receptive-field sizes. On HDM05, the best tested configuration was K1 = 2 and K2 = 6, adopted as the default.

  • K1 and K2 control receptive-field sizes in the temporal and spatial domains, respectively.
  • Increasing K1 and K2 expands local filtering toward farther neighboring nodes.
  • The HDM05 search tested K1 ∈ {1, 2, 3} and K2 ∈ {1, 2, 4, 6, 8} under Protocol 1.
  • The best performance occurred at K1 = 2 and K2 = 6, which became the default configuration.Because temporal filtering is continuous across time slices, the implementation uses a smaller K1 than K2.
  • K = 1 corresponds to STGC(w/o L), which convolves only each node’s own signal.

4.4 Verification of STGC Structure

Experiments compare progressively richer STGC configurations, from no adjacency to multi-scale and deep variants. Incorporating neighborhood structure helps, dependent signal interactions are slightly better, and two-layer deep STGC performs best across all four datasets.

  • The study compares six configurations, including adjacency-free, neighborhood-based, independent-signal, dependent-signal, and two deep STGCK variants.
  • Adding the 1-neighborhood convolution in STGC(w/ L) improves performance over the adjacency-free STGC(w/o L) baseline.
  • STGCK(dep.) is slightly superior to STGCK(indep.) because it models signal interaction.
  • Two-layer deep STGC achieves the best performance on all four benchmark datasets.

4.5 Comparisons with State-of-the-Art

STGC is compared with graph-, LSTM-, and CNN-based approaches across four benchmark datasets, improving state-of-the-art performance on most datasets. Results vary by dataset difficulty, with NTU presenting the greatest challenge, especially for cross-subject recognition.

  • Comparisons with State-of-the-Art: STGC outperforms recent graph-based and LSTM-based methods, improving by more than 7% over the graph-based method on Florence.The comparison distinguishes STGC's recursively convolutional architecture from graph-kernel matching and LSTM-only sequence modeling.
  • Comparisons with State-of-the-Art: 99.07% accuracy is achieved on Florence, while NTU performance reaches 86.28% and 74.85%, compared with 82.80% and 74.40% for GCA-LSTM.STGC remains more competitive than CNN-based methods without using extra training data.
  • Comparisons with State-of-the-Art: Deep learning methods outperform shallow methods because their nonlinear dynamic networks and robust representations improve action recognition performance.The discussion includes both LSTM variations and CNN-based models.
  • Comparisons with State-of-the-Art: Performance differs across datasets: Florence is simpler, whereas NTU contains 56,880 sequences and diverse daily and pair actions.NTU cross-subject accuracy remains below 80% because of entangled actions.
  • Comparisons with State-of-the-Art: Cross-subject recognition is harder than cross-view or cross-sample recognition because the testing set contains more unforeseeable information.This pattern is reported across the relevant benchmark tables.

5 Conclusion

The paper concludes that STGC combines local convolutional filtering with recursive learning for dynamic skeletal graphs. It provides theoretical guarantees, extends to deep architectures, and improves results across four public datasets, while broader dynamic-graph applications remain future work.

  • Conclusion: STGC combines local convolutional filtering with the recursive learning power of autoregressive moving average.Its multi-scale graphical kernels use adjacency-polynomial receptive fields and signal mappings.
  • Conclusion: The model recursively applies multi-scale kernels to hidden states and input states, encoding motion variations and extracting spatial graph features.The approach is extended from a basic model into a multi-layer deep architecture.
  • Conclusion: The paper proves convergence, provides an upper bound, and demonstrates improvements on four public skeletal datasets including NTU RGB+D.The conclusion reports verification of both STGC and its deep version.
  • Conclusion: Generalization of STGC to other dynamic-graph problems is identified as future work.
Loading 1802.09834v1…