PaperlayerView exact revision on arXiv

Source-linked AI summary

Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks

Lei Shi, Yifan Zhang, Jian Cheng, Hanqing Lu

arXiv:1912.06971v1cs.CV

TL;DR

Previous skeleton-based GCNs use fixed, heuristic graph topologies and often omit bone information, limiting how they represent varied actions and samples. MS-AAGCN learns adaptive graph structures, adds spatial-temporal-channel attention, and jointly models joint, bone, and motion streams; the paper reports strong results on NTU-RGBD and Kinetics-Skeleton, including improvements over STGCN.

  • Problem

    Previous GCN-based models use graph topologies that are heuristic and fixed across layers and input data, while bone information is rarely investigated despite its reported informativeness for action recognition.

  • Method

    MS-AAGCN learns global and individual graph topologies end-to-end, enhances graph-convolutional layers with STC attention, and fuses joint, bone, and corresponding motion streams.

  • Results

    The model achieves state-of-the-art performance on NTU-RGBD and Kinetics-Skeleton, with +7.9% and +8.5% over STGCN on NTU-RGBD CV and CS benchmarks, respectively.

  • Takeaways & Limitations

    Adaptive topology, attention, and joint-bone-motion modeling together provide a multi-stream approach that improves skeleton-based action-recognition performance.

Abstract

from arXiv · show

Graph convolutional networks (GCNs), which generalize CNNs to more generic non-Euclidean structures, have achieved remarkable performance for skeleton-based action recognition. However, there still exist several issues in the previous GCN-based models. First, the topology of the graph is set heuristically and fixed over all the model layers and input data. This may not be suitable for the hierarchy of the GCN model and the diversity of the data in action recognition tasks. Second, the second-order information of the skeleton data, i.e., the length and orientation of the bones, is rarely investigated, which is naturally more informative and discriminative for the human action recognition. In this work, we propose a novel multi-stream attention-enhanced adaptive graph convolutional neural network (MS-AAGCN) for skeleton-based action recognition. The graph topology in our model can be either uniformly or individually learned based on the input data in an end-to-end manner. This data-driven approach increases the flexibility of the model for graph construction and brings more generality to adapt to various data samples. Besides, the proposed adaptive graph convolutional layer is further enhanced by a spatial-temporal-channel attention module, which helps the model pay more attention to important joints, frames and features. Moreover, the information of both the joints and bones, together with their motion information, are simultaneously modeled in a multi-stream framework, which shows notable improvement for the recognition accuracy. Extensive experiments on the two large-scale datasets, NTU-RGBD and Kinetics-Skeleton, demonstrate that the performance of our model exceeds the state-of-the-art with a significant margin.

I. INTRODUCTION

The paper addresses fixed, heuristic skeleton-graph topologies, limited use of bone information, and insufficiently selective feature modeling. It proposes MS-AAGCN, combining adaptive graphs, STC attention, and four joint-, bone-, and motion-based streams, with strong reported benchmark improvements.

  • Motivation: ST-GCN uses a heuristically predefined graph that may miss task-relevant dependencies and remains fixed across hierarchical model layers.The paper gives interactions between distant hands during clapping or reading as an example of a dependency that the predefined graph may miss.
  • Adaptive graph modeling: MS-AAGCN learns global and input-specific graph structures end-to-end, fusing them with a gate and optimizing them separately across layers.The global graph is learned from dataset knowledge, while the individual graph is built from feature similarity between vertices.
  • Attention: The STC-attention module recalibrates activations over joints, frames, and channels within each graph-convolutional layer.Its design reflects that different joints, temporal stages, and feature channels can have different importance for an action or sample.
  • Multi-stream representation: The model explicitly represents bone vectors and coordinate-difference motion for both joints and bones, then fuses four modalities in a multi-stream framework.Bone vectors encode directions from source joints to target joints, while motion captures changes between consecutive frames.
  • RGB fusion: 99% and 96% accuracies are reported for the CV and CS NTU-RGBD benchmarks when the model is fused with skeleton-guided cropped RGB data.The reported fused model exceeds other methods with a significant margin.
  • Evaluation: +7.9% and +8.5% improvements over STGCN are reported on the NTU-RGBD CV and CS benchmarks, respectively.The paper also reports state-of-the-art performance on NTU-RGBD and Kinetics-Skeleton.

II. RELATED WORK

Prior skeleton-recognition methods represent sequences with RNNs or pseudo-images with CNNs, but these representations do not fully preserve skeleton structure. The paper motivates graph-based modeling and describes a spatiotemporal graph over joints and frames.

  • Prior representations: Conventional methods use handcrafted human-body features, while deep methods commonly model skeletons as coordinate sequences with RNNs or transformed pseudo-images with CNNs.RNNs process joint-coordinate sequences, whereas CNNs rely on manually designed transformation rules.
  • Graph-based modeling: RNNs and CNNs do not fully represent skeleton structure because skeleton data are naturally embedded as graphs rather than vector sequences or regular grids.ST-GCN instead directly models the skeleton as a graph and avoids handcrafted transformations into sequences or pseudo-images.
  • GCN background: GCNs operate on non-Euclidean graph data through spatial neighborhoods or spectral graph representations.Spatial methods construct local neighborhoods, while spectral methods use graph Laplacian eigenvalues and eigenvectors.
  • Graph construction: The paper constructs a spatiotemporal skeleton graph with joints as vertices, natural body connections as spatial edges, and corresponding joints across consecutive frames as temporal edges.Each joint’s coordinate vector is assigned as the attribute of its corresponding vertex.
  • Neighborhood mapping: The mapping strategy partitions each vertex’s one-distance neighborhood into the vertex, centripetal, and centrifugal subsets.The figure caption identifies the left panel as the spatiotemporal graph and the right panel as the mapping strategy, with colors denoting subsets.

B. Graph convolution

The basic graph convolution aggregates features from each vertex’s one-distance spatial neighborhood using mapped subsets and learned weights. Temporal convolution then processes the resulting features before pooling and classification.

  • Spatial aggregation: For vertex vi, graph convolution samples features from its one-distance neighboring vertices Bi and applies a weighting function after neighborhood mapping.The mapping function assigns neighbors to fixed-numbered subsets associated with unique weight vectors.
  • Feature representation: The feature map is represented as a tensor f ∈ R^C×T×N, where C, T, and N denote channels, temporal length, and vertices.This tensor formulation supports implementation of graph convolution over spatial and temporal dimensions.
  • Spatial implementation: The spatial kernel uses three mapped subsets, with adjacency-like matrices selecting connected vertices for the corresponding 1×1 convolution weights.The kernel size Kv is set to 3; the normalized diagonal matrix includes α = 0.001 to avoid empty rows.
  • Temporal aggregation: Temporal processing applies a Kt × 1 convolution because each vertex has two fixed neighbors corresponding to the same joint in adjacent frames.This temporal operation is performed on the spatial graph-convolution output.
  • Prediction: After stacked spatiotemporal graph-convolution layers extract high-level features, global average pooling and a softmax classifier predict action categories.The pipeline uses multiple graph-convolution operations before classification.

IV. MULTI-STREAM ATTENTION-ENHANCED ADAPTIVE GRAPH CONVOLUTIONAL NETWORK

The adaptive graph convolutional layer learns graph structure from data rather than fixing it heuristically, combining global and sample-specific graphs with layer-wise gating. It also uses residual connections and training stabilization strategies.

  • Adaptive graph convolutional layer: The adaptive graph topology is optimized end-to-end and can differ across model layers and input samples.This increases flexibility for hierarchical representations and diverse skeleton data.
  • Adaptive graph convolutional layer: The global graph Bk is learned from training data and is independently parameterized for each layer.It is initialized from the human-body graph but has no constraints on its learned values.
  • Adaptive graph convolutional layer: The individual graph Ck learns a unique topology for each sample from feature similarity between graph vertices.Embedded feature maps are compared through 1 × 1 convolutional embedding functions to form a normalized similarity matrix.
  • Adaptive graph convolutional layer: A gating mechanism fuses Bk and Ck while adjusting their relative importance separately in each layer.The individual graph is more strongly required in upper layers, where representations are more semantic and variable.
  • Adaptive graph convolutional layer: The AGCL uses a residual path and a learned coefficient α, while initialization strategies stabilize early graph changes and model convergence.A 1 × 1 convolution is inserted in the residual path when input and output channel counts differ.

B. Attention module

The STC-attention module sequentially applies spatial, temporal, and channel attention to refine skeleton features. Its ordering is selected empirically over parallel and alternative sequential arrangements.

  • Attention module: The STC-attention module contains spatial, temporal, and channel attention sub-modules arranged sequentially as SAM, TAM, and CAM.The module is designed to focus on important joints, frames, and feature channels.
  • Attention module: SAM generates joint-wise attention maps from features averaged over frames and applies them through residual feature refinement.The attention map has shape 1 × 1 × N.
  • Attention module: TAM produces temporal attention maps over frames, with shape 1 × T × 1.Its definitions otherwise follow the spatial attention module.
  • Attention module: CAM generates channel attention maps from features averaged over joints and frames to strengthen discriminative channels.The channel attention map has shape C × 1 × 1 and uses two fully connected layers.

C. Basic block

Each basic block combines spatial graph convolution, STC attention, and temporal graph convolution, with normalization, nonlinearities, and a residual connection. The network stacks nine such blocks before pooling and classification.

  • Basic block: A basic block serially applies spatial GCN, STC attention, and temporal GCN operations.The spatial and temporal graph convolutions are each followed by batch normalization and ReLU.
  • Basic block: Residual connections are added to every basic block to stabilize training and ease gradient propagation.The residual path supports the block’s optimization and information flow.
  • Basic block: The overall network stacks 9 basic blocks with output channels progressing through 64, 128, and 256-channel stages.Global average pooling produces a common feature size before the softmax classifier predicts the action.

E. Multi-stream networks

MS-AAGCN models joint coordinates, bone vectors, and their respective motions in four streams, then fuses the streams’ softmax scores. It is evaluated on NTU-RGBD and Kinetics-Skeleton using established benchmarks and released skeleton data.

  • E. Multi-stream networks: The multi-stream framework models four modalities: joints, bones, joint motion, and bone motion.Joint coordinates provide first-order information, while bone direction and length provide second-order information.
  • E. Multi-stream networks: Each bone is represented as a vector from a source joint nearer the skeleton’s center of gravity to a farther target joint.The vector is computed as the target coordinates minus the source coordinates.
  • E. Multi-stream networks: Motion information is computed as the difference between the same joints or bones in consecutive frames.For joints, this is the coordinate difference between frames t and t + 1.
  • E. Multi-stream networks: The four streams’ softmax scores are fused by weighted summation to predict the action label.The complete MS-AAGCN architecture is presented as a four-stream model.
  • E. Multi-stream networks: Experiments use NTU-RGBD and Kinetics-Skeleton, with exhaustive component ablations on NTU-RGBD and final evaluation on both datasets.NTU-RGBD reports top-1 accuracy under cross-subject and cross-view benchmarks; Kinetics-Skeleton uses released OpenPose-based skeleton data.

B. Training details

Training uses SGD with Nesterov momentum and dataset-specific schedules, while preprocessing normalizes NTU-RGBD skeleton inputs and improves recognition.

  • SGD with Nesterov momentum 0.9, batch size 64, cross-entropy loss, and weight decay 0.0001 are used for optimization.
  • NTU-RGBD samples are padded or repeated to two bodies and 300 frames, with learning-rate drops at epochs 30 and 40 and training ending at epoch 50.
  • Kinetics-Skeleton inputs contain 150 frames and two bodies, with random frame selection, coordinate perturbations, and learning-rate drops at epochs 45 and 55.
  • The rearranged learning-rate scheduler and preprocessing are evaluated against the original training setup on NTU-RGBD.
  • NTU-RGBD preprocessing selects two bodies by body energy, normalizes channel distributions, and aligns coordinates relative to the spine.

2) Adaptive graph convolutional block:

Ablation studies on NTU-RGBD show benefits from adaptive global and individual graphs, gating, attention submodules, and combining joint, bone, and motion modalities.

  • Adaptive graph convolutional block: The second initialization strategy performs slightly better than the first in the adaptive graph experiments.
  • Adaptive graph convolutional block: +3.1% on CS and +1.7% on CV are obtained by the complete adaptive graph convolutional layer.Both adaptive graphs improve recognition, and gating provides additional improvement.
  • STC-attention module: The spatial, temporal, and channel attention submodules each improve performance, and their sequential concatenation is evaluated.The comparisons use STGCN and AGCN baselines defined in the attention ablation table.
  • Multi-stream framework: The four-stream model outperforms single-stream models, while combining joints and bones brings notable improvement.Bone-only performance is slightly higher than joint-only performance on CS, with the reverse pattern on CV.
  • Multi-stream framework: Joint-motion and bone-motion streams are generally weaker than joint and bone streams individually, but adding motion still improves the combined model.

D. Comparisons with the state-of-the-art methods

MS-AAGCN achieves state-of-the-art performance with a large margin on NTU-RGBD and Kinetics-Skeleton, while RGB fusion with pose-guided cropping reaches high NTU-RGBD accuracy.

  • D. Comparisons with the state-of-the-art methods: MS-AAGCN achieves state-of-the-art performance with a large margin on both NTU-RGBD and Kinetics-Skeleton.The comparison includes handcrafted-feature, RNN-, CNN-, and GCN-based methods.
  • Fusion with the RGB modality: Skeleton data is robust to dynamic circumstances and complicated backgrounds but lacks appearance information needed for some action distinctions.The paper motivates RGB fusion to address this missing appearance information.
  • Fusion with the RGB modality: 96.1% CS and 99.0% CV are achieved by RNX3D101+MS-AAGCN-C on NTU-RGBD.The model combines RGB and skeleton streams with pose-guided cropping.
  • Fusion with the RGB modality: Pose-guided cropping notably improves RGB-only recognition, but its improvement decreases when RGB and skeleton data are fused.Skeleton data effectively avoids environmental interference, reducing the cropping strategy’s additional contribution.

VII. VISUALIZATION AND DISCUSSION

Visualizations show that learned graph topologies vary across layers and samples, with individual graphs becoming more important in deeper layers and attention emphasizing action-relevant joints and frames.

  • Learned graph topologies: Learned global graphs retain the human-body structure but change substantially, especially in higher layers.The visualization compares the body-based adjacency matrix with adaptive matrices across subsets and layers.
  • Learned graph topologies: Different input samples and layers produce different individual-graph topologies.
  • Graph fusion: The individual graph receives greater importance in top layers because their larger receptive fields and more informative features support sample-specific topology learning.
  • STC attention: Spatial attention emphasizes hands and head, while attention is less evident in lower layers.Circle size in the visualization represents joint importance.
  • STC attention: Temporal attention shifts across layers and actions, focusing on hand-raising during selfie-taking and final posture, while throwing emphasizes hand-related frames.

C. Multi-modalities

The paper examines how skeleton and RGB modalities complement each other, especially when similar skeletons are difficult to distinguish. RGB information can resolve ambiguities that skeleton data alone cannot.

  • Modality complementarity: Skeleton data helps RGB recognition for “rub the hands,” while RGB helps skeleton recognition for “reading” and “writing.”The comparison uses class-level accuracy differences between the two modalities on the NTU-RGBD CV benchmark.
  • Modality complementarity: Similar skeletons make “reading” and “writing” difficult to distinguish using skeleton information alone.The paper presents two examples illustrating this ambiguity.
  • Modality complementarity: RGB data distinguishes similar “reading” and “writing” skeletons by revealing whether a pen is present in the hands.This example demonstrates how visual appearance supplies information absent from nearly identical skeletal configurations.
  • Modality complementarity: MS-AAGCN combines skeleton information with skeleton-guided cropped RGB data, producing additional improvement.The conclusion identifies improved fusion of RGB and skeleton modalities as a direction for future work.
Loading 1912.06971v1…