Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition

1. Introduction

• Fixed, manually predefined graphs may miss important dependencies, cannot vary across hierarchical layers, and may not suit different action samples.Examples include hand dependencies for clapping, reading, wiping face, and touching head.Source: §1 · pp. 1–2
• 2s-AGCN adaptively learns graph topologies for different GCN layers and skeleton samples in an end-to-end manner.It uses jointly trained global and individual graphs to represent common and sample-specific patterns.Source: §1 · p. 2
• The method explicitly models second-order bone information and fuses it with first-order joint information in a two-stream framework.Bone information includes lengths and directions and is described as more informative and discriminative for action recognition.Source: §1 · p. 2
• 2s-AGCN achieves state-of-the-art performance on both NTU-RGBD and Kinetics-Skeleton.The contribution summary states that it exceeds the state of the art by a significant margin on both datasets.Source: §1 · p. 2

2.1. Skeleton-based action recognition

• RNNs represent skeletons as coordinate-vector sequences, whereas CNNs arrange them as pseudo-images using manually designed transformation rules.CNN-based methods are described as more popular because they are easier to train and more parallelizable than RNNs.Source: §2.1 · p. 2
• ST-GCN directly models skeletons as spatiotemporal graphs, avoiding handcrafted part-assignment or traversal rules and improving over earlier methods.It uses graph structure to represent the skeleton rather than a vector sequence or 2D grid.Source: §2.1 · p. 2

2.2. Graph convolutional neural networks

• Graph convolution methods are categorized into spatial approaches that aggregate graph neighbors and spectral approaches based on graph Laplacian eigenvalues and graph Fourier transforms.The proposed work follows the spatial perspective.Source: §2.2 · p. 2

3.1. Graph construction

• Each skeleton joint is a graph vertex with coordinate attributes, connected spatially to anatomical neighbors and temporally to corresponding joints in adjacent frames.The graph follows ST-GCN and captures structure across both spatial and temporal dimensions.Source: §3.1 · p. 3

3.2. Graph convolution

• Multiple spatiotemporal graph-convolution layers extract high-level features before global average pooling and softmax classification.The pipeline applies graph convolutions to the constructed graph and then predicts action categories.Source: §3.2 · p. 3
• Spatial graph convolution samples one-distance neighbors and partitions them into self, centripetal, and centrifugal subsets for distinct weight vectors.The kernel size is set to 3, and the mapping balances each subset's contribution.Source: §3.2 · p. 3

3.3. Implementation

• The implementation treats features as a C × T × N tensor, transforms spatial convolution using subset-specific adjacency matrices, and applies learned 1 × 1 weights and attention maps.N, T, and C denote vertices, temporal length, and channels; α is set to 0.001 to avoid empty rows.Source: §3.3 · p. 3
• Temporal convolution uses a K_t × 1 kernel because each vertex has two corresponding temporal neighbors in adjacent frames.This operation follows the spatial graph-convolution output.Source: §3.3 · p. 3

4.1. Adaptive graph convolutional layer

• Adaptive graph convolution learns layer- and sample-specific topology by combining the physical graph with globally learned and data-dependent graphs.The learned graph is optimized jointly with convolution parameters, while the residual branch preserves the original model’s behavior.Source: §4.1 · p. 4
• The data-dependent graph computes vertex similarities in an embedding space using two 1 × 1 convolutional embedding functions.The resulting N × N similarity matrix is normalized and used as soft edges between vertices.Source: §4.1 · p. 4

4.2. Adaptive graph convolutional block

• Each adaptive graph convolutional block combines spatial and temporal GCNs with batch normalization, ReLU, dropout, and a residual connection.The temporal convolution follows ST-GCN, using a Kt × 1 convolution on C ×T ×N feature maps with dropout set to 0.5.Source: §4.2 · p. 5, Figure 3 · p. 5

4.3. Adaptive graph convolutional network

• The AGCN stacks nine blocks, applies input batch normalization and global average pooling, and sends the pooled features to a softmax classifier.The block output channels progress through 64, 128, and 256 across the network.Source: §4.3 · p. 5, Figure 4 · p. 5

4.4. Two-stream networks

• The two-stream framework explicitly models bone vectors containing length and direction information alongside joint data.Joint and bone inputs are processed by J-stream and B-stream, and their softmax scores are added for prediction.Source: §4.4 · p. 5

• Experiments use the NTU-RGBD and Kinetics-Skeleton datasets to compare the proposed model with ST-GCN and other methods.Source: §5 · pp. 5–6
• Figure 5 depicts the overall 2s-AGCN architecture, where the scores from the two streams are added for the final prediction.Source: Figure 5 · p. 6
• Figure 6 labels joints separately for Kinetics-Skeleton and NTU-RGBD, reflecting their different skeleton definitions.Source: Figure 6 · p. 6, §5 · p. 6

• NTU-RGBD contains 56,000 clips across 60 classes, with 25 joints per subject and recordings from three cameras.Source: §5.1 · p. 6
• NTU-RGBD evaluation reports top-1 accuracy under cross-subject and cross-view benchmarks with prescribed training and validation splits.Source: §5.1 · p. 6
• Kinetics-Skeleton contains 300,000 clips in 400 classes, uses 18 OpenPose-estimated joints, and reports validation top-1 and top-5 accuracy.Source: §5.1 · p. 6

• Models are trained with SGD using Nesterov momentum, batch size 64, cross-entropy loss, and weight decay 0.0001.Source: §5.2 · p. 6
• NTU-RGBD samples are padded or repeated to two bodies and 300 frames, with learning-rate reductions at epochs 30 and 40 before stopping at epoch 50.Source: §5.2 · p. 6
• Kinetics-Skeleton inputs use 150 frames and two bodies, with random frame selection, coordinate perturbations, rotations, and translations.Source: §5.2 · pp. 6–7
• The Kinetics-Skeleton learning rate is reduced at epochs 45 and 55, and training ends at epoch 65.Source: §5.2 · p. 7

5.3.1 Adaptive graph convolutional block.

• With all three adaptive graph components A, B, and C, the model achieves the best validation performance, while removing any component harms recognition.Source: §5.3.1 · p. 7, Table 1 · p. 7

5.3.2 Visualization of the learned graphs

• The learned adjacency matrix is more flexible than the original graph and is not constrained to physical human-body connections.Source: §5.3.2 · p. 7, Figure 7 · p. 7
• Different layers learn different topologies: lower layers emphasize adjacent joints, whereas higher layers connect semantically related joints such as the two hands.Source: §5.3.2 · p. 7, Figure 8 · p. 7
• Graphs learned for different samples differ even within the same convolutional subset and layer, supporting data-dependent topology learning.Source: §5.3.2 · pp. 7–8, Figure 9 · p. 8

5.3.3 Two-stream framework

• The combined two-stream 2s-AGCN outperforms the single-stream Js-AGCN and Bs-AGCN variants.Source: §5.3.3 · p. 8, Table 2 · p. 8

• The model achieves state-of-the-art performance with a large margin on both NTU-RGBD and Kinetics-Skeleton.Source: §5.4 · p. 8
• The comparisons cover handcraft-feature-based, RNN-based, CNN-based, and GCN-based skeleton action-recognition methods.Source: §5.4 · p. 8
• The NTU-RGBD and Kinetics-Skeleton comparison results are presented in Tables 3 and 4, respectively.Source: §5.4 · p. 8

• 2s-AGCN parameterizes the skeleton graph so its structure is jointly learned and updated with the model.Source: §5.5 · p. 8
• The two-stream framework explicitly incorporates bone information alongside first-order skeleton information to enhance recognition performance.Source: §5.5 · p. 8
• The final model achieves state-of-the-art performance on both NTU-RGBD and Kinetics, with comparisons reported in Tables 3 and 4.Source: §5.5 · p. 8, Table 3 · p. 8, Table 4 · p. 8