Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition

Fanfan Ye; Shiliang Pu; Qiaoyong Zhong; Chao Li; Di Xie; Huiming Tang

Source-linked AI summary

Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition

Fanfan Ye, Shiliang Pu, Qiaoyong Zhong, Chao Li, Di Xie, Huiming Tang

arXiv:2007.14690v1Jul 29, 2020cs.CV

TL;DR

Skeleton-based action recognition depends on learning effective graph topology, but existing methods can neglect global contextual information when estimating joint dependencies. Dynamic GCN introduces a lightweight Context-encoding Network that learns sample- and layer-specific topology, achieving state-of-the-art benchmark performance with substantially lower computation.

Problem
Existing topology-learning methods estimate dependencies between joint pairs while ignoring the influence of other contextual joints, limiting the available topology information.
Method
Dynamic GCN embeds a data-driven Context-encoding Network into graph-convolutional layers to learn globally context-enriched, directed dynamic topology for each sample and layer.
Results
The final model achieves state-of-the-art performance on NTU-RGB+D, NTU-RGB+D 120, and Skeleton-Kinetics, while CeN adds ~7% extra FLOPs and Dynamic GCN uses 2×~4× fewer FLOPs than other GCN-based methods.
Takeaways & Limitations
Context-enriched topology learning combines with static physical connections and motion modalities to provide an effective and computationally competitive skeleton-recognition framework.
Takeaways & Limitations
The hybrid GCN-CNN design requires further exploration across tasks and alternative sequential or parallel arrangements of convolution types.

Abstract

from arXiv · show

Graph Convolutional Networks (GCNs) have attracted increasing interests for the task of skeleton-based action recognition. The key lies in the design of the graph structure, which encodes skeleton topology information. In this paper, we propose Dynamic GCN, in which a novel convolutional neural network named Contextencoding Network (CeN) is introduced to learn skeleton topology automatically. In particular, when learning the dependency between two joints, contextual features from the rest joints are incorporated in a global manner. CeN is extremely lightweight yet effective, and can be embedded into a graph convolutional layer. By stacking multiple CeN-enabled graph convolutional layers, we build Dynamic GCN. Notably, as a merit of CeN, dynamic graph topologies are constructed for different input samples as well as graph convolutional layers of various depths. Besides, three alternative context modeling architectures are well explored, which may serve as a guideline for future research on graph topology learning. CeN brings only ~7% extra FLOPs for the baseline model, and Dynamic GCN achieves better performance with $2\times$~$4\times$ fewer FLOPs than existing methods. By further combining static physical body connections and motion modalities, we achieve state-of-the-art performance on three large-scale benchmarks, namely NTU-RGB+D, NTU-RGB+D 120 and Skeleton-Kinetics.

1 INTRODUCTION

Dynamic GCN addresses limitations in skeleton topology learning by using a lightweight context-encoding network to construct sample- and layer-specific dynamic graphs. The approach combines global contextual topology learning with static connections and achieves strong accuracy and computational efficiency.

Existing non-local topology learning considers only pairs of joints, whereas reliable topology learning should incorporate contextual information from the remaining joints.
Dynamic GCN embeds the Context-encoding Network into graph convolutional layers to learn skeleton topology automatically from globally aggregated joint features.CeN is data-driven, supports directed dependencies, and can be trained end-to-end.
CeN predicts a unique graph topology for each input sample and each graph-convolutional layer, producing dynamic rather than static topology.The resulting topology is intended to enhance model capacity and expressiveness.
Figure 2 contrasts physical, masked physical, pairwise non-local, and CeN-based adjacency matrices; CeN incorporates features from all contextual joints.
~7% extra FLOPs is added by CeN, while Dynamic GCN requires 2×~4× fewer FLOPs than other GCN-based methods.Combining CeN-predicted topology with static topology further improves performance.
The final model achieves state-of-the-art performance on NTU-RGB+D, NTU-RGB+D 120, and Skeleton-Kinetics.The final model combines static physical body connections with spatial and motion modalities.

2 RELATED WORKS

Related work covers deep neural and graph-convolutional approaches to skeleton-based action recognition. GCN methods preserve irregular skeleton structure, but topology construction ranges from fixed physical graphs to learned sample-specific graphs with limited contextual modeling.

Skeleton-based action recognition research includes DNN-based and GCN-based methods, with deep learning methods outperforming hand-crafted-feature approaches.
CNN methods encode spatiotemporal information jointly but can lose skeleton topology when irregular joint data are converted into regular pseudo-images.CeN instead uses CNNs to learn graph topology rather than directly classify actions.
GCNs represent skeleton topology with an N × N adjacency matrix and can use fixed physical connections or learnable masks applied to them.
Non-local methods construct sample-specific graphs by measuring dependencies between arbitrary joints, but they use only the two underlying joints and ignore contextual joints.Dynamic GCN addresses this contrast by incorporating contextual-joint features.

3 METHOD

Dynamic GCN learns sample- and layer-specific skeleton topologies with the lightweight Context-encoding Network (CeN), then combines dynamic and static graph features in Dynamic GConv layers. The framework also supports joint-level aggregation and multiple spatial-motion modalities.

Context-encoding Network: CeN learns skeleton topology automatically by using a lightweight convolutional network that incorporates contextual information across joint, temporal, and feature dimensions.It takes the whole feature map as input and predicts the full adjacency matrix directly.
Context-encoding Network: CeN produces a unique adjacency matrix for each input sample and each GConv layer, rather than sharing topology across samples or depths.Its trainable kernels encode global context by treating the joint dimension as a channel.
Dynamic GConv Layer: Dynamic GConv fuses a static physical-graph branch with a CeN-predicted dynamic branch, then applies temporal convolution.The static branch uses a physical graph with a learnable mask, while the dynamic branch uses the context-enriched graph.
Dynamic GConv Layer: The fused branch outputs are combined by weighted summation, with λ balancing dynamic and static topology features before subsequent processing.The framework stacks 10 Dynamic GConv layers and appends global average pooling and a fully connected classifier.
Joint-level Feature Aggregation and Modalities: Joint-level feature aggregation progressively shrinks the joint dimension using a projection matrix P, while separate joint, bone, and motion streams provide multiple modalities.Motion is defined as the temporal movement of each joint or bone, and modality logits are fused by summation.

4 EXPERIMENTS

Dynamic GCN is evaluated on three large-scale skeleton-based action-recognition benchmarks with ablations and comparisons against current state-of-the-art methods.

Evaluation Protocol: The evaluation covers NTU-RGB+D, NTU-RGB+D 120, and Skeleton-Kinetics, with extensive ablations and comparisons to current state-of-the-art methods.The experiments assess different framework components before evaluating the final model.

4.1 Datasets

The experiments use three large-scale skeleton-action benchmarks and examine joint-level feature aggregation across GConv layers.

Datasets: NTU-RGB+D contains 56,880 skeleton clips across 60 action classes, captured from three camera views with 25 joints per subject.The dataset provides 3D joint locations and clips contain at most two subjects.
Datasets: The standard NTU-RGB+D cross-subject and cross-view evaluation protocols are followed.The supplied passage identifies these as the standard protocols for the dataset.

4.2 Implementation Details

The implementation follows MS-AAGCN processing and attention settings, trains for 65 epochs, and uses fixed sequence lengths with empirically selected aggregation settings.

Training Configuration: Training lasts 65 epochs on all three datasets, with λ set to 1 and joint aggregation rate α set to 0.6.The projection matrix is inserted after the 5th and 8th GConv layers.
Input Processing: Input sequences are resized to 64 frames for NTU-RGB+D and NTU-RGB+D 120, and 150 frames for Skeleton-Kinetics.These fixed lengths are used across the respective datasets.

4.3 Ablation Studies

Ablations show that CeN’s directed, context-enriched topology complements static topology, while modality fusion improves accuracy and Dynamic GCN remains computationally efficient.

4.3.2 Effectiveness of CeN: Directed graphs predicted by CeN outperform undirected CeN* graphs with or without the static branch, supporting directed topology for dynamic skeleton characteristics.The comparison explicitly evaluates static, CeN-predicted, non-local-based, and combined graph configurations.
4.3.2 Effectiveness of CeN: Static graphs alone reach 88.2% accuracy, whereas combining static and CeN-predicted graphs improves accuracy to 89.2%.The corresponding non-local dynamic graph improves the static result by only 0.2%.
4.3.3 Alternative Context-enriched Topology: CeN variants aggregate global context along feature or temporal dimensions instead of the joint dimension, providing alternative context-modeling architectures.CeN-F uses feature-dimension aggregation and CeN-T uses temporal-dimension aggregation.
4.3.4 Contribution of Model Ensemble: Spatial modality ensemble raises accuracy from 89.2% to 90.9%, and adding motion reaches 91.5% top-1 accuracy in NTU-RGB+D C-Subject.The 91.5% result is reported for the ensemble model after incorporating motion information.
4.3 Ablation Studies: CeN adds ~7% FLOPs to the baseline, while Dynamic GCN has a 2×~4× FLOPs advantage over other GCN-based methods with state-of-the-art performance.The baseline increases from ~1.86G to ~1.99G FLOPs.
4.3 Ablation Studies: Visualized learned topologies emphasize action-relevant joints, including hands for Wipe Face, knees and feet for Jump, and hands and feet for walking.Dynamic topology lines are superimposed on the physical human-body topology.

4.4 Comparison with the State-of-the-arts

Dynamic GCN achieves state-of-the-art results across NTU-RGB+D, NTU-RGB+D 120, and Skeleton-Kinetics, while retaining a substantial efficiency advantage over competing GCN methods.

NTU-RGB+D: Dynamic GCN achieves 91.5% C-Subject and 96.0% C-View top-1 accuracy on NTU-RGB+D.The paper reports state-of-the-art performance in both settings and comparable accuracy to MS-G3D Net with greater efficiency.
Comparison with the State-of-the-arts: Across the benchmark comparisons, Dynamic GCN combines state-of-the-art performance with substantially lower computational cost than competing GCN-based methods.The efficiency comparison is stated alongside the NTU-RGB+D results and references the FLOPs analysis.
Skeleton-Kinetics: Dynamic GCN achieves 37.9% top-1 and 61.3% top-5 accuracy on Skeleton-Kinetics.These results are reported as state-of-the-art performance on the dataset.
NTU-RGB+D 120: On NTU-RGB+D 120, Dynamic GCN reaches 87.3% C-Subject and 88.6% C-Setup top-1 accuracy, surpassing the listed baselines.The model also achieves slightly better accuracy than MS-G3D Net.

5 DISCUSSION

The hybrid GCN-CNN design uses CNN-based topology learning to expand graph-convolution capacity, while its broader application and architecture remain open for further exploration.

5.1 On Hybrid GCN-CNN: Dynamic GCN is a simple hybrid GCN-CNN architecture in which CNNs learn topology, producing a flexible and expressive graph topology.The discussion contrasts this hybrid design with GCN alone.
5.1 On Hybrid GCN-CNN: The hybrid GCN-CNN approach may extend beyond skeleton action recognition to graph-structured domains such as social network modeling.The paper presents this as a possible application rather than an evaluated result.
5.1 On Hybrid GCN-CNN: The specific form of hybrid GCN-CNN remains underexplored for different tasks, including whether normal and graph convolutions should be stacked sequentially or in parallel.The authors leave these architecture choices as future work.
5.1 On Hybrid GCN-CNN: CeN-learned topology is complementary to physical topology because it captures informative potential connections missed by physical body connections.The paper links this interpretation to experiments and topology visualizations.

6 CONCLUSION

Dynamic GCN addresses skeleton-topology learning with a context-enriched hybrid GCN-CNN framework and is validated on three large-scale datasets with reduced computation.

6 CONCLUSION: Dynamic GCN uses a context-encoding network to learn global context-enriched skeleton topology within a hybrid GCN-CNN framework.The framework targets effective topology extraction for skeleton-based action recognition.
6 CONCLUSION: Extensive experiments on three large-scale datasets validate the method’s superiority, while reduced FLOPs improve its competitiveness for edge-device deployment.The conclusion specifically connects computational reduction with deployment where computing power is limited.