Source-linked AI summary
Symbiotic Graph Neural Networks for 3D Skeleton-based Human Action Recognition and Motion Prediction
Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, Qi Tian
TL;DR
The paper addresses separate treatment of skeleton-based action recognition and motion prediction, along with insufficient modeling of body relations. Sym-GNN jointly trains mutually enhancing recognition and prediction heads using multi-scale graph operations and complementary bone features. Experiments on four datasets report improved performance over previous methods on both tasks.
Problem
Prior studies often treated action recognition and motion prediction separately and rarely exploited their mutual promotion, while body relations were insufficiently captured.
Method
Sym-GNN jointly models both tasks with multi-scale joint- and part-scale graphs, actional and structural relations, and a complementary bone-based network.
Results
Sym-GNN consistently improves over previous methods on both action recognition and motion prediction across experiments on four datasets.
Takeaways & Limitations
The results support using a shared symbiotic framework to learn action-recognition and motion-prediction representations together.
Takeaways & Limitations
The stability analysis evaluates robustness under input perturbations satisfying ∥X*−X∥≤ϵ.
Abstract
from arXiv · showhide
3D skeleton-based action recognition and motion prediction are two essential problems of human activity understanding. In many previous works: 1) they studied two tasks separately, neglecting internal correlations; 2) they did not capture sufficient relations inside the body. To address these issues, we propose a symbiotic model to handle two tasks jointly; and we propose two scales of graphs to explicitly capture relations among body-joints and body-parts. Together, we propose symbiotic graph neural networks, which contain a backbone, an action-recognition head, and a motion-prediction head. Two heads are trained jointly and enhance each other. For the backbone, we propose multi-branch multi-scale graph convolution networks to extract spatial and temporal features. The multi-scale graph convolution networks are based on joint-scale and part-scale graphs. The joint-scale graphs contain actional graphs, capturing action-based relations, and structural graphs, capturing physical constraints. The part-scale graphs integrate body-joints to form specific parts, representing high-level relations. Moreover, dual bone-based graphs and networks are proposed to learn complementary features. We conduct extensive experiments for skeleton-based action recognition and motion prediction with four datasets, NTU-RGB+D, Kinetics, Human3.6M, and CMU Mocap. Experiments show that our symbiotic graph neural networks achieve better performances on both tasks compared to the state-of-the-art methods.
1 INTRODUCTION
The paper targets the joint understanding of 3D skeleton-based actions and motion, addressing separate-task modeling and incomplete body relations with a symbiotic multi-scale graph framework.
- 3D skeleton-based action recognition classifies query actions, while motion prediction forecasts future movements from observations.
- Prior work commonly treated recognition and prediction separately, although the tasks can mutually promote one another.
- Its joint-scale and part-scale graph operators capture actional, structural, and body-part relations for multi-scale spatial features.
- A dual bone-based network learns complementary bone features for more effective classification and prediction.
- Experiments on NTU-RGB+D, Kinetics, Human 3.6M, and CMU Mocap report better performance than state-of-the-art methods on both tasks.
- Sym-GNN jointly trains an action-recognition head and a motion-prediction head whose outputs enhance each other.
2 RELATED WORKS
Related work spans recurrent, convolutional, generative, and graph-based approaches for skeleton action recognition and motion prediction, with graph methods modeling non-grid structure.
- 3D skeleton-based action recognition: RNN-based recognition methods model temporal dependencies between consecutive skeleton frames.
- 3D skeleton-based motion prediction: Motion-prediction research includes state models, RNNs, adversarial mechanisms, and geodesic losses for learning future dynamics.
- Graph deep learning: Graph deep learning propagates features over non-grid data structures using spectral- or vertex-domain operations.
3 PROBLEM FORMULATION
The formulation represents observed and future 3D joint positions with skeleton graphs and defines the model outputs as an action class and predicted future motion.
- At time t, an action pose X(t) contains M joint positions in 3D, with t≤0 denoting observations and t>0 denoting future frames.
- The skeleton graph uses adjacency matrix A, where connected joints have A_ij=1, disconnected joints have A_ij=0, and self-loops are included.
- The data comprise a previous-motion tensor, a future-motion tensor, and a one-hot class label across C possible classes.
- The overall model produces a discriminated class category and predicted future motion using trainable backbone, recognition-head, and prediction-head parameters.
4 BASIC COMPONENTS
The model uses joint-scale and part-scale graph operators to capture structural, actional, and higher-level body relations, while difference features provide motion information. These components are integrated with temporal processing and robustness analysis.
- Basic operators: Sym-GNN introduces joint-scale, part-scale, bidirectional-fusion, and difference operators for extracting spatial and temporal skeleton features.Joint-scale operators model body-joint relations, part-scale operators model body-part relations, fusion exchanges information across scales, and the difference operator extracts temporal features.
- Joint-scale graphs: AGIM learns actional graphs from observations by propagating features between joints and possible edges, then estimating pairwise relation strengths.After K propagation iterations, embedded joint features are used to construct the actional graph without requiring action-category labels.
- Joint-scale graphs: Structural graphs extend physical skeleton connectivity to longer ranges through multiple weighted graph orders.The model uses structural graphs of orders γ = 1, . . . , Γ; for example, a hand can be correlated with the entire arm.
- Joint-scale graphs: JGC combines actional and structural graph convolutions, with λact controlling the contribution of actional features relative to structural features.This aggregation updates each center joint using both learned action-based relations and structure-based relations.
- Stability: JGC outputs remain upper bounded under disturbed input skeletons, and experiments report an amplify factor q around 1.The bound extends to subsequent layers, supporting stable performance under small input noises.
- Temporal features: J-GTC blocks combine JGC with temporal convolution, aggregating spatial relations among joints and information across consecutive frames.The blocks also include batch normalization, dropout, and a residual connection preserving input features.
- Part-scale graphs: The part-scale graph contains 10 body-part nodes, propagates information over natural part connections, and omits a separate part-scale actional graph.Integrated part relations and shorter distances facilitate long-range links, while the P-GTC block extracts higher-scale spatial and temporal features.
- Temporal features: The difference operator uses orders β = 0, 1, 2 to represent positions, velocities, and accelerations, computed efficiently in parallel.High-order pose differences are used as motion information to guide spatial-temporal feature learning.
5 SYMBIOTIC GRAPH NEURAL NETWORKS
Sym-GNN combines a multi-branch, multi-scale backbone with task-specific recognition and motion-prediction heads trained through a multitasking scheme. Its branches model multiple motion differences and joint- and part-scale relations, while the prediction head sequentially generates future poses.
- Architecture: Sym-GNN comprises a deep backbone, an action-recognition head, and a motion-prediction head, with both tasks trained jointly.The model also includes a dual bone-based network for complementary downstream features.
- Backbone: Three parallel multi-scale GCN branches process position, velocity, and acceleration proxies derived by difference operators.The branches share identical architectures and concatenate their learned semantics for both tasks.
- Backbone: Each branch combines joint-scale and part-scale graphs through J-GTC and P-GTC blocks to learn spatial and temporal features at two body granularities.Joint-scale graphs capture actional and structural joint correlations, while part-scale graphs represent higher-level body parts.
- Backbone: Bidirectional fusion exchanges information between scales by pooling joint features into part representations and matching part features back to corresponding joints.For example, torso joints are averaged into a part-scale super node, while a thigh feature is copied to corresponding hip and knee joints.
- Action recognition: The recognition head concatenates the three branch outputs, applies an MLP and global average pooling, and produces class probabilities with a softmax classifier.Its loss is cross entropy between ground-truth categorical labels and inferred labels.
- Motion prediction: The motion-prediction head uses JGC, a GRU, and an output MLP to predict future poses sequentially from recurrently recycled predictions.The GRU receives motion-difference features and classified labels, while a residual connection predicts frame-wise displacement added to the previous pose.
- Training: Joint training minimizes cross-entropy recognition loss and L1 motion-prediction loss, with experiments reporting that the L1 norm yields more precise predictions than the common L2 norm.The two heads are described as mutually promoting, improving performance in both tasks.
6 EXPERIMENTS AND ANALYSIS
Experiments across recognition and prediction benchmarks show that Sym-GNN benefits from joint task learning, multi-scale relational graphs, and complementary motion representations. The model generally outperforms or competes strongly with prior methods, while exposing tradeoffs in prediction speed, graph complexity, and task-noise robustness.
- Sym-GNN outperforms state-of-the-art methods on both action recognition and motion prediction benchmarks.
- Graph Components: The complete model benefits from joint-scale and part-scale graphs, motion prediction, and dual bone-based features, whereas overly large graph hops introduce redundancy and confusion.
- Motion Prediction: Sym-GNN is faster and more precise than competitors for 400 ms prediction, although larger AGIM iteration counts lower MAE while reducing speed.
- Symbiosis of Recognition and Prediction: Joint training improves both tasks: recognition strengthens prediction, while prediction benefits recognition even with noisy motion targets near 50%.
- Symbiosis of Recognition and Prediction: With limited observations, predicting all future frames gives the best recognition performance at low observation ratios, while prediction variants outperform no prediction.
- Loss Balance: Recognition is strongest at λact = 0.5, while prediction remains robust across weights with MAEs around 0.615 and lowest errors at λact = 0.9 and 1.0.
7 CONCLUSIONS
The conclusion presents Sym-GNN as a joint graph-based framework for skeleton action recognition and motion prediction, evaluated across four datasets with consistent improvements over previous methods.
- Sym-GNN jointly handles action recognition and motion prediction through mutually enhancing task heads and graph-based action-pattern modeling.
- Learnable joint-scale and part-scale graphs extract spatial information, while experiments cover NTU-RGB+D, Kinetics, Human 3.6M, and CMU Mocap.
- Experiments show consistent improvements compared with previous methods.
APPENDIX A PROOF OF THEOREM 1
The appendix proves that the activated JGC operator is robust to input perturbations by bounding graph-convolution discrepancies and using the contractive property of ReLU.
- Theorem 1 establishes robustness of the activated JGC operator against input perturbations.
- The proof bounds the discrepancy after joint-scale graph convolution using norm inequalities and stated assumptions.
- ReLU is shown to be contractive through case analysis, supporting the perturbation bound.
APPENDIX B TRAINING ALGORITHM
Training uses KKT conditions to balance recognition and prediction losses in the shared backbone while optimizing task-specific heads separately.
- The multitask optimization imposes KKT conditions on both the shared backbone and task-specific heads.
- The backbone satisfies a convex combination of recognition and prediction gradients, λ∇θbkLrecg+(1−λ)∇θbkLpred = 0, with 0 ≤ λ ≤ 1.
- Task-specific parameters separately satisfy zero gradients for their respective recognition and prediction losses.
- λ is adaptively adjusted during training using multi-objective optimization to approach a stationary point.
APPENDIX C NETWORK ARCHITECTURE
The appendix provides detailed network-architecture information for Sym-GNN, including parameter sizes and corresponding operations.
- The appendix presents the network structure of Sym-GNN in greater detail.
- It lists parameter sizes for the actional graph inference module.
- It also lists corresponding operations for the AGIM and backbone networks.
C.1 Actional Graph Inference Module
The actional graph inference module learns action-based correlations among moving joints by propagating joint and arbitrary-link features for long-range relation capture and action-graph estimation.
- AGIM learns action-based correlations among different moving joints.
- It propagates joint and arbitrary-link features to aggregate long-range joint features.
- The module supports relation capturing and action-graph estimation, with its structure presented in Table 16.
C.2 Backbone (9 layers)
The NTU-RGB+D and Kinetics backbone uses nine J-GTC layers and eight P-GTC layers, with spatial-temporal operations and joint-part feature transformations documented in the architecture tables.
- C.2 Backbone (9 layers): The backbone contains 9 layers of J-GTC blocks and 8 layers of P-GTC blocks.
- C.2 Backbone (9 layers): Each block specifies spatial and temporal convolution operators together with kernel sizes, normalization, dropout, activations, and data dimensions.
- C.2 Backbone (9 layers): Joint2part pooling and part2joint matching are annotated as feature-concatenation operators that increase feature dimensions.
C.3 Backbone (4 layers light version)
For Human 3.6M and CMU Mocap, Sym-GNN uses a light backbone for action-feature extraction, documenting its convolutional operations and joint-part transformations in Table 18.
- C.3 Backbone (4 layers light version): Sym-GNN uses a light version to extract action features on Human 3.6M and CMU Mocap.
- C.3 Backbone (4 layers light version): The architecture specifies spatial and temporal convolution operations with kernel sizes, normalization, dropout, activations, and feature shapes.
- C.3 Backbone (4 layers light version): Joint2part pooling and part2joint matching are included in the light backbone architecture.