Source-linked AI summary
Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition
Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, Wanli Ouyang
TL;DR
Skeleton action recognition needs unbiased long-range aggregation and unobstructed cross-spacetime information flow, but existing approaches have limitations in both areas. The paper disentangles multi-scale graph convolutions and introduces G3D, combining them into MS-G3D. Experiments show that the resulting model outperforms existing methods on NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400.
Problem
Existing multi-scale operators bias long-range aggregation toward local joints, while factorized spatial-temporal modules hinder direct modeling of complex cross-spacetime dependencies.
Method
The paper removes redundant dependencies between neighborhoods and introduces G3D, whose dense cross-spacetime edges provide direct information flow; together they form MS-G3D.
Results
The resulting model outperforms existing methods on NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400.
Takeaways & Limitations
MS-G3D captures multi-scale spatial-temporal features that factorized modeling may overlook.
Takeaways & Limitations
G3D requires balancing temporal window size and dilation because larger windows increase computation and genericity, whereas larger dilations reduce temporal resolution.
Abstract
from arXiv · showhide
Spatial-temporal graphs have been widely used by skeleton-based action recognition algorithms to model human action dynamics. To capture robust movement patterns from these graphs, long-range and multi-scale context aggregation and spatial-temporal dependency modeling are critical aspects of a powerful feature extractor. However, existing methods have limitations in achieving (1) unbiased long-range joint relationship modeling under multi-scale operators and (2) unobstructed cross-spacetime information flow for capturing complex spatial-temporal dependencies. In this work, we present (1) a simple method to disentangle multi-scale graph convolutions and (2) a unified spatial-temporal graph convolutional operator named G3D. The proposed multi-scale aggregation scheme disentangles the importance of nodes in different neighborhoods for effective long-range modeling. The proposed G3D module leverages dense cross-spacetime edges as skip connections for direct information propagation across the spatial-temporal graph. By coupling these proposals, we develop a powerful feature extractor named MS-G3D based on which our model outperforms previous state-of-the-art methods on three large-scale datasets: NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400.
1. Introduction
Skeleton-based action recognition uses abstract, noise-resistant joint positions, but existing graph convolutions struggle with unbiased long-range modeling and direct cross-spacetime dependency capture. The paper introduces disentangled multi-scale aggregation and G3D, combined as MS-G3D, which outperforms prior methods on three large-scale datasets.
- Skeleton representations encode 2D or 3D human-joint positions while reducing environmental noise such as background clutter and lighting conditions.
- Skeleton spatial-temporal graphs model joint movement patterns as sequences of connected human-body graphs across time.
- Higher-order adjacency polynomials expand receptive fields but bias aggregation toward closer joints because cyclic walks overweight local neighborhoods.
- Factorized spatial-only and temporal-only modules hinder direct information flow across spacetime, weakening cues involving coordinated movements across body regions and future frames.
- The proposed disentangled aggregation removes redundant dependencies between neighborhoods, enabling multi-scale operators to model graphwide joint relationships.
- G3D introduces unified spatial-temporal graph convolution with cross-spacetime edges, and its integration with disentangled aggregation yields MS-G3D.MS-G3D provides multi-scale receptive fields across spatial and temporal dimensions.
2. Related Work
Related work develops graph neural networks and multi-scale skeleton models for structured human-joint data. Existing approaches use adjacency powering, temporal-spatial factorization, or cross-frame edges, while this paper targets weighting bias and richer cross-spacetime modeling.
- Architectures: Graph neural networks for arbitrarily structured graphs are broadly divided into spectral and spatial approaches.
- Multi-Scale Graph Convolutions: Multi-scale spatial GNNs commonly use higher-order adjacency polynomials to aggregate features from non-local neighbors.
- Multi-Scale Graph Convolutions: Adjacency powering can introduce weighting bias in long-range modeling, motivating disentangled multi-scale aggregators.
- Skeleton Action Recognition: Recent skeleton methods construct spatial-temporal graphs and apply GNNs to model semantic human-body connectivity for action prediction.
- Skeleton Action Recognition: GR-GCN adds sparsified edges between adjacent frames, whereas G3D uses semantically structured cross-spacetime edges and supports multiple temporal contexts with varied windows and dilations.
3. MS-G3D
MS-G3D combines disentangled multi-scale aggregation with unified spatial-temporal graph convolution to model long-range and cross-spacetime dependencies in skeleton sequences.
- Disentangled Multi-Scale Aggregation: Higher-order adjacency powers bias aggregation toward closer, higher-degree nodes, causing local signals to dominate and weakening long-range joint modeling.Cyclic walks and self-loops amplify this bias on skeleton graphs.
- Disentangled Multi-Scale Aggregation: The disentangled k-adjacency formulation removes redundant dependencies on closer neighborhoods, enabling additive aggregation across scales and more effective large-k modeling.The resulting adjacency matrices are also sparser than exponentiated alternatives.
- G3D: Unified Spatial-Temporal Modeling: G3D constructs spatial-temporal windows whose dense cross-frame edges connect each node to itself and its one-hop spatial neighbors across all selected frames.Dilated windows select frames at a fixed interval, expanding temporal coverage while reusing the same spatial-temporal structure.
- G3D: Unified Spatial-Temporal Modeling: MS-G3D integrates disentangled multi-scale aggregation directly into G3D for multi-scale reasoning in the spatial-temporal domain.This couples the two proposed mechanisms within a unified module.
- Model Architecture: G3D requires balancing temporal window size and dilation because larger windows increase receptive field and computation, while larger dilation reduces temporal resolution.The authors report best performance when G3D is augmented with long-range factorized modules.
- Model Architecture: The final architecture combines G3D pathways with factorized spatial and temporal pathways to capture regional spatial-temporal relationships and long-range dependencies simultaneously.The G3D pathway processes windows with disentangled multi-scale graph convolutions, while factorized modules provide extended spatial and temporal contexts.
4. Experiments
Experiments evaluate MS-G3D across datasets, component configurations, graph-connectivity choices, and input modalities. The results support disentangled multi-scale aggregation, G3D’s cross-spacetime modeling, and the full model’s state-of-the-art performance.
- Evaluation setup: NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 provide the three large-scale evaluation datasets.The experiments use standard classification settings, including Cross-Subject and Cross-Setup evaluations for NTU RGB+D datasets.
- Disentangled Multi-Scale Aggregation: 1.4% gain over adjacency powering at K = 4 demonstrates the benefit of disentangled multi-scale aggregation in spatial graph convolutions.The gain reflects correction of weighting imbalance among neighborhoods; on the G3D pathway, the gap reaches 0.8% at K = 12 even with residual masks.
- Effectiveness of G3D: Adding the G3D pathway consistently improves results with similar or fewer parameters, capturing regional spatial-temporal correlations missed by factorized modeling.The controlled comparison uses the joint stream of 2s-AGCN as a baseline and replaces its temporal convolutions with MS-TCN layers.
- Exploring G3D Configurations: τ = 5 performs slightly better than τ = 7, while combining two G3D pathways with d = (1, 2) and using τ = (3, 5) gives the best results.Larger temporal neighborhoods can produce overly generic features, whereas larger dilation can reduce temporal resolution.
- Cross-spacetime Connectivity: Dense cross-spacetime edges are essential for efficient information flow in unified spatial-temporal graph convolutions.The connectivity study contrasts dense cross-spacetime connections with grid-like temporal self-edges while holding other architectural components fixed.
- Comparison against the State-of-the-Art: The full model outperforms all existing methods under every evaluation setting across the three datasets, and joint-bone fusion gives the best performance on NTU RGB+D 60.The joint-bone result indicates that the method generalizes to another input modality.
5. Conclusion
The paper combines disentangled multi-scale graph aggregation with G3D, a unified spatial-temporal operator, to build MS-G3D for skeleton-based action recognition. Experiments on three large-scale datasets show that the resulting model outperforms existing methods by a sizable margin.
- The disentangled multi-scale aggregation scheme removes redundant dependencies between different neighborhoods in graph convolutions.
- G3D directly models spatial-temporal dependencies from skeleton graph sequences through a unified graph convolutional operator.
- Coupling both methods yields MS-G3D, a feature extractor that captures multi-scale spatial-temporal features overlooked by factorized modeling.
- Experiments on three large-scale datasets show that MS-G3D outperforms existing methods by a sizable margin.