Source-linked AI summary
Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognition
Jungho Lee, Minhyeok Lee, Dogyoon Lee, Sangyoun Lee
TL;DR
Skeleton-based GCNs need semantically meaningful edges but struggle to identify important distant relationships and sample-specific edge contributions. HD-GCN addresses this with HD-Graph, A-HA, and a six-way joint-and-bone ensemble. The model outperforms state-of-the-art methods on four skeleton-action-recognition benchmarks.
Problem
Existing skeleton-based GCNs rely on physically connected edges and may aggregate edge features without identifying important distant or sample-specific relationships.
Method
HD-GCN decomposes joints into hierarchy node sets, connects semantically related adjacent and distant nodes through HD-Graph, applies A-HA, and ensembles three center-of-mass variants using joint and bone streams.
Results
HD-GCN outperforms state-of-the-art methods on four benchmarks for skeleton-based action recognition.
Takeaways & Limitations
The combined HD-Graph, A-HA, and six-way ensemble forms the paper’s empirically verified feature extractor for skeleton-based action recognition.
Abstract
from arXiv · showhide
Graph convolutional networks (GCNs) are the most commonly used methods for skeleton-based action recognition and have achieved remarkable performance. Generating adjacency matrices with semantically meaningful edges is particularly important for this task, but extracting such edges is challenging problem. To solve this, we propose a hierarchically decomposed graph convolutional network (HD-GCN) architecture with a novel hierarchically decomposed graph (HD-Graph). The proposed HD-GCN effectively decomposes every joint node into several sets to extract major structurally adjacent and distant edges, and uses them to construct an HD-Graph containing those edges in the same semantic spaces of a human skeleton. In addition, we introduce an attention-guided hierarchy aggregation (A-HA) module to highlight the dominant hierarchical edge sets of the HD-Graph. Furthermore, we apply a new six-way ensemble method, which uses only joint and bone stream without any motion stream. The proposed model is evaluated and achieves state-of-the-art performance on four large, popular datasets. Finally, we demonstrate the effectiveness of our model with various comparative experiments.
1. Introduction
HD-GCN addresses limitations of skeleton-based GCNs by modeling semantically meaningful distant as well as adjacent joint relationships and selecting important edge sets per action. It combines HD-Graph, A-HA, and a six-way joint-and-bone ensemble, achieving state-of-the-art results on four benchmarks.
- Motivation: Skeleton-based methods are favored because skeleton coordinates are less affected by environmental noise than RGB video.They recognize actions from 2D or 3D coordinates of major human joints represented as time-series inputs.
- Limitations: Existing GCNs often use physically connected edges, leaving relationships between structurally distant joints unidentified and limiting long-range dependencies.Distant joints can be strongly correlated during action recognition, but handcrafted physical graphs are heuristically fixed.
- Limitations: Existing methods may aggregate edge features without accounting for each edge’s contribution, making sample-specific significant relationships difficult to identify.For actions such as squat down, interactions between legs and arms should receive emphasis.
- Proposed Method: HD-Graph decomposes joints into hierarchy node sets and connects neighboring sets to capture adjacent and distant relationships within shared semantic spaces.The hierarchy expands step by step from a center-of-mass node, while S-EdgeConv captures semantically close edges not represented by HD-Graph.
- Proposed Method: A-HA uses representative spatial average pooling and hierarchical edge convolution to highlight dominant hierarchy edge sets and joints for each input.H-EdgeConv compares hierarchical features in feature space to determine which should be emphasized.
- Evaluation: HD-GCN uses a six-way ensemble with three HD-Graphs, different center-of-mass nodes, and joint and bone streams without motion data.The framework is evaluated on NTU-RGB+D 60, NTU-RGB+D 120, Kinetics-Skeleton, and Northwestern-UCLA, where it outperforms state-of-the-art methods.
2. Related Work
GCN-based action-recognition methods use skeleton graphs to model spatial joint relationships, while attention mechanisms either construct graph topologies or emphasize features along selected dimensions. HD-Graph is introduced to retain skeletal structure while identifying meaningful distant connections.
- GCN-Based Action Recognition: Recent GCN methods extract spatial features from physically connected skeleton edges and outperform handcrafted-feature approaches.Their graphs commonly follow a handcrafted physical skeleton structure.
- HD-Graph: HD-Graph’s physically connected version decomposes the skeleton into a rooted tree, while its fully connected version links nodes in adjacent hierarchy edge sets within the same semantic space.The figure distinguishes physically connected and fully connected edges using blue and red lines.
- GCN-Based Action Recognition: Adaptive attention-based graph structures learn sample-wise topological features but may ignore physical skeletal priors and permit excessive flexibility during training.HD-Graph is presented as an alternative that references the skeletal structure while deriving a new graph organization.
- Attention Modules: Attention modules in GCNs include attention-based graph construction as well as spatial-, temporal-, and channel-wise feature attention.Graph-construction methods form topologies using non-local blocks or customized correlation matrices.
3. Methodology
HD-GCN constructs a hierarchically decomposed graph that connects semantically related distant joints, then uses hierarchy-aware convolution and attention-guided aggregation to learn sample-specific representations. A six-way ensemble combines joint and bone streams across graphs with different centers of mass.
- Hierarchically Decomposed Graph: HD-Graph decomposes the skeleton into hierarchy node sets formed by expanding stepwise from a center-of-mass node.Nodes in the same semantic space, such as elbows and knees or hands and feet, belong to one hierarchy node set.
- Hierarchically Decomposed Graph: Each hierarchy layer contains identity, centripetal, and centrifugal edge subsets, producing a graph with bidirectional and identity edges.The construction uses NL hierarchy layers and concatenates the three edge subsets for each layer.
- Fully Connected Inter-Hierarchy Edges: Fully connected edges between neighboring hierarchy sets capture meaningful distant-joint relationships, enlarge the receptive field, and remain learnable after degree normalization.This densifies the sparse physical-connection graph while retaining training adaptability.
- HD-Graph Convolution: HD-Graph convolution uses three subset-wise GCN branches plus a spatial EdgeConv branch, concatenating subset outputs rather than summing them.Spatial EdgeConv extracts sample-wise node connectivity in feature space that HD-Graph may not capture.
- Attention-Guided Hierarchy Aggregation: A-HA weights hierarchy-wise outputs using attention derived from representative spatial pooling and hierarchical EdgeConv.The module selects a representative temporal frame and nodes, learns similarities among hierarchy features, then performs a weighted sum across the hierarchy axis.
- Six-Way Ensemble: The six-way ensemble trains joint and bone streams on three HD-Graphs whose center-of-mass nodes are chest, belly, and hip.This design uses no motion streams and yields six graph-stream combinations.
4. Experiments
Experiments evaluate HD-GCN on four benchmark datasets, compare it with state-of-the-art methods, and test its graph, attention, ensemble, and complexity components. The results support hierarchical edge decomposition, sample-adaptive attention, and motion-free ensemble designs.
- Datasets and Experimental Settings: HD-GCN is evaluated on NTU-RGB+D 60, NTU-RGB+D 120, Kinetics-Skeleton, and Northwestern-UCLA.The experiments use the recommended dataset protocols and specified optimization settings.
- Comparison with State-of-the-Arts Methods: HD-GCN exceeds state-of-the-art recognition performance on all four datasets without motion streams.Its proposed ensemble also compares favorably with a six-way state-of-the-art method while using a four-way ensemble.
- Ablation Study: HD-Graph with only physically connected edges outperforms the conventional graph, while adding fully connected edges and S-EdgeConv improves performance across datasets.The ablation indicates that dividing joints into hierarchy edge sets is beneficial even when the underlying physical edges are shared.
- Ablation Study: Using three HD-Graphs with different center-of-mass nodes improves ensemble diversity without motion streams.The reported comparison shows that the four-way ensemble outperforms state-of-the-art four-way methods with motion data and is comparable to a six-way method.
- Complexity Analysis: HD-GCN achieves the best NTU-RGB+D 120 joint-stream performance with the lowest computational complexity among compared models.The complexity comparison fixes the window size at 64 and evaluates models in the same environment.
5. Conclusions
The paper presents HD-GCN for skeleton-based action recognition, combining hierarchical graph decomposition, attention-guided aggregation, and a six-way ensemble. It reports state-of-the-art performance on four benchmark datasets.
- Conclusions: HD-GCN decomposes joint nodes into hierarchy edge sets to model major distant-node connectivity in semantically related skeleton spaces.Its HD-Graph, A-HA module, and six-way ensemble are combined into the proposed feature extractor.
- Conclusions: The combined approach outperforms current state-of-the-art methods on four benchmark datasets.
A. Additional Details of HD-Graph
The additional HD-Graph details explain how hierarchy-wise node sets replace exhaustive handcrafted physical-edge specifications while enabling distant semantic relationships. The graph includes directed, reverse-directed, and identity edges, with flexible edge features and adaptive hierarchy weighting.
- Universality of HD-Graph: HD-Graph construction requires hierarchy-wise node sets rather than every physically adjacent human-joint edge.This makes the construction more universal than the existing handcrafted graph according to the authors.
- HD-Graph Convolution: The graph uses flexible edge features and A-HA to adaptively emphasize significant edge sets.A-HA applies hierarchy-wise attention after HD-Graph convolution rather than simply adding all hierarchy outputs.
B. Effectiveness of Six-way Ensemble
The six-way ensemble combines joint and bone streams across three HD-Graphs with different center-of-mass nodes, excluding motion streams. Different graphs produce distinct learning patterns, and the resulting ensemble outperforms competing methods in the reported comparisons.
- Six-Way Ensemble: The six-way ensemble trains joint and bone streams with HD-Graphs centered at chest, belly, and hip nodes.These six combinations are formed by pairing two data streams with three center-of-mass choices.
- Graph Diversity: Accuracy differences among the three center-of-mass graphs range from 4% to 13%, indicating different learning patterns.The comparison is shown for several action classes with different HD-Graphs.
- Ensemble Coefficients: Equal-contribution six-way ensembling avoids manually selected ensemble coefficients by using only joint and bone streams.The method replaces lower-performing motion streams and does not require coefficients controlling stream contributions.
- Additional Experimental Results: HD-GCN outperforms competing models even in single-stream experiments by a large margin.The additional results compare the reported single-stream outcomes with other models.
C. Architectures for Kinetics-Skeleton
The Kinetics-Skeleton graph is modified for HD-Graph by adding two manually generated center-of-mass nodes, increasing the architecture from 18 to 20 nodes. The modified architecture also includes newly generated edges.
- The original Kinetics-Skeleton architecture has 18 nodes but lacks hip and belly nodes for center-of-mass representation.These nodes are manually set using existing nodes.
- The CoM hip node is defined as the midpoint of the left and right hip nodes.
- The belly CoM node is defined as the midpoint of the chest and hip nodes.
- The modified skeleton architecture contains 20 nodes after generating the CoM hip and belly nodes.
- Figure 8 distinguishes newly generated edges and nodes in red in the original and modified architectures.