Source-linked AI summary
Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching
Wei Peng, Xiaopeng Hong, Haoyu Chen, Guoying Zhao
TL;DR
Existing skeleton-action GCNs commonly rely on fixed graphs and one-order approximations, motivating better architectures that capture implicit joint correlations and higher-order connections. The paper uses NAS to assemble dynamic spatial-temporal and higher-order graph modules with an efficient evolution strategy, and reports state-of-the-art performance on two large-scale datasets.
Problem
Fixed graph topologies can ignore implicit joint correlations, while one-order spectral approximations do not adequately involve higher-order connections.
Method
NAS searches a GCN space containing multiple dynamic spatial-temporal graph modules, higher-order connections, and a sampling- and memory-efficient evolution strategy.
Results
The searched model achieves the best performance across all evaluation metrics on both NTU RGB+D and Kinetics-Skeleton datasets.
Takeaways & Limitations
The searched architecture supports the effectiveness of higher-order approximation and temporal-based dynamic graph modeling for skeleton action recognition.
Abstract
from arXiv · showhide
Human action recognition from skeleton data, fueled by the Graph Convolutional Network (GCN), has attracted lots of attention, due to its powerful capability of modeling non-Euclidean structure data. However, many existing GCN methods provide a pre-defined graph and fix it through the entire network, which can loss implicit joint correlations. Besides, the mainstream spectral GCN is approximated by one-order hop, thus higher-order connections are not well involved. Therefore, huge efforts are required to explore a better GCN architecture. To address these problems, we turn to Neural Architecture Search (NAS) and propose the first automatically designed GCN for skeleton-based action recognition. Specifically, we enrich the search space by providing multiple dynamic graph modules after fully exploring the spatial-temporal correlations between nodes. Besides, we introduce multiple-hop modules and expect to break the limitation of representational capacity caused by one-order approximation. Moreover, a sampling- and memory-efficient evolution strategy is proposed to search an optimal architecture for this task. The resulted architecture proves the effectiveness of the higher-order approximation and the dynamic graph modeling mechanism with temporal interactions, which is barely discussed before. To evaluate the performance of the searched model, we conduct extensive experiments on two very large scaled datasets and the results show that our model gets the state-of-the-art results.
Introduction
Skeleton-based action recognition uses GCNs to model non-Euclidean data, but fixed graphs and limited one-order connections leave architecture design and implicit joint correlations unresolved. The paper addresses these gaps with NAS, dynamic spatial-temporal graphs, higher-order connections, and an efficient evolution strategy.
- Skeleton data compactly represents body movement and is robust to viewpoint variation, occlusion, and self-occlusion.
- Most existing GCN methods use pre-defined fixed graph topologies that ignore implicit correlations between joints.
- NAS searches GCN architectures with dynamic graph structures, reducing manual effort while addressing the mismatch between graph data and conventional convolution-oriented search spaces.
- The searched model achieves the best performance across all evaluation metrics on both NTU RGB+D and Kinetics-Skeleton datasets.
- The proposed search space combines multiple spatial-temporal dynamic graph modules with higher-order Chebyshev connections to enlarge GCN receptive fields.
- A sampling- and memory-efficient evolution-based NAS strategy searches the architecture by estimating architecture distributions and activating one function module per iteration.
Related work
Skeleton-based action recognition benefits from graph-based modeling because skeletons form non-Euclidean structures, while NAS automates neural network design under computational constraints. Prior work spans grid-based recurrent or convolutional processing, GCN methods, reinforcement learning, and evolutionary search.
- Skeleton data is increasingly used because it is robust to changes in body scales, viewpoints, and backgrounds.
- Because human skeletons form non-Euclidean graphs rather than grids, conventional methods rearrange them into grid-shaped structures for RNNs and CNNs.
- NAS automatically builds neural networks under limited computational resources through approaches including reinforcement learning, evolutionary search, and gradient-based methods.
Methodology
The method uses NAS to construct GCN architectures for skeleton sequences, replacing fixed graphs with searched dynamic spatial-temporal modules and higher-order graph convolutions. A sampling- and memory-efficient evolution strategy searches module combinations and architecture distributions.
- Chebyshev polynomial modules with different orders are searched across layers to model higher-order hop connections beyond the first-order approximation.
- Skeleton action recognition is modeled as supervised graph classification over a sequence of graphs whose nodes and edges represent joints and bones.
- The search space replaces fixed graph structures with multiple graph-generating modules that explore dynamic topology at different representation levels.
- The graph search space combines Chebyshev-based functions and three dynamic graph modules, weighted and summed using architecture parameters.
- Dynamic graphs capture spatial node similarity and temporal representation correlations, incorporating interactions between neighboring frames before computing node connections.
- CEIM estimates an architecture distribution with weighted samples, while activating one function module per search step improves memory efficiency.
Experiments
Experiments evaluate the searched GCN on two large-scale skeleton datasets, analyze selected modules and higher-order connections, and compare performance with existing methods. The searched model achieves the best performance across all reported evaluation metrics on both datasets.
- Datasets and comparisons: Experiments use NTU RGB+D and Kinetics-Skeleton for skeleton-based action recognition, with comparative evaluation against existing approaches.NTU RGB+D includes Cross-subject and Cross-view benchmarks, while the reported comparisons cover multiple state-of-the-art methods.
- Architecture search: The searched network selects among dynamic graph modules and Chebyshev approximations with different orders at each block layer.Table 1 defines eight module options, including M(S), M(T), M(ST), and higher-order approximations.
- Architecture search: The search runs for 70 epochs, sampling N = 50 architectures after the first 20 epochs and selecting modules with architecture parameters α > 0.1.The architecture distribution is updated with the CEIM algorithm after the initial random-update phase.
- Architecture search analysis: The 2-order hop connection is preferred over other higher-order choices, while the physical-structure graph L is not selected at any layer.This result indicates that the searched architecture favors a particular higher-order connection and does not use the physical graph module.
- Performance comparison: The searched model achieves the best performance on both NTU RGB+D and Kinetics-Skeleton across all reported evaluation metrics.The comparison reports score-level fusion on joints and bones against 14 state-of-the-art methods on NTU RGB+D and eight on Kinetics.
Conclusion
The paper uses NAS to construct GCNs with dynamic graph-generating modules and higher-order connections, optimized through a sampling- and memory-efficient CEIM strategy. Experiments on two large-scale datasets report strong performance relative to state-of-the-art approaches and support temporal-based dynamic graph modeling.
- Conclusion: NAS constructs a GCN search space with three dynamic graph modules based on spatial-temporal node correlations and modules with higher-order connections.The search space is designed to explore graph topology and enlarge the GCN receptive field.
- Conclusion: The CEIM strategy combines cross-entropy evolution strategy with importance mixing to search architectures efficiently in sampling and memory.The method searches for an optimal GCN architecture within the proposed module space.
- Conclusion: Experiments on two very large-scale datasets report overwhelming performance compared with state-of-the-art approaches and support the effectiveness of temporal-based dynamic graph modeling.The conclusion specifically identifies the temporal-based dynamic graph module as effective.