Source-linked AI summary
Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition
Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, Weiming Hu
TL;DR
Existing skeleton-based action-recognition GCNs have difficulty modeling channel-specific and sample-dependent joint correlations efficiently. The paper proposes CTR-GC, which refines a shared topology with dynamic channel-specific correlations, and combines it with temporal modules in CTR-GCN. CTR-GCN outperforms state-of-the-art methods on three skeleton-based action-recognition benchmarks.
Problem
Existing graph approaches struggle with efficient channel-wise topology modeling, while parameterized topologies remain fixed across samples and cannot model sample-dependent correlations.
Method
CTR-GC learns a shared topology and dynamically inferred channel-specific correlations, then refines the shared topology to obtain channel-wise topologies.
Results
CTR-GCN exceeds state-of-the-art methods on the NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets.
Takeaways & Limitations
CTR-GC provides stronger graph representation capability by relaxing constraints imposed by other graph convolutions.
Takeaways & Limitations
Parameterized topologies remain the same for all samples, limiting their ability to model sample-dependent correlations.
Abstract
from arXiv · showhide
Graph convolutional networks (GCNs) have been widely used and achieved remarkable results in skeleton-based action recognition. In GCNs, graph topology dominates feature aggregation and therefore is the key to extracting representative features. In this work, we propose a novel Channel-wise Topology Refinement Graph Convolution (CTR-GC) to dynamically learn different topologies and effectively aggregate joint features in different channels for skeleton-based action recognition. The proposed CTR-GC models channel-wise topologies through learning a shared topology as a generic prior for all channels and refining it with channel-specific correlations for each channel. Our refinement method introduces few extra parameters and significantly reduces the difficulty of modeling channel-wise topologies. Furthermore, via reformulating graph convolutions into a unified form, we find that CTR-GC relaxes strict constraints of graph convolutions, leading to stronger representation capability. Combining CTR-GC with temporal modeling modules, we develop a powerful graph convolutional network named CTR-GCN which notably outperforms state-of-the-art methods on the NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets.
1. Introduction
Skeleton-based action recognition benefits from modeling joint topology, but existing approaches struggle to represent channel-specific and sample-dependent correlations efficiently. CTR-GC addresses this with shared-topology refinement and supports CTR-GCN, which outperforms state-of-the-art methods on three benchmarks.
- Motivation: Skeleton-based recognition uses joint correlations because body topology provides important information that sequence- and image-based methods overlook.Earlier methods treat joints independently, while graph-based approaches model their relationships.
- Limitations of prior methods: Topology-shared graph convolutions aggregate different channels with the same topology, limiting channel-specific feature modeling.Topology-non-shared methods address this limitation but can require independently learned topologies.
- Limitations of prior methods: Independent channel-wise topology learning can make models too heavy and difficult to optimize, while fixed parameterized topologies cannot capture sample-dependent correlations.These constraints hinder effective channel-wise topology modeling.
- Proposed approach: CTR-GC learns a shared topology as a generic prior and refines it with dynamically inferred channel-specific correlations for each sample.The refinement approach avoids independently modeling every channel and introduces few extra parameters.
- Results: CTR-GCN combines CTR-GC with temporal modeling and outperforms state-of-the-art methods on NTU RGB+D, NTU RGB+D 120, and NW-UCLA.Experiments also report benefits from channel-wise topology and the refinement method.
- Proposed approach: CTR-GC relaxes strict constraints of other graph convolutions through a unified formulation, improving representation capability.The paper supports this conclusion through mathematical analysis and experiments.
2. Related Work
GCN-based skeleton action recognition models joint relationships through graph topology, with methods differing in whether topology is dynamic and whether it is shared across channels.
- Topology Modeling: Topology modeling methods vary by whether their graphs remain fixed or are dynamically adjusted during inference.Dynamic methods infer correlations from input features and generally provide stronger generalization than static methods.
- Topology Sharing: Topology-shared methods aggregate all channels with one topology, whereas topology-non-shared methods use different topologies across channels or channel groups.Shared topologies can limit channel-specific aggregation, while non-shared designs introduce additional modeling flexibility.
- Topology Sharing: Channel-wise topology modeling is difficult because independently parameterized topologies require many parameters and remain unchanged across samples.These issues increase optimization difficulty and prevent parameterized topologies from modeling sample-dependent correlations.
3. Method
CTR-GC transforms node features, refines a shared topology with input-dependent channel-specific correlations, and aggregates each channel using its corresponding topology. Its dynamic topology-non-shared formulation relaxes constraints that limit earlier graph convolutions.
- CTR-GC Framework: CTR-GC transforms input features into high-level representations before topology modeling and channel-wise aggregation.The framework consists of feature transformation, channel-wise topology modeling, and channel-wise aggregation.
- Channel-wise Topology Modeling: CTR-GC learns one shared topology as a generic prior and refines it with channel-specific correlations for each output channel.The shared adjacency is learned, while channel-specific correlations are inferred from vertex features.
- Channel-wise Topology Modeling: Correlation functions model pairwise vertex relationships using transformed features, with M1 based on nonlinear distances and M2 using concatenation followed by an MLP.Linear transformations reduce feature dimensions before correlation modeling, and a further transformation raises the channel dimension.
- Topology Refinement: The refinement topology is formed by adding scaled channel-specific correlations to the shared topology, with trainable scalar α controlling refinement intensity.The shared topology is broadcast across channels before addition.
- Channel-wise Aggregation: CTR-GC aggregates each channel on its own refined channel-graph and concatenates the resulting channel outputs.Each channel-graph represents vertex relationships under a particular type of motion feature.
- Dynamic Modeling: Because channel-specific correlations depend on input samples, CTR-GC is a dynamic graph convolution that adapts its topology across samples.The resulting topologies vary with different input samples rather than remaining fixed.
- Representation Capability: Unified reformulation identifies CTR-GC as dynamic topology-non-shared graph convolution, the least constrained category in the paper’s comparison.The analysis states that this formulation theoretically provides stronger representation capability than previous graph convolutions.
4. Experiments
Experiments evaluate CTR-GC through ablations, configuration studies, comparisons with other graph convolutions, topology visualizations, and state-of-the-art benchmarks. CTR-GCN consistently benefits from channel-wise topology refinement and achieves strong results across three datasets.
- Ablation Study: Replacing baseline graph convolutions with CTR-GCs steadily increases accuracy, with the largest improvement when all graph convolutions are replaced.This ablation validates the effectiveness of CTR-GC within CTR-GCN.
- Ablation Study: Removing channel-specific correlations reduces performance by 1.2%, while removing the shared topology reduces it by 0.9%.Both shared priors and channel-specific refinement contribute to CTR-GCN, with channel-wise modeling having the larger measured effect.
- Configuration Exploration: Correlation modeling remains effective across multiple functions, whereas excessive reduction to r = 16 weakens channel-specific correlation modeling.Models with r = 4 or 8 perform better, and r = 8 is slightly better with fewer parameters; Sigmoid and ReLU perform worse than Tanh.
- Comparison with Other Graph Convolutions: Topology-non-shared graph convolutions generally outperform topology-shared methods, while dynamic methods provide stronger performance in the comparison.CTR-GC is compared with other graph convolutions under a fixed backbone and comparable computational conditions.
- Visualization of Learned Topologies: Refined topologies differ across channels while some joint relationships remain consistently strong, showing both channel-specific and shared structural patterns.The visualization uses an action sample of typing on the keyboard; values near zero indicate weak joint relationships.
5. Conclusion
The conclusion presents CTR-GC as a channel-wise topology refinement method with stronger correlation modeling capability. CTR-GCN outperforms state-of-the-art methods on three skeleton-based action recognition datasets.
- Conclusion: CTR-GC refines channel-wise topologies and demonstrates stronger representation capability than other graph convolutions through mathematical analysis and experiments.The method is designed for skeleton-based action recognition.
- Conclusion: CTR-GCN outperforms state-of-the-art methods on three skeleton-based action recognition datasets.The conclusion summarizes the reported result across the evaluated datasets.
Supplemental Materials for Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition
The supplemental materials provide derivations, architecture details, visualizations, and additional ablation studies for CTR-GCN.
- Supplemental Materials: The supplement derives selected equations, documents CTR-GCN architecture and hyperparameters, visualizes shared and channel-specific topologies, and reports further ablations.It specifically mentions derivations from Equations 12 to 13 and 8 to 14, plus input and output settings for each block.
Formula Derivation
The supplemental derivation shows that the paper’s graph-convolution equations can be rewritten into equivalent unified forms. It relates channel-wise relationships and feature transformations across the reformulated equations.
- Derivation from Equation 12 to Equation 13: The derivation identifies each output feature as a weighted aggregation involving channel-wise relationships, input features, and the weight matrix.The passages define p_ij as the channel-wise relationship, x_j as the input feature, and W as the weight matrix.
- Equation Reformulation: Equation 20 has the same form as Equation 16 and can therefore be reformulated into the form of Equation 17.The supplemental text presents this as an intermediate equivalence in the derivation.
- Equation Reformulation: Equation 21 is equivalent to Equation 14, showing that Equation 8 can be reformulated into Equation 14.This establishes the final equivalence stated in the supplemental derivation.
Detailed Architecture
CTR-GCN processes skeleton sequences through ten basic blocks and a classification layer, while CTR-GC learns shared and channel-specific topologies for feature aggregation.
- Detailed Architecture: CTR-GCN contains ten basic blocks followed by global average pooling, a fully connected layer, and softmax classification.The architecture processes skeleton sequences independently before averaging their classification scores.
- Detailed Architecture: Figure 5 visualizes shared topologies and channel-specific correlations for a “typing on a keyboard” sample.Green lines represent natural skeleton connections, while red-line intensity indicates correlation strength.
- Detailed Architecture: The model uses M=2, 2, and 1 people for NTU RGB+D, NTU RGB+D 120, and NW-UCLA, respectively.
Visualization
Visualization shows that CTR-GCN combines coarse, dense shared topologies with channel-specific relationships that emphasize motion-dependent hand interactions.
- Visualization: Shared topologies in three layers are coarse and dense, capturing global features for action recognition.
- Visualization: Channel-specific correlations vary across channels, indicating that CTR-GCN models joint relationships for different motion features.
- Visualization: Most channel-specific correlations focus on the two hands, helping capture subtle hand interactions in “typing on a keyboard.”
Ablation Study
The ablation evidence examines the number of CTR-GCs and performance on difficult actions involving subtle finger interactions.
- Ablation Study: CTR-GCN accuracy initially increases as the number of CTR-GCs grows, but drops at six CTR-GCs, possibly because of overfitting.The reported comparison tests models with 1–6 CTR-GCs and uses three CTR-GCs for fair comparison with other methods.
- Ablation Study: 7.03% is the minimum advantage of CTR-GC over other graph convolutions on “cut nails.”
- Ablation Study: 4.36% is the minimum advantage of CTR-GC over other graph convolutions on “open bottle.”
- Ablation Study: CTR-GC outperforms other graph convolutions across all evaluated hard action classes.These actions involve subtle finger interactions that are difficult to recognize correctly.