Source-linked AI summary
Topology-aware Convolutional Neural Network for Efficient Skeleton-based Action Recognition
Kailin Xu, Fanfan Ye, Qiaoyong Zhong, Di Xie
TL;DR
Pure CNNs have been underused for skeleton-based action recognition because they are considered weak at modeling irregular skeleton topology. Ta-CNN addresses this with topology-aware feature augmentation, convolutional analysis, and SkeletonMix, achieving competitive performance with lower complexity. The paper also notes that direct image mixup can severely deform skeleton topology.
Problem
Pure CNN methods have limited ability to model irregular skeleton topology and have rarely achieved performance competitive with GCN-based methods.
Method
Ta-CNN uses cross-channel feature augmentation through CAG and VAG, treats joint dimensions as channels for CNN topology modeling, and applies SkeletonMix for skeleton-specific augmentation.
Results
Ta-CNN surpasses existing CNN-based methods and achieves comparable performance to GCN-based methods with significantly fewer parameters and GFLOPs.
Takeaways & Limitations
Graph convolution can be implemented as a special case of normal convolution, supporting lightweight CNN-based topology modeling for skeleton action recognition.
Takeaways & Limitations
Applying vanilla mixup directly to skeleton data causes severe deformation of skeleton topology and poor performance.
Abstract
from arXiv · showhide
In the context of skeleton-based action recognition, graph convolutional networks (GCNs) have been rapidly developed, whereas convolutional neural networks (CNNs) have received less attention. One reason is that CNNs are considered poor in modeling the irregular skeleton topology. To alleviate this limitation, we propose a pure CNN architecture named Topology-aware CNN (Ta-CNN) in this paper. In particular, we develop a novel cross-channel feature augmentation module, which is a combo of map-attend-group-map operations. By applying the module to the coordinate level and the joint level subsequently, the topology feature is effectively enhanced. Notably, we theoretically prove that graph convolution is a special case of normal convolution when the joint dimension is treated as channels. This confirms that the topology modeling power of GCNs can also be implemented by using a CNN. Moreover, we creatively design a SkeletonMix strategy which mixes two persons in a unique manner and further boosts the performance. Extensive experiments are conducted on four widely used datasets, i.e. N-UCLA, SBU, NTU RGB+D and NTU RGB+D 120 to verify the effectiveness of Ta-CNN. We surpass existing CNN-based methods significantly. Compared with leading GCN-based methods, we achieve comparable performance with much less complexity in terms of the required GFLOPs and parameters.
1 Introduction
The paper proposes Ta-CNN, a lightweight pure CNN that explicitly enhances coordinate and joint topology features for skeleton-based action recognition. Its CAG/VAG modules, theoretical convolution analysis, and SkeletonMix strategy target competitive accuracy with reduced complexity.
- Motivation: Skeleton-based action recognition favors GCNs for irregular topology, while pure CNNs have rarely achieved competitive performance.GCN-based methods can rely on heavy models, motivating a lighter CNN alternative.
- Method: Ta-CNN enhances coordinate and joint features through a cross-channel map-attend-group-map module implemented as CAG and VAG.CAG targets coordinate features, while VAG targets joint features; joint transposition enables CNN-based topology modeling.
- Method: Ta-CNN uses two input streams, skeleton sequence and motion information, which are fused in the middle of the network.The streams support the framework shown in Figure 2.
- Data augmentation: SkeletonMix combines upper- and lower-body parts from different persons to enrich training samples and improve robustness and accuracy.The strategy is specialized for skeleton data rather than directly applying image mixup.
- Results: The method surpasses CNN-based methods and matches GCN-based performance with significantly fewer parameters and GFLOPs.The contribution summary reports comparable GCN performance alongside reduced complexity.
2 Related Work
Prior skeleton-action-recognition research includes RNN, CNN, and GCN approaches. GCNs lead because they handle irregular graphical structures, while image-oriented mixup methods cannot be directly transferred to skeleton data without distorting topology.
- Task overview: Deep learning methods have surpassed hand-crafted features, with skeleton-action-recognition models commonly grouped into RNN, CNN, and GCN categories.These categories reflect distinct approaches to modeling skeleton sequences and their spatiotemporal structure.
- RNNs and CNNs: RNNs model temporal dynamics, while CNN methods encode spatiotemporal features either through skeleton-to-image conversion or direct 3D skeleton input.The cited CNN approaches differ in how skeleton information is represented before or during convolution.
- GCNs: GCNs became dominant because they handle irregular graphical structures, with later work improving local/global perception and reducing model complexity.ST-GCN established graph convolution for spatial and temporal skeleton patterns, followed by many extensions.
- Data augmentation: Image mixup, CutMix, and related strategies cannot be directly adopted for skeleton data because they cause unreasonable deformation of skeleton topology.SkeletonMix is introduced as a topology-aware alternative specialized for skeleton augmentation.
3 Method
Ta-CNN treats skeleton sequences as structured tensors and enhances their topology with CNN modules that operate on coordinate and joint features. The framework also establishes a convolutional interpretation of graph convolution and introduces SkeletonMix for skeleton-specific augmentation.
- 3.1 Preliminaries: Skeleton sequences are represented as 3D tensors, with coordinates analogous to channels and temporal and joint dimensions analogous to image height and width.
- 3.1 Preliminaries: The coordinate-scaling experiment tests whether the three coordinate dimensions contribute unequally to recognition accuracy.Each coordinate is multiplied by a random scale factor from 0 to 1 in the toy experiment.
- 3.2 Enhancing Coordinate Feature via CAG: CAG combines feature mapping, channel attention, and dual coordinate-wise grouped convolution to enhance coordinate features.The grouped convolutions use 3×1 and 1×1 kernels with different group counts, then fuse their outputs by element-wise summation.
- 3.3 Graph Convolution Is a Special Convolution: Graph convolution is equivalent to normal convolution when the joint dimension is treated as channels, the output and input channel counts match, and adjacency values serve as weights.Larger kernels can jointly perceive topology, temporal features, and coordinate features, while output channels can expand or reduce the joint dimension.
- 3.4 Enhancing Joint Feature via VAG: VAG extends the map-attend-group-map design to virtual-part-aware grouping, while Ta-CNN uses CAG and VAG in its input streams.The framework processes skeleton sequences and skeleton motion, fusing multiple-person features by element-wise maximum at a late stage.
- 3.6 SkeletonMix: SkeletonMix combines the upper body of one person with the lower body of another to augment skeleton data.A selected batch fraction is mixed, while the remaining samples stay unchanged; direct image-style mixup can severely deform skeleton topology.
4 Experiments
Experiments evaluate Ta-CNN across four datasets, ablate its components, and compare accuracy and efficiency with CNN- and GCN-based methods. Results show gains from the proposed modules and SkeletonMix, strong accuracy, and competitive efficiency.
- Experimental Setup: Ta-CNN is evaluated on NTU RGB+D, NTU RGB+D 120, N-UCLA, and SBU using ablations and state-of-the-art comparisons.The experiments also report model complexity and modality-specific results.
- Ablation Studies: CAG and VAG improve the baseline, with the ablation reporting a 2.3% gain in HCN using a more efficient model.The modules enhance coordinate- and joint-level feature modeling.
- Attention Visualization: Learned channel attention varies across coordinates, joints, and action classes, supporting the claim that feature dimensions have unequal importance.Figure 6 visualizes coordinate and joint attention responses for the NTU RGB+D test set.
- Efficiency: Ta-CNN achieves state-of-the-art performance while balancing parameters and GFLOPs, and Ta-CNN+ remains competitive despite doubled computational complexity.This efficiency is highlighted as suitable for deployment on resource-constrained edge devices.
- Comparison with the State-of-the-arts: Ta-CNN obtains 97.2% on N-UCLA and 98.9% on SBU, surpasses CNN-based methods on NTU datasets, and reaches comparable performance to GCNs with smaller models.The reported comparisons cover the four evaluated datasets.
5 Conclusion
The paper presents Ta-CNN as a pure CNN for skeleton action recognition that enhances topology features through cross-channel augmentation and SkeletonMix. It reports strong accuracy with a tiny model and relates graph convolution theoretically to normal convolution.
- Conclusion: Ta-CNN enhances irregular skeleton topology using cross-channel feature augmentation and a skeleton-specific mixup strategy.The architecture applies the augmentation to coordinate and joint features.
- Conclusion: The paper proves that graph convolution is a special case of normal convolution when the joint dimension is treated as channels.This theoretical result is presented as consistent with the model design.
- Conclusion: Ta-CNN achieves state-of-the-art performance with a tiny model and is described as suitable for real-world deployment.The conclusion connects its performance and model size to deployment suitability.
A.1 Notes on the Data Modalities We Used
The experiments use one joint modality for ablations, complexity evaluation, and visualizations, while final Ta-CNN and Ta-CNN+ results use four modalities.
- Data Usage: Joint data are used for ablations, model-complexity evaluation, and visualizations.These include the listed tables and Figures 6 and 7.
- Data Usage: Joint, bone, joint motion, and bone motion are used only for final Ta-CNN and Ta-CNN+ results.These multimodal results are reported on N-UCLA, NTU RGB+D, and NTU RGB+D 120.
A.2 Per-class Performance Improvement
Ta-CNN+ improves performance most clearly on difficult action classes involving similar actions or subtle movements, consistent with its cross-channel feature enhancements.
- Per-class Improvements: Ta-CNN+ improves most classes, especially drinking, brushing teeth, reading, wearing or removing a shoe, and sneezing.These six classes are identified as showing particularly strong improvements.
- Per-class Improvements: The largest improvements occur on baseline-difficult classes with similar actions or subtle movements.Examples include distinguishing wearing from taking off a shoe and recognizing reading.
- Per-class Improvements: CAG and VAG enhance cross-channel features in low-dimensional subspaces and virtual parts, supporting recognition of nuanced actions.The passage links this design to separation from similar classes.
A.3 Impact of the Groups in CAG and VAG
The study evaluates how the group-count hyper-parameter n affects CAG and VAG, with the strongest combinations being (10,6) and (10,10). These settings align with interpretable coordinate and joint groupings.
- The top-performing CAG–VAG group-count combinations are (10,6) and (10,10).
- For CAG, n = 10 divides the feature space into 10 three-dimensional subspaces analogous to the x, y, and z coordinates.
- For VAG, n = 10 or n = 6 corresponds to selecting three or five joints as one part.
A.4 SkeletonMix
SkeletonMix combines upper and lower body parts from different people and improves recognition across input modalities. Its best reported interpolation setting on NTU RGB+D 120 is α = 1/16.
- SkeletonMix improves recognition accuracy across different input modalities on NTU RGB+D and NTU RGB+D 120.
- α = 1/16 produces the best result among the tested SkeletonMix settings on the NTU RGB+D 120 cross-subject benchmark.
A.5 Architecture of Ta-CNN
Ta-CNN uses separate coordinate and joint feature processing, repeats its grouping modules across skeleton and motion streams, and dynamically fuses multiple-person inputs. Its per-class accuracies are compared against a strong baseline on NTU RGB+D.
- CAG and VAG are repeated for the skeleton-sequence and skeleton-motion input streams.
- Dynamic fusion makes the actual GFLOPs approximately proportional to the number of input persons and variable across samples.
- Table 14 reports each module's data shape, parameter count, and GFLOPs, while Maxout is applied dynamically to a varying number of persons.
- Figure 7 plots per-class accuracies for the strong baseline and Ta-CNN+ on NTU RGB+D cross-subject data, with classes sorted by baseline accuracy.