Source-linked AI summary
Revisiting Skeleton-based Action Recognition
Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, Bo Dai
TL;DR
Skeleton-based action recognition relies heavily on GCNs, whose coordinate-based graph representations have limitations in robustness, interoperability, and scalability. PoseConv3D instead uses 2D-pose heatmaps stacked into 3D volumes and classified with 3D-CNNs. Across multiple benchmarks, it achieves state-of-the-art performance for skeleton and multimodal recognition while improving robustness, interoperability, and scalability.
Problem
GCN-based skeleton recognition is limited by sensitivity to pose-estimation changes, difficult early multimodal fusion, and person-count scalability.
Method
PoseConv3D converts 2D poses into temporally stacked 3D heatmap volumes and applies a 3D-CNN, with RGBPose-Conv3D extending this to RGB–pose fusion.
Results
PoseConv3D achieves state-of-the-art performance across multiple benchmarks for skeleton-based and multimodal action recognition.
Takeaways & Limitations
3D heatmap volumes provide a competitive alternative to graph-based skeleton representations with improved robustness, interoperability, and scalability.
Takeaways & Limitations
In FineGYM, identifying the relevant athlete among dozens of audience members requires priors such as the interested person's location in the first frame.
Abstract
from arXiv · showhide
Human skeleton, as a compact representation of human action, has received increasing attention in recent years. Many skeleton-based action recognition methods adopt graph convolutional networks (GCN) to extract features on top of human skeletons. Despite the positive results shown in previous works, GCN-based methods are subject to limitations in robustness, interoperability, and scalability. In this work, we propose PoseC3D, a new approach to skeleton-based action recognition, which relies on a 3D heatmap stack instead of a graph sequence as the base representation of human skeletons. Compared to GCN-based methods, PoseC3D is more effective in learning spatiotemporal features, more robust against pose estimation noises, and generalizes better in cross-dataset settings. Also, PoseC3D can handle multiple-person scenarios without additional computation cost, and its features can be easily integrated with other modalities at early fusion stages, which provides a great design space to further boost the performance. On four challenging datasets, PoseC3D consistently obtains superior performance, when used alone on skeletons and in combination with the RGB modality.
1. Introduction
Skeletons provide compact, action-focused representations, but GCN-based methods face robustness, interoperability, and scalability limitations. PoseConv3D replaces graph sequences with 3D heatmap volumes processed by a 3D-CNN.
- Skeleton sequences preserve action information while reducing sensitivity to background and lighting changes.
- GCNs model joints as spatiotemporal graph nodes and have become a standard skeleton-based action-recognition approach.
- Limitations: Coordinate distribution shifts can substantially affect GCN predictions, even when caused by small pose-estimation perturbations.
- Limitations: Irregular skeleton graphs complicate early fusion with modalities such as RGB and optical flow, which are commonly represented on regular grids.
- PoseConv3D: PoseConv3D stacks 2D joint heatmaps over time into 3D heatmap volumes and applies a 3D-CNN for action recognition.
- PoseConv3D: PoseConv3D generalizes across input skeletons, supports multimodal convolutional integration, and handles varying person counts without added volume-computation cost.
2. Related Work
Related work centers on GCNs and CNNs for skeleton-based action recognition. PoseConv3D extends CNN-based processing by applying 3D-CNNs directly to 3D heatmap volumes.
- 3D-CNNs: 3D-CNNs are established for RGB action recognition but had been less explored for skeleton-based recognition because of their parameter demands.
- PoseConv3D: PoseConv3D uses 3D-CNNs with 3D heatmap volumes and reports state-of-the-art skeleton-based action-recognition performance.
- GCNs: GCN methods represent skeleton sequences as spatiotemporal graphs, with later work adding multiscale adjacency modeling and self-attention.
- GCNs: GCN-based methods remain limited in robustness and scalability, while multimodal feature fusion may require careful design.
- CNNs: CNN-based methods transform skeleton sequences into pseudo-images or aggregate heatmaps, but temporal aggregation can lose information and reduce recognition performance.
3. Framework
The framework extracts 2D poses, converts joint or limb evidence into temporally stacked 3D heatmap volumes, and processes them with specialized 3D-CNN architectures. It also supports efficient preprocessing, multi-person inputs, and early RGB–pose fusion.
- Good Practices for Pose Extraction: Pose extraction quality strongly affects recognition accuracy, and FineGYM may require priors to identify the relevant athlete among many people.
- Good Practices for Pose Extraction: PoseConv3D uses 2D top-down pose extraction, motivated by the generally better quality of 2D poses than 3D poses.
- Good Practices for Pose Extraction: PoseConv3D can also process 3D skeletons by decomposing their coordinates into three 2D skeleton projections.
- From 2D Poses to 3D Heatmap Volumes: A 2D pose is represented as K × H × W joint heatmaps, using estimator heatmaps directly or composing Gaussian maps from coordinate-triplets.
- From 2D Poses to 3D Heatmap Volumes: Joint or limb heatmaps from all frames are stacked temporally into a K × T × H × W volume, with multi-person maps accumulated without enlarging the heatmap.
- Heatmap Processing: Subjects-centered cropping reduces redundant spatial content, while uniform sampling better preserves global action dynamics.
- 3D-CNN for Skeleton-based Action Recognition: PoseConv3D adapts C3D, SlowOnly, and X3D with reduced early downsampling and smaller networks suited to mid-level heatmap features.
- 3D-CNN for Skeleton-based Action Recognition: RGBPose-Conv3D uses asymmetric RGB and pose pathways with bidirectional lateral connections for early-stage multimodal fusion.
4. Experiments
Experiments evaluate PoseConv3D across datasets, robustness settings, efficiency comparisons, multimodal fusion, and heatmap-processing choices. The results show strong recognition performance, cross-pose generalization, efficient scaling, and benefits from fusion and uniform sampling.
- Dataset Preparation: Experiments cover six datasets, including FineGYM, NTURGB+D, Kinetics400, UCF101, HMDB51, and Volleyball.FineGYM uses Mean Top-1 accuracy; the other datasets use Top-1 accuracy.
- Good properties of PoseConv3D: PoseConv3D is lighter than the GCN counterpart and consistently outperforms the state-of-the-art GCN with 10-clip testing.It uses the same architecture and hyperparameters across datasets, whereas GCN requires heavier dataset-specific tuning.
- Good properties of PoseConv3D: PoseConv3D loses less accuracy than GCN when training and testing use lower-quality poses or different person-box sources.The cross-PoseAnno evaluation compares high-quality and low-quality pose estimators and bounding-box annotations.
- Multi-Modality Fusion with RGBPose-Conv3D: Bi-directional lateral connections outperform uni-directional connections, and early+late fusion improves performance on both RGB-dominant NTU-60 and pose-dominant FineGYM.With bi-directional fusion, 1-clip early+late fusion can outperform 10-clip late fusion.
- Comparisons with the state-of-the-art: Multi-modality fusion achieves state-of-the-art results across eight video-recognition benchmarks and raises Kinetics400 accuracy by 0.6% beyond the state-of-the-art.The reported fusion settings include RGBPose-Conv3D and late fusion with PoseConv3D predictions.
- Ablation on Heatmap Processing: Subjects-centered cropping improves FineGYM Mean-Top1 by 1.0%, from 91.7% to 92.7%, while uniform sampling consistently outperforms fixed temporal strides.Uniform sampling draws frames across the entire clip and can make 1-clip testing outperform fixed-stride sampling.
5. Conclusion
The paper concludes that PoseConv3D uses 3D heatmap volumes with lightweight 3D-CNNs to address limitations of GCN-based skeleton recognition. It reports state-of-the-art performance for skeleton and multimodal action recognition across multiple benchmarks.
- PoseConv3D takes 3D heatmap volumes as input for 3D-CNN-based skeleton-based action recognition.
- The approach is presented as resolving GCN limitations in robustness, interoperability, and scalability while improving accuracy and efficiency.
- PoseConv3D achieves state-of-the-art results for skeleton-based and multimodality-based action recognition across multiple benchmarks.
A. Visualization
The visualization shows pose-extraction quality across four datasets, from near-perfect NTURGB+D and accurate Volleyball poses to challenging FineGYM errors and variable Kinetics400 conditions.
- NTURGB+D extraction is almost perfect because scenes are simple, with at most two people and little occlusion.
- FineGYM poses remain imperfect despite ground-truth athlete boxes, especially for rare poses and motion-blurred frames.The extracted poses are nevertheless described as discriminative enough for skeleton-based action recognition.
- Figures 7 and 8 visualize extracted skeletons from Kinetics400 and Volleyball, respectively.
- Kinetics400 is harder than NTURGB+D or FineGYM because person locations, scales, and counts vary substantially.The pose estimator still accurately predicts skeletons in the illustrated examples.
- Volleyball contains roughly a dozen people per frame, yet the pose extractor accurately predicts each person’s pose.Most poses are regular compared with FineGYM.
B. Generating Pseudo Heatmap Volumes.
PoseConv3D converts RGB videos into compact 3D pseudo-heatmap volumes through pose extraction, coordinate-triplet storage, temporal sampling, cropping, and heatmap generation.
- A Top-Down pose estimator extracts each person’s 2D pose and stores joints as coordinate-triplets (x, y, score).The pipeline uses HRNet for pose extraction.
- Joint and limb pseudo-heatmaps are generated from the coordinate-triplets and stacked across time into 3D heatmap volumes.
- Subjects-centered cropping reduces spatial redundancy while retaining the poses and their motion.
- Uniform sampling selects T frames from the video before generating the compact heatmap representation.
C.1. Different variants of PoseConv3D.
PoseConv3D adapts established 3D-CNN backbones—C3D, X3D, and SlowOnly—for skeleton action recognition, with lightweight variants and a dual-modality RGBPose-Conv3D design.
- PoseConv3D experiments instantiate three backbones: C3D, X3D, and SlowOnly.
- C3D: C3D is made more efficient by halving channel width from 64 to 32, while Pose-C3D-s removes its final two convolution layers.
- X3D: Pose-X3D removes the original first stage, and Pose-X3D-s uniformly removes convolution layers by changing γ_d from 2.2 to 1.
- SlowOnly: Pose-SlowOnly halves channel width and removes the original first stage; wider and deeper variants add no performance despite heavier backbones.
- RGBPose-Conv3D: RGBPose-Conv3D uses separate SlowOnly RGB and pose pathways with lateral connections, late fusion, and individual pathway losses.The RGB pathway uses lower frame rate and wider channels, whereas the pose pathway uses higher frame rate and narrower channels.
D.1. Ablation Study on Pose Extraction
The pose-extraction ablations examine 2D versus 3D representations, estimator choices, person-selection priors, and heatmap compression, showing that pose quality and representation format materially affect recognition.
- Experimental setup: The ablation uses an input volume of T × H × W = 48 × 56 × 56 and reports pose-estimator comparisons including COCO AP and NTU-60 accuracy.
- 2D versus 3D skeletons: Estimated 2D keypoints consistently outperform low-quality alternatives under the compared MS-G3D configuration.
- 2D versus 3D skeletons: Lifted 3D poses provide no additional information and perform worse than the original 2D poses for action recognition.
- Pose estimators: Top-Down and Bottom-Up pose extraction are compared using HRNet-w32, with MobileNet-v2 also evaluated as a Top-Down backbone.
- Interested person versus all persons: A prior identifying the interested person is extremely important, and even one ground-truth box per video can improve performance by a large margin.
- Coordinates versus heatmaps: Compressing heatmaps into coordinate-triplets causes a 2% Mean-Top1 drop for low-quality pose estimators.The experiment evaluates information loss on FineGYM.
D.2. Multi-Modality Action Recognition Results on UCF101 and HMDB51
PoseConv3D improves skeleton-based recognition on UCF101 and HMDB51, including when combined with RGB predictions. Additional experiments examine pretraining, projected 2D skeletons, multi-person inputs, and sampling strategies.
- Pretraining: Pretraining PoseConv3D on Kinetics400 skeleton data benefits downstream recognition on smaller UCF101 and HMDB51 datasets.The gains occur under both Linear and Finetune paradigms.
- Multi-modality fusion: PoseConv3D outperforms prior approaches by a large margin on skeleton recognition and improves overall performance after fusion with I3D predictions.The comparison uses I3D predictions rather than OmniSource for fairness.
- Projected 2D skeletons: Using projected 2D skeletons causes a 2% Top-1 accuracy drop for MS-G3D, while PoseConv3D exceeds the comparable GCN model by 2.4%.PoseConv3D also surpasses MS-G3D using 3D skeletons by 0.4%, suggesting stronger spatiotemporal modeling can compensate for projection loss.
- Group activity recognition: Representing all people with a single heatmap volume is the best practice for group activity recognition with PoseConv3D.The authors compare this representation with alternatives that process different persons’ heatmaps separately on Volleyball.
- Sampling: Uniform sampling outperforms fix-stride sampling for RGB recognition on NTU-60 and GYM, although it slightly reduces accuracy on Kinetics400.On the first two datasets, one-clip uniform sampling exceeds ten-clip fix-stride testing, which the authors mainly attribute to variable video lengths.
D.6. NTU-60 Error Analysis
NTU-60 reaches 94.1% Top-1 accuracy, but errors concentrate in a small set of confusing action pairs that skeletons alone cannot reliably distinguish. RGB fusion improves such cases, while Kinetics-based evaluation remains limited by pose extraction quality and weaker in-the-wild performance.
- NTU-60 performance: 94.1% Top-1 accuracy on NTU-60 X-Sub exceeds the current state-of-the-art result by 2.6%.Over 27% of failure cases belong to the five most confusing action pairs, which are difficult to distinguish using human skeletons alone.
- Error resolution: RGBPose-Conv3D substantially improves recognition on confusing action pairs by exploiting appearance cues such as visible keyboards.Typing and writing can be distinguished when the model recognizes the keyboard.
- Kinetics400 limitations: Kinetics400 skeleton accuracy remains low because videos often omit humans, show very small people, or contain only partial human regions that defeat pose estimators.These pose-extraction problems arise because Kinetics400 is not human-centric.
- Kinetics-Motion: Skeleton recognition on Kinetics-Motion rises from 47.7% to 81.9%, but remains below RGB recognition at 92.7% mean class accuracy.The subset contains 30 Kinetics classes strongly related to body motions, yet the authors identify in-the-wild skeleton recognition as requiring further work.