Source-linked AI summary
Graph Stacked Hourglass Networks for 3D Human Pose Estimation
Tianhan Xu, Wataru Takano
TL;DR
2D-to-3D human pose estimation with graph convolutions is limited by oversimple single-scale and single-level representations, while graph irregularity complicates image-style scaling. The paper proposes Graph Stacked Hourglass Networks with skeletal pooling and unpooling across three scales and intermediate features across depths; it reports accurate estimation that outperforms the state of the art. The architecture remains bounded by the challenges and scope of graph-based skeletal representations, and future work targets temporal multi-frame features.
Problem
Existing GCN architectures use oversimple representations and do not adequately exploit multi-scale and multi-level features for 2D-to-3D human pose estimation.
Method
Graph Stacked Hourglass Networks use repeated graph encoder-decoder modules, Skeletal Pool and Skeletal Unpool, and intermediate features across depths.
Results
The proposed architecture achieves accurate 2D-to-3D human pose estimation and outperforms the state of the art.
Takeaways & Limitations
Multi-scale and multi-level feature extraction provides the paper’s supported basis for more expressive graph representations in 2D-to-3D pose estimation.
Takeaways & Limitations
Image-style multi-scale and multi-level methods remain difficult to apply to graph-structured data because graph structures are irregular.
Abstract
from arXiv · showhide
In this paper, we propose a novel graph convolutional network architecture, Graph Stacked Hourglass Networks, for 2D-to-3D human pose estimation tasks. The proposed architecture consists of repeated encoder-decoder, in which graph-structured features are processed across three different scales of human skeletal representations. This multi-scale architecture enables the model to learn both local and global feature representations, which are critical for 3D human pose estimation. We also introduce a multi-level feature learning approach using different-depth intermediate features and show the performance improvements that result from exploiting multi-scale, multi-level feature representations. Extensive experiments are conducted to validate our approach, and the results show that our model outperforms the state-of-the-art.
1. Introduction
The paper identifies limited multi-scale and multi-level representation in existing GCN architectures for 2D-to-3D human pose estimation. It proposes Graph Stacked Hourglass Networks with skeletal pooling, unpooling, and intermediate features to improve graph representation.
- Existing GCN approaches process skeletal features at one scale, limiting representation of spatially local and global information.
- Sequential graph-convolution architectures discard intermediate features at different depths, limiting the benefits of model depth.
- Graph Stacked Hourglass Networks use repeated encoder-decoder processing for multi-scale features and intermediate features for multi-level extraction.
- Graph Hourglass modules introduce Skeletal Pool and Skeletal Unpool to extract multi-scale human skeletal features while respecting skeletal structure.
- GraphSH incorporates multi-scale and multi-level representations with prior knowledge of the human skeleton for 2D-to-3D pose estimation.
2. Related Work
Prior work applies deep learning, CNNs, and GCNs to 3D human pose estimation, including two-stage methods that regress 3D pose from detected 2D joints. The paper addresses the limited use of multi-scale and multi-level features in graph-based architectures by introducing graph hourglass processing.
- 3D human pose estimation has progressed from handcrafted geometric methods toward deep neural networks for image-to-3D prediction.
- Two-stage approaches first predict 2D joints from images and then regress the 3D human pose from those joints.
- GCNs exploit human-skeleton topology for 2D-to-3D pose estimation, with variants learning semantic relationships or using node-specific transformations.
- Existing graph-based pose architectures use straightforward sequential designs that do not exploit multi-scale and multi-level features.
- Image-based methods obtain multi-scale features by resizing images and multi-level features from intermediate depths, but graph irregularity makes analogous scaling nontrivial.
- The proposed architecture integrates multi-scale and multi-level features into graph-structured data.
3. Graph Stacked Hourglass Networks
Graph Stacked Hourglass Networks extend stacked hourglass processing to graph-structured human skeletons, extracting multi-scale features through skeletal pooling and unpooling. The architecture also integrates intermediate features across depths before SE-based channel weighting and final 3D pose prediction.
- Graph Hourglass Module: The architecture extends repeated encoder-decoder hourglass processing to graph-structured human skeletal data for multi-scale feature extraction.The graph hourglass structure is designed to process information across three skeletal scales.
- Multi-scale and Multi-level Features: The overall network maintains the graph structure throughout, feeds intermediate outputs into later hourglass modules, and uses a 1x1 convolution for final prediction.The last hourglass output is not passed to a subsequent hourglass module.
- Graph Hourglass Module: Skeletal pooling fuses paired body-node features into lower-scale skeletons with 16, 8, and 4 nodes, while skeletal unpooling restores higher-scale representations.Unpooling duplicates each lower-scale node feature onto its corresponding pair of higher-scale nodes.
- Graph Hourglass Module: Residual connections preserve information within each skeletal scale and help prevent vanishing gradients, while the graph hourglass remains compatible with arbitrary graph convolution operations.The design does not depend on a specific graph convolution layer.
- Graph Hourglass Module: Unlike Graph U-Nets, the method uses fixed human-skeleton pooling based on prior structure and increases channels at lower scales to reduce information loss.These choices distinguish the architecture from input-dependent pooling and uniform channel allocation across scales.
- Multi-scale and Multi-level Features: Intermediate features from different depths are transformed and concatenated, then an SE block learns channel-wise weights before the output layer predicts the final 3D pose.The network stacks four hourglass modules; its experiments use 64 latent channels and PreAggr graph convolution.
4. Experiments
The experiments evaluate Graph Stacked Hourglass Networks for 2D-to-3D human pose estimation through ablations, architectural comparisons, and state-of-the-art benchmarks. Results support the value of skeletal multi-scale processing and show improved performance across graph convolution choices and datasets.
- Experimental setup: The experiments predict 3D joint positions from 2D joint positions using Human3.6M and evaluate with MPJPE under Protocol #1.MPI-INF-3DHP evaluation additionally uses 3D-PCK and AUC.
- Feature ablations: Ablations show that skeletal pooling and unpooling improve performance over processing features only at the highest scale.The multi-level-feature study also compares removing multi-level features and removing the SE block.
- Pool/unpool ablation: Skeletal Pool and Skeletal Unpool exploit fixed human-skeleton topology, preserve information between paired nodes, and reduce pooling-layer computational complexity.Skeletal Pool uses maximum pooling between paired nodes, while Skeletal Unpool copies lower-scale features to corresponding higher-scale nodes.
- Architecture comparison: GraphSH outperforms Sequential Residual architectures even with fewer channels, and this improvement holds across Vanilla, Semantic, and Pre-aggregation Graph Convolution layers.The comparison controls parameter scale by using 64 channels in the proposed architecture.
- State-of-the-art comparison: On Human3.6M with CPN detections and ground-truth 2D keypoints, the approach outperforms state-of-the-art methods; with precise 2D inputs, it exceeds other GCN methods by a large margin.The model uses 3.70M parameters versus 4.38M for the second-place method cited in the comparison.
- State-of-the-art comparison: A model trained only on Human3.6M outperforms other methods on the MPI-INF-3DHP test set, indicating strong generalization to an unseen dataset.This evaluates transfer across datasets without training on MPI-INF-3DHP.
5. Conclusions
Graph Stacked Hourglass Networks uses skeletal pooling and unpooling within a graph architecture to extract multi-scale and multi-level features for 2D-to-3D pose estimation, achieving accurate results that outperform the state-of-the-art.
- GraphSH combines skeletal pooling and unpooling with graph hourglass architecture for multi-scale and multi-level feature extraction.The method processes graph-structured human skeleton data across different scales and architectural depths.
- The method achieves accurate 2D-to-3D human pose estimation while outperforming the state-of-the-art.
- Future work will introduce temporal multi-frame features to further improve the architecture.