Source-linked AI summary
Semantic Graph Convolutional Networks for 3D Human Pose Regression
Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, Dimitris N. Metaxas
TL;DR
The paper addresses limitations of GCNs for graph regression, particularly shared node transformations and small receptive fields. It proposes SemGCN to learn local and global semantic relations and applies it to 2D-to-3D human pose regression, achieving state-of-the-art results with 90% fewer parameters.
Problem
GCN regression is limited by shared transformation matrices and small receptive fields, restricting how graph information is exchanged.
Method
SemGCN learns local and global semantic node relations for graph regression and uses them in 2D-to-3D human pose regression.
Results
90% fewer parameters achieved state-of-the-art performance on 3D human pose regression, with image features further improving performance.
Takeaways & Limitations
SemGCN captures graph semantics for pose regression and can incorporate image content within an end-to-end framework.
Takeaways & Limitations
The method fails on extreme poses that differ substantially from those in Human3.6M, although it still produces reasonable 3D poses.
Abstract
from arXiv · showhide
In this paper, we study the problem of learning Graph Convolutional Networks (GCNs) for regression. Current architectures of GCNs are limited to the small receptive field of convolution filters and shared transformation matrix for each node. To address these limitations, we propose Semantic Graph Convolutional Networks (SemGCN), a novel neural network architecture that operates on regression tasks with graph-structured data. SemGCN learns to capture semantic information such as local and global node relationships, which is not explicitly represented in the graph. These semantic relationships can be learned through end-to-end training from the ground truth without additional supervision or hand-crafted rules. We further investigate applying SemGCN to 3D human pose regression. Our formulation is intuitive and sufficient since both 2D and 3D human poses can be represented as a structured graph encoding the relationships between joints in the skeleton of a human body. We carry out comprehensive studies to validate our method. The results prove that SemGCN outperforms state of the art while using 90% fewer parameters.
1. Introduction
The paper targets graph regression, where baseline GCNs use shared node transformations and one-step receptive fields that limit information exchange. It proposes SemGCN to learn local and global semantic relations and applies it to 2D-to-3D human pose regression, matching state-of-the-art performance with 90% fewer parameters and improving results with image features.
- Motivation: GCNs are investigated for regression, extending their graph-based modeling beyond prior classification-oriented computer-vision applications.The paper identifies regression as another core computer-vision problem with real-world applications.
- Motivation: Baseline GCNs share one transformation matrix across nodes and restrict convolution filters to one-step neighborhoods, limiting deeper information exchange.The shared matrix handles varying neighborhood sizes, while the one-step receptive field can impair efficiency as networks deepen.
- Approach: SemGCN learns local and global node relations from graph structure without hand-crafted constraints, supporting generalization across tasks.Its semantic information is learned end-to-end rather than specified by application-specific rules.
- Approach: SemGConv learns channel-wise edge weights as graph-implied priors and combines them with kernel matrices to strengthen graph convolution.The operation is derived from CNNs and is a central architectural contribution.
- Approach: SemGCN interleaves SemGConv with non-local layers to capture both local and global relationships among nodes.The architecture is studied for 2D-to-3D human pose regression, where poses naturally form skeleton graphs.
- Results: 90% fewer parameters matched state-of-the-art performance on Human3.6M using only 2D joint coordinates, while image features further improved results beyond state of the art.The evaluation used comprehensive experiments, ablations, and comparisons on standard 3D benchmarks.
2. Related Work
Related work establishes GCNs as graph-processing models used mainly to represent relations for visual or temporal tasks, while this paper focuses on regression. Prior 3D pose methods include direct 2D-to-3D prediction, refinement, hand-crafted features, sparse pose bases, synthetic data, high-level losses, and temporal information, with temporal methods outside this paper’s single-image scope.
- Graph convolutional networks: GCN research follows spectral and spatial perspectives; this paper uses the spatial stream, applying filters directly to nodes and their neighbors.The related-work discussion presents GNNs as a general solution for arbitrary graph data.
- Graph convolutional networks: Recent computer-vision studies use GCNs to model relations among visual objects or temporal sequences, whereas this paper explores GCN-based regression.The most relevant application here is 2D-to-3D human pose regression.
- 3D pose estimation: 3D pose estimation methods have progressed from nearest-neighbor refinement and hand-crafted features to deep networks mapping 2D joint locations to 3D poses.Other approaches construct over-complete bases for representing human poses as sparse combinations.
- 3D pose estimation: Later work showed that accurate 3D pose estimation can use only 2D pose information, including direct prediction and pose grammar networks.These methods are identified as most relevant to the paper’s 2D-to-3D regression setting.
- 3D pose estimation: Other studies add synthetic data or high-level-knowledge losses, while separate methods exploit temporal information for 3D pose regression.Temporal approaches are outside this paper’s scope because it handles a 2D pose from one single image.
3. Semantic Graph Convolutional Networks
SemGCN extends graph convolutions by learning semantic, channel-wise edge weights while combining local graph aggregation with non-local global relations. Its design targets the shared-kernel and one-step receptive-field limitations of conventional GCNs.
- Limitations of conventional GCNs: Conventional GCNs share one transformation matrix across edges and aggregate only first-order neighbors, limiting structural modeling and receptive field.These constraints motivate a more expressive graph convolution for regression.
- Semantic Graph Convolution: SemGConv adds a learnable weighting matrix M to conventional graph convolution, assigning semantic weights to neighboring-node relationships.The adjacency matrix masks computation to graph neighbors, while Softmax normalizes their weights.
- Semantic Graph Convolution: Channel-wise weighting extends SemGConv by applying a distinct weighting matrix M_d to each output feature channel.This increases the operation’s capacity beyond a single shared edge-weighting pattern.
- Comparison with previous GCNs: The method learns input-independent edge priors from graph structure rather than input-dependent attention weights.This distinguishes SemGCN from self-attention approaches such as aGCN and GAT.
- Semantic Graph Convolutional Networks: SemGCN interleaves SemGConv and non-local layers to capture local and global semantic relationships among graph nodes.The example architecture uses residual blocks containing two 128-channel SemGConv layers followed by one non-local layer, repeated four times.
- Semantic Graph Convolutional Networks: The architecture alternates local message updates with global-state refinement, improving information exchange across the graph.This provides a neural message-passing interpretation of the alternating SemGConv and non-local phases.
4. 3D Human Pose Regression
The framework regresses 3D human joints from 2D poses, optionally incorporating aligned image features through a backbone and SemGCN. It combines joint and bone constraints for training, and image content further boosts performance.
- Image Features: Image content is added as an additional constraint because it can provide cues for ambiguous pose cases.The image contains the aligned human pose associated with the input 2D joints.
- Framework Overview: A convolutional backbone predicts 2D joints and supplies pooled intermediate image features, which SemGCN combines with the 2D pose to predict 3D coordinates.The framework pools hierarchical features from ResNet layers conv 1 through conv 4 using RoIAlign, then concatenates them with joint coordinates.
- Framework Overview: The complete framework is end-to-end trainable and reduces to the 2D-to-3D formulation when image features are omitted.SemGCN captures spatial patterns in the human-joint skeleton while predicting 3D coordinates.
- Loss Function: The loss combines errors on predicted joint positions and bone vectors.Bones are directed vectors computed from predicted joints and compared with corresponding ground-truth bones.
5. Experiments
The experiments evaluate SemGCN for 2D-to-3D human pose regression on Human3.6M using standardized protocols and two input configurations. Results show that combining local and global graph modeling improves training and accuracy, while image features and SemGCN deliver strong performance with fewer parameters.
- Datasets and Evaluation Protocols: The evaluation uses Human3.6M under Protocol #1, with MPJPE in millimeters after pelvis alignment as the metric.Protocol #1 trains on subjects S1, S5, S6, S7, and S8 and tests on S9 and S11.
- Configurations: Configuration #1 predicts 3D joints from 2D joint coordinates using SemGCN with ground-truth or HourGlass-detected 2D inputs.Configuration #2 additionally uses image features through a ResNet50 backbone and feature pooling.
- Ablation Study: The full SemGCN model outperforms variants with fewer components, while SemGConv produces smoother training curves that indicate more stable learning of local node relations.The ablation compares SemGConv-only, non-local-only, baseline, and full-model variants using ground-truth 2D joints.
- 2D to 3D Pose Regression: 20%: SemGCN advances other GCN-based approaches and achieves state-of-the-art performance with 90% fewer parameters than [34].Its runtime is also reported as 10% lower than [34], at around 1.8ms per forward pass on a Titan Xp GPU.
- Comparison with the State of the Art: Using only 2D joints, SemGCN matches state-of-the-art performance; incorporating image features sets a new state of the art on Human3.6M.The authors report especially large improvements on several actions with severe self-occlusions.
- Qualitative Results: SemGCN predicts accurate 3D poses for Human3.6M and most in-the-wild MPII images, but fails on extreme poses unlike those in Human3.6M.Even in these failure cases, the method still produces reasonable 3D poses.
6. Conclusions
The paper concludes that SemGCN addresses key GCN limitations by learning local and global semantic relations for 3D human pose regression. Combining SemGCN with pooled image features further improves performance, while temporal inputs remain a future direction.
- Conclusions: SemGCN learns local and global semantic relations among graph nodes to address key challenges in graph convolutional networks.The model is presented as a neural architecture for 3D human pose regression.
- Conclusions: 90% fewer parameters: the network reaches state-of-the-art performance compared with the closest work.The conclusion attributes the result to comprehensive evaluation of the proposed network.
- Future Work: Temporal information such as video sequences is identified as a natural direction for extending SemGCN.The paper leaves incorporating temporal information to future work.
Appendix A. Supplementary Material
The supplementary material adds technical details and further results supporting SemGCN. It covers graph construction, graph convolutions, non-local layers, and visualizations of learned weights.
- Supplementary Material: The supplement details the skeleton representation used to build the graph and the implementations of graph convolutions and non-local layers.It also provides visualizations of learned weights associated with Semantic Graph Convolutions.
A.1. Skeleton Representation
SemGCN builds on a common 16-joint human skeleton with the pelvis as root, initially modeled as an undirected graph and transformed into a weighted directed graph after training.
- The graph uses a common 16-joint skeleton representation for Human3.6M and MPII, with the pelvis defined as the root joint.
- The skeleton is initialized as an undirected graph before training and becomes a weighted directed graph through learned edge weights.
- Bone vectors are directed from each joint’s parent to the target joint for computing the bone loss.
- The bone loss is omitted in Configuration #1 to match prior experimental settings for fair comparison.
A.2. Implementation of Graph Convolutions
The implementation compares a baseline graph convolution using separate transformations for each node and its neighbors with the proposed SemGConv operation.
- The baseline applies one transformation matrix to a node and another shared matrix to all of its neighbors.
- The implementation also applies the proposed SemGConv operation in the same general manner as the baseline graph convolution.
A.3. Non-local Layers
SemGCN’s non-local layers reduce feature and node dimensions for efficiency, using concatenated embeddings and max pooling to capture broader graph relationships.
- The non-local layers down-sample both feature dimensions and graph nodes when computing node embeddings for efficiency.
- Feature embeddings use concatenation, with 1 × 1 convolutions reducing each node’s features from 128 to 64 channels.
- A learned parameter projects the concatenated feature vector to a scalar used by the embedding function.
- Max pooling reduces the graph from 16 to 8 nodes in every non-local layer.
- The pooling operation speeds runtime without affecting final regression accuracy in the experiments.
A.4. Visualization of Weights in SemGCN
Visualization shows that SemGConv layers learn distinct weighting matrices and assign larger weights to joints farther from the skeleton’s center, especially distal joints.
- The eight SemGConv layers learn different weighting matrices despite sharing the same graph structure.
- Learned weights are generally higher for joints farther from the human skeleton’s gravity center.
- The wrists, ankles, and head have average learned weights above 0.4, while the neck, thorax, and pelvis have weights below 0.3.
- The average joint weights are visualized as a regional skeleton map grouping joints into three weight-based regions.
- The learned spatial relationships use only ground-truth supervision, without additional hand-crafted constraints or rules.