Source-linked AI summary
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose
Hongsuk Choi, Gyeongsik Moon, Kyoung Mu Lee
TL;DR
Existing image-based methods face an appearance gap between controlled training data and in-the-wild data, while 3D rotation parameters are difficult regression targets. Pose2Mesh instead uses 2D poses and a model-free GraphCNN to directly estimate mesh coordinates while exploiting mesh topology. It outperforms previous 3D pose and mesh estimation methods across benchmark datasets.
Problem
3D pose and mesh methods face an appearance domain gap and representation issues when regressing human-mesh pose parameters.
Method
Pose2Mesh uses a model-free GraphCNN to directly regress 3D mesh coordinates from 2D human poses while exploiting mesh topology.
Results
Pose2Mesh outperforms previous 3D human pose and mesh estimation methods on various publicly available body and hand datasets.
Takeaways & Limitations
The 2D-pose input lets Pose2Mesh benefit from controlled-setting 3D data without the appearance domain gap, while GraphCNN avoids rotation-parameter representation issues.
Takeaways & Limitations
Recovering diverse 3D shapes solely from pose remains challenging, although the authors report that 2D poses contain necessary shape information.
Abstract
from arXiv · showhide
Most of the recent deep learning-based 3D human pose and mesh estimation methods regress the pose and shape parameters of human mesh models, such as SMPL and MANO, from an input image. The first weakness of these methods is an appearance domain gap problem, due to different image appearance between train data from controlled environments, such as a laboratory, and test data from in-the-wild environments. The second weakness is that the estimation of the pose parameters is quite challenging owing to the representation issues of 3D rotations. To overcome the above weaknesses, we propose Pose2Mesh, a novel graph convolutional neural network (GraphCNN)-based system that estimates the 3D coordinates of human mesh vertices directly from the 2D human pose. The 2D human pose as input provides essential human body articulation information, while having a relatively homogeneous geometric property between the two domains. Also, the proposed system avoids the representation issues, while fully exploiting the mesh topology using a GraphCNN in a coarse-to-fine manner. We show that our Pose2Mesh outperforms the previous 3D human pose and mesh estimation methods on various benchmark datasets. For the codes, see https://github.com/hongsukchoi/Pose2Mesh_RELEASE.
1 Introduction
Pose2Mesh addresses appearance-domain and 3D-rotation representation challenges by recovering mesh vertices from 2D poses with a topology-aware GraphCNN. It reports stronger performance than previous methods across public body and hand datasets, including an in-the-wild dataset trained only on controlled data.
- Motivation: 3D human pose and mesh estimation must recover joint and mesh-vertex locations despite depth, scale, and articulation ambiguities.
- Motivation: Image-based methods face an appearance domain gap between controlled training data and in-the-wild test data, while mesh-model pose parameters create representation difficulties.
- Pose2Mesh: Pose2Mesh recovers 3D pose and mesh from 2D pose input in a model-free fashion, using relatively homogeneous geometric information across domains.
- Pose2Mesh: GraphCNN directly regresses mesh-vertex coordinates while exploiting predefined mesh topology and avoiding pose-parameter representation issues.
- Architecture: PoseNet lifts 2D pose to 3D pose, then MeshNet combines 2D and 3D poses to estimate the mesh coarse-to-fine.
- Results: Pose2Mesh outperforms previous methods across public 3D body and hand datasets, achieving a state-of-the-art result on an in-the-wild dataset when trained only on controlled data.
3 PoseNet
PoseNet is designed to lift normalized 2D human poses into root-relative 3D poses while improving robustness to errors in estimated 2D inputs. It uses error synthesis during training and predicts a 3J-dimensional pose representation.
- PoseNet: PoseNet estimates the root-joint-relative 3D pose P3D from an estimated or synthesized 2D pose P2D.
- Input errors: Training synthesizes realistic errors on ground-truth 2D poses to improve robustness to inaccurate inputs, especially under occlusion or challenging poses.
- Input normalization: Standard normalization subtracts the 2D-pose mean and divides by its standard deviation, removing location and scale effects before prediction.
- Network architecture: The normalized 2D pose is mapped through a fully connected layer and two residual blocks to a 3J-dimensional vector representing P3D.
- Training objective: PoseNet training minimizes the L1 distance between predicted and ground-truth 3D poses.
4 MeshNet
MeshNet estimates a root-relative 3D mesh from combined 2D and 3D pose features using spectral graph convolutions over progressively coarsened and upsampled mesh graphs. Its training combines coordinate, joint, surface-normal, and edge-length losses.
- MeshNet architecture: MeshNet concatenates the 2D and 3D pose representations, then estimates root-relative 3D mesh vertices with spectral graph convolution.The input feature map has five channels per joint in practice.
- MeshNet architecture: The graph convolution uses Chebyshev polynomials, making the operation K-localized so nodes within at most K hops can affect each other.The polynomial formulation is designed to reduce computational complexity.
- MeshNet architecture: The mesh graph is coarsened into multiple resolutions, forming a balanced binary-tree structure in which each coarse vertex represents two finer vertices.This hierarchy supports coarse-to-fine processing across mesh resolutions.
- MeshNet architecture: MeshNet first maps pose features to the coarsest mesh graph, applies graph convolutions, and progressively upsamples features to finer mesh graphs.Nearest-neighbor upsampling copies parent-vertex features to corresponding child vertices, with residual connections between resolutions.
- Loss functions: Training minimizes four losses: vertex coordinates, joint coordinates, surface normals, and edge lengths.Surface-normal consistency improves surface smoothness and local details, while edge-length consistency is effective for dense regions such as hands, feet, and the mouth.
- Loss functions: The total mesh loss weights vertex and joint losses by 1, surface-normal loss by 0.1, and edge loss by 20.These coefficients define the weighted combination of the four loss terms.
5 Implementation Details
The authors pre-train PoseNet before training the complete Pose2Mesh network end-to-end, using RMSprop optimization with mini-batches of 64.
- Training strategy: PoseNet is pre-trained for 60 epochs before the whole Pose2Mesh network is trained end-to-end.The authors report that this two-step strategy performs better empirically than one-step training.
- Optimization: Training uses RMSprop with a mini-batch size of 64 and an initial learning rate of 10^-3 for PoseNet.The learning rate is reduced by a factor of 10 after the 30th epoch.
6 Experiment
Experiments evaluate Pose2Mesh across controlled and in-the-wild body and hand benchmarks, ablations, training-set variants, and qualitative outputs. Pose2Mesh generally outperforms prior methods, while GraphCNN regression and cascaded 2D–3D processing are supported by component analyses.
- Datasets and evaluation metrics: Human3.6M uses motion-capture 3D poses and newly generated pseudo-groundtruth meshes, with five training subjects and two test subjects.The evaluation uses subjects S1, S5, S6, S7, and S8 for training and S9 and S11 for testing.
- Datasets and evaluation metrics: MPJPE measures root-aligned Euclidean joint error in millimeters, while PA-MPJPE additionally applies Procrustes alignment.Only 14 of the 17 estimated joints are evaluated, following prior work.
- Ablation study: Vertex-GraphCNN substantially improves joint errors over vertex-FC and outperforms SMPL-parameter networks with considerably fewer parameters.This ablation supports exploiting mesh topology while directly regressing mesh vertex coordinates.
- Ablation study: The coarse-to-fine upsampling strategy is supported by lower joint error in the comparison of two upsampling schemes.Table 2 also compares training GPU memory and inference fps.
- Ablation study: Concatenating 2D and predicted 3D poses gives the lowest errors, indicating that retained 2D geometry and 3D depth jointly benefit mesh estimation.A direct 2D-pose GraphCNN outperforms a cascaded network using only pretrained 3D-pose output.
- Comparison with state-of-the-art methods: Pose2Mesh outperforms prior methods across body and hand benchmarks, including state-of-the-art performance on in-the-wild 3DPW when trained only on controlled data.It remains strongest on 3DPW across different training-set comparisons and produces visually decent meshes without model-fitting post-processing.
7 Discussion
Recovering varied 3D shapes solely from 2D pose remains challenging, although the pose contains enough information to reason about corresponding shape.
- Recovering various 3D shapes solely from 2D pose could be challenging.
- The authors found that 2D pose still carries necessary information for reasoning about corresponding 3D shape.
- Supplementary experiments show that Pose2Mesh can recover various body shapes from 2D pose.
8 Conclusion
Pose2Mesh estimates human pose and mesh from 2D pose, using controlled-setting 3D data without the appearance-domain-gap issue. Its GraphCNN design exploits mesh topology while avoiding 3D-rotation representation issues.
- Pose2Mesh is a general system for 3D human mesh and pose estimation from a 2D human pose.
- Its 2D-pose input enables use of controlled-setting 3D data without the appearance domain gap issue.
- The model-free GraphCNN approach exploits mesh topology while avoiding representation issues of 3D rotation parameters.
3D Human Pose and Mesh Recovery from a 2D Human Pose”
The supplementary material reports additional experimental results omitted from the main manuscript because of limited space.
- The supplementary material presents additional experimental results.
- These results were not included in the main manuscript because of limited space.
- The supplementary material extends the experiments beyond those included in the main manuscript.
9 Qualitative results
Qualitative results show that Pose2Mesh can recover body-shape features from 2D pose and produces visually improved meshes over GraphCMR, while failures remain on overweight appearance and some arm poses.
- 9.1 Shape recovery: Pose2Mesh recovers body-shape features such as bone-length ratios and fatness from input joint locations.Experiments on SURREAL show these features are expressed in the predicted outputs.
- 9.2 Additional results: Some qualitative failures predict overweight people closer to the average shape.
- 9.2 Additional results: A right arm may be predicted bent when it appears straight in the input.
- 9.3 Comparison with the state-of-the-art: Pose2Mesh provides much more visually pleasant mesh results than GraphCMR in qualitative comparison.
10 Details of PoseNet
PoseNet converts normalized 2D human poses into lifted 3D poses, using residual fully connected blocks and dataset-specific joint representations as part of Pose2Mesh.
- PoseNet architecture: PoseNet converts a normalized 2D pose into a 4096-dimensional feature through a fully connected layer before residual processing.Each residual block uses a fully connected layer, 1D batch normalization, ReLU activation, and dropout with probability 0.5.
- Evaluation: PoseNet is evaluated with MPJPE and PA-MPJPE across Human3.6M, 3DPW, and FreiHAND benchmarks using benchmark-specific joint protocols.The evaluated joints and test pose sources differ across datasets.
- Dataset-specific graphs: Pose2Mesh uses different predefined joint sets and graph structures for Human3.6M, 3DPW, SURREAL, and FreiHAND.These correspond to Human3.6M, COCO, SMPL, and MANO joints, respectively; COCO additionally includes pelvis and neck joints.
12 Pseudo-groundtruth SMPL parameters of Human3.6M
Because Human3.6M lacks publicly accessible mesh annotations from Mosh, the study constructs alternative SMPL-based supervision and examines synthetic-data and input-pose choices.
- Supervision construction: SMPLify-X is applied to Human3.6M ground-truth 3D joint coordinates to obtain alternative SMPL parameters for mesh supervision.The fitted parameters are not perfectly aligned with the joint coordinates, but their fitting error is reported as lower than errors from current state-of-the-art pose estimators.
- Supervision limitation: Human3.6M training tends to produce female-shaped meshes because fitting yields subject-specific body shapes and most training subjects are female.Fixing the SMPL identity code increased errors, so it was not fixed for each subject.
- Additional data: AMASS motion-capture data is converted into paired 2D-pose and 3D-mesh training data by projecting mesh-derived 3D poses with Human3.6M camera parameters.Adding AMASS decreases both joint error and surface error on 3DPW.
- Input-pose synthesis: Training synthesizes input 2D poses by adding realistic errors to ground-truth poses, while FreiHAND uses detector outputs because verified synthetic errors are unavailable.The study compares training with ground-truth versus synthesized input poses using MPJPE and PA-MPJPE.
15 Train/test with groundtruth input poses
Ground-truth-pose experiments compare Pose2Mesh, PoseNet, and MeshNet upper bounds and separately assess how losses affect numerical accuracy and mesh appearance.
- Ground-truth input upper bounds: Pose2Mesh and PoseNet receive ground-truth 2D poses, whereas MeshNet receives ground-truth 3D poses in the upper-bound experiments.These upper bounds are reported on Human3.6M and 3DPW.
- Ground-truth input upper bounds: Pose2Mesh has an upper bound similar to PoseNet, indicating that its 3D pose errors follow those of PoseNet.Pose2Mesh and MeshNet are evaluated on 3D poses regressed from mesh outputs, while PoseNet is evaluated on its lifted 3D pose.
- Loss ablation: Removing the joint-coordinate loss has a relatively distinctive effect on MPJPE and PA-MPJPE, while other loss combinations show numerically negligible differences.The comparison uses Human3.6M for training and testing.
- Loss ablation: Removing normal or edge losses visibly degrades mesh quality, whereas removing the joint loss has nearly no visual effect.Without normal loss, smoothness and local facial, hand, and foot details deteriorate; without edge loss, dense-vertex regions develop flying-vertex artifacts.