Source-linked AI summary
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, Zhenan Sun
TL;DR
Regression-based human mesh recovery can produce coarse mesh-image alignment because small parametric errors cause noticeable projection drift and standard feedback may reuse uninformative global features. PyMAF progressively samples mesh-aligned evidence from multi-scale spatial features to rectify parameters, with experiments showing mesh-aligned features perform best and the method remains limited by depth ambiguity and extreme shapes.
Problem
Regression-based mesh recovery remains vulnerable to coarse mesh-image alignment because small pose errors can accumulate into noticeable projection drift, while IEF reuses global features during feedback.
Method
PyMAF progressively extracts mesh-aligned evidence from spatial feature maps according to the estimated mesh projection and feeds it back to update parameters.
Results
Mesh-aligned features yield the best performance in both non-pyramidal and pyramidal designs, with pyramidal maps providing additional performance gains.
Takeaways & Limitations
Explicit mesh-alignment feedback provides the regressor with informative evidence for rectifying current mesh parameters and improving image-plane alignment.
Takeaways & Limitations
PyMAF improves alignment on 2D image planes but does not resolve depth ambiguity in 3D space and fails on extreme shapes because training data are insufficient.
Abstract
from arXiv · showhide
Regression-based methods have recently shown promising results in reconstructing human meshes from monocular images. By directly mapping raw pixels to model parameters, these methods can produce parametric models in a feed-forward manner via neural networks. However, minor deviation in parameters may lead to noticeable misalignment between the estimated meshes and image evidences. To address this issue, we propose a Pyramidal Mesh Alignment Feedback (PyMAF) loop to leverage a feature pyramid and rectify the predicted parameters explicitly based on the mesh-image alignment status in our deep regressor. In PyMAF, given the currently predicted parameters, mesh-aligned evidences will be extracted from finer-resolution features accordingly and fed back for parameter rectification. To reduce noise and enhance the reliability of these evidences, an auxiliary pixel-wise supervision is imposed on the feature encoder, which provides mesh-image correspondence guidance for our network to preserve the most related information in spatial features. The efficacy of our approach is validated on several benchmarks, including Human3.6M, 3DPW, LSP, and COCO, where experimental results show that our approach consistently improves the mesh-image alignment of the reconstruction. The project page with code and video results can be found at https://hongwenzhang.github.io/pymaf.
1. Introduction
Regression-based human mesh recovery is efficient but can produce coarse mesh-image alignment because small parameter errors create visible projection errors. PyMAF addresses this with a mesh-alignment feedback loop, feature pyramids, and auxiliary pixel-wise supervision.
- Regression-based methods directly predict model parameters from images but can suffer coarse alignment between estimated meshes and image evidences.
- Relative rotations along SMPL kinematic chains can accumulate minor errors into noticeable joint-position drifts and projection misalignment.
- Iterative Error Feedback reuses global features, limiting regressors’ ability to perceive mesh-image misalignment during inference.
- PyMAF extracts mesh-aligned evidences from spatial features using the estimated mesh projection and feeds them back to correct parameters progressively.
- A feature pyramid supplies multi-scale alignment contexts, while auxiliary pixel-wise supervision makes spatial features more informative and evidences more reliable.
2. Related Work
Related work spans optimization-based fitting, direct regression, non-parametric reconstruction, and iterative refinement strategies. PyMAF contributes a closed-loop regressor that uses spatial evidences during both training and inference.
- Optimization-based approaches fit parametric models to 2D observations using priors and data terms that measure mesh reprojection errors.
- Regression-based approaches predict human models directly from monocular images, using neural networks, varied supervision signals, and architectural designs.
- Unlike open-loop or global-feature feedback regressors, PyMAF operates as a closed loop during training and inference and leverages spatial evidences for alignment.
- Non-parametric methods regress volumetric, implicit, vertex-based, or position-map representations to facilitate high-resolution reconstruction.
- Prior work also combines regression with iterative fitting or mesh deformation to refine intermediate human-mesh estimates.
3. Methodology
PyMAF combines coarse-to-fine spatial features with mesh-conditioned feedback to progressively rectify parameters according to mesh-image alignment. Auxiliary pixel-wise supervision preserves correspondence-relevant spatial cues for more reliable mesh-aligned evidence.
- Feature Pyramid: The image encoder produces a feature pyramid from coarse to fine granularities for predicting SMPL pose, shape, and camera parameters.The pyramid describes the posed person at different scale levels and supports subsequent SMPL predictions.
- Feature Pyramid: At each level, sampled spatial features are processed by an MLP and concatenated before entering the parameter regressor.Feature sampling uses bilinear interpolation, followed by dimension reduction and concatenation.
- Parameter Regression: The regressor updates parameters from the current estimate and extracted features, beginning from mean training parameters at level t = 0.At later levels, the current parameter estimate is refined by adding the predicted residual.
- Parameter Regression: Training combines 2D keypoint, 3D joint, and parameter supervision to penalize discrepancies between estimated outputs and ground truth.The regression loss weights squared L2 terms for projected keypoints, 3D joints, and model parameters.
- Mesh Alignment Feedback: Mesh-aligned sampling points are obtained by down-sampling the current mesh and projecting it into the image, providing position-sensitive evidence for parameter updates.Unlike global or uniformly sampled features, these evidences reflect the current mesh-image alignment status.
- Auxiliary Pixel-wise Supervision: An auxiliary pixel-wise task predicts dense correspondence maps from final spatial features, guiding the encoder to preserve mesh-image-relevant information.The maps use part-index classification and UV-channel regression; auxiliary prediction is required only during training.
4. Experiments
Experiments evaluate PyMAF on 3D reconstruction, 2D alignment, and ablations, showing improvements over a matched baseline and benefits from mesh-aligned feedback, auxiliary supervision, and progressive refinement.
- Implementation Details: The ResNet-50 implementation uses spatial feature maps at 14 × 14, 28 × 28, and 56 × 56 resolutions for pyramidal feedback.The backbone receives 224 × 224 images and initially produces 2048 × 7 × 7 features.
- Datasets: Training mixes Human3.6M, MPI-INF-3DHP, LSP, LSP-Extended, MPII, and COCO, while 3DPW is reserved for test evaluation.Pseudo-ground-truth SMPL parameters are used for the last five training datasets.
- 3D Human Pose and Shape Estimation: PyMAF reduces MPJPE by 5.7 mm on 3DPW and 7.1 mm on Human3.6M versus a baseline trained under the same setting.The baseline uses the HMR architecture with 6D rotation representation; the authors caution that comparisons with prior methods are not strictly comparable.
- 2D Segmentation and Pose Estimation: On COCO, PyMAF improves AP by 7.8% and AP50 by 10.7% over the baseline, while auxiliary supervision adds a 3.9% AP gain.Qualitative results are reported as better-aligned and natural than SPIN and the baseline.
- Ablation Study: Mesh-aligned features perform best in both non-pyramidal and pyramidal feedback designs, with greater gains when multi-scale alignment evidence is available.Uniform grid features outperform global features but cannot reflect the current estimation's alignment status.
- Ablation Study: Feature maps outperform intermediate auxiliary predictions for point-wise extraction, while dense correspondence supervision preserves richer foreground information than part segmentation.Using part-segmentation masking degrades MPJPE from 75.5 mm to 77.6 mm.
- Ablation Study: PyMAF improves reconstruction from both grid-point and mean-pose initialization, progressively correcting body-part drift even when the initial mean-pose mesh is far from ground truth.This supports robustness to the feedback loop's initialization choice.
5. Limitations and Future Work
PyMAF improves mesh-image alignment on 2D image planes but remains limited by depth ambiguity and insufficient training data for extreme shapes. Future work targets 3D alignment and more precise pseudo 3D ground-truth labels.
- PyMAF can hardly address depth ambiguity in 3D space despite improving mesh-image alignment on 2D image planes.
- PyMAF fails to handle extreme shapes because of insufficient training data.
- Future work includes extending PyMAF with methods that improve alignment in 3D space.
- Another proposed direction combines PyMAF with HoloPose, SPIN, or EFT to generate more precise pseudo 3D ground-truth labels.
A.1. More Experimental Details
The implementation trains PyMAF with Adam on staged mixtures of 3D and 2D datasets, using fully connected residual regressors and a feedback loop with modest runtime overhead.
- Training uses Adam with batch size 64 and learning rate 5e−5 without learning rate decay.
- The network trains for 60 epochs on Human3.6M, followed by 60 epochs on a mixture of 2D and 3D datasets.
- Each PyMAF parameter regressor has two 1024-neuron fully connected layers, dropout, and a 157-dimensional residual output.
- PyMAF processes one sample in about 30 ms on a single 2080 Ti GPU, with each feedback iteration taking about 6 ms.
A.2. Datasets
PyMAF is trained on multiple 2D and 3D datasets with real or pseudo annotations, while 3DPW and several benchmarks provide distinct evaluation settings. Figure 6 contrasts alignment with PA-MPJPE.
- Training uses Human3.6M, MPI-INF-3DHP, LSP, LSP-Extended, MPII, and COCO, including pseudo-ground-truth SMPL parameters for the last five datasets.
- 3DPW contains challenging outdoor scenes with accurate shape and pose ground truth, but is used only for test-set evaluation.
- Human3.6M contains 3.6 million controlled-environment video frames, with five subjects used for training and two for evaluation.
- MPI-INF-3DHP covers more actor subjects and poses than Human3.6M across indoor and outdoor scenes, with some noise in its 3D annotations.
- LSP-Extended contains challenging-pose images with 14 visible keypoints and 9,428 training samples.
- LSP contributes 1,000 test samples for silhouette and parts segmentation evaluation.
- MPII provides 25,000 YouTube images spanning activities, reduced to 14,810 training samples after excluding incomplete annotations.
- COCO provides 28,344 training samples and a 50,197-sample validation set for 2D keypoint localization, using persons with at least 12 keypoints.
A.3. Evaluation Metrics
The evaluation uses 3D reconstruction errors, segmentation measures, and COCO keypoint AP to assess pose, shape, and mesh-image alignment. PA-MPJPE alone may not reveal reprojection alignment.
- PVE, MPJPE, and PA-MPJPE evaluate 3D reconstruction and pose estimation on 3DPW and Human3.6M, reported in millimeters.
- PVE is mean per-vertex Euclidean error, while MPJPE measures mean per-joint position error.
- PA-MPJPE is MPJPE after rigid Procrustes alignment and cannot fully reveal mesh-image alignment.
- LSP segmentation evaluation measures silhouette foreground/background and part segmentation using accuracy and F1 scores.
- COCO keypoint localization uses mean AP and AP variants based on Object Keypoint Similarity, including AP50, AP75, APM, and APL.
A.4. More Experimental Results
Additional experiments show that PyMAF progressively improves mesh-image alignment and converges faster than SPIN in qualitative comparisons. Its corrections remain limited when initial reconstructions contain severe deviations, heavy occlusions, or ambiguous limb connections.
- Additional datasets: On 3DOH50K and SSP-3D, the paper evaluates reconstruction performance on occluded images and varied body shapes without using those datasets for training.The 3DOH50K test set contains 1,290 images, while SSP-3D contains 311 images.
- Qualitative comparisons: PyMAF converges faster than SPIN and corrects mesh parameters more effectively during iterative reconstruction.The comparison contrasts SPIN’s global-feature updates with PyMAF’s spatial-feature updates.
- Qualitative comparisons: Across indoor and in-the-wild examples, PyMAF produces natural reconstructions with better image alignment than SPIN under challenging cases.Examples include COCO, 3DPW, and Human3.6M.
- Iterative refinement: On COCO and 3DPW examples, PyMAF progressively improves mesh-image alignment by correcting predictions from current observations.Figures 11 and 12 show the results after each iteration.
- Failure cases: PyMAF may fail to correct body parts when initial reconstructions have severe deviations caused by heavy occlusions or ambiguous limb connections.These failure cases are illustrated in Figure 13.