PaperlayerView exact revision on arXiv

Source-linked AI summary

PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images

Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, Yebin Liu

arXiv:2207.06400v4cs.CV

TL;DR

Monocular full-body mesh recovery is difficult because small parameter errors misalign meshes and part integration can produce unnatural wrists. PyMAF uses mesh-alignment feedback with multi-scale evidence, while PyMAF-X adds adaptive full-body integration; the paper reports improved alignment and new state-of-the-art results across body, hand, face, and full-body benchmarks.

  • Problem

    Regression-based monocular human mesh recovery often yields coarse mesh-image alignment, while integrating part-specific estimates can degrade alignment or create unnatural wrist poses.

  • Method

    PyMAF progressively feeds mesh-aligned evidence from a feature pyramid back to regressors for parameter rectification, and PyMAF-X adds adaptive integration with elbow-twist compensation.

  • Results

    PyMAF and PyMAF-X improve mesh-image alignment and achieve new state-of-the-art results across body, hand, face, and full-body benchmark datasets.

  • Takeaways & Limitations

    The framework supports well-aligned and natural recovery of expressive full-body models from monocular images within the evaluated benchmark scope.

  • Takeaways & Limitations

    PyMAF-X fails on interacting hands and leaves strong perspective distortions and upper-torso-only observations with bent legs or erroneous limb poses unresolved.

Abstract

from arXiv · show

We present PyMAF-X, a regression-based approach to recovering parametric full-body models from monocular images. This task is very challenging since minor parametric deviation may lead to noticeable misalignment between the estimated mesh and the input image. Moreover, when integrating part-specific estimations into the full-body model, existing solutions tend to either degrade the alignment or produce unnatural wrist poses. To address these issues, we propose a Pyramidal Mesh Alignment Feedback (PyMAF) loop in our regression network for well-aligned human mesh recovery and extend it as PyMAF-X for the recovery of expressive full-body models. The core idea of PyMAF is to leverage a feature pyramid and rectify the predicted parameters explicitly based on the mesh-image alignment status. Specifically, given the currently predicted parameters, mesh-aligned evidence will be extracted from finer-resolution features accordingly and fed back for parameter rectification. To enhance the alignment perception, an auxiliary dense supervision is employed to provide mesh-image correspondence guidance while spatial alignment attention is introduced to enable the awareness of the global contexts for our network. When extending PyMAF for full-body mesh recovery, an adaptive integration strategy is proposed in PyMAF-X to produce natural wrist poses while maintaining the well-aligned performance of the part-specific estimations. The efficacy of our approach is validated on several benchmark datasets for body, hand, face, and full-body mesh recovery, where PyMAF and PyMAF-X effectively improve the mesh-image alignment and achieve new state-of-the-art results. The project page with code and video results can be found at https://zhanghongwen.cn/pymaf-x.

1 INTRODUCTION

PyMAF addresses coarse mesh-image alignment in regression-based human mesh recovery through explicit mesh-alignment feedback and multi-scale features. PyMAF-X extends this approach to full-body recovery with adaptive integration that preserves alignment while producing natural wrist poses.

  • Regression-based human mesh recovery is efficient but typically produces coarse alignment between predicted meshes and image observations.
  • Relative rotations along kinematic chains allow minor parametric errors to create noticeable joint-position drifts and mesh-image misalignment.
  • Iterative Error Feedback reuses a global feature, making it difficult for the regressor to perceive mesh-image misalignment during inference.
  • PyMAF extracts mesh-aligned evidence from spatial features using projected mesh locations and feeds it back progressively for explicit parameter correction.
  • PyMAF combines a feature pyramid, auxiliary pixel-wise supervision, and spatial alignment attention to provide multi-scale, informative, and globally aware alignment features.
  • PyMAF-X uses body-, hand-, and face-specific PyMAF experts with adaptive integration and elbow-twist compensation to avoid unnatural wrists while maintaining part-specific alignment.

2 RELATED WORK

Prior work includes optimization-based fitting and regression-based prediction for monocular human mesh recovery, alongside specialized efforts for full-body integration and iterative alignment. PyMAF builds on these directions by combining fine-grained evidence extraction with a mesh-alignment feedback loop.

  • Optimization-based Approaches: Optimization-based approaches fit parametric models to 2D observations using priors and data terms that measure reprojection errors.
  • Regression-based Approaches: Regression-based approaches predict human models directly from monocular images, learning model priors under varied supervision and proxy representations.
  • Regression-based Approaches: Non-parametric regression targets such as volumetric representations, mesh vertices, and position maps facilitate high-resolution features but require further processing.
  • Research has also addressed multi-person recovery, video inputs, occlusion, shape accuracy, ambiguity, camera estimation, imbalanced data, pseudo-ground-truth generation, and clothed reconstruction.
  • Full-body Mesh Recovery: Full-body recovery receives less attention than body-, hand-, and face-only recovery because of its difficulty and limited annotated datasets.
  • Full-body Mesh Recovery: Full-body regression methods commonly use body, hand, and face experts on cropped part images, differing mainly in expert architectures and integration strategies.
  • Iterative Alignment and Integration: PyMAF contributes fine-grained evidence extraction and mesh-alignment feedback, while its full-body extension integrates part predictions with natural wrist rotations and preserved part alignment.

3 METHOD

PyMAF uses a coarse-to-fine feature pyramid and mesh-aligned feedback to progressively rectify model parameters for better mesh-image alignment. PyMAF-X extends this design with part-specific experts and adaptive elbow-twist integration for full-body recovery.

  • Feature Pyramid: A feature pyramid supports coarse-to-fine mesh recovery, while progressively finer spatial features provide evidence for improving current predictions.The pyramid describes the posed person at multiple scales and feeds subsequent SMPL parameter predictions.
  • Mesh Alignment Feedback Loop: Auxiliary dense supervision improves the reliability of mesh-aligned cues by guiding spatial features toward pixel-wise correspondence prediction.The paper visualizes spatial features learned without and with auxiliary supervision alongside predicted dense correspondence maps.
  • Feature Pyramid: The encoder produces spatial feature maps, from which point-wise features are sampled, processed by an MLP, concatenated, and passed to a parameter regressor.At each level, the regressor takes sampled features and current parameters, then predicts residuals for iterative updates.
  • Mesh Alignment Feedback Loop: Mesh-aligned sampling projects a downsampled current mesh onto the image plane, yielding position-sensitive evidence for parameter rectification.Unlike global or uniformly sampled features, these samples reflect the current mesh-image alignment status.
  • Mesh Alignment Feedback Loop: Spatial alignment attention fuses grid and mesh-aligned features so the network can relate localized alignment cues to broader image context.A self-attention module processes features extracted from the same spatial feature map and captures relationships among all elements.
  • PyMAF-X for Full-body Mesh Recovery: PyMAF-X combines body-, hand-, and face-specific PyMAF experts and integrates their predictions into SMPL-X full-body parameters.Its adaptive integration corrects elbow twist using solved wrist poses, producing more natural wrists while maintaining body and hand alignment.

4 EXPERIMENTS

The experiments use ResNet-50 or HRNet-W48 encoders with multiresolution spatial features, iterative regressors, specified camera settings, and measured runtimes for body-only and full-body recovery.

  • Implementation: Part-specific PyMAF uses ResNet-50 primarily, while body regression also adopts HRNet-W48 following ExPose and PIXIE.The encoder processes 224 × 224 images into 14 × 14, 28 × 28, and 56 × 56 spatial feature maps.
  • Implementation: Mesh-aligned features are reduced to five channels per sampled point, producing a 2155-dimensional body feature vector from 431 downsampled vertices.The feedback loop runs for a maximum of three iterations, matching HMR.
  • Implementation: The regressors use a 21 × 21 grid at t = 0, continuous 6D pose representations, a [−72°, 72°] twist-angle constraint, and a 2D-pose-pretrained body encoder.The grid contains 441 points, approximately matching the 431-vertex downsampled mesh.
  • Camera Setting: Training and evaluation use a weak-perspective camera with focal length 5,000 by default, switching to SPEC-estimated perspective focal lengths on AGORA.Using SPEC camera settings for more accurate mesh recovery is left for future work.
  • Runtime: PyMAF processes one body-only sample in about 22 ms, while PyMAF-X takes about 80 ms and runs part-specific backbones sequentially.Parallel execution would further reduce full-body runtime.

4.2 Datasets

The study trains body, hand, and face experts on distinct mixtures of annotated datasets and pseudo-ground-truth model fits, while generating dense correspondence supervision from meshes.

  • Datasets: The body expert combines Human3.6M, MPI-INF-3DHP, MPII, LSP, LSP-Extended, and COCO data with 3D and 2D annotations.The hand expert uses FreiHAND, InterHand2.6M, and COCO-Wholebody, while the face expert uses VGGFace2.
  • Pseudo Ground-truth: SMPL/SMPL-X fits from EFT and ExPose provide pseudo-ground-truth annotations for body and full-body regressor training.These fitted models are used as training annotations rather than direct ground-truth measurements.
  • Evaluation Data: Table 1 reports reconstruction errors on the 3DPW and Human3.6M datasets, with backbone architectures highlighted in brackets.The caption identifies the datasets and the table’s backbone notation.
  • Pseudo Ground-truth: Face-expert training uses pseudo-ground-truth FLAME models and facial landmarks generated by DECA and FAN on VGGFace2.The two tools supply model and landmark supervision for the face expert.
  • Dense Correspondence: Dense correspondence maps are rendered from pseudo-ground-truth meshes instead of using DensePose annotations from COCO.This establishes the source of auxiliary dense supervision.

4.3 Evaluation Metrics

The evaluation reports standard 3D pose and shape errors in millimeters, including raw and Procrustes-aligned variants, while noting that aligned metrics omit global rotation and scale errors.

  • Metrics: Results are computed using the same evaluation procedures as prior work, enabling quantitative comparison with existing methods.The paper reports multiple evaluation metrics across its experiments.
  • Metrics: PVE measures mean vertex distance, MPJPE measures mean joint-position error, and PA-PVE and PA-MPJPE apply Procrustes rigid alignment before measurement.All four metrics are reported in millimeters by default.
  • Metric Caveat: PA-PVE and PA-MPJPE are insensitive to global rotation and scale errors because they are computed after rigid alignment.These metrics therefore assess errors after removing rigid alignment differences.

4.4 Comparison with the State of the Art

PyMAF improves body, hand, and face recovery across dedicated benchmarks, while PyMAF-X achieves leading full-body results and more accurate wrist poses in qualitative comparisons.

  • 3D Human Pose and Shape Estimation: 4.7 mm and 5.5 mm: PyMAF reduces MPJPE over the matched baseline on 3DPW and Human3.6M, respectively, using ResNet-50 without 3DPW training.The authors caution that broader Table 1 comparisons are not strictly comparable because methods use different training settings and supervision.
  • 2D Human Pose Estimation: PyMAF outperforms previous regression-based methods by notable margins in COCO keypoint-localization AP, directly evaluating mesh-image alignment in real-world images.SMPLify is included as an optimization-based reference but is described as slower and potentially unnatural despite alignment.
  • 2D Human Pose Estimation: Table 2 reports keypoint-localization APs on the COCO validation set, including SMPLify and methods evaluated from publicly released code and models.The caption specifies the evaluation sources and implementation basis for the comparisons.
  • Face Reconstruction: Qualitative face comparisons on in-the-wild images use parametric FLAME models for both DECA and PyMAF.The figure compares reconstructed face results rather than different face-model parameterizations.
  • Hand Reconstruction: On FreiHAND, hand-only PyMAF surpasses the baseline and previous full-body methods while remaining comparable with recent hand-only approaches.The comparison distinguishes parametric hand representations used by full-body methods from non-parametric representations used by some hand-only methods.
  • Face Reconstruction: On Stirling3D and NoW, PyMAF outperforms ExPose and PIXIE’s face experts and achieves similar results to DECA under PA-P2S.The metric is Point-to-Surface after Procrustes Alignment.
  • Full-body Reconstruction: PyMAF-X is evaluated on EHF and AGORA, two benchmark datasets for full-body mesh recovery.These datasets support comparisons with optimization-based and regression-based full-body methods.
  • Full-body Reconstruction: PyMAF-X achieves the best results among existing solutions on most Table 5 metrics, especially for body and full-body reconstruction.Table 5 compares optimization-based and regression-based full-body methods on EHF.

4.5 Ablation Study

The ablations evaluate mesh-aligned feedback, pyramidal features, auxiliary supervision, spatial alignment attention, and adaptive integration. Mesh-aligned evidence and adaptive integration improve alignment while supporting natural wrist poses.

  • Feedback Features: Mesh-aligned features achieve the best performance under both non-pyramidal and pyramidal feedback designs.Grid features outperform global features but remain worse than mesh-aligned features.
  • Feedback Features: Pyramidal feature maps increase the performance gain from mesh-aligned feedback by providing multi-scale alignment evidence.Uniformly sampled grid features provide spatial cues but do not reflect the current mesh-image alignment status.
  • Auxiliary Supervision: Auxiliary pixel-wise supervision improves the reliability of mesh-aligned evidence, with dense correspondences outperforming part segmentation in the compared settings.The auxiliary predictions are used for supervision during training, while point-wise features are extracted from feature maps.
  • Spatial Alignment Attention: Spatial alignment attention is more effective than fully connected fusion for combining grid and mesh-aligned features.Simply fusing grid features yields only marginal improvement compared with spatial alignment attention.
  • Adaptive Integration: The proposed adaptive integration produces well-aligned body, hand, and wrist parts while maintaining natural wrist poses.Learned integration can produce natural wrists but degrades hand alignment; the comparison uses EHF evaluations of joint alignment and wrist-vertex plausibility.

5 CONCLUSION

PyMAF introduces pyramidal mesh-alignment feedback for human mesh recovery, and PyMAF-X extends it to expressive full-body recovery. The methods improve mesh-image alignment, but challenging interactions, viewpoints, partial observations, and body-expert errors remain limitations.

  • Conclusion: PyMAF uses feature-pyramid spatial information and mesh-aligned feedback to correct parameter deviations explicitly and progressively.Mesh-aligned features are extracted from spatial maps using the estimated mesh and fed back for parameter rectification.
  • Conclusion: PyMAF-X adds adaptive integration with elbow-twist compensation to produce natural wrist poses while preserving part-specific alignment.The full-body model combines body, hand, and face expertise through this integration strategy.
  • Conclusion: PyMAF and PyMAF-X improve mesh-image alignment over baseline and previous regression-based solutions on indoor and in-the-wild datasets.The conclusion reports validation across the paper’s evaluated settings without specifying a single representative metric here.
  • Limitations and Future Work: PyMAF-X fails on interacting hands, while strong perspective distortion and upper-torso-only observations can still produce bent legs and erroneous limb poses.These issues are identified as future-work targets.
  • Limitations and Future Work: Full-body alignment heavily relies on the body expert, whose inaccurate shapes in challenging cases can cause erroneous arm bone lengths and coarse hand alignment.The authors suggest improved pseudo full-body annotations as a future direction.

APPENDIX A MORE IMPLEMENTATION DETAILS

The implementation uses a single attention block, mesh down-sampling or selected face vertices for feature extraction, and rendered dense supervision from pseudo ground-truth meshes. Hand visibility is supervised through visible-keypoint proportions.

  • Attention Module: A single attention block fuses grid and mesh-aligned features at each iteration because additional blocks mainly increase memory consumption.The implementation follows Mesh Graphormer for the attention module.
  • Mesh Sampling: Body and hand meshes use provided down-sampling matrices for mesh-aligned feature extraction, while face meshes use manually selected front-face vertices.The face selection reflects the concentration of expression information in the front face region.
  • Auxiliary Dense Supervision: Rendered dense correspondences and part segmentations are generated from pseudo ground-truth meshes for auxiliary supervision.Dense correspondences use IUV for the body expert and PNCC for hand and face experts.
  • Auxiliary Dense Supervision: Rendered dense correspondences provide finer-grained and dataset-consistent supervision without requiring manually defined mesh parts.Rendering costs for part segmentation and dense correspondences are reported as nearly identical with PyTorch3D.
  • Auxiliary Dense Supervision: Pseudo ground-truth meshes are used because relatively few datasets provide annotated part segmentation or dense correspondence.Such pseudo meshes are commonly available through prior work on datasets used by the method.
  • Hand Visibility: Hand visibility is predicted from hand-only mesh-aligned features and supervised by the proportion of visible annotated hand keypoints.For example, 15 visible keypoints out of 21 yield a pseudo visibility confidence of about 0.714.

APPENDIX B ABOUT METRICS

PA-PVE and PA-MPJPE do not fully reveal mesh-image alignment because rigid alignment is applied before computing them. Consequently, lower PA-MPJPE can coexist with worse image alignment.

  • Metric Limitations: PA-PVE and PA-MPJPE cannot fully measure mesh-image alignment because both are calculated after rigid alignment.The metrics are widely used for 3D pose estimation but may obscure reprojection discrepancies.
  • Metric Limitations: A reconstruction with lower PA-MPJPE can have higher MPJPE and worse alignment between the reprojected mesh and image.The appendix illustrates this metric-alignment mismatch with reconstruction examples.

APPENDIX C ABOUT DATASETS

The appendix describes the datasets used to train and evaluate the body, hand, face, and full-body experts, including their annotations, scales, and evaluation protocols.

  • Training datasets: Training combines 3D- and 2D-annotated body datasets with FreiHAND, InterHand2.6M, FFHQ, and COCO-WholeBody for hand and full-body regression.Body training uses Human3.6M, MPI-INF-3DHP, LSP, MPII, and COCO; hand and full-body training additionally uses FreiHAND, InterHand2.6M, FFHQ, and COCO-WholeBody.
  • Training datasets: Human3.6M provides 312,188 down-sampled training frames from controlled video, with SMPL parameters generated from sparse MoCap markers.The experiments follow common protocols and use five subjects for training.
  • Training datasets: MPI-INF-3DHP contributes 96,507 training frames spanning more subjects, poses, and indoor or outdoor scenes, although its 3D annotations contain noise.
  • Training datasets: LSP and LSP-Extended provide 10,428 training samples with 14 visible 2D keypoints per image.
  • Training datasets: MPII contributes 14,667 filtered training images, while COCO and COCO-WholeBody provide body, hand, and face keypoint annotations for model training.MPII images without complete keypoint annotations are discarded; COCO is used for body-only PyMAF and COCO-WholeBody hand keypoints are leveraged during training.
  • Evaluation datasets: Evaluation uses EHF and AGORA for full-body recovery, FreiHAND for hand recovery, and Stirling3D and NoW for face recovery.EHF contains 100 testing images with scans and SMPL-X meshes; AGORA evaluation uses its official platform, while NoW evaluation follows its official instructions.

APPENDIX D MORE QUALITATIVE RESULTS

The qualitative results show progressive alignment correction, strong visual comparisons on COCO, and reasonable full-body recovery under occlusion, while also exposing failure cases.

  • Feedback refinement: Across feedback iterations, PyMAF progressively corrects body-part drift and produces better-aligned human models.
  • COCO comparisons: On COCO qualitative comparisons, PyMAF produces better-aligned and more natural body reconstructions than competitors and the baseline.
  • Occlusion cases: Under body or hand occlusions, PyMAF-X produces reasonable full-body meshes by handling invisible hands with predicted visibility, default hand poses, and body-expert wrist poses.
  • Failure cases: Misalignment remains possible with inaccurate bone length, inaccurate body pose, challenging hand poses, occlusions, and interacting hands.The appendix attributes full-body alignment heavily to body-pose accuracy and notes that inaccurate shape affects bone length.
  • Full-body visualization: PyMAF-X full-body reconstructions are visualized from multiple viewpoints, including input images, overlays, and vertical-axis rotations.
Loading 2207.06400v4…