Source-linked AI summary
Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flows
Andrei Zanfir, Eduard Gabriel Bazavan, Hongyi Xu, Bill Freeman, Rahul Sukthankar, Cristian Sminchisescu
TL;DR
Monocular 3D human pose and shape estimation lacks diverse 3D supervision for large-scale learning in real-world images and video. The paper combines kinematic normalizing-flow representations with differentiable semantic body-part alignment losses, and reports state-of-the-art results across motion-capture and in-the-wild datasets, supporting accurate learning from large-scale weak supervision.
Problem
Monocular 3D pose and shape estimation is difficult because human bodies have many degrees of freedom and large-scale supervised training data for complex visual scenes is limited.
Method
The paper combines latent kinematic normalizing-flow representations and dynamics with differentiable semantic body-part alignment losses for weakly and self-supervised learning.
Results
The methodology achieves state-of-the-art results in images and video across motion-capture datasets and in-the-wild repositories, including COCO.
Takeaways & Limitations
The results support constructing accurate 3D human pose and shape models from large-scale, diverse, incompletely labeled image and video data.
Abstract
from arXiv · showhide
Monocular 3D human pose and shape estimation is challenging due to the many degrees of freedom of the human body and thedifficulty to acquire training data for large-scale supervised learning in complex visual scenes. In this paper we present practical semi-supervised and self-supervised models that support training and good generalization in real-world images and video. Our formulation is based on kinematic latent normalizing flow representations and dynamics, as well as differentiable, semantic body part alignment loss functions that support self-supervised learning. In extensive experiments using 3D motion capture datasets like CMU, Human3.6M, 3DPW, or AMASS, as well as image repositories like COCO, we show that the proposed methods outperform the state of the art, supporting the practical construction of an accurate family of models based on large-scale training with diverse and incompletely labeled image and video data.
1 Introduction
The paper targets limited and mismatched 3D supervision for monocular human pose and shape estimation by combining normalizing-flow priors with semantic alignment losses for weakly and self-supervised learning.
- 1 Introduction: Limited diversity of 3D supervision motivates combining in-the-wild 2D annotations with motion-capture 3D data and strong pose and shape priors.In-the-wild datasets often lack 3D labels, while motion-capture datasets have less varied imagery, clothing, and body shapes.
- 1 Introduction: The proposed framework introduces normalizing-flow-based kinematic priors and semantic alignment losses to make large-scale weakly and self-supervised learning more accurate and efficient.The components are designed for integration into deep-learning losses and direct nonlinear model refinement.
- 1 Introduction: The evaluation spans Human3.6M, COCO, and 3DPW, with ablations indicating good weakly supervised performance for 3D reconstruction.The framework also uses CMU motion-capture data to construct kinematic priors and unlabeled COCO images for predict-and-reproject losses.
- 1 Introduction: Prior work includes keypoint-, segmentation-, and 3D-joint-based methods, while this paper focuses on learning under weak supervision with learned priors and image-based feedback.The related-work discussion positions weak supervision as the next frontier addressed by the paper.
- 1 Introduction: Semantic alignment provides differentiable, long-range body-part correspondences, including gradients for points that are not rendered, while remaining parallelizable and easy to implement.This is presented as an alternative to technically more general differentiable rendering models for obtaining useful image-model correspondences.
2 Methodology
The methodology combines kinematic body representations with normalizing-flow priors, differentiable semantic alignment, and multistage prediction to support weakly supervised 3D pose and shape estimation. It compares latent-space optimization with ambient-space optimization and uses projected mesh structure to exploit 2D supervision.
- 3D Pose and Shape Representations: The pose prior uses kinematic rotation representations including angle-axis variables and a 6D rotation-matrix over-parameterization, while related priors include Gaussian mixtures and adversarial discriminators.The 6D representation is tested for optimization, and quaternions perform at least 10% worse than angle-axis in the reported experiments.
- 3D Normalizing Flow-based Representations: Directly predicting or optimizing the latent representation yields better results than working in the ambient kinematic space.Both parameterizations are differentiable, but the empirical comparison favors latent-space inference.
- 3D Normalizing Flow-based Representations: Normalizing flows provide bijective mappings between kinematic pose variables and Gaussian latent variables, enabling smooth probabilistic priors without reconstruction error.The latent variable is modeled with a spherical Gaussian, while the pose is recovered through the inverse bijection.
- Differentiable Semantic Alignment Loss: Semantic alignment compares projected 3D body-part vertices with 2D segmentation regions through forward and backward losses with exact gradients.The loss is designed for categorical masks, supports parallel GPU computation, and is intended to provide large basins of attraction.
- Network Architecture: A multistage convolutional network predicts joints, semantic body-part masks, and 3D pose and shape using different subsets of losses under varying supervision levels.In the weakly supervised regime, projected meshes are matched to 2D joints and semantic masks while 3D mocap data supplies kinematic priors.
3 Experiments
Experiments evaluate normalizing-flow priors and semantic alignment losses for supervised, weakly supervised, and self-supervised 3D pose and shape estimation. Across H80K, COCO, and 3DPW, latent-space optimization, weak supervision, and temporal refinement improve or preserve reconstruction performance.
- 3 Experiments: Normalizing-flow priors optimized in latent space with semantic alignment perform best on H80K, with 6D rotations slightly better than angle-axis.The strongest objectives use normalizing-flow priors trained on H80K or AMASS, while VPoser performs slightly worse.
- 3 Experiments: Weakly supervised losses improve H80K performance whenever additional image data is available, including experiments across 20%, 40%, 60%, 80%, and 100% supervision partitions.Table 2 reports MPJPE/MPVPE ablations and highlights effects for small training sets and random initialization.
- 3 Experiments: 84mm MPJPE versus HMR’s 88mm shows the proposed model compares favorably on H80K.The comparison uses an H80K-retrained HMR model; the authors could not reproduce HMR’s original data split and retargeting.
- 3 Experiments: COCO fine-tuning consistently decreases projected-joint pixel error and improves body-part IoU, with keypoint and body-part alignment losses evaluated separately and together.The weakly supervised models start from H80K pretraining and use COCO images lacking 3D ground truth.
4 Conclusions
The paper presents scalable weakly supervised models for monocular 3D human pose and shape estimation by combining diverse image/video evidence with pose and shape priors. Latent normalizing flows, dynamical models, and differentiable semantic alignment losses support state-of-the-art results under large-scale weak supervision.
- Large-scale weakly supervised training exploits diverse image repositories that lack detailed 3D pose or shape supervision.The approach combines image and video evidence with existing human pose and shape prior knowledge.
- Latent normalizing flow representations, dynamical models, and differentiable semantic body-part alignment losses form the paper’s core methodology.The alignment losses provide informative feedback for self-supervised learning.
- Experiments across motion-capture datasets and in-the-wild repositories achieve state-of-the-art results for images and video.The reported datasets include CMU, Human3.6M, AMASS, 3DPW, and COCO.
5 Appendix
The appendix details flow architectures, training settings, translation estimation, and experiments evaluating priors and self-supervision. These studies report better latent-space optimization, lower reconstruction error with normalizing-flow priors, and improved 2D predictions with more self-supervised data.
- Normalizing-flow architectures: The appendix uses 138-dimensional 6D rotation representations for 23 SMPL joints when constructing normalizing-flow priors.The low-capacity architecture has 95,914 parameters, while the Real-NVP version has 331,462.
- Translation Estimation from 2D Keypoints: The translation procedure estimates global translation with weak-perspective least squares, then uses the estimated translation for perspective keypoint alignment while preserving gradients.The perspective projection is retained for the subsequent alignment loss.
- Normalizing Flows and VPoser on 3DPW: Normalizing-flow priors reduce MPJPE and MPVPE reconstruction errors across all tested priors and loss functions.Figure 5 evaluates the comparison on 500 randomly sampled 3DPW images using identical fitting conditions for both priors.
- Self-supervised Learning on COCO and OpenImages: Increasing self-supervised OpenImages data decreases 2D joint error and increases mIOU on the 3DPW test set.The additional training uses predicted 2D keypoints and semantic segmentation, and the authors report corresponding improvement in 3D predictions.