Source-linked AI summary
Synthetic Training for Accurate 3D Human Pose and Shape Estimation in the Wild
Akash Sengupta, Ignas Budvytis, Roberto Cipolla
TL;DR
Existing methods often predict inaccurate body shapes, and prevalent training datasets have limited body-shape diversity. STRAPS trains a regressor using synthetic data with varied shapes and evaluates shape prediction on SSP-3D. It outperforms other approaches on SSP-3D for shape accuracy while remaining competitive on pose-centric datasets.
Problem
Current approaches often predict inaccurate body shapes, while prevalent 3D human datasets have limited body-shape diversity.
Source: §1 · pp. 1–2Method
STRAPS generates synthetic training data with diverse body shapes on the fly for a regressor, and the paper evaluates it on SSP-3D, an in-the-wild dataset with pseudo-ground-truth SMPL labels.
Source: §1 · p. 2, §6 · p. 10Results
STRAPS outperforms other approaches on SSP-3D in shape prediction accuracy while remaining competitive with the state of the art on pose-centric datasets.
Source: §6 · p. 11Takeaways & Limitations
SSP-3D provides a challenging evaluation dataset of tightly clothed sports-persons with diverse body shapes and corresponding pseudo-ground-truth SMPL parameters.
Source: §1 · p. 2For outlier body shapes, STRAPS may produce erroneous reconstructions when DensePose predicts an inaccurate silhouette; substantial self-occlusion is another cited failure case.
Source: §5 · p. 10
Abstract
from arXiv · showhide
This paper addresses the problem of monocular 3D human shape and pose estimation from an RGB image. Despite great progress in this field in terms of pose prediction accuracy, state-of-the-art methods often predict inaccurate body shapes. We suggest that this is primarily due to the scarcity of in-the-wild training data with diverse and accurate body shape labels. Thus, we propose STRAPS (Synthetic Training for Real Accurate Pose and Shape), a system that utilises proxy representations, such as silhouettes and 2D joints, as inputs to a shape and pose regression neural network, which is trained with synthetic training data (generated on-the-fly during training using the SMPL statistical body model) to overcome data scarcity. We bridge the gap between synthetic training inputs and noisy real inputs, which are predicted by keypoint detection and segmentation CNNs at test-time, by using data augmentation and corruption during training. In order to evaluate our approach, we curate and provide a challenging evaluation dataset for monocular human shape estimation, Sports Shape and Pose 3D (SSP-3D). It consists of RGB images of tightly-clothed sports-persons with a variety of body shapes and corresponding pseudo-ground-truth SMPL shape and pose parameters, obtained via multi-frame optimisation. We show that STRAPS outperforms other state-of-the-art methods on SSP-3D in terms of shape prediction accuracy, while remaining competitive with the state-of-the-art on pose-centric datasets and metrics.
1 Introduction
Current methods can predict inaccurate body shapes, particularly when test subjects differ from the training data’s shape distribution. STRAPS uses synthetic training data to address limited shape diversity, and SSP-3D provides an evaluation dataset for shape prediction.
1 Introduction
- Inaccurate body-shape predictions are linked to limited shape diversity in prevalent training datasets, with errors increasing as test subjects depart from the training-set mean shape.The introduction notes that models can struggle to generalize when test and training distributions differ.Source: §1 · pp. 1–2
- STRAPS achieves greater body-shape accuracy than SPIN [15], CMR [16], and HMR [8] on SSP-3D without requiring training images annotated with 3D labels.The paper presents SSP-3D as its evaluation dataset for comparing shape prediction accuracy.Source: Figure 1 · p. 2, §1 · p. 2
- STRAPS trains a regressor on synthetic inputs generated from sampled SMPL shape and pose parameters, using silhouettes and 2D joints as proxies and augmentation to handle noisy test inputs.The proxy representations are predicted at test time using segmentation and keypoint-detection CNNs.Source: §1 · p. 2
- SSP-3D contains tightly clothed sports-person images with varied body shapes and environments, paired with pseudo-ground-truth SMPL parameters obtained through multi-frame optimization.The dataset supports evaluation of shape prediction using per-vertex Euclidean error in a neutral pose.Source: §1 · p. 2
- The paper’s two stated contributions are a synthetic-training framework and the SSP-3D evaluation dataset for comparing body-shape prediction accuracy.The authors report that the dataset demonstrates better accuracy by their regressor than competing approaches.Source: §1 · p. 2
2 Related Work
Related work spans optimization-based fitting and learning-based prediction, with learning methods further divided by their output representation. The paper identifies limited shape diversity and in-the-wild coverage in existing training datasets as persistent constraints.
2 Related Work
- Optimization-based methods fit parametric body models to 2D observations and avoid 3D-labelled training data, but are slow, initialization-sensitive, and can get stuck in local minima.Their observations can include joints, surface landmarks, silhouettes, or body-part segmentations.Source: §2 · p. 3
- Learning methods either predict non-parametric body representations or regress statistical body-model parameters, sometimes from proxy inputs such as silhouettes, keypoints, segmentations, or IUV maps.Voxel resolution and mesh surface artifacts are cited as drawbacks of non-parametric representations.Source: §2 · p. 3
- Existing 3D human datasets offer varied, accurate poses but limited body-shape diversity; baggy clothing and indoor capture settings also constrain their relevance to in-the-wild images.The paper proposes synthetic training data and SSP-3D to address these dataset limitations and evaluate shape prediction.Source: §2 · p. 3
3 Method
STRAPS trains a pose-and-shape regressor on synthetic proxy representations, augmenting them to address limited body-shape diversity and noisy real inputs. SSP-3D supplies in-the-wild sports images with pseudo-ground-truth SMPL labels obtained through multi-frame fitting.
3.1 STRAPS
- STRAPS generates synthetic proxy inputs by sampling SMPL pose and shape, rendering silhouettes and 2D joints, and corrupting them to model noisy test-time predictions.Training uses shape augmentation, noise on joint centres and silhouette edges, and simulated occlusion; paired training images are not required.Source: §3.1 · pp. 4–5
- The regressor predicts SMPL pose, shape, and camera parameters from proxy features, with supervision on pose and shape parameters, 3D joints and vertices, and projected 2D joints.The multi-task objective uses homoscedastic uncertainty to adaptively weight its losses; redundant supervision was found to improve convergence and final performance.Source: Figure 2 · p. 4, §3.1 · p. 5
3.2 SSP-3D
- SSP-3D contains 311 in-the-wild images of 62 tightly-clothed sports-persons with diverse body shapes and pseudo-ground-truth SMPL shape and pose labels.Its multi-frame optimisation uses silhouettes and 2D joints, with pose regularisation to prevent perspective effects from substituting for shape updates.Source: §3.2 · pp. 5, 7
- SSP-3D fits a shared subject shape across frames while optimising per-frame pose and camera parameters against 2D joints and silhouettes.VIBE initialises the fits, and suitable frames are selected by human annotators.Source: §3.2 · pp. 5–6
4 Implementation Details
Training uses SMPL poses sampled from existing datasets to generate synthetic inputs, while evaluation covers shape- and pose-centric benchmarks. The regressor consumes predicted silhouettes and 2D joints at inference.
4 Implementation Details
- Synthetic training poses come from UP-3D, 3DPW, and selected Human3.6M subjects; baseline runs without shape augmentation also use their SMPL shape parameters.Synthetic silhouettes are cropped and resized to 256×256.Source: §4 · p. 7
- Evaluation measures pose with MPJPE-PA on Human3.6M and 3DPW, and shape with PVE-T-SC on BML-MoVi and SSP-3D; SSP-3D also uses silhouette mIOU.Human3.6M uses Protocol 2 subjects S9 and S11; 3DPW uses its test set, and MoVi uses F-PG1 videos.Source: §4 · p. 7
- A ResNet-18 encoder feeds an iterative regressor, while DensePose and Keypoint-RCNN provide the silhouette and 2D-joint inputs at inference.Inference runs at approximately 4 fps, with 90% of the time spent on silhouette and joint prediction.Source: §4 · p. 7
- Figure 4 compares augmentation variants and tests synthetic proxy inputs against proxies predicted from RGB images using DensePose and Keypoint-RCNN.Its panels report pose MPJPE-PA and shape PVE-T-SC metrics, and illustrate shape-diversity and noisy-input effects.Source: Figure 4 · p. 8
5 Empirical Evaluation
STRAPS improves shape prediction over competing methods on shape-centric benchmarks and remains competitive on pose-centric datasets. Shape and proxy-representation augmentation address distinct weaknesses, while difficult shapes, occlusion, and overlapping people remain challenging.
5 Empirical Evaluation
- STRAPS with shape and proxy-representation augmentation surpasses the state of the art in PVE-T-SC and mIOU on SSP-3D and MoVi.Figure 5 compares shape metrics on SSP-3D and MoVi and pose MPJPE-PA on Human3.6M and 3DPW.Source: Figure 5 · p. 8, §5 · p. 9
- Proxy-representation augmentation reduces the performance drop from synthetic to predicted inputs by modeling input noise and occlusion during synthetic training.Shape augmentation alone improves synthetic-input metrics but worsens real-input metrics, particularly with occluded silhouettes.Source: §5 · p. 9
- Across SSP-3D samples, STRAPS maintains shape accuracy on challenging cases where competing methods degrade on non-average body shapes.Erroneous reconstructions can still occur for outlier shapes when DensePose predicts an inaccurate silhouette, or with substantial self-occlusion.Source: §5 · p. 10
- The method is competitive with the state of the art in MPJPE-PA on Human3.6M and 3DPW, and outperforms methods not trained on images paired with 3D labels.On 3DPW, it matches or surpasses the state of the art up to the median MPJPE-PA, but severe occlusion and overlapping people degrade results in the 75–100% quartile.Source: §5 · p. 10
6 Conclusion
The paper presents STRAPS as a response to inaccurate shape predictions associated with limited body-shape diversity in prevalent datasets. It evaluates the framework on SSP-3D and reports stronger shape accuracy there while remaining competitive on pose-centric datasets.
6 Conclusion
- STRAPS generates synthetic training data with diverse body shapes on the fly, exposing the regressor to a new body shape at every training iteration.The conclusion identifies limited body-shape diversity in prevalent 3D human datasets as a source of inaccurate predictions, particularly for non-typical subjects.Source: §6 · pp. 10–11
- SSP-3D contains tightly clothed sports-person images with varied body shapes and pseudo-ground-truth SMPL shape and pose parameters, obtained through multi-frame optimisation.The paper reports that STRAPS outperforms other approaches in shape prediction accuracy on SSP-3D while remaining competitive on pose-centric datasets.Source: §6 · p. 11