Source-linked AI summary
Unite the People: Closing the Loop Between 3D and 2D Human Representations
Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael J. Black, Peter V. Gehler
TL;DR
Detailed human-body learning lacks large-scale labeled data, while existing representations and datasets serve different tasks and label sets. The paper builds UP-3D from semi-automatically validated 3D fits, trains detailed discriminative models, and uses their predictions to improve fitting and scale the dataset. The resulting system supports 31-part segmentation, 91-landmark estimation, improved 3D pose results, and faster direct pose prediction.
Problem
Learning-based human representations are trained on separate datasets, while detailed 2D annotations are difficult to acquire at large scale.
Method
An improved SMPLify method creates validated 3D fits for multiple pose datasets, which provide rich labels for training discriminative models and iteratively improving the fits.
Results
The models predict 31 body segments and 91 landmarks, improve 3D fitting, achieve state-of-the-art results on HumanEva and Human3.6M, and increase high-quality LSP fits by 9.3%.
Takeaways & Limitations
Self-improvement and easy integration of new data make the system deployable on large scale with limited human feedback.
Takeaways & Limitations
The approach assumes a full human body model with shape and pose priors, and its pose experiments use the DeeperCut CNN.
Abstract
from arXiv · showhide
3D models provide a common ground for different representations of human bodies. In turn, robust 2D estimation has proven to be a powerful tool to obtain 3D fits "in-the- wild". However, depending on the level of detail, it can be hard to impossible to acquire labeled data for training 2D estimators on large scale. We propose a hybrid approach to this problem: with an extended version of the recently introduced SMPLify method, we obtain high quality 3D body model fits for multiple human pose datasets. Human annotators solely sort good and bad fits. This procedure leads to an initial dataset, UP-3D, with rich annotations. With a comprehensive set of experiments, we show how this data can be used to train discriminative models that produce results with an unprecedented level of detail: our models predict 31 segments and 91 landmark locations on the body. Using the 91 landmark pose estimator, we present state-of-the art results for 3D human pose and shape estimation using an order of magnitude less training data and without assumptions about gender or pose in the fitting procedure. We show that UP-3D can be enhanced with these improved fits to grow in quantity and quality, which makes the system deployable on large scale. The data, code and models are available for research purposes.
1. Introduction
The paper unifies human representations and tasks by using 3D body fits to create rich labels from existing datasets. A self-improving pipeline produces detailed predictions and improves 3D fitting at scale.
- Human understanding spans representations from bounding boxes to detailed 3D models, but learning methods are typically trained separately on independent datasets.
- The method applies an improved SMPLify procedure to pose-dataset keypoints, while annotators only sort good and bad 3D fits.The resulting fits form the initial UP-3D dataset.
- 3D fits provide consistent labels across tasks and integrate datasets with different label sets, including alternative keypoint locations.
- 9.3%: improved 91-landmark predictions increase the ratio of high-quality LSP fits compared with fits using 14 ground-truth keypoints.
2. Related Work
Related work covers 3D pose annotation, detailed body representations, body-part segmentation, intermediate-level person representations, and efforts to unify human-body annotations. Existing datasets trade off label fidelity, diversity, realism, or annotation detail.
- 3D human pose annotation remains a longstanding problem, while single-image 3D pose estimation remains challenging despite progress in 2D keypoint prediction.
- HumanEva and Human3.6M provide accurate 3D labels but limited diversity in backgrounds and person appearance because they use motion-capture environments.
- Natural-scene 3D pose ground truth has used vision systems prone to failure or inertial suits that alter body appearance and suffer motion drift.
- Detailed body representations include fitted surfaces, which connect more closely to physical human bodies and image formation than 3D skeletons alone.
- Prior datasets address fine-grained body-part segmentation and intermediate-level representations through part labels, sitting-person annotations, or puppets.
- Earlier unification efforts mainly targeted human kinematics by transferring motion-capture marker sets across different targets.
3. Building the Initial Dataset
The initial dataset maps heterogeneous 2D human annotations into SMPL fits using improved fitting objectives and robust initialization, then validates those fits with human judgments and richer labels.
- 3. Building the Initial Dataset: A common 3D representation maps diverse dataset annotations into consistent labels for supervised training.The authors motivate this representation as a way to unify label types and generate detailed annotations.
- 3.1. Improving Body Shape Estimation: SMPLify is extended with a silhouette-matching term because 2D keypoint connection lengths alone inadequately estimate body shape.The objective compares image and model silhouettes using a bidirectional distance, with different L1 and squared L2 contributions.
- 3.2. Handling Noisy Ground Truth Keypoints: Robust initialization addresses SMPLify’s sensitivity to missing torso joints and perspective-induced foreshortening.A corrected-mean estimator over connection-length ratios provides a simple, robust person-size estimate.
- 3. Building the Initial Dataset: Human annotators reviewed rendered fits from multiple perspectives, reducing fit-labeling time to ∼12s per image and exposing 3D errors hidden by 2D keypoints.The review selected fits whose body-part rotation and location matched image evidence; head and foot rotation were excluded as criteria.
- 3. Building the Initial Dataset: The validated fits produced UPI-3D with 5,569 training images and 1,208 test images.A held-out validation set of 1,112 training images was used in the experiments.
- 3.3. Exploring the Data: Label quality analysis found joint-location variance, including around the hips, indicating labeling noise even among images with very good 3D fits.Part-segmentation density also showed mixed annotations from arm occlusion and varying torso-to-leg boundaries around the hips.
4. Label Generation and Learning
UPI-3D supports learning detailed 2D representations and 3D body estimates from projected body-model annotations. Across segmentation, pose, fitting, and direct prediction, the resulting models generalize, improve fitting, and offer faster inference.
- Semantic Body Part Segmentation: The experiments cover 31-part segmentation, 2D pose and shape estimation, and direct 3D body prediction from 2D landmarks.The methods deliberately use 2D inputs rather than 2.5D data, preserving applicability to outdoor images and regular photo datasets.
- Semantic Body Part Segmentation: 0.4432 IoU and 0.9331 accuracy are achieved for 31-part segmentation on size-normalized crops using a multiscale ResNet101.The high-resolution input helps recover small body parts.
- Human Pose Estimation: The 91-landmark representation places markers around joints and across the body surface to disambiguate pose configuration and estimate shape.The core 14 skeleton keypoints remain included for comparability with existing methods.
- Human Pose Estimation: 0.9028 PCK@0.2 is obtained on the human-labeled FashionPose test set, compared with 0.897 for DeeperCut.The estimator is trained on landmarks projected from the full UP-3D dataset.
- 3D Human Pose Estimation: 3.6 f1 score points are gained by using the 91-landmark model for fitting, although extreme poses missed by the estimator prevent matching DeepCut-based fits.The reported boost follows training on generated 14-joint data and then applying SMPLify.
- 3D Human Pose Estimation: 5.4mm and 1.6mm average improvements are reported on HumanEva and Human3.6M, respectively, without actor- or gender-specific body models.The approach uses one hybrid human model and additional landmarks for shape inference.
- Direct 3D Prediction: 0.13s per decision forest enables full pose and shape prediction one to two orders of magnitude faster than SMPLify.Together with a ResNet101 CNN, full 3D body configuration prediction takes 0.378s, with the CNN as the bottleneck.
5. Closing the Loop
The improved 91-landmark predictions are used to re-fit previously rejected LSP images, increasing both the usable data quantity and fit quality. Annotation noise, perspective, and skeleton correspondence are identified as major sources of improvement.
- 5. Closing the Loop: 185 additional LSP images are accepted, a 20% increase over the initial accepted fits and a 9.3% absolute increase across the dataset.These images come from the formerly unused 54.75% of LSP, comprising 1,095 images.
- 5. Closing the Loop: 308 of 1,095 formerly unused LSP images show improved six-part segmentation f1 scores when refit from 91 predicted landmarks.Examples attribute improvement to left-right label noise, depth ambiguity, and perspective resolution.
- 5. Closing the Loop: The most common improvement sources are noisy annotations, better perspective resolution, and better correspondence between keypoints and the SMPL skeleton.The authors expect higher improvement ratios for noisier datasets such as LSP-extended and MPII-HumanPose.
- 5. Closing the Loop: The enlarged dataset can be used to retrain estimators and continue the improvement process iteratively.Figure 5 compares fits from 91 predicted keypoints with fits from 14 ground-truth keypoints.
6. Discussion
The method supports a holistic view of human prediction by using detailed representations to connect tasks and datasets. Its iterative generative–discriminative design enables faster 3D inference and continued improvement with limited human feedback.
- 31-part segmentation and 91-landmark pose estimation provide a level of human representation detail beyond previous work.
- The detailed representation enables integration of datasets with different annotations and supports established tasks at greater detail.
- A regression tree predicts 3D body configuration directly from 2D keypoints, avoiding optimization-based inference.
- The regression-tree method runs orders of magnitude faster than optimization-based methods while capturing overall pose reasonably well from simple 2D input.
- Improved 3D fits enlarge the training set, supporting iterative learning with limited human feedback and potential large-scale deployment.