PaperlayerView exact revision on arXiv

Source-linked AI summary

DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare

Yuanlu Xu, Song-Chun Zhu, Tony Tung

arXiv:1910.00116v2cs.CVcs.LGeess.IV

TL;DR

Monocular RGB images make 3D human pose and body-shape recovery difficult because sparse traditional representations are ambiguous. DenseRaC addresses this with an end-to-end two-step IUV-based framework, differentiable render-and-compare learning, and synthetic MOCA data; it reports superior performance across three evaluated tasks and benchmarks. The authors identify IUV-estimation errors as a failure source and plan future work on occlusion and interaction.

  • Problem

    Sparse landmarks, silhouettes, and body-part masks provide limited information for recovering 3D human pose and body shape from monocular images.

  • Method

    DenseRaC estimates IUV correspondences, regresses parametric pose and shape, compares rendered outputs with inputs, and trains with hybrid data including MOCA synthetic examples.

  • Results

    DenseRaC outperforms existing methods on public benchmarks and the proposed synthetic dataset across 3D pose estimation, semantic body segmentation, and 3D body reconstruction.

  • Takeaways & Limitations

    The framework provides an end-to-end pixel-to-surface correspondence and dense render-and-compare approach for reconstructing 3D human bodies from monocular RGB images in the wild.

  • Takeaways & Limitations

    DenseRaC suffers from errors in IUV estimation, and the authors identify handling occlusion and interaction as future work.

Abstract

from arXiv · show

We present DenseRaC, a novel end-to-end framework for jointly estimating 3D human pose and body shape from a monocular RGB image. Our two-step framework takes the body pixel-to-surface correspondence map (i.e., IUV map) as proxy representation and then performs estimation of parameterized human pose and shape. Specifically, given an estimated IUV map, we develop a deep neural network optimizing 3D body reconstruction losses and further integrating a render-and-compare scheme to minimize differences between the input and the rendered output, i.e., dense body landmarks, body part masks, and adversarial priors. To boost learning, we further construct a large-scale synthetic dataset (MOCA) utilizing web-crawled Mocap sequences, 3D scans and animations. The generated data covers diversified camera views, human actions and body shapes, and is paired with full ground truth. Our model jointly learns to represent the 3D human body from hybrid datasets, mitigating the problem of unpaired training data. Our experiments show that DenseRaC obtains superior performance against state of the art on public benchmarks of various humanrelated tasks.

1. Introduction

DenseRaC addresses monocular 3D human pose and body-shape estimation by using pixel-to-surface correspondences as an intermediate representation and learning through dense render-and-compare. It combines hybrid supervision with large-scale synthetic data and reports superior performance across evaluated benchmarks and tasks.

  • Motivation: Sparse landmarks, body masks, and silhouettes leave 3D pose and body shape ambiguous under monocular inputs and natural-image nuisances.These nuisances include illumination, occlusion, and texture.
  • Framework: DenseRaC first estimates pixel-to-surface correspondences, or IUV images, and then uses them to estimate 3D human pose and body shape.The framework is designed for monocular RGB inputs.
  • Framework: The framework integrates parametric pose and shape representations with dense render-and-compare losses between inputs and rendered outputs.Training jointly minimizes parameter-regression, 3D-reconstruction, landmark-reprojection, body-part-segmentation, and adversarial losses.
  • Data: MOCA is a large-scale synthetic dataset covering diversified human poses, body shapes, and camera views to support learning with incomplete or unpaired real-world supervision.The data uses web-crawled 3D animations and scanned all-gender body shapes.
  • Experiments: DenseRaC is evaluated on 3D pose estimation, semantic body segmentation, and 3D body reconstruction.The experiments include public benchmarks and the proposed synthetic dataset.
  • Results: The authors report that DenseRaC outperforms existing methods on public benchmarks and the newly proposed synthetic dataset.The paper presents the framework as an end-to-end pixel-to-surface correspondence and dense render-and-compare approach.

2. Related Work

Related work covers monocular 3D pose estimation, full 3D human-body reconstruction, and synthetic-data generation. Existing approaches use direct regression, lifted 2D poses, parametric body models, or rendered synthetic examples, but commonly face in-the-wild or data limitations.

  • Monocular 3D pose estimation: Monocular 3D pose estimation methods regress 3D joints directly or map lifted 2D poses into 3D.State-of-the-art performance is reported mainly on laboratory benchmarks, with unsatisfactory results on in-the-wild images.
  • Monocular 3D pose estimation: Some existing pose methods lack physically plausible skeleton constraints and therefore require extra post-processing for graphics applications.
  • 3D human body reconstruction: 3D human-body reconstruction seeks full body meshes from single RGB images or video rather than only major skeleton joints.Prior work uses realistic body models, part-based graphical models, and SMPL blend-shape representations.
  • Representation: Dense 3D landmarks can connect RGB pixels to points on a body model surface, illustrating a denser alternative to major skeleton joints.
  • Synthetic data: Synthetic datasets are generated by fitting or rendering 3D humans across virtual viewpoints and by varying factors such as pose, shape, texture, and background.The cited datasets are described as unable to solely serve for training generalizable models.

3. DenseRaC Framework

DenseRaC estimates 3D human pose and body shape by converting RGB images into IUV pixel-to-surface correspondences, then fitting a parametric body model with dense render-and-compare supervision. Its losses combine image-aligned landmarks and body-part masks with reconstruction, parameter, and adversarial constraints.

  • Framework overview: The framework first estimates IUV correspondences and then uses them to infer parametric 3D human pose and body shape.IUV maps link image pixels to unique points on the body surface, providing the intermediate representation for reconstruction.
  • Network architecture: The network predicts pose, shape, and orthographic camera parameters from IUV inputs, using an iterative regressor and a differentiable renderer.The renderer produces projected mesh outputs and differentiable IUV images for gradient-based learning.
  • Motivation: The approach addresses the ambiguity of recovering 3D pose and shape from sparse joints, silhouettes, or body-part masks by using denser pixel-to-surface information.The paper identifies monocular ambiguity, acquisition difficulty, and natural-image nuisances as limitations of prior proxy representations.
  • Loss terms: Training jointly minimizes parameter regression, 3D reconstruction, render-and-compare, and adversarial losses across paired and unpaired supervision.Rotation parameters are compared after conversion to rotation matrices, avoiding singularities associated with XYZ-Euler representations.
  • Dense render-and-compare: Dense render-and-compare losses compare rendered and input IUV maps through landmark reprojection, body-part mask, and adversarial terms.The IUV representation is less sensitive to background, lighting, and clothing texture while retaining dense geometric information.

4. MOCA Synthetic Dataset

MOCA is a large-scale synthetic dataset for joint 3D pose and body-shape learning, built from diverse animations, scanned body shapes, and rendered camera views. It provides fully paired ground truth to address weaknesses in existing datasets.

  • Dataset motivation: Existing joint pose-and-shape datasets are limited because SURREAL has only rough skeletons, while UP-3D uses potentially imprecise weakly supervised shapes.MOCA is presented as a response to these annotation and coverage limitations.
  • Pose and action diversity: MOCA draws 2,446 Mixamo animation sequences covering sports, combat, daily, and social activities.The sequences contain 261,138 frames at 30 fps and use a refined skeleton with fingers and facial bones.
  • Synthesis process: The dataset combines Mixamo poses with 2,781 CAESAR-derived body shapes and renders randomized camera views as IUV image sequences.Each selected body shape is paired with action sequences and camera views through standard linear blend skinning and IUV rasterization.

5. Experiments

DenseRaC is evaluated across 3D pose estimation, segmentation, and 3D body reconstruction using public benchmarks and MOCA. Its components each improve performance, while qualitative and quantitative results compare favorably with existing methods.

  • Evaluation setup: DenseRaC is evaluated on 3D pose estimation, body shape estimation, body semantic segmentation, and mesh-level reconstruction across public datasets and MOCA.The experiments use LSP, MPII, COCO, H3.6M, MPI-INF-3DHP, UP-3D, and the synthetic MOCA dataset.
  • 3D Pose Estimation: The pose evaluation compares MPJPE, PCK, and AUC on H3.6M under three protocols and MPI-INF-3DHP under two protocols.Protocols vary camera views, rigid alignment, subjects, sequences, and trials.
  • Ablation studies: Each DenseRaC component contributes to final performance, and the full method outperforms state-of-the-art parametric body-model estimators by a large margin.The evaluated components include dense render-and-compare and joint learning with synthetic data.
  • 3D Human Body Reconstruction: DenseRaC significantly outperforms a competitive HMR variant on whole-body reconstruction metrics evaluated on MOCA.The metrics include mean per mesh vertex position error and pose-and-shape parameter error.
  • Ablation studies: Dense render-and-compare losses provide richer information than sparse joints and greatly reduce impossible 3D body configurations.When task-oriented losses are present, dense supervision still helps the model reach a better local optimum.
  • Empirical Studies: Qualitative results show more plausible and natural bodies, while failures arise from erroneous IUV estimates, orthographic projection, and the SMPL representation.The reported failure cases include occlusions, long-tail data, and under-represented body shapes such as children.

6. Conclusion

DenseRaC reconstructs 3D human bodies from monocular RGB images using pixel-to-surface correspondences, dense render-and-compare, and large-scale synthetic data. The framework obtains superior performance, while occlusion and interaction remain directions for future work.

  • 6. Conclusion: DenseRaC reconstructs 3D human bodies from monocular RGB images using pixel-to-surface correspondence maps and dense render-and-compare.The correspondence map serves as a proxy representation between RGB input and 3D reconstruction.
  • 6. Conclusion: Figure 7 identifies heavy occlusions, incorrect IUV estimations, and under-represented body shapes such as children as current limitations.Each triplet contains the original image, the IUV input, and the model output.
  • 6. Conclusion: Large-scale MOCA synthetic data mitigates the problem of unpaired training data.MOCA is used to boost model training alongside the proposed framework.
  • 6. Conclusion: The framework obtains superior performance, while future work will address occlusion and interaction.The authors mention multi-view fusion and temporal smoothing as possible directions.
Loading 1910.00116v2…