PaperlayerView exact revision on arXiv

Source-linked AI summary

Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction

Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll

arXiv:2007.11432v2cs.CV

TL;DR

Implicit reconstructions preserve detail but are static and difficult to control, while parametric models enable control without fully capturing clothing and surface detail. The paper combines both through IP-Net, which predicts inner and outer surfaces and semantic correspondences from sparse point clouds, then registers and deforms a parametric model. Experiments report detailed controllable reconstructions, body-shape-under-clothing estimation, and effective registration from incomplete and single-view point clouds.

  • Problem

    Implicit human reconstructions are detailed but not controllable, while parametric models do not adequately capture clothing and surface detail.

  • Method

    IP-Net jointly predicts dressed outer and underlying inner surfaces plus body-part correspondences, which guide parametric fitting and deformation to the outer surface.

  • Results

    The method produces detailed controllable reconstructions and accurate body-shape-under-clothing estimation, including from sparse and single-view point clouds.

  • Takeaways & Limitations

    Combining implicit detail with parametric modelling yields controllable dressed-human reconstructions across full-body and hand settings.

  • Takeaways & Limitations

    Body-shape-under-clothing estimation has significantly higher errors for female subjects because training uses a male SMPL model.

Abstract

from arXiv · show

Implicit functions represented as deep learning approximations are powerful for reconstructing 3D surfaces. However, they can only produce static surfaces that are not controllable, which provides limited ability to modify the resulting model by editing its pose or shape parameters. Nevertheless, such features are essential in building flexible models for both computer graphics and computer vision. In this work, we present methodology that combines detail-rich implicit functions and parametric representations in order to reconstruct 3D models of people that remain controllable and accurate even in the presence of clothing. Given sparse 3D point clouds sampled on the surface of a dressed person, we use an Implicit Part Network (IP-Net)to jointly predict the outer 3D surface of the dressed person, the and inner body surface, and the semantic correspondences to a parametric body model. We subsequently use correspondences to fit the body model to our inner surface and then non-rigidly deform it (under a parametric body + displacement model) to the outer surface in order to capture garment, face and hair detail. In quantitative and qualitative experiments with both full body data and hand scans we show that the proposed methodology generalizes, and is effective even given incomplete point clouds collected from single-view depth images. Our models and code can be downloaded from http://virtualhumans.mpi-inf.mpg.de/ipnet.

1 Introduction

The paper combines implicit and parametric representations to reconstruct detailed dressed humans from sparse point clouds while retaining pose and shape control. IP-Net predicts inner and outer surfaces plus body-part correspondences, enabling parametric registration and detailed controllable models.

  • Motivation: Sparse or incomplete point clouds make dressed-human reconstruction difficult because clothing occludes the body and body-part identities become ambiguous.The target setting includes scans and depth-camera captures, including incomplete observations.
  • Motivation: Implicit representations preserve resolution and complex surfaces but produce static, uncontrollable outputs, whereas parametric models provide control but lack clothing and surface detail.The paper targets the complementary strengths of both representations.
  • Method: IP-Net jointly predicts the outer dressed surface, inner body surface, and semantic body-part labels for subsequent parametric registration.The approach uses two implicit functions: one for inner/outer occupancy and another for body-part labels.
  • Results: Experiments on Renderpeople, BUFF, and MANO show detailed controllable reconstructions, body-shape-under-clothing estimation, and accurate registration from sparse or single-view point clouds.The method performs comparably on BUFF without BUFF training or temporal information and generalizes to hand scans.
  • Contribution: The unified formulation combines implicit functions with parametric modelling to obtain high-quality controllable reconstructions from partial, sparse, or dense point clouds.The contribution covers articulated dressed humans and includes full dressed reconstruction together with body shape under clothing.

2 Related Work

Prior work balances controllability, detail, completeness, and computational practicality imperfectly across parametric, implicit, voxel, depth-map, and correspondence-based approaches. The paper positions its method as combining detailed continuous reconstruction with controllable, complete human models from incomplete point clouds.

  • Correspondence-Based Methods: Earlier correspondence methods use depth maps or random forests, whereas this approach predicts body shape under clothing and avoids generating multiple depth maps.The paper also claims complete controllable detailed surfaces from incomplete point clouds.
  • Implicit Functions for Humans: Voxel and depth-map reconstructions can provide more detail than parametric models, but limbs may be missing and voxel quality is limited by cubically growing memory costs.Increasing voxel resolution is difficult because the memory footprint grows cubically.
  • Implicit Functions for Humans: IP-Net predicts local and global evidence from point clouds through multiscale deep features and continuous query-point occupancy decisions.Its double-layer prediction supports articulated and fine-structure reconstruction while distinguishing inner and outer surfaces.
  • Parametric Modelling for Humans: Parametric human models provide pose and shape control and retain body parts, but their feed-forward predictions are difficult and their reconstructions lack detail.Optimization-based fitting can also be slow and susceptible to local minima.
  • Implicit Functions for Humans: Implicit functions preserve detail, support continuous surfaces, and represent complex topologies, but their outputs are static and may omit body parts.Naive fitting of a body model to an implicit surface can become trapped in local minima for difficult poses or occluded clothing.

3 Method

IP-Net combines implicit predictions with parametric registration to reconstruct detailed, controllable dressed-human models from sparse point clouds. It predicts inner and outer surfaces plus body-part labels, then fits SMPL and deforms it with per-vertex displacements.

  • IP-Net overview: IP-Net jointly predicts inner-body and outer dressed surfaces, body-part labels, and SMPL correspondences from sparse point clouds.The network processes approximately 5k points and predicts labels for 14 SMPL body parts.
  • Occupancy prediction: Three-region occupancy classification separates points inside the body, between body and clothing, and outside, enabling extraction of two surfaces.An ensemble of part-specific occupancy classifiers combines predictions using body-part scores, reducing bias toward larger parts.
  • Surface generation: Marching cubes converts predicted occupancies into triangulated mesh surfaces after the double-layer implicit surface is extracted.
  • SMPL+D registration: After fitting SMPL shape, pose, and translation to the inner surface, the method jointly optimizes per-vertex displacements to fit the outer surface.SMPL+D represents garment and hair detail through displacements, while a Laplacian regularizer is included in the registration objective.
  • SMPL registration: IP-Net uses predicted body-part labels to constrain SMPL registration to the inner surface and prevent incorrect body-part correspondences.The part term is reported as critical for correct registration.

4 Dataset and Experiments

The experiments train IP-Net on sparse point clouds and evaluate registration against scans and implicit reconstructions using vertex-based errors.

  • Dataset: IP-Net is trained on 700 scans and tested on 50 held-out scans.Sparse inputs contain 5k randomly sampled points voxelized into a 128x128x128 grid.
  • Experiments: The evaluation compares registration to point clouds, IF-Net predictions, directly regressed SMPL+D parameters, and IP-Net predictions.Table 1 reports vertex-to-vertex error in centimeters, with NP* indicating not possible.
  • Experiments: Table 2 compares registration using the outer implicit surface, the predicted body surface, and both body and part predictions.Errors are measured as vertex-to-vertex distances between ground-truth and predicted registered meshes.
  • Experiments: Figure 4 visualizes SMPL+D registration from the input point cloud, IF-Net prediction, and IP-Net prediction against a color-coded reference.Sitting poses are difficult to register without explicit SMPL correspondences.

4.2 Outer surface reconstruction.

IP-Net reconstructs outer surfaces competitively with implicit baselines while supplying predictions that enable controllable SMPL+D registration from sparse data.

  • Outer surface reconstruction: 4.86mm and 4.95mm are the average bidirectional vertex-to-surface errors for IF-Net and IP-Net, respectively, versus 9.86mm for Occ.Net.IP-Net performs better or on par with the compared implicit reconstruction methods while also predicting body shape and part labels.
  • Controllable reconstruction: IP-Net predictions make implicit reconstructions controllable by enabling SMPL+D registration to predicted outer and inner surfaces with part correspondences.The resulting model retains detailed outer geometry while allowing pose and shape control.
  • Registration comparison: Figure 5 compares registration without and with IP-Net correspondences, highlighting flips and incorrect body-part assignments without them.The figure uses color-coded SMPL vertices and three comparative examples.
  • Registration comparison: Registration using IP-Net predictions is significantly better than the alternatives summarized in Table 1.Figure 4 compares registration from sparse point clouds, IF-Net predictions, and IP-Net predictions.

4.4 Body Shape under Clothing

The method estimates body shape under clothing from sparse point clouds and remains competitive despite using less information than the BUFF baseline.

  • Body shape under clothing: IP-Net estimates body shape under clothing from sparse BUFF point clouds by predicting inner and outer surfaces with correspondences.SMPL is fitted to the predicted inner surface for evaluation.
  • Evaluation conditions: The BUFF comparison favors because it uses 4D scans, temporal optimization, and training data unavailable to IP-Net.IP-Net uses sparse point clouds, was not trained on BUFF, and predicts each frame independently.
  • Reposing: Figure 8 shows registration and two novel poses from single-depth-view point clouds.Figure 7 likewise illustrates reposing from point-cloud inputs using SMPL+D registration.
  • Evaluation conditions: Female-subject errors are higher because training data were generated by registering scans to a male SMPL model.The authors suggest gender-specific fitting during training-data generation could address this issue.

4.5 Why is correspondence prediction important?

Correspondence prediction resolves ambiguous registration by constraining which SMPL parts explain the predicted inner surface before deforming the model to the outer surface.

  • Registration strategy: Registering directly to the outer implicit surface is a baseline that can also be applied to other implicit reconstruction methods.The alternative does not use the predicted inner body surface or part correspondences.
  • Registration strategy: The proposed pipeline fits SMPL to the predicted inner surface using correspondences, then non-rigidly registers SMPL+D to the outer surface.This combines body prediction with part correspondences rather than relying on outer-surface fitting alone.
  • Why correspondences matter: Without inner-body and part predictions, registrations exhibit incorrect arm poses, torso vertices explaining arms, and 180° flips.These failures occur when pose or clothing makes correspondence ambiguous.

4.6 Why not independent networks for inner & outer surfaces?

IP-Net jointly predicts inner and outer surfaces, avoiding the severe inter-penetrations produced when two independent implicit networks cannot reason about each other.

  • 2000.71mm2 versus 0.65mm2 average intersecting surface area distinguishes independent networks from IP-Net.Joint prediction lets the two surfaces reason about each other during reconstruction.

4.7 Using IP-Net Correspondences to Register Scans

IP-Net provides a feed-forward alternative for scan registration by predicting the inner body surface and SMPL correspondences before fitting and non-rigid deformation.

  • IP-Net predicts the inner surface and SMPL correspondences, replacing accurate 3D joints for registration.SMPL is fitted to the inner surface, then SMPL+D is non-rigidly registered to the scan surface.
  • The pipeline uses scan points and an IP-Net voxel grid, followed by parametric fitting and non-rigid registration.This replaces tedious bottlenecks in existing scan-registration approaches.

4.8 Registration From Point Clouds Obtained from a Single View

IP-Net can process sparse single-view point clouds, where its predictions support successful fitting, and the same correspondence strategy extends to hand registration.

  • Single-view point clouds from depth cameras yield successful IP-Net fitting in the reported qualitative and quantitative evaluation.The experiment demonstrates applicability across dense scans, sparse clouds, and single-view clouds.
  • 4.80mm and 4.87mm average vertex-to-vertex errors are reported for full and single-view hand-point-cloud registration, respectively.IP-Net was trained on MANO and evaluated on both full and single-view point clouds.

5 Conclusions

The paper combines implicit and parametric representations to reconstruct detailed clothed humans while retaining control and reposing capabilities. Experiments report improvements from double-layer surfaces and semantic parts, including generalization to hands and incomplete single-view inputs.

  • 5 Conclusions: Implicit representations provide expressive surfaces, while parametric body models provide control and reposing for clothed-human reconstruction.These capabilities are relevant to animation and inference in computer vision.
  • 5 Conclusions: 3mm reconstruction improvement comes from predicting a double-layer surface for subsequent model fitting.The method jointly predicts dressed outer and underlying inner surfaces before deformation to capture garment, face, and hair details.
  • 5 Conclusions: 8.17cm improvement follows from leveraging semantic body parts during subsequent fitting.The conclusion identifies part classification as crucial for successful fitting.
  • 5 Conclusions: The method generalizes to 3D hands and incomplete single-view point clouds, including difficult poses.The conclusion identifies double-layer reconstruction and part classification as essential for successful registration in these settings.
Loading 2007.11432v2…