Source-linked AI summary
Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing
Thiemo Alldieck, Mihai Zanfir, Cristian Sminchisescu
TL;DR
PHORHUM addresses limitations of monocular dressed-human reconstruction by jointly estimating geometry, unshaded albedo, and scene illumination end-to-end from one RGB image. Rendering losses improve perceptual color fidelity, while the method reconstructs plausible visible and non-visible appearance and geometry.
Problem
Prior reconstruction methods may bake shading into appearance, omit color, or rely on impractical multi-step pipelines that cannot be trained end-to-end.
Method
PHORHUM uses an end-to-end composite network to jointly predict 3D geometry, albedo surface color, and per-image scene illumination from a single RGB image, with sparse geometry and ray-traced patch losses.
Results
Rendering losses significantly improve visual fidelity, while the method produces accurate, detailed geometry and appearance for visible and non-visible parts.
Takeaways & Limitations
PHORHUM enables jointly estimated, relightable human scans whose surface color is separated from scene-specific illumination in principle.
Takeaways & Limitations
Performance can degrade when clothing or pose deviates substantially from the training distribution, especially for loose, oversized, or non-Western clothing.
Abstract
from arXiv · showhide
We present PHORHUM, a novel, end-to-end trainable, deep neural network methodology for photorealistic 3D human reconstruction given just a monocular RGB image. Our pixel-aligned method estimates detailed 3D geometry and, for the first time, the unshaded surface color together with the scene illumination. Observing that 3D supervision alone is not sufficient for high fidelity color reconstruction, we introduce patch-based rendering losses that enable reliable color reconstruction on visible parts of the human, and detailed and plausible color estimation for the non-visible parts. Moreover, our method specifically addresses methodological and practical limitations of prior work in terms of representing geometry, albedo, and illumination effects, in an end-to-end model where factors can be effectively disentangled. In extensive experiments, we demonstrate the versatility and robustness of our approach. Our state-of-the-art results validate the method qualitatively and for different metrics, for both geometric and color reconstruction.
1. Introduction
PHORHUM targets photorealistic 3D reconstruction of clothed people from one RGB image by jointly estimating geometry, albedo, shading, and illumination end-to-end. It addresses prior limitations in appearance disentanglement, training practicality, and color fidelity.
- Single-image human reconstruction supports applications including immersive AR/VR, games, telepresence, virtual try-on, visualization, and creative editing.
- Prior methods often bake shading into appearance, omit color, use impractical multistage pipelines, or fail to recover true geometry for measurement-oriented applications.
- PHORHUM jointly predicts geometry and appearance in one end-to-end composite network with inter-dependent parameters.
- Compared with other methods, PHORHUM uniquely predicts albedo and shading, avoids test-time image matting, and uses signed distances rather than binary occupancy.
- The model estimates unshaded albedo, scene illumination, and shading, enabling scans to be re-shaded or composited into existing scenes.
- Rendering losses significantly improve visual fidelity, while the method reports more accurate and more detailed results than current state-of-the-art methods.
2. Related Work
Single-image human-shape reconstruction evolved from optimization and video-based methods toward deep models that predict geometry directly from one image. Representations include voxels, visual hulls, depth maps, template meshes, implicit functions, and neural radiance fields.
- Early optimization-based methods integrated information across videos of moving subjects to reconstruct complete 3D shape.
- Deep learning reduced reliance on video and enabled approaches predicting 3D human geometry from a single image.
- Voxel-based methods predict occupied spatial segments but require high memory, limiting the spatial resolution of shape estimates.
- Alternative representations include visual hulls, front-and-back depth maps, augmented template meshes, implicit function networks, and neural radiance fields.
3. Method
PHORHUM reconstructs dressed humans from a single RGB image by jointly estimating geometry, unshaded albedo, and illumination. It combines sparse 3D supervision with rendering-based losses to improve appearance fidelity and model visible and non-visible surfaces.
- Representation: PHORHUM represents the subject surface as the zero-level set of a neural signed distance function and estimates unshaded albedo and per-image lighting.Pixel-aligned features from the input image condition the geometry and appearance predictions.
- Representation: The feature extractor projects spatial samples into image-aligned features, while the implicit network predicts signed distance and surface color.The feature extractor uses bilinear pixel access for projected points, and the implicit network returns distance and albedo.
- Shading: The shading network predicts surface shading from normals and illumination, and the final shaded color is the element-wise product of shading and albedo.Image-color supervision is applied both at traced surface intersections and across image pixels using ground-truth normals and albedo.
- 3D supervision: Geometry is supervised with surface distance, normal, inside/outside, and signed-distance regularization losses, while albedo is supervised on and near the surface.The method uses mesh samples and enforces zero distance, matching normals, sign labels, unit gradient norm, and texture-derived albedo.
- Rendering losses: Patch-based rendering losses constrain visible and backside colors by comparing rendered intersections and patches with synthesized unshaded albedo images.The training data includes front and back albedo images; back views are generated by inverting the rendering Z-buffer.
- Dataset: Training examples pair scanned human meshes with HDRI-rendered images, alpha masks, front and back albedo images, and normal maps.Samples are drawn on and near mesh surfaces for geometric and appearance supervision.
4. Experiments
PHORHUM is evaluated against single-view reconstruction methods and ablations for geometry, surface color, and practical scene-aware reconstruction. It achieves strong geometric and color results, with qualitative advantages on detailed visible and unseen regions.
- Experimental setup: The evaluation covers geometric and color reconstruction on the authors’ dataset, plus qualitative comparisons on real images.The experiments include quantitative metrics, ablations, and real-image comparisons.
- Experimental setup: Perspective projection avoids the unnatural head proportions produced by orthographic camera assumptions.Orthographic models can produce heads that extend too far in depth and appear unnaturally large.
- Quantitative results: PHORHUM produces the most natural surface colors for both visible front and unseen back renderings.Table 2 reports Inception Scores for front- and back-side renderings.
- Quantitative results: PHORHUM produces the most accurate 3D reconstructions and is the only compared method that computes surface albedo.Its surface-normal reconstruction is on par with PIFuHD, while competing methods do not decompose albedo and shading.
- Qualitative results: On real images, PHORHUM reconstructs facial detail, hair, clothing wrinkles, sharp unseen backs, and small clothing or hair details.The albedo preserves small color patterns without strong shading effects, while the reconstructed back remains detailed and plausible.
- Ablations: Joint shading estimation improves albedo reconstruction without impairing reconstruction accuracy.The ablation results support the authors’ hypothesis that shading estimation helps decouple shading effects from albedo.
5. Discussion and Conclusions
PHORHUM reconstructs detailed geometry, albedo, shading, and plausible unseen appearance from a single image, while rendering losses improve color fidelity. The method remains limited by clothing and pose distributions represented in its dataset.
- Limitations: Loose, oversized, non-Western clothing and complex poses remain challenging, sometimes causing missing parts or front–back semantic mismatches.The authors identify broader geographic and culturally diverse training data as a way to alleviate these problems.
- Discussion: Compared with PIFuHD, PHORHUM achieves similar detail with less noise and more reliable body-pose reconstruction while also producing albedo and shaded color.
- Discussion: Estimated illumination can be transferred between images to create composites with matching shaded surfaces.
- Conclusions: The authors expect practical performance to improve as the diverse training set grows.
- Conclusions: PHORHUM reconstructs detailed geometry, albedo, shading, and plausible non-visible appearance from a single photograph.The method supports applications including immersive visualization, virtual try-on, AR/VR, telepresence, and creative image editing.
- Conclusions: Rendering losses are essential for perceptually accurate surface color, whereas sparse 3D supervision constrains geometry effectively.
Supplementary Material
The supplementary material expands the paper with implementation details, inference timings, reposing demonstrations, additional comparisons, and further results.
- Supplementary Material: The supplementary material reports hyper-parameters, inference times, reposing demonstrations, further comparisons, and additional results.
A. Implementation Details
The implementation details section describes hyper-parameters and timing measurements for mesh reconstruction using Marching Cubes.
- A. Implementation Details: The implementation details include the used hyper-parameters and timings for mesh reconstruction via Marching Cubes.
A.1. Hyper-parameters
Training minimizes a weighted combination of geometry, appearance, rendering, color, surface, and perceptual losses, with empirically selected weights and a scheduled surface-loss increase.
- A.1. Hyper-parameters: Training minimizes a weighted combination of geometry, lighting, appearance, rendering, color, surface, and VGG losses.
- A.1. Hyper-parameters: The experiments use λe = 0.1, λl = 0.2, λr = 1.0, λc = 1.0, λs = 50.0, and λVGG = 1.0.
- A.1. Hyper-parameters: The surface loss weight λg1 increases linearly from 1.0 to 15.0 over 100k interactions.
- A.1. Hyper-parameters: Marching Cubes extracts meshes from the distance field using coarse bounding-box probing and progressively refined Octree sampling.
- A.1. Hyper-parameters: Reconstructions can also be rendered through sphere tracing without generating an explicit mesh.
B. Additional Results
The paper extends its evaluation with automatic rigging and animation of reconstructed meshes using a fitted statistical body model.
- B. Additional Results: PHORHUM reconstructions are automatically rigged by fitting GHUM to triangulated keypoints and mesh surfaces with ICP.Joints and blend weights are then transferred from GHUM to the reconstructed meshes.
- B. Additional Results: The rigged reconstructions can be animated using motion-capture data.
B.2. Comparisons on the PeopleSnapshot Dataset
The PeopleSnapshot comparison evaluates reconstructions from single video frames against PIFuHD and Tex2Shape, with the latter estimating shape but not pose.
- B.2. Comparisons on the PeopleSnapshot Dataset: The comparison uses only the first frame of each PeopleSnapshot video, whose subjects rotate before the camera in an A-pose.
- B.2. Comparisons on the PeopleSnapshot Dataset: PHORHUM is compared with PIFuHD and the model-based Tex2Shape approach.Tex2Shape estimates the observed subject’s shape but not its pose.
B.3. Qualitative Results
Qualitative results show reconstructions across varied subjects, clothing, backgrounds, and illumination, including white-background images and post-processed animation.
- B.3. Qualitative Results: PHORHUM performs well across subjects, outfits, backgrounds, and illumination conditions in qualitative evaluations.
- B.3. Qualitative Results: Despite no training on this data type, the method performs extremely well on images of people with solid white backgrounds.The results suggest that matting can be applied before reconstruction when the model has difficulty with the image background.
- B.3. Qualitative Results: Reconstructions can be rigged and animated, then rendered under novel illumination.
- B.3. Qualitative Results: The qualitative comparisons display input images, reconstructed geometry, and PHORHUM albedo results against Tex2Shape and PIFuHD geometry.