Source-linked AI summary
Tex2Shape: Detailed Full Human Body Geometry From a Single Image
Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, Marcus Magnor
TL;DR
Detailed full-body reconstruction from a single photograph remains difficult when fine clothing, hair, facial detail, and occluded regions must be recovered quickly. Tex2Shape addresses this by translating a partial UV texture into normal and displacement maps for a smooth body model. It produces detailed one-shot reconstructions, generalizes from synthetic training to real photographs, and operates at interactive speed.
Problem
Single-image human shape reconstruction must recover detailed hair, face, clothing wrinkles, and occluded body regions while remaining quick and easy for applications such as virtual try-on and scene analysis.
Method
Tex2Shape maps visible image pixels into SMPL UV space and uses an image-to-image network to predict complete normal and vector displacement maps that augment a smooth body model.
Results
Tex2Shape produces one-shot full 3D clothing, hair, facial, and occluded-region detail, generalizing robustly to real-world footage despite synthetic-only training.
Takeaways & Limitations
The pose-independent translation formulation supports detailed single-image reconstruction without requiring high mesh resolution and provides a foundation for broader 3D reconstruction applications.
Takeaways & Limitations
Performance is limited for hair and clothing absent from training, especially long hair and dresses that cannot be modeled as vector displacement fields.
Abstract
from arXiv · showhide
We present a simple yet effective method to infer detailed full human body shape from only a single photograph. Our model can infer full-body shape including face, hair, and clothing including wrinkles at interactive frame-rates. Results feature details even on parts that are occluded in the input image. Our main idea is to turn shape regression into an aligned image-to-image translation problem. The input to our method is a partial texture map of the visible region obtained from off-the-shelf methods. From a partial texture, we estimate detailed normal and vector displacement maps, which can be applied to a low-resolution smooth body model to add detail and clothing. Despite being trained purely with synthetic data, our model generalizes well to real-world photographs. Numerous results demonstrate the versatility and robustness of our method.
1. Introduction
Tex2Shape reconstructs detailed full-body geometry from one image by translating a partial UV texture into complete normal and displacement maps. It produces clothing, hair, and facial detail, including on occluded regions, at interactive speed.
- The method targets detailed reconstruction because visual fidelity and rapid, easy acquisition matter for applications including virtual try-on and scene analysis.Fine detail includes hair, face, clothing folds, and wrinkles.
- A partial texture map is translated into UV-space normal and vector displacement maps that augment a smooth SMPL body.Each pixel stores surface normals and displacement vectors relative to the smooth body shape.
- 2043 3D scans support synthetic training across varying clothing, poses, and body shapes.The training pipeline registers scans to SMPL and renders varied poses, camera views, and illumination.
- Synthetic-only training still yields robust one-shot full 3D shapes with varied clothing, hair, and occluded geometry.Tex2Shape combines translation and completion, rather than predicting detail only on visible regions.
- Tex2Shape reframes detailed full-body shape regression as a pose-independent aligned image-to-image translation problem.The method maps visible image pixels and outputs into the SMPL UV space.
- 50 milliseconds is sufficient to regress full 3D clothing, hair, and facial details from a single image.
2. Related Work
Prior work addresses monocular pose, body shape, face geometry, clothing, and hair through model fitting, templates, optimization, and learned representations. These approaches often impose pose, template, visibility, or garment-scope restrictions.
- Clothing and hair: Video-based clothing and hair reconstruction can be automatic and fast but often requires A-poses, multiple images, or robust pose detection.
- Shading-based refinement: Shading cues have been used to refine high-frequency human geometry from stereo or multi-view imagery.
- Face reconstruction: Monocular face methods improve geometry through shading-based refinement, differentiable rendering, or learned geometry and albedo representations.
- Garment reconstruction and modeling: Clothing methods model garments from scans or RGB-D, while wrinkle predictors commonly target pose-dependent animation or garments in isolation.
3. Method
The method represents detailed human geometry in the SMPL UV domain rather than directly on a coarse mesh. A CNN maps visible texture evidence to canonical normal and displacement maps, while SMPL supplies the underlying body shape.
- The target is an animatable 3D subject model from one photograph, including hair, clothing wrinkles, and details on unseen body parts.
- Parametric body model: SMPL provides a pose- and shape-parameterized body mesh, while Tex2Shape adds details beyond its shape space through UV maps.The model estimates standardized-height shape parameters independently of pose.
- UV parameterization: UV mapping unwraps the body surface onto a regular 2D grid, enabling image-based CNN processing and detail resolution proportional to the map.
- UV shape-images: The network receives visible pixels as a partial texture map and predicts normal and vector displacement maps in canonical SMPL T-pose.
- UV shape-images: The normal map adds shading detail, whereas the vector displacement map changes the underlying surface geometry.
4. Dataset Generation
The dataset is built by registering clothed-person scans to SMPL, rendering varied poses and illumination, and converting detailed registrations into coherent UV-space training targets.
- Synthetic rendering: Synthetic training images vary human pose, camera view, and realistic illumination, paired with normal maps, displacement maps, and SMPL shape parameters.Spherical harmonic lighting uses nine components per color, with coefficients derived from indoor HDR imagery and randomly rotated around the Y-axis.
- Scan registration: 2043 manually selected high-quality registrations form the dataset after non-rigidly registering SMPL to scans of clothed people.The scans include 1826 from Twindom, 163 from Renderpeople, and 54 from Axyzdesign.
- Dataset boundaries: The dataset reserves 20 scans for validation and 55 for testing, but is slightly biased toward men because registration fails more often for women with long hair, skirts, and dresses.The authors identify non-rigid registration of clothed people as challenging and sometimes producing unnatural shapes.
- Scan registration: The registration pipeline uses a subdivided SMPL mesh with 27554 vertices and 55104 faces to represent fine geometric details.Registration is performed in stages: pose, body shape, then non-rigid details.
- UV target generation: Detailed UV displacement maps are synthesized by rendering registered meshes into UV space and filling information between vertices through barycentric interpolation.Each mesh face has a 2D UV counterpart, enabling per-vertex geometric information to become image-space supervision.
5. Model and Training
Tex2Shape converts DensePose-derived partial textures into detailed normal and displacement maps with a Pix2Pix-style CNN, while a separate network predicts SMPL shape parameters.
- Shape prediction: A separate β-network processes 1024×1024 DensePose detections and predicts 10 SMPL shape parameters.It uses seven convolution-ReLU-batchnorm down-sampling layers followed by a fully connected layer.
- Training objective: Training maximizes structural similarity between ground-truth and predicted normal and displacement maps by minimizing MS-DSSIM.The objective is motivated by perceived image quality across multiple scales.
- Input construction: DensePose detections are mapped from 24 body-part UV parameterizations into a single SMPL UV map to create the partial-texture input.A precomputed lookup table converts DensePose coordinates into the joint SMPL UV parameterization.
6. Experiments
Experiments evaluate Tex2Shape across datasets, reconstruction baselines, UV-mapping choices, supervision losses, visibility, and unseen poses. The results show detailed reconstructions, real-world generalization, and robustness beyond training conditions.
- 6.1. Qualitative results and comparisons: Compared with BodyNet, HMR, SiCloPe, and Video Shapes, Tex2Shape preserves the highest level of detail from a single image.The comparison reports higher detail than Video Shapes despite that method using 120 frames.
- 6.1. Qualitative results and comparisons: Tex2Shape generalizes from synthetic training to real-world footage across 3DPW, DeepFashion, and PeopleSnapshot, hallucinating details on unseen backs.Synthetic results are compared with ground truth, while real-world datasets are evaluated qualitatively.
- 6.2. Supervision losses: MS-DSSIM supervision reconstructs complex clothing more reliably than L1 and GAN losses alone.The comparison is made by visual inspection because improved structure is not straightforward to quantify.
- 6.3. Impact of UV mapping: DensePose and ground-truth UV mappings produce nearly identical results, while HMR-based mapping lacks detail and introduces facial noise from imperfect alignment.DensePose can be used directly on real-world footage despite losing texture in partially mapped hair and clothing regions.
- 6.4. Impact of visibility: Across visibility settings, displacement error increases linearly with texture occupancy, with the minimum occurring near trained occupancy margins.At higher occupations, error rises slightly because fully covered subjects were not included in training scenarios.
- 6.4. Impact of visibility: For unseen Y-axis rotations, error remains nearly constant from 0° to 30° and then increases linearly beyond 30°.The network was trained with rotations between ±20°; the authors attribute degradation at larger angles to untrained poses.
7. Discussion and Conclusion
Tex2Shape reconstructs detailed full-body human shape from one image, including fine details on occluded parts, and generalizes from synthetic training to real-world footage. Its main scope limitation is hair and clothing absent from the training set.
- Discussion and Conclusion: Tex2Shape infers full-body shape with fine details on occluded parts from a single input image.It estimates UV-space normal and displacement maps from partial DensePose texture maps and augments SMPL without high mesh resolution.
- Discussion and Conclusion: The method generalizes robustly to real-world footage despite being trained only on synthetic data.
- Discussion and Conclusion: Hair and clothing not covered by the training set remain limitations, especially long hair and dresses that cannot be modeled as vector displacement fields.The paper attributes typical failures to garment-type or gender confusion caused by missing training samples.
- Discussion and Conclusion: The authors argue that transferring full-body reconstruction into a simpler formulation can outperform complex models and support broader reconstruction applications.
A. Appendix
The appendix evaluates illumination, camera-intrinsic robustness, UV-map resolution, and qualitative results to analyze the model’s behavior.
- Appendix: The appendix studies illumination effects, camera-intrinsic robustness, β-regression, UV-map resolution, and additional qualitative results.
A.1. Influence of Illumination
The appendix examines illumination as a cue and tests whether the model remains consistent under changed lighting. Controlled experiments indicate more consistent details with illumination augmentation and limited variation across photos.
- Influence of Illumination: Shading is treated as a potentially strong cue, motivating evaluation of illumination augmentation and robustness to varying illumination.
- Influence of Illumination: Training with constant ambient illumination cannot completely remove shading because wrinkles and smaller structures still contain shading effects.
- Influence of Illumination: Across 9 photos of two subjects under rotating illumination, the model shows a consistent picture, with variation mainly in areas of likely fabric movement.
A.2. Influence of Camera Intrinsics
Because focal length is often unknown for in-the-wild photos, the appendix tests the model under varying focal length and camera distance while preserving their ratio.
- Influence of Camera Intrinsics: The focal-length experiment renders A-pose test images with different focal lengths and camera distances while keeping their ratio fixed to create a Vertigo Effect.
A.3. Numerical Comparison with HMR
Tex2Shape is evaluated against HMR on naked body shape and through ablations of UV resolution, illumination augmentation, and in-the-wild reconstruction.
- Numerical Comparison with HMR: 10.57 ± 10.68mm mean bi-directional vertex-to-surface error outperforms HMR’s 16.28 ± 17.05mm on clothed scans after pose and scale alignment.The comparison uses results without added displacements and factors out pose before evaluation.
- UV Resolution Ablation: 256 × 256px UV maps preserve quality surprisingly well, but lose details and produce smoother edges than 512 × 512px maps.The experiment demonstrates that lower-resolution training does not largely decrease overall quality.
- In-the-Wild Results: The method produces 3D reconstruction results on the in-the-wild PeopleSnapshot and MonoPerfCap datasets.Figure 17 presents results for PeopleSnapshot in the first row and MonoPerfCap in the second.
- Illumination Augmentation: Illumination augmentation improves reconstruction relative to training without it, especially for face, legs, and chest regions.Without augmentation, the figure reports worse facial performance and visible artifacts.