PaperlayerView exact revision on arXiv

Source-linked AI summary

TeCH: Text-guided Reconstruction of Lifelike Clothed Humans

Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, Justus Thies

arXiv:2308.08545v2cs.CVcs.AIcs.GR

TL;DR

Recovering detailed unseen regions from a single clothed-human image is difficult because those areas are unobserved. TeCH combines image-derived descriptions and personalized diffusion guidance to optimize a 3D representation, and reports detailed geometry and improved rendering quality over prior methods.

  • Problem

    Reconstructing detailed unseen regions from a single clothed-human image remains challenging because non-visible areas lack observations.

    Source: §1 · p. 2

  • Method

    TeCH combines parsed text descriptions with a personalized diffusion model and optimizes an SMPL-X-initialized hybrid DMTet human using SDS losses.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    TeCH reports detailed full-body geometry and consistent, intricate textures, with evaluations finding stronger geometric-detail reconstruction and rendering quality than state-of-the-art methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present image and textual descriptions as a paradigm they believe can support reconstruction tasks beyond human bodies.

    Source: §7 · p. 10

    TeCH occasionally fails on extremely loose clothing, may produce mismatched patterns such as tattoos, and relies on robust SMPL-X pose estimation, which remains unsolved for challenging poses.

    Source: §6 · p. 10

Abstract

from arXiv · show

Despite recent research advancements in reconstructing clothed humans from a single image, accurately restoring the "unseen regions" with high-level details remains an unsolved challenge that lacks attention. Existing methods often generate overly smooth back-side surfaces with a blurry texture. But how to effectively capture all visual attributes of an individual from a single image, which are sufficient to reconstruct unseen areas (e.g., the back view)? Motivated by the power of foundation models, TeCH reconstructs the 3D human by leveraging 1) descriptive text prompts (e.g., garments, colors, hairstyles) which are automatically generated via a garment parsing model and Visual Question Answering (VQA), 2) a personalized fine-tuned Text-to-Image diffusion model (T2I) which learns the "indescribable" appearance. To represent high-resolution 3D clothed humans at an affordable cost, we propose a hybrid 3D representation based on DMTet, which consists of an explicit body shape grid and an implicit distance field. Guided by the descriptive prompts + personalized T2I diffusion model, the geometry and texture of the 3D humans are optimized through multi-view Score Distillation Sampling (SDS) and reconstruction losses based on the original observation. TeCH produces high-fidelity 3D clothed humans with consistent & delicate texture, and detailed full-body geometry. Quantitative and qualitative experiments demonstrate that TeCH outperforms the state-of-the-art methods in terms of reconstruction accuracy and rendering quality. The code will be publicly available for research purposes at https://huangyangyi.github.io/TeCH

1. Introduction

Single-image clothed-human reconstruction leaves unseen regions ill-posed, and prior methods often produce blurry textures and smoothed geometry. TeCH combines image-derived text and personalized diffusion guidance to reconstruct detailed clothed humans; the related work spans image-based reconstruction, 3D human generation, and general image-to-3D methods.

1. Introduction

  • TeCH addresses unseen-region ambiguity by combining parsed garment, color, hairstyle, and facial descriptions with a personalized diffusion model that captures distinctive appearance difficult to describe in text.It optimizes geometry and texture using multi-view SDS, observation-based reconstruction losses, normal regularization, and a hybrid DMTet representation.Source: §1 · p. 2
  • TeCH reports detailed full-body geometry and consistent, intricate textures, with quantitative geometric-detail results and qualitative rendering comparisons favoring it over state-of-the-art methods.Evaluations cover CAPE and THuman2.0 datasets and in-the-wild images, with a perceptual study of rendering quality.Source: §1 · p. 2

2. Related Work

  • The related work groups image-based human reconstructors into explicit-shape, implicit-function, and NeRF-based approaches, alongside categories of 3D human generators.The generator categories distinguish methods learned from real 3D captures from those learned from large-scale 2D images.Source: §2 · p. 2

2.1. Image-based Clothed Human Reconstruction

  • Image-based methods use explicit shapes, topology-agnostic implicit fields, or NeRFs, but prior approaches face topology limits or depth and texture inconsistencies without multi-view supervision.NeRF approaches cited here incorporate SMPL-X priors, partial-observation completion, or a pretrained CLIP appearance prior.Source: §2.1 · pp. 2–3

2.2. Generative Modeling of 3D Clothed Humans

  • 3D-trained human generators depend on limited, expensive 3D data, while 2D-trained generators still have a gap in lifelike geometry and input-consistent texture; TeCH targets that gap from one image.The paper describes TeCH as using a pretrained diffusion model and descriptive prompts to imagine non-visible regions while reconstructing visible regions from the input.Source: §2.2 · p. 3

2.3. Image-to-3D for General Objects

  • General-object image-to-3D methods use pretrained diffusion guidance, including multi-view SDS and direct view-conditional generation, but TeCH targets pixel-aligned models with intricate texture in non-visible regions.The cited related work also includes textual inversion, DreamBooth, CLIP-guided priors, depth priors, and reconstruction losses.Source: §2.3 · pp. 3–4

3. Method

TeCH reconstructs a clothed human from one image by pairing extracted semantic prompts and a personalized appearance model with staged optimization of a hybrid 3D representation.

3.1. Extracting Text-guidance from the Observation

  • SegFormer and fine-grained BLIP VQA describe garments and facial attributes, while few-shot DreamBooth learns an identity token; combining them yields consistent patterns and supports garment-color editing.Detailed clothing descriptions and added colors and hairstyles improve semantic consistency; VQA descriptions alone are insufficient for a convincingly realistic appearance.Source: §3.1 · pp. 4–5

3.2. Hybrid 3D Representation

  • The hybrid representation embeds DMTet around an SMPL-X body mesh, using a tetrahedral grid and an SDF-predicting geometry network for efficient high-resolution modeling.The outer-shell grid is described as more computationally efficient than DMTet’s cubic-based grid for high-resolution human geometry.Source: §3.2 · p. 5

3.3. Multi-stage Optimization

  • TeCH first optimizes geometry, then recovers appearance on the extracted mesh, with personalized DreamBooth SDS supervision from sampled camera views.The method uses a multi-stage, coarse-to-fine optimization process.Source: §3 · p. 4, §3.3 · p. 6

3.3.1 Geometry Stage

  • Geometry combines input-silhouette alignment, normal-image SDS, predicted-normal and Laplacian regularization, then refines DMTet coarse-to-fine.Normal regularization addresses noisy surfaces, particularly for subjects wearing complex clothing.Source: §3.3.1 · p. 6

3.3.2 Texture Stage

  • Texture optimization combines input-view reconstruction, text-guided color SDS, and color-consistency regularization, training on both the input pose and an A-pose.An occlusion-aware mask excludes self-occluded regions from the reconstruction loss; the color-consistency loss is added after texture convergence.Source: §3.3.2 · p. 7
  • Chamfer-based color consistency addresses SDS-generated oversaturation, while multi-pose training improves texture quality under self-occlusion.The paper reports improved frontal-to-back color consistency and texture quality across all 2D image quality metrics.Source: §3.3.2 · p. 7, Figure 7 · p. 7

3.3.3 Camera sampling during optimization

  • Camera sampling covers the body with random views and adds view-aware prompts to mitigate mirrored appearance artifacts; face-focused views and prompts target facial detail.The view-aware prompts specify front, side, back, or overhead views.Source: §3.3.3 · p. 7

4. Experiments

TeCH is evaluated against reconstruction baselines on scanned and in-the-wild humans, with quantitative, perceptual, and qualitative comparisons. The experiments report advantages on CAPE and in user preferences, while ablations examine the contributions of its guidance and optimization choices.

4.1. Models and Datasets

  • Evaluation uses 150 CAPE scans, 150 THuman2.0 scans, and 90 randomly sampled SHHQ images; comparisons include body-agnostic and SMPL(-X)-prior methods.PHORHUM is used only for qualitative comparisons because its camera model is incompatible with the test data.Source: §4 · p. 7, §4.1 · p. 8

4.2. Quantitative Comparison

  • TeCH surpasses baselines across 3D and 2D image-quality metrics on CAPE, while achieving accuracy comparable to prior-based methods on THuman2.0.The authors note that realistic hallucinated backs may differ from ground truth, and use a perceptual study for additional clarification.Source: §4.2 · p. 8, Table 1 · p. 8

4.3. Perceptual Evaluation

  • In 3,150 pairwise judgments from 63 participants, TeCH is preferred for both geometry and texture over the compared methods.The study covers 90 SHHQ subjects; Figure 8 shows in-the-wild results with diverse clothing and textures.Source: §4.3 · p. 8, Table 2 · p. 9, Figure 8 · p. 9

4.4. More Qualitative Results

  • Qualitative comparisons show detailed garment and facial features across difficult poses, complex textures, loose clothing, self-occlusion, and in-the-wild images.Comparisons visualize multi-view surface normals, color renderings, and zoomed-in details.Source: §4.4 · p. 9

4.5. Ablation Studies

  • Combining VQA prompts with DreamBooth guidance gives the best results; VQA helps recover clothing structure, while DreamBooth enhances fine texture patterns.Either guidance source alone reduces reconstruction accuracy and texture quality.Source: §4.5 · p. 9, Table 3 · p. 9
  • Geometry-only SDS produces noisy artifacts, especially on loose clothes, and incorporating Lnorm at optimization’s beginning can mitigate the issue.The study also reports a corresponding increase in Normal error when using only Lnorm SDS.Source: §4.5 · p. 9
  • LCD enhances color consistency between front and back, while multi-pose training improves texture quality under self-occlusion across 2D image-quality metrics.Source: §4.5 · p. 9
  • Joint optimization drops across 3D and 2D metrics relative to decoupled stages; the separate texture stage also saves 20% runtime.The authors attribute the drop possibly to entangled geometry and texture gradients; the separate stage renders color images from the extracted mesh.Source: §4.5 · pp. 9–10

5. Applications

TeCH supports animation by aligning the reconstructed clothed mesh with SMPL-X, and supports texture stylization while retaining the character’s inherent appearance traits.

5.1. Avatar animation

5.2. Avatar editing

  • Text-guided texture generation enables stylization in pop art, pixel art, and Van Gogh styles while preserving the character’s inherent appearance traits.Source: §5.2 · p. 10, Figure 10 · p. 10

6. Discussion

TeCH has failure cases, remains time-consuming per subject, and leaves controllability and component-wise generation as future work; the authors also note potential misuse and intellectual-property concerns.

6. Discussion

  • Failures include occasional problems with extremely loose clothing, mismatched patterns or tattoos, and errors propagated from incorrect SMPL-X pose estimates.The paper also notes possible noisy surfaces and pattern mismatches.Source: §6 · p. 10, Figure 11 · p. 10
  • Per-subject optimization takes approximately 4.5 hours on a V100 GPU.The reported stages include DreamBooth training, geometry optimization, mesh post-processing, and texture optimization.Source: §6 · p. 10
  • Future work includes using controllable text-to-image models and solving compositional generation of separate components such as haircuts, accessories, and decoupled outfits.Source: §6 · p. 10
  • The authors identify potential facilitation of deep-fake avatars and intellectual-property concerns, alongside benefits in entertainment, as broader-impact issues requiring regulation.Source: §6 · p. 10

7. Conclusion

The conclusion presents TeCH as a single-image approach for reconstructing lifelike clothed humans, with detailed geometry and consistent texture, and reports superiority over existing methods in geometry and rendering quality. The appendices provide additional preliminaries, implementation details, and qualitative results.

7. Conclusion

  • TeCH reconstructs lifelike clothed humans from a single image, including detailed full-body geometry and high-quality, consistent texture, and experiments report superiority over existing methods in geometry and rendering quality.Source: §7 · p. 10

Appendices

  • The appendices add preliminaries, VQA questions, outer-shell construction and camera-sampling details, network and optimization settings, and qualitative results on benchmark datasets and in-the-wild photos.Source: Appendices · p. 11

A. Preliminaries

The preliminaries describe DreamBooth personalization, SDS-based 3D optimization, and DMTet as the representation used for TeCH’s shape. Together, these background components explain how the paper adapts image diffusion guidance to 3D reconstruction.

A. Preliminaries

  • DreamBooth personalizes a pretrained diffusion model using 3–5 images of one subject and denoising losses, with a prior-preservation term intended to mitigate language drift and augment output diversity.The prior term uses self-generated images conditioned on a class-specific embedding.Source: A. Preliminaries · p. 11
  • SDS optimizes a rendered image through a differentiable renderer so it resembles a sample from a frozen text-to-image diffusion model; TeCH applies SDS guidance across random camera poses.TeCH parameterizes its 3D shape with improved DMTet rather than NeRF.Source: A. Preliminaries · p. 11
  • DMTet learns signed-distance values on a deformable tetrahedral grid, refines the surface and grid, and extracts a triangular mesh through Marching Tetrahedra.The passage identifies fast optimization and high resolution as advantages over NeRF, enabled by differentiable rasterization of mesh patches.Source: A. Preliminaries · p. 11

B. VQA Questions Q

The VQA prompt combines general descriptions of a person with attributes parsed for segmented garment categories. Short answers are concatenated with duplicates removed, with a dataset-specific hairnet description added for CAPE.

B. VQA Questions Q

  • BLIP questions parse gender, facial appearance, hair, facial hair, and pose, while SegFormer identifies ten garment categories whose attributes are queried separately.Source: B. VQA Questions Q · p. 11
  • The prompt PVQA concatenates BLIP’s typically 1–3-word answers and removes repeated words; CAPE adds “hairnet” because BLIP has difficulty recognizing it.Source: B. VQA Questions Q · p. 11

C. Construction of the Outer SMPL-X Shell

TeCH constructs a compact tetrahedral shell from the estimated SMPL-X body mesh. The shell is simplified and tetrahedralized to support the representation.

C. Construction of the Outer SMPL-X Shell

  • A coarse outer shell is calculated from the estimated SMPL-X body mesh and dilated by an offset of 0.1.The mesh is simplified by reducing its triangle count by 90% using quadric decimation.Source: C. Construction of the Outer SMPL-X… · pp. 11–12
  • The attribute-parsing table defines g as a garment-part segmentation category and d as a garment category recognized from an answer to the second garment question.Source: Table 4 · p. 12
  • TetGen generates the tetrahedral grid from the simplified outer shell with a maximum volume size of 5 × 10−8.Source: C. Construction of the Outer SMPL-X… · p. 12

D. Camera Sampling

TeCH samples virtual cameras for body coverage and face detail during optimization. View-aware prompts are used to mitigate mirrored-appearance artifacts.

D. Camera Sampling

  • Two virtual-camera groups cover the full body or main body parts and zoom in on the face.Source: D. Camera Sampling · p. 12
  • Body cameras are sampled with probability 0.7 across azimuths from −180° to 180°; view-aware front, side, back, or overhead prompts mitigate Janus-head artifacts.The body-camera elevation is 0°; height and radius ranges are (−0.4, 0.4) and (0.7, 1.3), respectively.Source: D. Camera Sampling · p. 12
  • Face cameras are sampled with probability 0.3 around the SMPL-X head keypoint, using the additional prompt “face of”.The face-camera radius is [0.3, 0.4] and its rotation range is [−90°, 90°].Source: D. Camera Sampling · p. 12

E.1. Network Structure

TeCH uses separate networks for geometry and albedo texture, then optimizes them in geometry and texture stages. The appendix also reports qualitative comparisons across datasets and in-the-wild images.

E.1. Network Structure

  • The geometry network predicts signed distance values at tetrahedral-grid vertices using a two-layer MLP and hash positional encoding.The encoding has 16 resolution levels and maximum resolution 1028.Source: §E.1 · p. 12
  • The albedo network queries color in canonical A-pose space, using a one-layer MLP and hash positional encoding with maximum resolution 2048.The mesh is registered with SMPL-X and reposed to the standard A-pose for this representation.Source: §E.1 · p. 12

E.2. Optimization Details

  • Both optimization stages use Adam with base learning rate 1 × 10−3 and weight decay 5 × 10−4.Source: §E.2 · p. 12
  • Geometry optimization runs coarse-to-fine for 5,000 steps per stage, with mesh subdivision in the fine stage and a two-round cosine schedule for the normal-loss weight.The schedule begins with a higher normal-loss weight and gradually decreases it to support robust convergence.Source: §E.2 · p. 12
  • Texture optimization runs for 7,000 steps, with color-consistency enforcement during its final 2,000 iterations.Source: §E.2 · p. 12
  • Qualitative comparisons report better performance on challenging poses in CAPE and across hard poses, complex textures, and loose clothing in THuman2.0.On in-the-wild SHHQ images, TeCH is reported to recover overall clothed-body structure and realistic texture consistent with clothing color patterns and material.Source: Figure 12 · p. 13, Figure 13 · p. 14, Figure 14 · p. 15

Related papers

Loading 2308.08545v2…