PaperlayerView exact revision on arXiv

Source-linked AI summary

ECON: Explicit Clothed humans Optimized via Normal integration

Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, Michael J. Black

arXiv:2212.07422v2cs.CVcs.AIcs.GR

TL;DR

Clothed-human reconstruction must handle both novel poses and clothing that departs from body topology, while existing implicit and explicit approaches each compromise one of these goals. ECON combines body-guided regularization with detailed 2D normal-map reconstruction, front/back d-BiNI surfaces, and shape completion. On CAPE, Renderpeople, and in-the-wild images, it is reported as quantitatively superior and perceptually preferred, especially for challenging poses and loose clothing.

  • Problem

    Existing implicit methods can fail on novel poses, while explicit body regularization can limit recovery of free-form clothing such as loose garments.

  • Method

    ECON predicts front and back normal maps, lifts them into body-registered d-BiNI surfaces, and inpaints missing geometry using SMPL-X-conditioned IF-Nets and stitching.

  • Results

    ECON is reported as quantitatively superior on CAPE and Renderpeople and significantly preferred perceptually on challenging poses and loose clothing.

  • Takeaways & Limitations

    ECON recovers detailed clothed humans with both pose robustness and topological flexibility, including loose clothing and geometric details.

  • Takeaways & Limitations

    ECON depends on an estimated SMPL-X body and predicted normal maps; failures in either can cause reconstruction failures or intersecting surfaces.

Abstract

from arXiv · show

The combination of deep learning, artist-curated scans, and Implicit Functions (IF), is enabling the creation of detailed, clothed, 3D humans from images. However, existing methods are far from perfect. IF-based methods recover free-form geometry, but produce disembodied limbs or degenerate shapes for novel poses or clothes. To increase robustness for these cases, existing work uses an explicit parametric body model to constrain surface reconstruction, but this limits the recovery of free-form surfaces such as loose clothing that deviates from the body. What we want is a method that combines the best properties of implicit representation and explicit body regularization. To this end, we make two key observations: (1) current networks are better at inferring detailed 2D maps than full-3D surfaces, and (2) a parametric model can be seen as a "canvas" for stitching together detailed surface patches. Based on these, our method, ECON, has three main steps: (1) It infers detailed 2D normal maps for the front and back side of a clothed person. (2) From these, it recovers 2.5D front and back surfaces, called d-BiNI, that are equally detailed, yet incomplete, and registers these w.r.t. each other with the help of a SMPL-X body mesh recovered from the image. (3) It "inpaints" the missing geometry between d-BiNI surfaces. If the face and hands are noisy, they can optionally be replaced with the ones of SMPL-X. As a result, ECON infers high-fidelity 3D humans even in loose clothes and challenging poses. This goes beyond previous methods, according to the quantitative evaluation on the CAPE and Renderpeople datasets. Perceptual studies also show that ECON's perceived realism is better by a large margin. Code and models are available for research purposes at econ.is.tue.mpg.de

1. Introduction

ECON combines explicit body regularization with implicit-function flexibility by reconstructing detailed front and back surfaces before completing missing geometry. It is designed for realistic clothed humans across novel poses, loose clothing, and complex topology, and is reported to outperform prior methods quantitatively and perceptually.

  • Motivation: Existing implicit methods capture clothing detail but can produce disembodied limbs or degenerate shapes in novel poses, while body-regularized methods can over-constrain loose clothing.The motivation is to combine pose robustness with free-form clothing topology.
  • Method: ECON takes an RGB image and an image-estimated SMPL-X body as input, then reconstructs a detailed free-form clothed human.ECON stands for Explicit Clothed humans Optimized via Normal integration.
  • Step 1: Front & back normal reconstruction: It predicts front- and back-side clothed-human normal maps conditioned on the estimated body.A standard image-to-image translation network performs this first step.
  • Step 2: Front & back surface reconstruction: It converts the predicted normals and SMPL-X-rendered depths into coherent detailed front and back d-BiNI surfaces.The optimization aligns high-frequency clothing detail with normals and low-frequency structure, discontinuities, and silhouettes with SMPL-X.
  • Step 3: Full 3D shape completion: ECON completes missing geometry by conditioning IF-Nets on SMPL-X and stitching the resulting infilling patches to the two d-BiNI surfaces.The body regularizes infilling, while Poisson reconstruction joins surfaces across small remaining holes.
  • Results: Quantitative evaluation on CAPE and Renderpeople and perceptual studies report ECON as superior or significantly preferred on challenging poses and loose clothing.Qualitative results also report better generalization across varied poses, extreme looseness, and complex topology.
  • Conclusion: ECON combines pose robustness and topological flexibility to recover detailed clothed humans with realistic poses.Its reconstruction is not over-constrained to body topology, allowing loose clothing and geometric detail.

2. Related Work

Prior clothed-human reconstruction methods use explicit representations for body structure or implicit representations for arbitrary topology, with each family retaining important limitations. ECON is situated among hybrid approaches that combine parametric body models with expressive implicit representations and among front/back surface reconstruction methods.

  • Explicit-shape-based approaches: Explicit approaches use parametric bodies, depth maps, or point clouds, often adding clothing offsets to body meshes.Body-based methods remain compatible with animation pipelines through inherited skeletons and skinning weights.
  • Explicit-shape-based approaches: Body-offset representations struggle with clothing that deviates substantially from body topology, such as dresses and skirts.Cloth-aware models improve flexibility but are difficult to scale across many clothing styles and in-the-wild outfit variation.
  • Implicit-function-based approaches: Implicit-function approaches represent arbitrary clothed-human topology, but learned clothing models may misalign with images and lack geometric detail.These methods commonly estimate a parametric body and optimize a clothing latent space.
  • Implicit-function-based approaches: PIFu and PIFuHD improve pixel-aligned reconstruction and geometric detail, yet lack human-body structure and can fail on novel poses with broken or disembodied limbs.Their pose generalization problem is linked in the passage to overfitting training-data poses.
  • Hybrid representations: Hybrid methods condition implicit representations on parametric body models or transform posed queries into canonical space.Examples include PaMIR, DeepMultiCap, JIFF, ARCH, ARCH++, and CAR.
  • Front/back surface reconstruction: Sandwich-like monocular methods estimate visible and invisible depth or use normal losses to improve front/back reconstruction.Moduling Humans, FACSIMILE, and Any-Shot GIN are cited as representative approaches.

3. Method

ECON reconstructs clothed humans by predicting front and back normal maps, lifting them into d-BiNI partial surfaces, and completing missing geometry with body-guided implicit inference and Poisson stitching.

  • 3.1. Detailed normal map prediction: ECON predicts front- and back-side clothed-human normal maps from RGB, conditioned on normal maps rendered from an estimated SMPL-X body.The body estimate is additionally optimized using normal-map, silhouette, and 2D joint losses.
  • 3.2. Front and back surface reconstruction: d-BiNI jointly reconstructs detailed front and back 2.5D surfaces from clothing normals and SMPL-X depth maps.Its objective combines bilateral normal integration, depth-prior, and front-back silhouette-consistency terms.
  • 3.2. Front and back surface reconstruction: The depth prior regularizes global front-back offsets by encouraging clothed surfaces to remain consistent with the coarse SMPL-X body.This resolves a coherence problem left by normal-integration terms alone.
  • 3.2. Front and back surface reconstruction: A silhouette-consistency term aligns front and back depths at the clothing silhouette, preventing intersections that otherwise create blobby artifacts.The term is computed on the silhouette domain ∂Ωn.
  • 3.3. Human shape completion: For self-occluded poses, ECON completes missing geometry using SMPL-X-guided implicit infilling and stitches the resulting patches with d-BiNI surfaces via Poisson reconstruction.The SMPL-X body regularizes infilling while preserving clothing surfaces; ECONEX instead uses visible-body triangle soup with Poisson reconstruction, which can remain incoherent for missing clothing and hair.
  • 3.3. Human shape completion: The final reconstruction may optionally replace noisy face and hand regions with corresponding geometry from SMPL-X.The raw reconstruction and the body-replaced version are distinguished in the completion pipeline.
  • 3.3. Human shape completion: Lossy voxelization and limited-resolution Marching cubes can smooth details from the optimized d-BiNI surfaces, although ECON variants preserve them better than the implicit completion output.The evaluation uses ECONIF and ECONEX rather than the intermediate RIF output.

4. Experiments

ECON is evaluated on established benchmarks and in-the-wild images, where quantitative, perceptual, and qualitative results support its robustness to challenging poses and clothing.

  • Evaluation setup: ECON is evaluated on CAPE, Renderpeople, and in-the-wild images using geometric, normal-based, and perceptual measures.The benchmark comparison includes body-agnostic and body-aware methods, while the perceptual study covers challenging poses, loose clothing, and fashion images.
  • Quantitative evaluation: ECONEX matches ICON on standard performance and achieves below 1cm distance error on out-of-distribution poses in CAPE.For out-of-distribution outfits in Renderpeople, ECONEX/IF performs on par with PaMIR and much better than PIFuHD.
  • Quantitative evaluation: ECONEX achieves state-of-the-art normal performance on both CAPE and Renderpeople, measuring high-frequency reconstruction details.The reported normal difference uses rendered normal images to assess local detail and projection consistency.
  • Perceptual study: ECON is significantly preferred over PIFuHD and outperforms ICON for challenging poses, while it is preferred over ICON by a large margin for loose clothing.On fashion images, PIFuHD receives a slight preference over ECON.
  • Ablation study: d-BiNI improves reconstruction accuracy by about 50% over BiNI and runs 33% faster, while IF-Nets+ improves robustness to out-of-distribution poses.The ablation attributes d-BiNI’s improvement to coarse-body regularization and front-back surface consistency, and conditions IF-Nets+ on SMPL-X.
  • Additional results: ECON reconstructs multiple detailed clothed humans despite inter-person occlusions, although the method was not trained for multiple people.Three examples show occluded parts successfully recovered.

5. Discussion

ECON generalizes to challenging poses, loose clothing, and occluded multi-person scenes, while its practical use remains bounded by reconstruction and input-quality limitations.

  • Limitations: ECON depends on an estimated SMPL-X body, so errors such as bent legs or wrong limb poses can propagate into reconstruction failures.The paper identifies single-image SMPL-X recovery as an unsolved problem.
  • Limitations: Poor normal maps can produce overly close or intersecting front and back surfaces, degrading ECON’s reconstruction quality.These artifacts are illustrated in the cited failure examples.
  • Future work: ECON currently reconstructs geometry only; future extensions target animatable, fully textured, and separately editable avatars.Proposed directions include skeleton and skinning recovery, back-view texture, and disentangling clothing, hair, and accessories.
  • Generalization: Qualitative results show detailed reconstructions for challenging poses and loose clothing, including extreme looseness and complex topology.The in-the-wild examples show front and rotated views of reconstructed humans.
  • Occlusions: ECON can reconstruct multiple detailed clothed humans with inter-person occlusions despite not being trained on multiple people.The shape completion module recovers the occluded parts in the shown examples.

6. Conclusion

ECON combines detailed 2.5D front and back surfaces with an underlying parametric body to reconstruct clothed humans that are robust to novel poses and flexible clothing topology.

  • Conclusion: ECON combines estimated 2.5D front and back surfaces with an underlying 3D parametric body for clothed-human reconstruction.It extends variational normal integration and shape completion to image-based 3D human reconstruction.
  • Conclusion: ECON is robust to novel poses while recovering loose clothing and geometric details without over-constraining the reconstruction to body topology.The conclusion attributes this combination to the method’s use of recent normal-integration and shape-completion advances.
  • Resources: Code and models are available for research purposes at econ.is.tue.mpg.de.

A.1. Normal map prediction

The normal-map prediction setup adapts inference and optimization to loose clothing and occlusion conditions while using standard software components for segmentation and reconstruction.

  • Inference settings: For loose clothing, ECON increases the 2D-joint loss weight from 5.0 to 50.0 when clothing-body overlap falls below 0.5.This adjustment gives greater trust to 2D joints in loose-clothing cases.
  • Inference settings: For occlusions, ECON sets the silhouette-loss weight to 0.0 when the specified body-mask overlap falls below 0.98.The stated purpose is to avoid limb self-intersection.
  • Inference settings: During inference, ECON iteratively refines SMPL-X and clothed-body normals for 50 iterations at 1.10 iterations per second on a Quadro RTX 5000 GPU.
  • Implementation: The implementation uses multi-person segmentation, full-body landmarks, Poisson surface reconstruction, implicit surface queries, and marching cubes.The cited components include Mask R-CNN, Mediapipe, Open3D, MonoPort, and PyTorch3D.

A.2. d-BiNI

d-BiNI jointly reconstructs front and back clothed depth surfaces from normal maps while enforcing depth-prior and silhouette-consistency constraints.

  • A.2. d-BiNI: d-BiNI jointly solves for front and back clothed depths by minimizing an objective with normal, depth-prior, and silhouette-consistency terms.The formulation uses front and back normal-map constraints together with prior-depth agreement and cross-surface consistency.
  • A.2. d-BiNI: With fixed weights, d-BiNI reduces to a convex least-squares problem and updates the weights iteratively from the resulting depth.The optimization alternates between solving for depths and recomputing weights.
  • A.2. d-BiNI: Equation (S.5) is solved as a large-scale sparse linear system using a CUDA-accelerated conjugate-gradient solver with Jacobi preconditioning.Its coefficient matrix is symmetric positive definite.
  • A.2. d-BiNI: The depth-prior term controls agreement between d-BiNI surfaces and the SMPL-X mesh, while silhouette consistency couples the front and back surfaces.The hyperparameters λd and λs separately control these two influences.
  • A.2. d-BiNI: The chosen d-BiNI settings are λd = 1e−4, λs = 1e−6, and k = 2.k controls surface stiffness, whereas λd and λs control depth-prior and silhouette-consistency influence.
  • A.2. d-BiNI: Small k produces tougher surfaces with unrecovered discontinuities, whereas large k introduces redundant discontinuities and noisy artifacts.Similarly, overly small λd causes misalignment, while overly large λd smooths high-frequency normal details.

A.3. IF-Nets+

IF-Nets+ combines multiscale voxelized representations of d-BiNI and SMPL-X surfaces with an implicit occupancy regressor.

  • A.3. IF-Nets+: IF-Nets+ encodes voxelized d-BiNI and SMPL-X surfaces using multi-scale 3D CNN feature grids.The features capture both local and global information across six resolutions.
  • A.3. IF-Nets+: An MLP implicit-function regressor consumes the multiscale features and predicts each point’s occupancy value using BCE loss.The feature grids use decreasing spatial resolutions and variable channel dimensions.
  • A.3. IF-Nets+: Both IF-Nets and IF-Nets+ voxelize SMPL-X and d-BiNI surfaces at 256^3 resolution and train for 20 epochs on an NVIDIA A100.Training uses RMSprop, learning rate 1e−4, and batch size 48.
  • A.3. IF-Nets+: THuman2.0 augmentation produces 18,900 samples by yaw rotation and randomly erases d-BiNI depth-map regions with probability p = 0.8.The rotations use 10-degree increments, and erased-region proportions are 0.01, 0.05, or 0.2.
  • A.3. IF-Nets+: For one image, ECONIF takes 112 seconds and ECONEX takes 97 seconds, compared with 78 seconds for ICON without cloth refinement.d-BiNI takes 6.2 seconds, while ECONIF’s Poisson step takes 10.7 seconds.

B. Qualitative results

The qualitative evaluation compares ECON with prior methods across reconstructed views, clothing conditions, poses, and intermediate processing choices.

  • B. Qualitative results: Qualitative comparisons show ECON, PaMIR, ICON, and PIFuHD on in-the-wild images with challenging poses, loose clothing, and fashion poses.Each example is rendered from virtual-camera rotations of 0°, 90°, 180°, and 270°.
  • B. Qualitative results: On SHHQ examples, ECON reconstructions are shown from front and side views alongside SMPL-X fits.The comparison provides a visual reference between ECON’s reconstruction and the parametric body fit.
  • B. Qualitative results: ECON is compared with PaMIR on loose clothing, with front and back 3D details highlighted for closer inspection.The figure directs readers to zoom in when assessing clothing geometry.
  • B. Qualitative results: The supplementary ablations examine how k affects d-BiNI surface smoothness and how λd affects alignment with SMPL-X and retained detail.Silhouette consistency is presented as preventing front/back surface intersections that otherwise cause blobby Poisson-reconstruction artifacts.
Loading 2212.07422v2…