Source-linked AI summary
ICON: Implicit Clothed humans Obtained from Normals
Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, Michael J. Black
TL;DR
Existing methods struggle to reconstruct detailed clothed humans from unconstrained images because their global features are pose-sensitive and their training data are limited. ICON uses SMPL-guided local features, implicit surfaces, and iterative refinement, then combines reconstructed frames into avatars; it outperforms prior methods with limited data and generalizes to challenging inputs.
Problem
Prior implicit-function methods often produce broken or disembodied limbs, missing details, noise, or non-human shapes, while existing datasets and global features limit robustness to varied poses.
Method
ICON combines SMPL-guided clothed-normal prediction, a visibility-aware implicit surface regressor, iterative SMPL-normal refinement, and modified SCANimate for avatar construction.
Results
ICON outperforms state-of-the-art methods, reaches state-of-the-art performance with 12% of the training data, and generalizes better to in-the-wild and out-of-distribution inputs.
Takeaways & Limitations
The method supports robust clothed-human reconstruction from in-the-wild images and avatar creation from monocular video frames with personalized pose-aware clothing deformation.
Takeaways & Limitations
Loose clothing far from the body, substantial body-fit errors, and strong perspective effects can cause reconstruction difficulties or failure.
Abstract
from arXiv · showhide
Current methods for learning realistic and animatable 3D clothed avatars need either posed 3D scans or 2D images with carefully controlled user poses. In contrast, our goal is to learn an avatar from only 2D images of people in unconstrained poses. Given a set of images, our method estimates a detailed 3D surface from each image and then combines these into an animatable avatar. Implicit functions are well suited to the first task, as they can capture details like hair and clothes. Current methods, however, are not robust to varied human poses and often produce 3D surfaces with broken or disembodied limbs, missing details, or non-human shapes. The problem is that these methods use global feature encoders that are sensitive to global pose. To address this, we propose ICON ("Implicit Clothed humans Obtained from Normals"), which, instead, uses local features. ICON has two main modules, both of which exploit the SMPL(-X) body model. First, ICON infers detailed clothed-human normals (front/back) conditioned on the SMPL(-X) normals. Second, a visibility-aware implicit surface regressor produces an iso-surface of a human occupancy field. Importantly, at inference time, a feedback loop alternates between refining the SMPL(-X) mesh using the inferred clothed normals and then refining the normals. Given multiple reconstructed frames of a subject in varied poses, we use SCANimate to produce an animatable avatar from them. Evaluation on the AGORA and CAPE datasets shows that ICON outperforms the state of the art in reconstruction, even with heavily limited training data. Additionally, it is much more robust to out-of-distribution samples, e.g., in-the-wild poses/images and out-of-frame cropping. ICON takes a step towards robust 3D clothed human reconstruction from in-the-wild images. This enables creating avatars directly from video with personalized and natural pose-dependent cloth deformation.
1. Introduction
ICON targets detailed, animatable clothed-human reconstruction from single RGB images in arbitrary poses, addressing artifacts and pose sensitivity in prior implicit-function methods. It combines local features, SMPL guidance, iterative refinement, and SCANimate-based avatar construction.
- Existing implicit-function methods capture detailed topology but often produce broken limbs, missing details, noise, or non-human shapes on in-the-wild data.
- Prior methods are limited by small, curated datasets and global image or voxel features that are sensitive to global pose.
- ICON uses local features and SMPL guidance to infer front/back clothed-human normals and a visibility-aware implicit surface from a segmented RGB image.
- ICON alternates between refining SMPL shape, pose, and translation from inferred normals and refining the normals to improve the reconstructed implicit shape.
- ICON reaches state-of-the-art performance with as little as 12% of the training data while generalizing better to in-the-wild images, unusual poses, clothing, and out-of-frame cropping.
- Meshes reconstructed from video frames are used with visibility-aware SCANimate modifications to produce a clothed avatar that moves and deforms naturally.
2. Related work
Related work contrasts easily animated statistical body meshes with expressive but less regularized implicit functions. ICON combines both representations while optimizing the body model in the loop and using local features to improve pose robustness.
- Mesh-based statistical models are well regularized and compatible with graphics pipelines, but body meshes generally lack clothing and body-to-cloth offsets poorly model skirts and dresses.
- Deep implicit functions represent detailed shapes with arbitrary topology, yet prior clothed-human methods often generate broken limbs, missing details, or geometric noise.
- Hybrid methods combine statistical models and implicit functions, but canonicalization can require accurate body fits and alter clothing details through undressed-model skinning weights.
- ICON combines SMPL with an implicit function, optimizes SMPL during inference, and replaces global body features with local features that are agnostic to global pose variations.
- With heavily limited data, ICON achieves state-of-the-art performance and robustness to out-of-distribution poses, linking monocular reconstruction with scan- and depth-based avatar modeling.
3. Method
ICON reconstructs clothed humans using SMPL-guided normal prediction and local-feature implicit surface reconstruction. A feedback loop refines the body fit and normals, while pose-independent local features improve robustness to unusual poses.
- 3. Method: ICON takes a segmented RGB image and an estimated SMPL body to predict clothed-body normals and reconstruct a pixel-aligned 3D shape.Its two modules perform SMPL-guided normal prediction and local-feature implicit surface reconstruction.
- 3.1. Body-guided normal prediction: SMPL body normals are rendered from opposite views, and separate normal networks predict front- and back-side clothed-body normals.The predicted normals use the color image together with rendered SMPL-body normal maps.
- 3.1. Body-guided normal prediction: During inference, ICON optimizes SMPL shape, pose, and translation against predicted clothed-body normals, then alternates SMPL refinement with normal re-inference.This feedback loop produces more reliable normal maps for both front and back sides.
- 3.2. Local-feature based implicit 3D reconstruction: The implicit regressor uses local features comprising signed distance, barycentric body normals, and visibility-selected clothed-body normals at each query point.Signed distance and barycentric surface normals regularize the reconstruction against self-occlusions.
- 3.2. Local-feature based implicit 3D reconstruction: Because the local feature F_P is independent of global body pose, experiments identify it as key to robustness on out-of-distribution poses and training-data efficacy.An MLP estimates occupancy from F_P, and a surface-localization algorithm extracts the reconstructed mesh.
4. Experiments
ICON outperforms state-of-the-art reconstruction methods across training-data scales and generalizes better to out-of-distribution poses. Ablations show benefits from SMPL-guided normals, local features, and the inference feedback loop.
- Evaluation: The evaluation uses Chamfer, P2S, and normal differences to measure large-scale geometry, scan-to-surface distance, and high-frequency details.Chamfer can miss smaller geometric details, while normal differences capture them when Chamfer and P2S errors are small.
- ICON -vs- SOTA: ICON generalizes well to out-of-distribution poses, whereas PaMIR* remains sensitive to global body pose despite ground-truth SMPL-X conditioning.PaMIR* performs better on in-distribution poses but fails to generalize comparably on CAPE-NFP.
- Body-guided normal prediction: Body-normal conditioning substantially improves inferred normals, especially in occluded body regions.Without conditioning, CAPE errors increase slightly, while qualitative improvements are strongest for unseen regions.
- Local-feature based implicit reconstruction: Replacing ICON’s local features with global convolutional features worsens reconstruction on out-of-distribution poses.The global model uses a receptive field spanning 46% of the image and becomes sensitive to global body pose.
- Dataset size: ICON outperforms all methods consistently as training data ranges from 56 to 3,709 scans, achieving state-of-the-art performance even with limited data.The authors attribute this to local point features that improve pose-space generalization and data efficiency.
- Robustness to SMPL-X noise: ICON remains effective with perturbed SMPL-X input because the feedback refinement loop improves performance.With noisy SMPL-X, ICON+BR performs comparably to PaMIR* using ground-truth SMPL-X.
5. Applications
ICON reconstructs clothed humans from in-the-wild images and supports animatable avatar creation from multiple frames. Its perceptual realism is evaluated against competing reconstruction methods.
- In-the-wild reconstruction: ICON handles 200 unseen in-the-wild images depicting parkour, sports, street dance, and kung fu.The images were collected from Pinterest and were not used during training.
- Perceptual evaluation: In a perceptual study, participants compared ICON with PIFu*, PaMIR*, and PIFuHD on how well reconstructions represented human shape.The study reports the percentage of trials in which participants preferred competing methods over ICON.
- Limitations: ICON’s failure cases occur with extreme clothing, poses, or camera views.The paper visualizes front and rotated views for these cases.
- Perceptual evaluation: ICON is judged significantly more realistic than the competing methods in the in-the-wild perceptual study.Table 3 reports the chance that participants preferred a competing reconstruction over ICON.
- Animatable avatars: SCANimate combines ICON reconstructions from multiple poses into animatable avatars with pose-dependent clothing deformation.The pipeline reconstructs one clothed-human mesh per frame before feeding the meshes to SCANimate.
6. Conclusion
ICON robustly recovers detailed 3D clothed humans from single images and produces results sufficient for avatar construction from monocular image sequences. Its body-model regularization, iterative optimization, and local features support accuracy, realism, and robustness, while limitations remain for loose clothing, failed body fits, and strong perspective effects.
- ICON combines 3D body-model regularization with iterative body-model optimization and local features to improve reconstruction accuracy and realism.
- Loose clothing far from the body may be difficult to reconstruct, and significant body-fit failures lead to reconstruction failure.
- Training on orthographic views causes trouble with strong perspective effects, including asymmetric limbs or anatomically improbable shapes.
- As virtual-human technology matures, it may enable full-body deep fakes, creating risks that require legal boundaries.
Appendices
The appendix extends the main paper with additional methodological, experimental, quantitative, and qualitative details.
- The appendix provides extended method and experiment details along with additional quantitative and qualitative results.
A. Method & Experiment Details
The appendix examines dataset scale, simulated body-fit perturbations, SMPL refinement, and perceptual evaluation of in-the-wild reconstructions and normal maps.
- Dataset size: The combined AGORA and THuman training set contains 3,709 scans, eight times the 450 Renderpeople scans used by prior methods.
- Dataset and perturbations: The simulated perturbation uses pose and shape scales sθ = 0.15 and sβ = 0.5 with noise parameter µ ∈ [−1, 1].
- Dataset limitations: The CLOTH3D++ domain gap from real images remains large, and training implicit functions on multi-layer non-watertight meshes is left for future research.
- SMPL refinement: LN diff + LS diff achieves the smallest Chamfer-distance error across all tested pose-noise levels during SMPL refinement.
- Perceptual studies: The in-the-wild reconstruction study evaluates ICON against PIFu, PIFuHD, and PaMIR on 200 unseen internet images.
- Reported results: The appendix includes quantitative-error results for ICON variants conditioned on perturbed SMPL-X fits and perceptual-study results for normal prediction.
- Perceptual studies: Twenty-eight of 36 participants remained valid after the perceptual-study catch-trial filter.
- Normal-map prediction: Body-prior normal prediction usually produces better back-side normal maps, while front-side predictions are generally similar.
A.4. Implementation details (Sec. 4.1)
Implementation details cover network architecture, feature dimensions, training, inference iterations, occupancy resolution, and receptive-field analysis.
- Network architecture: The body-guided normal-prediction network uses PIFuHD’s residual architecture, while image encoders use two-stack modified hourglass networks.
- Network architecture: Feature tables distinguish pixel and point dimensions, with MLP layer widths specified for the evaluated approaches.
- Training: GN excludes THuman because of low-quality texture, whereas IF trains on both AGORA and THuman.
- Inference: Inference performs 50 SMPL and normal-map refinement iterations, each taking approximately 460 ms on a Quadro RTX 5000 GPU.
- Inference: The queried occupancy space has resolution 256^3, and in-the-wild humans are segmented with rembg.
- Receptive-field analysis: Reducing PaMIR’s receptive field alone does not improve performance, indicating that informative 3D features and normal maps also matter.
B. More Quantitative Results (Sec. 4.3)
The extended quantitative results examine reconstruction error as training-data size changes and compare ICON with ground-truth SMPL-X baselines.
- Table 4 compares ICON variants conditioned on perturbed SMPL-X meshes, while Table 9 extends the training-data-size analysis from Figure 6.
- Table 9 reports reconstruction error in centimeters across different training-data sizes, with “Training set scale” measured relative to 450 scans.The 8x setting uses all 3,709 AGORA and THuman scans.
- 1.158 cm Chamfer and 1.125 cm P2S are the ground-truth SMPL-X results reported for comparison in Table 2.
C. More Qualitative Results (Sec. 5)
The qualitative results compare ICON with state-of-the-art methods on in-the-wild images, multiple viewpoints, and out-of-frame cropping, while also showing representative failures and animation examples.
- In-the-wild comparisons: Figures 14–16 render in-the-wild reconstructions from four viewpoints, with surface normals color coded.
- Out-of-frame cropping: Figure 17 compares ICON with state-of-the-art methods on images containing out-of-frame cropping.
- Failures and animation: Figure 18 shows additional representative ICON failures, while the accompanying video demonstrates animation examples created with ICON and SCANimate.
- Additional evaluations: Figures 10–13 present representative dataset poses and perceptual or qualitative evaluations of in-the-wild reconstructions and the body prior for normal prediction.