Source-linked AI summary
ARCH: Animatable Reconstruction of Clothed Humans
Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, Tony Tung
TL;DR
Single-image reconstruction of detailed, animation-ready clothed humans remains difficult because existing approaches struggle with pose variation, detail recovery, and automatic rigging. ARCH canonicalizes arbitrary poses with Semantic Space and Semantic Deformation Field representations, learns implicit geometry and appearance, and refines results through opacity-aware differentiable rendering. It reports over 50% average improvement over PIFu on RenderPeople and over 60% on BUFF, while producing avatars that can be animated.
Problem
Detailed 3D clothed-human reconstruction from unconstrained single images remains challenging, and existing approaches do not reliably handle pose variation, recover details, or produce animation-ready models.
Method
ARCH uses Semantic Space and Semantic Deformation Field representations to canonicalize arbitrary poses, then learns implicit surface geometry and appearance with differentiable rendering refinement.
Results
ARCH reports over 50% average improvement over PIFu on RenderPeople and over 60% on BUFF, with further gains from Granular Render-and-Compare.
Takeaways & Limitations
The resulting clothed-human models are animation-ready, can be driven by arbitrary motion sequences, and support relighting through estimated albedo.
Takeaways & Limitations
Heavy occlusion in in-the-wild images remains a future challenge, and unusual poses absent from training data can produce inaccurate body priors.
Abstract
from arXiv · showhide
In this paper, we propose ARCH (Animatable Reconstruction of Clothed Humans), a novel end-to-end framework for accurate reconstruction of animation-ready 3D clothed humans from a monocular image. Existing approaches to digitize 3D humans struggle to handle pose variations and recover details. Also, they do not produce models that are animation ready. In contrast, ARCH is a learned pose-aware model that produces detailed 3D rigged full-body human avatars from a single unconstrained RGB image. A Semantic Space and a Semantic Deformation Field are created using a parametric 3D body estimator. They allow the transformation of 2D/3D clothed humans into a canonical space, reducing ambiguities in geometry caused by pose variations and occlusions in training data. Detailed surface geometry and appearance are learned using an implicit function representation with spatial local features. Furthermore, we propose additional per-pixel supervision on the 3D reconstruction using opacity-aware differentiable rendering. Our experiments indicate that ARCH increases the fidelity of the reconstructed humans. We obtain more than 50% lower reconstruction errors for standard metrics compared to state-of-the-art methods on public datasets. We also show numerous qualitative examples of animated, high-quality reconstructed avatars unseen in the literature so far.
1. Introduction
ARCH addresses the challenge of reconstructing detailed, animation-ready clothed humans from unconstrained single images. It canonicalizes arbitrary poses, learns detailed implicit geometry and appearance, and produces rigged avatars that outperform prior methods.
- 3D human reconstruction from in-the-wild images remains challenging because accurate sensor-based methods require controlled environments.
- Existing representations trade off body structure, topology, resolution, or animation readiness, while automatic rigging remains unavailable.
- ARCH uses body pose and shape estimation to normalize clothed humans into a canonical space with Semantic Space and Semantic Deformation Field representations.These representations are built by densely sampling points around the clothed body surface and assigning skinning weights.
- Its implicit occupancy representation reconstructs high-frequency clothing and hair details, while differentiable rendering refines normal and color information.The framework combines correspondence estimation, canonical-space reconstruction, and rendering-based refinement.
- ARCH outperforms state-of-the-art reconstruction methods on public 3D scan benchmarks and in-the-wild 2D images, with reconstructed clothed humans animated by motion-capture data.
2. Related Work
Prior clothed-human reconstruction methods use richer inputs or single-image body estimators but remain limited in deployment cost and clothing-detail recovery. ARCH instead canonicalizes arbitrary poses and learns an implicit representation with opacity-aware differentiable rendering.
- Video, RGB-D, and multi-view methods provide richer information than single images but impose hardware and deployment-time costs.
- Single-image parametric body estimation has advanced, but the extent of reconstructable 3D clothing detail remains under-explored.
- ARCH transforms people in arbitrary poses into canonical space using body-prior knowledge before learning an implicit representation.
- The proposed differentiable-rendering formulation revisits continuous rendering to handle opacity while optimizing geometric primitives at points of interest.
3. Proposed Framework
ARCH reconstructs clothed humans in a canonical space using semantic deformation, then learns implicit occupancy, normals, and color for detailed, rigged avatars. Differentiable rendering supplies image-level supervision, while skinning weights and a skeleton support animation across poses.
- Semantic Space and Deformation Field: ARCH uses Semantic Space and Semantic Deformation Field to transform arbitrary-pose clothed humans into a normalized canonical space.The field performs inverse skinning, extending the transformation beyond structured surfaces to arbitrary 3D points.
- Semantic Space and Deformation Field: The canonical representation associates 3D points with skinning weights tied to body parts and a skeletal rig.Weights are approximated from the closest template-surface point, with distant points removed using a threshold.
- Implicit Surface Reconstruction: ARCH implicitly represents clothed-human geometry with an occupancy map and extracts a surface by thresholding occupancy values.The network takes canonical 3D points, corresponding 2D positions, and the input image to estimate occupancy, normals, and color.
- Implicit Surface Reconstruction: Separate occupancy, normal, and color sub-networks combine spatial features with pixel-aligned image features to predict the human representation.The sub-networks use image encoders and MLP regressors, with normal and color estimation receiving features from earlier predictions.
- Training: Training jointly optimizes occupancy, normal, and color models with 3D and rendered 2D losses, including pixel-wise supervision through differentiable rendering.Opacity-aware rendering is designed to remain numerically stable while optimizing per-point color and normal values.
- Inference: The skinning-equipped implicit representation can be warped to arbitrary poses, producing detailed avatars that can be animated with unseen motion sequences.This animation capability follows from embedding skinning weights and a skeleton rig in the representation.
4. Experiments
ARCH is evaluated on public 3D scan benchmarks and in-the-wild images using quantitative comparisons, ablations, and qualitative demonstrations of reconstruction and animation. Results show improved fidelity, detailed clothing reconstruction, and direct avatar animation, while rare poses and occluded details remain challenging.
- Datasets and Metrics: The evaluation compares normal, P2S, and Chamfer errors against posed ground-truth surfaces on RenderPeople and BUFF.Lower values are better, and reconstructions are reposed to the input images' original poses before comparison.
- Evaluation Design: The experiments include state-of-the-art comparisons and ablations of Semantic Deformation Field and Granular Render-and-Compare.The ablations separately evaluate a direct implicit reconstruction baseline, SemDF, and differentiable render-and-compare refinement.
- Animation: A predicted avatar from one frontal view is skinned and driven by off-the-shelf motion-capture data, with plausible unseen hair and dress-back geometry.Because reconstruction occurs in canonical space, the avatar is automatically rigged and can be directly animated from a single view.
- Qualitative Reconstruction: ARCH reconstructs larger wrinkles and clothing seams geometrically, while predicted normals reproduce finer wrinkles and support plausible rendered appearance.The examples include details such as pants and shoe seams, although the zipper is not reproduced in the predicted normal map.
- Quantitative Results: Adding SemDF improves average performance by over 50% on RenderPeople and over 60% on BUFF compared with PIFu metrics.Adding Granular Render-and-Compare improves the numbers slightly further, especially for normal estimation, while increasing visual fidelity and removing artifacts.
- Limitations: Rare poses insufficiently represented in training, such as kneeling, produce inaccurate body priors and challenging reconstructions; occluded normals also need improved treatment.The authors identify occlusion-aware estimation as a potential way to improve details in occluded areas.
5. Conclusion
ARCH reconstructs clothed humans from unconstrained photos by using semantic canonicalization and render-and-compare supervision. The resulting models provide detailed, animation-ready avatars, while heavy occlusion remains future work.
- ARCH uses Semantic Space and Semantic Deformation Field representations to reconstruct clothed humans from arbitrary poses.
- ARCH produces higher-fidelity clothing details, including pixel-aligned colors and normals, across a wider range of body configurations.
- The reconstructed models are animation-ready and can be driven by arbitrary motion sequences.
- Heavy occlusion cases in in-the-wild images remain an area for future exploration.