PaperlayerView exact revision on arXiv

Source-linked AI summary

ARCH++: Animation-Ready Clothed Human Reconstruction Revisited

Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, Tony Tung

arXiv:2108.07845v4cs.CVcs.GR

TL;DR

Image-based clothed-human reconstruction must balance posed-space fidelity, canonical-space completeness, and detailed, view-consistent surfaces. ARCH++ addresses these issues with learned body-semantic encoding, joint-space occupancy co-supervision, and image-based surface refinement, reporting improvements over prior work on public benchmarks and user studies.

  • Problem

    Existing image-based avatar methods have limitations in body-semantic representation, occupancy estimation across topology-changing spaces, and detailed reconstruction of occluded regions.

  • Method

    ARCH++ combines a PointNet++ geometry encoder, cross-space co-supervision of posed and canonical occupancy, and image-to-image normal and texture refinement.

  • Results

    ARCH++ reports improved reconstruction quality and realism over prior work on public benchmarks and user studies.

  • Takeaways & Limitations

    The reconstructed avatars are reported to be highly realistic and animation-ready in visible and unseen regions, supporting arbitrary clothing styles.

  • Takeaways & Limitations

    The method assumes the input image can be directly used as the front texture map, and future work identifies environment information such as lighting as an unresolved limitation.

Abstract

from arXiv · show

We present ARCH++, an image-based method to reconstruct 3D avatars with arbitrary clothing styles. Our reconstructed avatars are animation-ready and highly realistic, in both the visible regions from input views and the unseen regions. While prior work shows great promise of reconstructing animatable clothed humans with various topologies, we observe that there exist fundamental limitations resulting in sub-optimal reconstruction quality. In this paper, we revisit the major steps of image-based avatar reconstruction and address the limitations with ARCH++. First, we introduce an end-to-end point based geometry encoder to better describe the semantics of the underlying 3D human body, in replacement of previous hand-crafted features. Second, in order to address the occupancy ambiguity caused by topological changes of clothed humans in the canonical pose, we propose a co-supervising framework with cross-space consistency to jointly estimate the occupancy in both the posed and canonical spaces. Last, we use image-to-image translation networks to further refine detailed geometry and texture on the reconstructed surface, which improves the fidelity and consistency across arbitrary viewpoints. In the experiments, we demonstrate improvements over the state of the art on both public benchmarks and user studies in reconstruction quality and realism.

1. Introduction

ARCH++ targets accessible, realistic, and animatable clothed-human avatars from images by revisiting limitations in prior reconstruction formulations. It introduces revised geometry, joint-space occupancy, and surface-refinement components, with reported gains on public benchmarks.

  • Motivation: Photorealistic digital humans are important for AR/VR applications, but manual authoring and controlled capture systems limit accessibility and increase cost.The motivation includes video games, social telepresence, and virtual try-on.
  • Prior limitations: Prior approaches trade off clothing fidelity and variation against intuitive animation: template methods struggle with clothing, while non-parametric methods lack direct animation mechanisms.ARCH is described as reconstructing non-parametric humans in canonical space.
  • Contributions: ARCH++ replaces hand-crafted 3D features with an end-to-end PointNet++ geometry encoder that describes the underlying human body.The encoder is intended to address depth ambiguity and missing human-body semantic information.
  • Contributions: ARCH++ jointly predicts occupancy in posed and canonical spaces with cross-space consistency, preserving posed-space detail while supporting canonical-space completeness.The method addresses topology changes introduced during unposing.
  • Contributions: ARCH++ refines occluded-region geometry and texture through image translation networks and moulding-inpainting, targeting sharper and more view-consistent avatars.The refinement is part of the method’s effort to improve surface fidelity across viewpoints.
  • Evaluation: Experiments on photorealistically rendered synthetic images and in-the-wild images report performance exceeding prior implicit-function methods and other design choices on public benchmarks.The supplied passage does not provide numerical benchmark values.

2. Related Work

Related work spans template-based, non-parametric, and hybrid reconstruction. These approaches combine different strengths but leave clothing variation and target-space choices as important challenges.

  • Template-based reconstruction: Template-based methods use parametric body models such as SCAPE and SMPL to provide shape and pose priors for ill-posed reconstruction problems.Extensions model clothing using displacements, external clothing templates, scans, videos, or single images.
  • Template-based reconstruction: Template-based clothing representations fail to capture real-world clothing variations because clothing topology can substantially diverge from the underlying body mesh.2D UV maps were proposed to address limited mesh detail, but topology and variation remain difficult.
  • Non-parametric reconstruction: Non-parametric capture supports highly detailed 3D shapes with arbitrary topology, including increasingly sparse-view and monocular settings.Single-view methods use representations including voxels, depth maps, visual hulls, and implicit functions.
  • Hybrid reconstruction: Hybrid approaches combine template-based structural priors with non-parametric arbitrary-topology representations, but some single-view methods choose the input view space for shape modeling.This choice motivates examining alternative target spaces for clothed-human reconstruction.

3. Proposed Methods

ARCH++ uses a coarse-to-fine pipeline: joint-space implicit reconstruction first, followed by mesh refinement in posed and canonical spaces. Its components encode body semantics, appearance, occupancy, normals, and textures.

  • Initial reconstruction: The initial reconstruction uses three components: a semantic-aware geometry encoder, a pixel-aligned appearance encoder, and a joint-space occupancy estimator.The pipeline is described as an initial joint-space implicit surface reconstruction.
  • Geometry encoding: The semantic-aware geometry encoder addresses the need for spatial features informed by the underlying 3D human-body semantics.Prior hand-crafted features ignore shapes, while voxel features face memory and quantization constraints.
  • Geometry encoding: PointNet++ learns semantic-aware geometry features from sampled parametric-body surface points using neighborhood pooling and progressively downsampled point sets.The described example uses N0 = 7324, with later point-set sizes N1 = 2048, N2 = 512, and N3 = 128.
  • Appearance encoding: The pixel-aligned appearance encoder projects posed-space query points into the image and bilinearly samples latent image features.The encoder maps a 512×512×3 image to 128×128×256 latent feature maps.
  • Occupancy estimation: Joint supervision addresses cases where mapped posed and canonical points have different ground-truth occupancies because of self-contact and topology changes.The posed space retains input-detail fidelity, while canonical reconstruction supports avatar completeness.
  • Occupancy estimation: Joint-space occupancy represents corresponding posed and canonical points, with separate implicit functions estimating occupancy in each space.Marching Cubes extracts isosurfaces at occupancy threshold τ = 0.
  • Mesh refinement: Mesh refinement uses front- and back-side normals and textures estimated in image space to add surface detail to the initially reconstructed meshes.The moulding-inpainting strategy is applied in joint spaces and is described as a coarse-to-fine refinement stage.
  • Mesh refinement: Compared with direct point-wise regression, inpainting-based refinement produces sharper details, fewer artifacts, and better preservation of fine-scale details across views.The paper reports that moulding-based refinement maintains fine-scale surface details across different views.

4. Training Losses

ARCH++ trains joint-space occupancy estimators alongside image-translation networks for normal and texture refinement. Its losses supervise occupancy in both spaces, enforce cross-space consistency, and combine reconstruction, perceptual, and selectively applied adversarial terms.

  • Training objectives: ARCH++ jointly learns occupancy estimation and normal/texture estimation through losses Lo, Ln, and Lt.Lo is the occupancy regression loss, while Ln and Lt are image-translation losses for normal and texture estimation.
  • Occupancy losses: Training samples 20480 paired query points per iteration after adding 5 cm diagonal Gaussian perturbations around mesh points.The perturbations increase coverage of close-to-surface regions in both spaces.
  • Occupancy losses: The occupancy objective combines canonical-space regression, posed-space regression, and a contrastive consistency term between mapped occupancy predictions.The consistency penalty treats occupancy pairs differently when their ground-truth labels agree or disagree.
  • Image-space refinement: Normal and texture estimation uses image-to-image translation losses combining L1 reconstruction and VGG-perceptual terms, with adversarial loss included for back-side texture.The input image can be directly used as the front texture map.
  • Image-space refinement: Adversarial loss is excluded from normal-map estimation because it counteracts performance there, whereas texture maps have greater variation.The authors attribute this difference to the more constrained normal-map space.
  • Evaluation: Table 1 evaluates proposed components across posed and canonical target and evaluation spaces, while Table 2 compares posed reconstruction errors on RenderPeople and BUFF.Table 1 distinguishes posed-space, canonical-space, and joint-space supervision; Table 2 reports normal, P2S, and Chamfer errors.

5. Experiments

ARCH++ is evaluated on synthetic and in-the-wild images using benchmark metrics, qualitative comparisons, and ablations. The results show stronger reconstruction quality, detail, and cross-space performance than prior designs and methods.

  • Results and Comparisons: The evaluation uses RenderPeople and BUFF benchmark settings alongside synthetic and in-the-wild images, with P2S, Chamfer distance, and L2 normal reprojection error metrics.Training uses 375 RenderPeople and 205 AXYZ scans; testing includes RenderPeople, AXYZ, BUFF, and Internet images.
  • Results and Comparisons: ARCH++ outperforms ARCH by large gaps on the reported RenderPeople and BUFF benchmark results.ARCH and PIFuHD are the two main comparison methods, with ARCH serving as the second-best baseline in the authors’ comparison.
  • Results and Comparisons: Qualitative comparisons show robust handling of arbitrary poses, self-contact, and occlusions, with more detailed posed-space and canonical-space reconstructions than existing methods.The comparisons include input, color, shape, and canonical-space avatar reconstructions.
  • Ablation Studies: Joint-space co-supervision balances posed-space input fidelity with canonical-space completeness, avoiding distortions associated with selecting only one reconstruction space.Single-space alternatives exhibit surface overstretching, intersecting surfaces, missing surfaces, topology distortions, broken manifolds, or unnatural deformations.
  • Ablation Studies: Learned point-based geometry features improve surface reconstruction over handcrafted RBF features while avoiding voxel-based quantization errors and computation overhead.The learned features encode both pose and shape priors, whereas handcrafted RBF features model only pose priors.
  • Ablation Studies: Image-space front/back normal regression and moulding-based refinement produce rich details across views, including sharper clothing wrinkles and fewer artifacts.The approach outperforms object-space regression and image-space input variants in visual detail, while image-space input achieves comparable quantitative performance.

6. Conclusion

The conclusion presents ARCH++ as a high-fidelity, animation-ready reconstruction method for many AR/VR applications. It validates the approach through comparisons and analysis, while identifying environment information as future work.

  • ARCH++ produces high-fidelity, animation-ready results for many AR/VR applications.
  • The authors validate their findings through comparisons with and analysis of state-of-the-art methods.
  • Future work will incorporate environment information such as lighting and affordance to further understand body pose and appearance and address current limitations.

Supplementary

The supplementary implementation details describe body-scale normalization for input images and training/test preprocessing.

  • The implementation details section documents the configuration of the proposed method.
  • Input images are normalized to human body scale using 3D skeleton estimates, with the pelvis centered and each pixel representing 1 cm under orthographic projection.

A.2. Network Architectures

ARCH++ combines point-based geometry and pixel-aligned appearance features with joint-space occupancy estimation and image-based surface refinement. Its staged inference reconstructs a fully colored animation-ready avatar in about 5 seconds on one Tesla V100 GPU.

  • Semantic-Aware Geometry Encoder: The geometry encoder uses PointNet++ Set Abstraction and Feature Propagation layers to produce 96-dimensional spatially aligned geometry features.The encoder progressively samples 2048, 512, and 128 points, while concatenating three 32-channel propagated feature sets.
  • Pixel-Aligned Appearance Encoder: The appearance encoder adopts a four-stack Stack Hourglass Network with two residual blocks per stack and 256-dimensional output features.
  • Joint-Space Occupancy Estimator: The two-branch occupancy estimator predicts canonical-space and posed-space occupancies from geometry and appearance features using multilayer perceptrons.Each branch uses four fully connected layers with hidden sizes 1024, 512, 256, and 128.
  • Surface Refinement: The pipeline refines normals and textures with image-to-image translation networks, extending the architecture with GAN losses for back-side texture inference.The normal and texture networks use nine residual blocks and four downsampling layers.
  • Inference Cost: 5 seconds is the total inference time for reconstructing a fully colored animation-ready avatar from an unconstrained RGB image, compared with over 40 seconds for PaMIR.
  • Inference Cost: The current sequential implementation exchanges intermediate results through CPU memory and file I/O, leaving room for parallelizing several reconstruction modules.The proposed framework could run the geometry, appearance, and refinement networks in parallel.

C. Extended Experiments

The extended experiments report additional conclusions, comparisons, applications, and failure cases, with backgrounds removed from images for visualization.

  • C. Extended Experiments: The extended experiments include user studies, avatar animation applications, video-based fusion, and failure-case analyses.

C.1. 3D Human Body Prior is Crucial for Our Task

The experiments examine the role of body priors, cross-view surface consistency, and image-based refinement for realistic clothed-human reconstruction. ARCH++ uses learned image-space refinements to recover details and improve unseen-surface realism.

  • 3D Human Body Prior is Crucial for Our Task: PIFuHD can fail to capture correct underlying body shapes and poses despite sharing the RenderPeople training set with ARCH++.Parametric body-shape estimation networks provide an alternative source of pose and shape information because varied-pose clothed-avatar ground truth remains difficult to obtain.
  • Cross-View Reconstruction Quality: Moulding-based refinement maintains fine-scale surface details across different views and produces more full-body details than competing methods.Examples include belts around the waist and clothing wrinkles on the back and legs.
  • Normal Refinement: Normal refinement enhances photorealistic rendering at different viewpoints while keeping mesh textures fixed.
  • 3D Human Body Prior is Crucial for Our Task: Geometry features encode body shape and pose priors, while appearance features provide image evidence for fine-scale clothing wrinkles and surface details.
  • Cross-View Reconstruction Quality: Prior methods often produce plausible details only from the input camera view, with fewer and less realistic details from other viewpoints.This cross-view inconsistency limits their applicability to AR/VR applications requiring free-viewpoint rendering.
  • Texture Refinement: GAN-trained back-side textures contain more plausible surface details and better lighting effects than textures trained without GAN losses.
  • Texture Refinement: Predicting albedos loses substantial clothing details and textured patterns, motivating texture reconstruction compatible with the shaded input image.The paper attributes the difficulty to insufficient data for implicitly compensating shading from shaded inputs.
  • Occluded-Surface Completion: Image-space refinement treats surfaces missing from front- and back-side ray tracing as an inpainting problem, producing more complete canonical-space avatars than ARCH.

C.6. More Normal Refinement Results

Additional results across several datasets show high-fidelity normal refinement for people viewed from different cameras, poses, and clothing configurations.

  • C.6. More Normal Refinement Results: Normal refinement achieves high fidelity across DeepHuman, AXYZ, RenderPeople, and Unsplash test sets spanning different views, poses, and clothes.

C.7. User Study on Reconstruction Photorealism Shows Superior Quality of Our Method

A user study with 22 participants found that ARCH++ was preferred over prior methods for normal and texture reconstruction, with statistically significant pairwise differences. The method also supports multi-view fusion and practical avatar applications, while remaining challenged by strong directional lighting and rare poses.

  • User-study setup: The study compared reconstructed avatars from ARCH++, PIFuHD, ARCH, and PIFu using front- and back-view realism judgments on 30 test examples.PIFu replaced PIFuHD for the texture study because PIFuHD does not reconstruct texture.
  • Normal reconstruction: For normal reconstruction, ARCH++ significantly outperformed PIFuHD and ARCH, while PIFuHD significantly outperformed ARCH.The reported comparisons were Ours vs. PIFuHD (mean = 0.39697, p < .001), Ours vs. ARCH (mean = 0.83636, p < .001), and PIFuHD vs. ARCH (mean = 0.32121, p < .001).
  • Texture reconstruction: For texture reconstruction, ARCH++ significantly outperformed PIFu and ARCH, while ARCH significantly outperformed PIFu.The reported comparisons were Ours vs. PIFu (mean = 0.49091, p < .001), Ours vs. ARCH (mean = 0.74545, p < .001), and ARCH vs. PIFu (mean = 0.22424, p < .001).
  • Multi-view fusion: With multi-view inputs, canonical-space fusion combines normals and textures across poses and viewpoints to produce photorealistic, animatable avatars.Views facing the surface receive higher fusion weights, and the method is demonstrated in a social telepresence application with predefined motions.
  • Failure cases and limitations: Typical failure cases occur under strong directional lighting and challenging poses, while the authors identify the relatively small training scan set as a broader limitation.The training set contains only hundreds of 3D scans, motivating more lighting augmentation and larger, more varied clothed-human datasets.
Loading 2108.07845v4…