PaperlayerView exact revision on arXiv

Source-linked AI summary

SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks

Shunsuke Saito, Jinlong Yang, Qianli Ma, Michael J. Black

arXiv:2104.03313v2cs.CV

TL;DR

Realistic animatable clothed avatars are difficult to learn from raw scans because scans have missing data, changing topology, and clothing that can diverge from the body. SCANimate uses weakly supervised, cycle-consistent canonicalization with implicit skinning and local pose-aware shape modeling, and reports superior generality and accuracy across settings. Its main scope boundaries are clothing topology and deterministic wrinkle modeling.

  • Problem

    Raw clothed-human scans contain missing data, changing topology, multiple layers, and sliding motion, while accurate template registration is often infeasible.

  • Method

    SCANimate combines fitted-body guidance, bidirectional implicit skinning, geometric cycle consistency, and locally pose-aware implicit surfaces to learn avatars from raw scans.

  • Results

    SCANimate is reported as superior to existing solutions in generality and accuracy, while local pose conditioning improves reconstruction accuracy and fidelity for interpolation and extrapolation.

  • Takeaways & Limitations

    The framework enables weakly supervised learning of realistic pose-dependent clothing deformation from raw scans without template-based surface registration.

  • Takeaways & Limitations

    The representation works best for clothing topologically similar to the body and is deterministic despite stochastic pose-dependent wrinkles.

Abstract

from arXiv · show

We present SCANimate, an end-to-end trainable framework that takes raw 3D scans of a clothed human and turns them into an animatable avatar. These avatars are driven by pose parameters and have realistic clothing that moves and deforms naturally. SCANimate does not rely on a customized mesh template or surface mesh registration. We observe that fitting a parametric 3D body model, like SMPL, to a clothed human scan is tractable while surface registration of the body topology to the scan is often not, because clothing can deviate significantly from the body shape. We also observe that articulated transformations are invertible, resulting in geometric cycle consistency in the posed and unposed shapes. These observations lead us to a weakly supervised learning method that aligns scans into a canonical pose by disentangling articulated deformations without template-based surface registration. Furthermore, to complete missing regions in the aligned scans while modeling pose-dependent deformations, we introduce a locally pose-aware implicit function that learns to complete and model geometry with learned pose correctives. In contrast to commonly used global pose embeddings, our local pose conditioning significantly reduces long-range spurious correlations and improves generalization to unseen poses, especially when training data is limited. Our method can be applied to pose-aware appearance modeling to generate a fully textured avatar. We demonstrate our approach on various clothing types with different amounts of training data, outperforming existing solutions and other variants in terms of fidelity and generality in every setting. The code is available at https://scanimate.is.tue.mpg.de.

1. Introduction

SCANimate learns animatable clothed-human avatars directly from raw scans, avoiding template registration while preserving detailed, pose-dependent clothing deformation. It combines weakly supervised canonicalization with implicit skinning and locally pose-aware shape learning.

  • SCANimate targets realistic, reposable clothed avatars with natural clothing motion, detailed wrinkles, and textured-image rendering.
  • The framework learns high-quality animatable clothed humans from raw scans without custom-template registration or synthetic clothing simulation data.
  • Its parametric representation retains wrinkles, garment sliding, and arbitrary clothing topology while supporting direct animation with SMPL pose parameters.
  • Because traditional surface registration is generally infeasible for clothed bodies, SCANimate learns continuous functions that transform posed scans to canonical space and back.
  • Implicit skinning predicts spatially continuous joint-weight vectors for arbitrary 3D locations, extending mesh-based skinning to scans with unknown topology.
  • The method uses fitted body models, geometric cycle consistency, and locally pose-aware implicit surfaces to disentangle articulated deformation and model pose-dependent clothing.

2. Related Work

Prior clothed-human models often depend on registered templates, fixed topologies, or independently composed implicit functions. SCANimate instead uses weak supervision, fitted body models, and a learnable LBS-based continuous skinning formulation for arbitrary-topology scans.

  • Parametric body models capture high-fidelity pose and shape variation but generally focus on minimally clothed bodies, while clothed extensions add garment layers or body-mesh displacements.
  • SCANimate builds parametric clothed models from raw scans without templates or surface registration, using canonicalization and pose-conditioned implicit surfaces.
  • Compared with NASA, SCANimate’s LBS-based formulation is reported to produce fewer artifacts and better generalization to unseen poses.
  • Unlike fully supervised methods that predict weights on neutral templates, SCANimate learns skinning weights throughout continuous 3D space.
  • Jointly learning skinning weights improves canonicalization and reproducibility compared with weights uniquely determined by an underlying body model.

3. Method

SCANimate learns bidirectional, continuous skinning fields to canonicalize raw clothed scans without template registration, then fits a locally pose-aware implicit surface model. Geometric cycle consistency provides weak supervision, while implicit regularization completes missing scan regions and local pose conditioning improves unseen-pose generalization.

  • Pipeline overview: The pipeline uses fitted SMPL models, learned bidirectional transformations, cycle consistency, and a locally pose-aware signed distance function.Raw scans are first canonicalized, then used to learn a pose-aware clothed-human model.
  • Canonicalization: Continuous skinning functions map arbitrary 3D points between scan-specific posed spaces and a shared canonical space without fixed topology.The functions predict skinning weights throughout 3D space rather than assigning them only to mesh vertices.
  • Weak supervision: SMPL-guided losses, cycle consistency, and regularization jointly learn skinning fields without ground-truth scan weights.Cycle constraints penalize weight assignments that fail to return posed points to their original locations; nearest-body assignments alone can cause severe artifacts.
  • Canonicalization: Inverse LBS canonicalizes all scans, after which highly distorted triangles are removed before implicit shape learning.The resulting canonical scans provide the training data for the pose-aware clothed-human model.
  • Implicit shape learning: The implicit surface model handles arbitrary topology and fills missing scan regions using surface, normal-alignment, Eikonal, and off-surface regularization.It does not require ground-truth signed distances for non-surface points.
  • Pose-dependent deformation: Local pose conditioning limits each joint’s influence to nearby kinematic-tree neighborhoods, reducing spurious correlations and overfitting on unseen poses.The paper uses a four-ring neighborhood rule and reports better generalization with few input scans.

4. Experimental Results

Experiments evaluate canonicalization, local pose conditioning under reduced data, and pose-aware shape modeling against prior methods. SCANimate preserves coherent details, generalizes with limited data, and achieves superior qualitative and reported comparative performance.

  • Dataset and Metric: The evaluation uses CAPE raw scan sequences and SMPL fits, testing unseen-pose interpolation and extrapolation.Motion sequences are split into 80% training and 20% testing; interpolation and extrapolation use distinct frame sampling protocols.
  • Canonicalization: Canonicalization baselines using nearest-neighbor or k-nearest-neighbor skinning associations break cycle consistency and create noticeable artifacts.These inaccurate canonicalizations substantially degrade reconstructed avatar quality.
  • Canonicalization: SCANimate’s cycle-consistent canonicalization normalizes scans while retaining coherent geometric details for parametric clothed-avatar modeling.Figure 4 compares canonicalization results at the top and reposed canonicalization results at the bottom.
  • Locally Pose-aware Shape Learning: Local pose conditioning maintains roughly equivalent reconstruction accuracy with only 5% of the original training data, unlike global conditioning’s severe overfitting below 10%.The comparison covers 100%, 50%, 10%, and 5% training-set sizes for interpolation and extrapolation.
  • Comparison with SoTA: Against state-of-the-art methods, SCANimate achieves superior reconstruction accuracy and perceptual realism while producing detailed, globally coherent deformations without template registration.Prior reconstructions are described as less detailed, unable to capture topological changes, or affected by discontinuities and ghosting artifacts.
  • Learning a Fully Textured Avatar: The pose-aware model extends to appearance modeling, producing high-resolution texture without 2D texture mapping.This demonstrates textured-avatar modeling without a template-mesh requirement.

5. Discussion and Future Work

SCANimate learns realistic, pose-driven clothed avatars directly from raw 3D scans using weak supervision and cycle consistency. The authors identify boundaries involving clothing topology, stochastic wrinkles, texture realism, and clothing variety.

  • Discussion: SCANimate directly learns high-quality, pose-driven avatars with realistic clothing deformations from raw 3D scans.The framework is fully automatic and uses weakly supervised decomposition of articulated deformations with body-guided supervision and cycle-consistency regularization.
  • Future Work: The approach can automatically build textured Scanimats with high-resolution realistic texture.The texture extension is presented as an application of the framework to texture modeling.
  • Discussion: The method enables learning physically plausible clothing deformations from raw scans, including complex multi-garment deformations.This addresses limitations imposed by accurate and coherent surface registration in prior approaches.
  • Limitations and Future Work: The representation works best for clothing topologically similar to the body and may fail for garments such as skirts.A synthetic-skirt example shows severe stretching artifacts between the legs during canonicalization.
  • Limitations and Future Work: The deterministic model cannot represent stochastic wrinkle variation that depends on preceding pose sequences.Future work should model this temporal variability rather than assigning one deterministic wrinkle configuration to each pose.

A.1. Network Architectures

SCANimate uses neural implicit fields for skinning and canonical geometry, with positional encoding and multilayer perceptrons. Pose information is incorporated through a learned pose feature for skinning and local pose conditioning for geometry.

  • Network Architectures: Forward and inverse skinning networks use multilayer perceptrons with skip connections and LeakyReLU activations.The networks have intermediate sizes (256, 256, 256, 24), and the final layer uses softmax to produce normalized skinning weights.
  • Network Architectures: Skinning weights are regressed from positionally encoded 3D coordinates, with a 64-dimensional pose feature added for scans in different poses.The pose feature is denoted i ∈ R64 in the supplied architecture description.
  • Network Architectures: The canonical geometry module is a multilayer perceptron with Fourier positional encoding, a skip connection, and softplus activations.Its listed intermediate sizes are (512, 512, 512, 343, 512, 512, 1), with β = 100 for softplus.

A.2. Training Procedure

Training proceeds in stages: skinning networks are pretrained, jointly optimized with cycle consistency, and then fixed while the geometry module is trained. Sampling and optimization schedules are specified for each stage.

  • Training Procedure: SCANimate first pretrains the skinning networks before introducing the proposed cycle-consistency constraint.The pretraining stage uses zero weights for the cycle-consistency terms, which are enabled during joint training.
  • Training Procedure: During joint skinning training, the cycle-consistency weights are λC′ = 1.0 and λC′′ = 1.0, with λC′′ multiplied by 10 in the second half.The same stage uses λB = 10.0, λS = 1.0, λSp = 0.001, λSm = 0.1, and λZ = 0.01.
  • Training Procedure: The first two stages use 6890 SMPL vertices and 8000 uniformly sampled scan points, with scan samples dynamically updated each iteration.These samples support training of the skinning networks across the first two stages.
  • Training Procedure: After skinning training, the weights are fixed and the geometry module is optimized using surface, regularization, and occupancy-related samples.The procedure samples 5000 scan-surface points per iteration and additional points from a bounding box, perturbed scans, and SMPL vertices.
  • Training Procedure: The three stages use Adam learning rates of 0.004, 0.001, and 0.001, with decay by 0.1 at one-half and three-quarters of training.The first and second stages run for 80 and 200 epochs, respectively.

A.3. Texture Inference

Texture is inferred with a neural texture field trained from scan colors, while canonical body construction uses pose-aware SMPL blend shapes. Additional processing removes problematic self-intersection geometry and detects concave regions for constraint handling.

  • Texture Inference: A neural texture field f^c(x) maps surface locations to RGB color and is trained by minimizing an L1 reconstruction loss.The texture network is optimized from 5000 sampled scan points per iteration using Adam.
  • Other Details: Concave regions are excluded from the smoothness constraint to avoid propagating incorrect skinning weights near self-intersections.The regions are detected from scan mean curvature using a threshold of 0.2.
  • Canonical Body: Canonical body geometry includes pose correctives obtained by activating pose-aware SMPL blend shapes for each frame’s body pose.The resulting body model is in a canonical pose while retaining pose-dependent deformations.
  • Other Details: Triangles with any edge longer than four times its initial length are removed after canonicalization.This targets highly distorted triangles in self-intersection regions that must separate in the canonical pose.
  • Other Details: Canonicalization failure on a synthetic skirt produces severe stretching artifacts between the legs.The figure highlights these artifacts using surface triangles in zoomed-in views.

B.1. Latent Autodecoding

Latent autodecoding learns per-scan embeddings to distinguish frames beyond pose similarity, and it outperforms pose regression under the canonicalization energy.

  • B.1. Latent Autodecoding: Auto-decoding discovers latent embeddings that distinguish scans even when their pose parameters are similar.The embeddings can encode pose information and other information needed to distinguish each frame.
  • B.1. Latent Autodecoding: 0.025 canonicalization energy: auto-decoding outperforms pose regression at 0.043.Both methods use Ecano without the EZ term.

B.2. CAPE Dataset Limitation

The CAPE evaluation includes erroneous wrist and ankle body fits that can distort nearby regions, while shape learning can reconstruct missing areas from other canonicalized scans.

  • B.2. CAPE Dataset Limitation: Erroneous CAPE body fitting around wrists and ankles creates unnecessary local distortions.These fitting errors occur in some frames and affect the canonicalized geometry.
  • B.2. CAPE Dataset Limitation: Smoothness regularization can propagate fitting distortions into nearby regions, causing a larger region to be discarded.The limitation concerns the extent of discarded geometry rather than only the original fitting location.
  • B.2. CAPE Dataset Limitation: Shape learning complements missing regions using information from other canonicalized scans.The reconstructed Scanimats are reported not to suffer from the small pose-fitting errors.
  • B.2. CAPE Dataset Limitation: The skinning networks may be implemented separately or combined through shared and per-frame latent vectors.The separate formulation defines gc and gs for forward and inverse transformations, respectively.

C. Additional Qualitative Results

Additional qualitative figures examine local versus global pose encoding, comparisons with state-of-the-art methods, and textured Scanimats. They emphasize local encoding under limited data and qualitative extrapolation behavior.

  • Locally Pose-aware Shape Learning: Local pose encoding generalizes well with severely limited training data, unlike global pose encoding, which shows severe overfitting artifacts.Figure C.1 compares local encoding in the top row with global encoding in the bottom row.
  • Comparison with the SoTA methods: Qualitative extrapolation results compare SCANimate with state-of-the-art methods and illustrate the advantages and limitations of existing approaches.Figure C.2 provides the additional comparison.
  • Textured Scanimats: Additional examples show that SCANimate can build Scanimats with high-resolution realistic texture.Figure C.3 presents textured Scanimats produced by the method.
Loading 2104.03313v2…