PaperlayerView exact revision on arXiv

Source-linked AI summary

The Power of Points for Modeling Humans in Clothing

Qianli Ma, Jinlong Yang, Siyu Tang, Michael J. Black

arXiv:2109.01137v2cs.CVcs.GR

TL;DR

Turning a static scan into an animatable clothed avatar remains difficult: existing methods require substantial artist work or 4D scans, and outfit-specific models do not generalize across garments. POP learns pose-dependent clothing geometry from dense point clouds and a geometric feature tensor, then fits that tensor to an unseen scan. The authors report state-of-the-art clothing modeling and unseen-outfit generalization on captured and synthetic data.

  • Problem

    Existing approaches do not provide realistic pose-dependent clothing deformation from a single static scan, and most data-driven models are outfit-specific.

    Source: §1 · pp. 1–2

  • Method

    POP learns pose-dependent clothing geometry across outfits using dense point clouds and a geometric feature tensor optimized to fit an unseen scan.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    POP shows state-of-the-art performance in clothing modeling and generalization to unseen outfits on captured and synthetic datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A fitted POP model can animate a single scan of an unseen subject and outfit with pose-dependent clothing variation.

    Source: §5 · p. 8

    POP assumes the minimally clothed body is given and training scans are noise-free and complete.

    Source: §5 · p. 8

Abstract

from arXiv · show

Currently it requires an artist to create 3D human avatars with realistic clothing that can move naturally. Despite progress on 3D scanning and modeling of human bodies, there is still no technology that can easily turn a static scan into an animatable avatar. Automating the creation of such avatars would enable many applications in games, social networking, animation, and AR/VR to name a few. The key problem is one of representation. Standard 3D meshes are widely used in modeling the minimally-clothed body but do not readily capture the complex topology of clothing. Recent interest has shifted to implicit surface models for this task but they are computationally heavy and lack compatibility with existing 3D tools. What is needed is a 3D representation that can capture varied topology at high resolution and that can be learned from data. We argue that this representation has been with us all along -- the point cloud. Point clouds have properties of both implicit and explicit representations that we exploit to model 3D garment geometry on a human body. We train a neural network with a novel local clothing geometric feature to represent the shape of different outfits. The network is trained from 3D point clouds of many types of clothing, on many bodies, in many poses, and learns to model pose-dependent clothing deformations. The geometry feature can be optimized to fit a previously unseen scan of a person in clothing, enabling the scan to be reposed realistically. Our model demonstrates superior quantitative and qualitative results in both multi-outfit modeling and unseen outfit animation. The code is available for research purposes.

1. Introduction

The paper targets animatable clothed avatars from a single static scan, a task existing approaches handle through substantial artist work or 4D training scans. POP combines dense point clouds with a learned geometric feature tensor to model clothing across outfits and poses, including unseen outfits.

1. Introduction

  • Existing solutions require heavy artist work or 4D scans for training, while automatic rigging cannot produce pose-dependent clothing deformation.Physics-based simulation requires expert reverse-engineering of a simulation-ready clothing mesh from the scan.Source: §1 · pp. 1–2
  • Most existing data-driven clothing models are outfit-specific, leaving cross-garment modeling and generalization to unseen outfits unresolved.The goal is a model that handles different outfit types and styles while producing pose-dependent deformations.Source: §1 · p. 2
  • POP uses dense point clouds with smooth local features on a 2D manifold, avoiding patches while enabling arbitrarily dense up-sampling at inference.The representation is proposed to model varied clothing topology and fine geometric detail.Source: §1 · p. 2
  • A learned geometric feature tensor separates intrinsic outfit shape from the decoder, which can then model pose-dependent effects shared across outfits.At inference, the tensor is optimized to fit a scan in an unseen outfit, allowing the decoder to predict its pose-dependent deformation.Source: §1 · p. 2
  • POP is evaluated on captured and synthetic data, with state-of-the-art performance in clothing modeling and generalization to unseen outfits.Its contributions include detailed clothing geometry and an application that animates a static scan with reasonable pose-dependent deformations.Source: §1 · p. 2

2. Related Work

Prior work trades off topology flexibility, detail, and generalization across clothing categories. POP uses dense structured point clouds with fine-grained local features to model varied outfit styles and generalize to unseen outfits.

Shape Representations for Clothed Human Modeling

  • Mesh methods are efficient and graphics-compatible, but fixed topology and template registration limit generalization across clothing categories.Even adaptable templates retain the challenge of registering complex garment geometry to the mesh template.Source: §2 · p. 2
  • Implicit surfaces handle varied topology without predefined templates, but thin structures remain challenging and explicit surface reconstruction scales cubically with resolution.This reconstruction cost limits practical applications.Source: §2 · pp. 2–3
  • Patch-based point clouds support varied topology, but patch overlap or separation can reduce geometric fidelity and visual quality.POP instead decodes dense point clouds from fine-grained local features without patches.Source: §2 · p. 3

Modeling Outfit Shape Space

  • Outfit-specific models train separately per outfit or category, while multi-outfit models combine predefined templates or categories but do not handle garments beyond them.Outfit-specific template-free models store intrinsic shape in their parameters, restricting test-time generalization to unseen outfits.Source: §2 · p. 3
  • Arbitrary-outfit approaches include sewing-pattern images and implicit surfaces, but are described as limited in garment complexity or geometric detail compared with POP.POP is reported to generalize to unseen outfits and achieve state-of-the-art garment-space modeling performance.Source: §2 · p. 3

3. Method

POP represents clothing as a dense point cloud whose local features vary continuously over the body surface. A shared decoder combines pose and garment geometry features to model outfit-dependent deformations and can be fitted to scans of unseen outfits.

3.1. Representing Humans with Point Clouds

  • POP maps body-surface coordinates and local pose and shape features to clothing displacement vectors, then transforms those vectors into posed space.The decoder also predicts point normals, which are transformed using the rotation component of the local body transformation.Source: §3.1 · pp. 4–5
  • Fine-grained local features are bilinearly interpolated across a denser body manifold, allowing the decoder to generate clothing points at arbitrary surface resolution.The displacement field is predicted on the canonically posed body, leaving articulation to explain a large portion of shape variation while avoiding a fixed-topology assumption.Source: §3.1 · p. 4

3.2. Cross-garment Modeling with a Single Model

  • A shared decoder combines pose features from a UV positional map with an outfit-specific geometry tensor that stores pose-independent garment shape.The same geometry tensor is used across an outfit's training poses, while the decoder is shared across garments and poses.Source: §3.2 · p. 5

3.3. Training and Inference

  • Training combines point-cloud Chamfer distance, normal discrepancy, and regularizers; at test time, the geometry tensor can be optimized to fit an unseen-outfit scan and then used for animation.For a previously seen outfit, unseen poses can instead be handled with a forward pass using its trained geometry tensor.Source: §3.3 · pp. 5–6

4. Experiments

The experiments assess POP on captured CAPE and synthetic ReSynth data, including representation quality, cross-outfit modeling, and animation of unseen scans. The cross-outfit model outperforms both baselines on both datasets, while single-scan tests show reasonable pose-dependent clothing animation, including on real captures.

4. Experiments

  • CAPE includes 14 outfits from three subjects, while synthetic ReSynth includes 24 varied outfits, including skirts and jackets, with poses generated through physics simulation.The experiments compare NASA and SCALE for representation power, per-outfit and cross-outfit POP for cross-outfit modeling, and POP with LBS for unseen-scan animation.Source: §4 · p. 6

4.1. Representation Power

  • For representation power, the study evaluates pose-dependent garment reconstruction on seen outfits and unseen motion sequences, reporting overall mean error, median per-outfit mean error, and maximum per-outfit error.The three statistics reflect variation in modeling difficulty across outfit types.Source: §4.1 · p. 6

4.2. Cross-Outfit Modeling

  • On both CAPE and ReSynth, the cross-outfit model consistently outperforms NASA and SCALE, with performance comparable to per-outfit POP on ReSynth and lower error on CAPE.Table 2 reports pose-dependent garment-shape reconstruction on unseen test sequences.Source: §4.2 · p. 7, Table 2 · p. 7
  • Cross-outfit POP shares information among similar CAPE outfits with differing subject motions, and remains comparable to full-data outfit-specific SCALE when trained with one-quarter of each outfit’s data.ReSynth uses consistent motions across outfits and varied geometry, limiting inter-outfit knowledge sharing and yielding similar performance for per-outfit and cross-outfit POP.Source: §4.2 · pp. 7–8

4.3. Single Scan Animation

  • Fitted to an unseen CAPE scan, POP produces reasonable clothing variation on unseen motions, such as lifted hems, unlike LBS’s rigid body-part transformations; tests also show high-quality ReSynth geometry and reasonable animation on a real capture.The real-capture test is more challenging because of the domain gap from synthetic training data.Source: §4.3 · p. 8

5. Conclusion

POP uses a template-free, topologically flexible point-cloud representation to model clothing across outfits and produce pose-dependent deformations. Its scan-animation scope has important assumptions and limitations, including clean scans, possible UV-map seams, and simplified treatment of body-shape effects.

5. Conclusion

  • POP is a dense structured point-cloud representation that models varied clothing styles with high detail and realistic pose-dependent deformation, and can animate a scan of an unseen person and outfit.The conclusion reports evaluations on captured and synthetic data, including an advantage over traditional subject-specific models.Source: §5 · p. 8
  • POP assumes the minimally clothed body is given and training scans have no noise or missing data.Handling partial, noisy scans and combining the method with automatic body-shape and pose estimation are future directions.Source: §5 · p. 8
  • UV-map pose conditioning can sometimes produce seams, and the model simplifies clothing deformation by factoring out body-shape dependencies.The conclusion suggests more continuous body-manifold parameterizations and learning body-shape effects given sufficient training data.Source: §5 · p. 8

A.1. Model Architecture

The architecture combines pose and geometric features at UV query locations to predict a dense point cloud, with separate heads for displacements and normals.

A.1. Model Architecture

  • SMPL or SMPL-X UV maps at 128 × 128 × 3 serve as pose input, and the pose encoder maps each pixel to 64 channels.The encoder is a seven-block convolutional and seven-block transposed-convolutional UNet.Source: §A.1 · p. 9
  • The geometric feature tensor has resolution 128 × 128 × 64 and is auto-decoded as a free variable optimized with network weights during training.Three learnable convolutions with receptive field 5 smooth the features spatially and reduce output noise.Source: §A.1 · p. 9
  • Pose and geometry features are concatenated, queried on a 256 × 256 UV map, and used to generate a 50K-point output cloud.The concatenated tensor is bilinearly upsampled by a factor of four before querying.Source: §A.1 · p. 9
  • At each query, the 64-dimensional pose and geometry features plus the UV coordinate enter an eight-layer MLP that predicts point displacements and normals through separate heads.The MLP uses a skip connection to its fourth layer; predicted normals are normalized to unit length.Source: §A.1 · p. 9

A.2. Training

POP is trained jointly for displacement and normal prediction, with the normal loss introduced after displacement loss stabilizes.

A.2. Training

  • Training uses Adam for 400 epochs, and the normal loss is enabled from epoch 250 after displacement loss roughly stabilizes.The displacement and normal modules are trained jointly.Source: §A.2 · p. 9

A.3. Data Processing

The data are normalized and sampled as point clouds; the supplement describes baseline evaluation and a large-scale visual-quality user study.

A.3. Data Processing

  • Each example is normalized by removing body translation and global orientation, then sampled at 40K surface points without using mesh connectivity.Source: §A.3 · p. 9

A.4. Baselines

  • NASA outputs are sampled from Marching Cubes surfaces, SCALE uses 798 patches with 64 points each, and LBS reposes a fitted SMPL model.Source: §A.4 · p. 9

A.5. User Study

  • The user study compares POP and SCALE renderings on more than 6,000 unseen CAPE and ReSynth examples, with each pair rated by three users.Images were shown side-by-side with randomized left-right order, and users could zoom in without a time limit.Source: §A.5 · p. 9

B. Datasets

ReSynth broadens outfit variation with designer-created clothing simulated on SMPL-X bodies, while CAPE supplies captured clothed-body motion sequences.

B. Datasets

  • ReSynth contains 24 designer-created outfits spanning varied types and styles, draped and simulated on fitted SMPL-X bodies.The examples are based on real-world scans [61], using the SMPL-X [53] body model and Deform Dynamics.Source: B. Datasets · pp. 9–10, Figure B.1 · p. 10
  • ReSynth provides 984 training and 347 test frames per outfit after manual removal of problematic simulated frames.Source: B. Datasets · p. 10
  • CAPE experiments use three subjects with 14 outfits, splitting each outfit’s motion sequences randomly into 70% training and 30% test.Source: B. Datasets · p. 10

C. Extended Results and Discussions

POP improves on SCALE’s average errors across CAPE and ReSynth, while the extended discussion examines NASA’s errors and inference speed. The supplement also presents qualitative pose-generalization comparisons.

C. Extended Results and Discussions

D. Run-time Comparison

  • POP generates 50K points in 48.8 ms versus SCALE’s 42.4 ms; NASA takes 12.2 s per example, while optional Poisson meshing takes 1.1 s.The passage characterizes POP and SCALE as having comparable runtime and states that explicit representations are faster than the implicit baseline.Source: D. Run-time Comparison · p. 10

E. Limitations and Failure Cases

The UV-map representation can produce discontinuities, while the discussion situates POP’s canonical-body displacement field and surface-coordinate decoder among related approaches.

E. Limitations and Failure Cases

F. Further Discussions on Related Work

Related papers

Loading 2109.01137v2…