Source-linked AI summary
Learning-Based Animation of Clothing for Virtual Try-On
Igor Santesteban, Miguel A. Otaduy, Dan Casas
TL;DR
Virtual try-on needs responsive clothing animation that handles diverse body shapes and motion while avoiding unrealistic deformation from prior approaches. The paper learns separate nonlinear models for garment fit and wrinkles from physics-based simulations, using recurrent dynamics for wrinkles; it reports 4ms-per-frame animation and over 1000x speed-up over full simulation, while requiring independent training per garment size and material.
Problem
Virtual try-on requires garment deformation to respond realistically to body shape and motion, whereas prior methods are limited in shape handling or produce unrealistic deformation.
Method
The method trains nonlinear models on physics-based dressed-character simulations, separately regressing garment fit from shape and wrinkles from shape and motion with recurrent neural networks.
Results
4ms per frame, including collision postprocessing, delivers cloth animation for meshes with thousands of triangles and more than 1000x speed-up over full simulation.
Takeaways & Limitations
The two-level nonlinear and dynamic regression scheme enables efficient virtual try-on animation while overcoming limitations of previous data-driven approaches.
Takeaways & Limitations
The method requires independent training for each garment size and material, and the dataset contains only one one-size garment with a specific fabric material.
Abstract
from arXiv · showhide
This paper presents a learning-based clothing animation method for highly efficient virtual try-on simulation. Given a garment, we preprocess a rich database of physically-based dressed character simulations, for multiple body shapes and animations. Then, using this database, we train a learning-based model of cloth drape and wrinkles, as a function of body shape and dynamics. We propose a model that separates global garment fit, due to body shape, from local garment wrinkles, due to both pose dynamics and body shape. We use a recurrent neural network to regress garment wrinkles, and we achieve highly plausible nonlinear effects, in contrast to the blending artifacts suffered by previous methods. At runtime, dynamic virtual try-on animations are produced in just a few milliseconds for garments with thousands of triangles. We show qualitative and quantitative analysis of results
1. Introduction
The paper targets responsive virtual try-on by learning clothing deformation from body shape and motion, separating garment fit from dynamic wrinkles. Its nonlinear, recurrent approach achieves plausible animations with millisecond-scale runtime.
- Virtual try-on aims to improve online shopping and other interactive applications, but existing solutions lack the responsiveness of physical try-on.
- Physics-based simulation produces detailed clothing results but incurs significant runtime computational cost, while interactive approximations sacrifice accuracy.
- Previous data-driven methods can produce plausible wrinkles under strong pose-cloth correlation but struggle with nonlinear deformation, contact, and blending artifacts.
- The proposed method models a garment as a function of body shape and motion, addressing the need to animate one garment across diverse body shapes.
- A two-level strategy separately learns garment fit from body shape and local wrinkles from body shape and motion, disentangling deformation sources.
- 4ms per frame, including collision postprocessing, enables cloth animation with thousands of triangles and more than 1000x speed-up over full simulation.
2. Related Work
Related work spans physics-based simulation, data-driven deformation, captured-cloth re-animation, and image-based methods. The paper emphasizes the challenge of combining realistic responses to body shape and pose with efficient animation.
- Fast Cloth Simulation: Physics-based cloth methods model internal forces and collision processes, but their computational cost motivates approximations, faster solvers, and adaptive remeshing.
- Data-Driven Models: Data-driven cloth animation uses pose or low-dimensional models, example-based detail transfer, and precomputed samples to synthesize garment deformation.
- Data-Driven Models: Virtual try-on requires cloth models responsive to both body pose and shape, a feature described as scarce among existing data-driven methods.
- Data-Driven Models: Prior approaches may use linear models, scaling factors, or wrinkle retargeting, but these do not provide realistic deformation across body shapes.
- Performance Capture Re-Animation: Performance-capture re-animation can edit shape and pose while retaining captured garments, but unseen poses or shapes lack realistic nonrigid clothing behavior.
- Image-Based Methods: Image-based methods generate compelling dressed-character images without operating on a 3D model or cloth simulation.
3. Clothing Animation
The method models clothing deformation through a pose-space pipeline that separates body-shape-dependent garment fit from shape-and-pose-dependent wrinkles. Nonlinear regressors and skinning produce efficient cloth animation, with collision postprocessing for tight garments.
- Clothing Model: The cloth skinning weights are obtained by projecting each cloth vertex onto the closest template-body triangle and interpolating the body's skinning weights.This lets the garment follow the deformations of the rigged parametric body model.
- Clothing Model: The pipeline deforms a template cloth mesh by first computing an unposed state, then applying skinning to produce the full cloth deformation.The unposed state incorporates corrective displacements before the final skinning step.
- Clothing Model: For tight garments, collision postprocessing pushes cloth vertices outside the body because the learned model does not guarantee collision-free outputs.Figure 3 contrasts apparent body-cloth collisions with the corrected result after postprocessing.
- Garment Fit Regressor: Garment fit captures global stretch or relaxation as a static corrective displacement determined by body shape alone.Fit displacements are learned from simulations of garments on unposed bodies and applied to the template cloth mesh.
- Garment Wrinkle Regressor: Garment wrinkles capture local dynamic deformation as a corrective displacement conditioned on both body shape and pose.The wrinkle regressor operates in the body's rest pose after removing the global garment fit from the simulated cloth.
- Garment Wrinkle Regressor: Nonlinear regressors model garment fit and wrinkles, while a recurrent neural network captures the history-dependent dynamics of wrinkle deformation.The wrinkle model uses a GRU-based RNN, avoiding an external feedback loop and its associated dimensionality-reduction requirement.
4. Training Data and Regressor Settings
The method trains nonlinear regressors on physics-based simulations of garments across diverse body shapes and motions. Separate networks model global garment fit and local, history-dependent wrinkles.
- Training Data: A novel dressed-character dataset provides ground-truth animations across diverse motions and body shapes.The prototype uses one garment but is intended to extend to other garments or combinations.
- Training Data: The simulations use 17 SMPL body shapes and 56 CMU motion sequences totaling 7,117 frames for an 8,710-triangle T-shirt.Each sequence is simulated for every selected body shape using ARCSim.
- Ground-Truth Extraction: The training pipeline generates garment-fit data by interpolating body shape, relaxing the cloth, and extracting fit displacements.Fit ground truth is computed after the body reaches its target shape and the garment settles.
- Ground-Truth Extraction: For animated data, shape and pose are interpolated from the template state before cloth relaxation and motion simulation produce wrinkle displacements.The resulting cloth meshes provide dynamic training targets for wrinkle regression.
- Regressor Settings: The garment-fit MLP uses 20 hidden neurons, whereas the garment-wrinkle GRU uses 1500 hidden neurons and dropout regularization.Both networks are implemented in TensorFlow.
- Regressor Settings: Training uses Adam for 2000 epochs; the wrinkle GRU trains on 52 sequences while reserving 4 sequences for testing, with TBPTT limited to 90 time steps.The GRU uses batches of 128 during training.
5. Evaluation
The evaluation compares the data-driven method with physics-based simulation and several linear or retargeting baselines. Results show plausible garment fit and wrinkles on unseen shapes and poses, while retaining a major runtime advantage.
- Evaluation Overview: The evaluation measures visual fidelity and runtime performance against state-of-the-art methods and full physics-based simulation.It includes quantitative and qualitative tests for virtual try-on.
- Generalization to New Poses: The method retains rich, history-dependent wrinkles, whereas linear regression produces smoothing and blending artifacts even on training sequences.This comparison isolates the benefit of nonlinear regression for cloth deformation.
- Generalization to New Shapes: For unseen body shapes, retargeting error increases as shape deviates from the nominal shape, while the proposed method’s error remains stable.The comparison evaluates both static poses and dynamic sequences using per-vertex mean error.
- Generalization to New Poses: The two-step nonlinear approach outperforms Linear Blend Skinning and Linear Regression on test sequences with complex, unseen poses.The linear regressor exhibits blending artifacts, especially in the accompanying animation.
- Generalization to New Shapes: On unseen shapes, the method reproduces physics-based overall drape and mid-scale wrinkles, while retargeting suffers noticeable artifacts away from the base shape.The evaluation changes body shape over time in a static pose.
- Comparison with Other Methods: The method realistically adapts one garment to avatars with substantially different body shapes, unlike DRAPE’s scaling-based deformation.DRAPE produces plausible but unrealistic results when fitting the garment to target shapes.
- Qualitative Evaluation: On an unseen test sequence, the method predicts overall fit and mid-scale wrinkles while running three orders of magnitude faster than physics-based simulation.Additional frames show realistic wrinkles and dynamics during crouching.
6. Conclusions and Future Work
The method enables efficient virtual try-on by separating garment fit from dynamic wrinkles while modeling nonlinearities and dynamics. Its main limitations concern garment/material coverage, collisions, high-frequency wrinkles, and loose clothing.
- Conclusions: The method enables virtual try-on at over 250 fps by independently modeling garment fit from body shape and wrinkles from shape and pose.The regressors represent nonlinearities and dynamics, addressing limitations of previous data-driven approaches.
- Future Work: Training is independent for each garment size and material, and the dataset contains only one one-size garment with a specific fabric.Including material as an input would make wrinkle regression more challenging and may produce over-smoothed results with the proposed architecture.
- Future Work: Collisions are not fully handled because the regressors learn from collision-free data and are not guaranteed to produce collision-free results.The authors suggest adding low-level collision constraints as an explicit regressor objective.
- Future Work: The method predicts overall drape and mid-scale wrinkles but excessively smooths high-frequency wrinkles in space and time.Future work will investigate recursion methods for history-dependent draping and highly dynamic wrinkles.
- Future Work: The static-fit and dynamic-wrinkle decomposition may be inaccurate for loose clothing because the model assumes garments closely follow the body.The authors identify testing under such conditions as future work.