Source-linked AI summary
PBNS: Physically Based Neural Simulator for Unsupervised Garment Pose Space Deformation
Hugo Bertiche, Meysam Madadi, Sergio Escalera
TL;DR
Garment animation faces a trade-off between realistic but costly PBS and efficient LBS/PSD methods that need substantial data for learned deformations. PBNS learns PSD for rigged garments through an implicit PBS formulation, without simulation or scan data. The resulting garments show cloth-consistent, pose-dependent wrinkles, and the method handles multiple layers and custom avatars.
Problem
PBS is computationally expensive, while learning realistic LBS/PSD garment deformations commonly depends on large volumes of simulation or scan data.
Source: §1 · p. 2Method
PBNS uses deep learning formulated as an implicit PBS to unsupervisedly learn PSD for rigged garments without PBS or scan data.
Source: §1 · p. 2Results
PBNS produces cloth-consistent garments with pose-dependent wrinkles and temporal coherence on unseen pose sequences.
Source: §1 · p. 2Takeaways & Limitations
PBNS handles multiple cloth layers and can be adapted to any 3D avatar, broadening its applicability to complete outfits.
Source: §7 · p. 15The pose-to-mesh mapping is unique and omits temporal behavior, limiting its ability to reproduce dynamic motion in very loose garments such as dresses and long skirts.
Source: §7 · p. 15
Abstract
from arXiv · showhide
We present a methodology to automatically obtain Pose Space Deformation (PSD) basis for rigged garments through deep learning. Classical approaches rely on Physically Based Simulations (PBS) to animate clothes. These are general solutions that, given a sufficiently fine-grained discretization of space and time, can achieve highly realistic results. However, they are computationally expensive and any scene modification prompts the need of re-simulation. Linear Blend Skinning (LBS) with PSD offers a lightweight alternative to PBS, though, it needs huge volumes of data to learn proper PSD. We propose using deep learning, formulated as an implicit PBS, to unsupervisedly learn realistic cloth Pose Space Deformations in a constrained scenario: dressed humans. Furthermore, we show it is possible to train these models in an amount of time comparable to a PBS of a few sequences. To the best of our knowledge, we are the first to propose a neural simulator for cloth. While deep-based approaches in the domain are becoming a trend, these are data-hungry models. Moreover, authors often propose complex formulations to better learn wrinkles from PBS data. Supervised learning leads to physically inconsistent predictions that require collision solving to be used. Also, dependency on PBS data limits the scalability of these solutions, while their formulation hinders its applicability and compatibility. By proposing an unsupervised methodology to learn PSD for LBS models (3D animation standard), we overcome both of these drawbacks. Results obtained show cloth-consistency in the animated garments and meaningful pose-dependant folds and wrinkles. Our solution is extremely efficient, handles multiple layers of cloth, allows unsupervised outfit resizing and can be easily applied to any custom 3D avatar.
1. Introduction
PBS can produce realistic cloth but is computationally expensive, while LBS and PSD are efficient alternatives with lower realism. PBNS learns physically consistent PSD for rigged garments without simulation or scan data, targeting efficient, compatible animation with realistic cloth behavior.
1. Introduction
- PBS trades increased realism for greater computational cost, whereas LBS and PSD use fewer resources but compromise animation realism.LBS and PSD are suited to low-computing or real-time settings, including portable devices and videogames.Source: §1 · p. 2
- PBNS learns PSD for rigged garments through deep learning formulated as an implicit PBS, enabling unsupervised training without simulation or scan data.The authors report training in minutes, reducing the time from garment design to deployment.Source: §1 · p. 2
- PBNS produces cloth-consistent garments with pose-dependent wrinkles and temporal coherence on unseen pose sequences.The paper reports these outcomes in its animated garments and supplementary video.Source: §1 · p. 2
- PBNS enforces physical consistency during training, removing dependence on PBS or scan data and supporting application to garments, body shapes, and poses without data-generation costs.This is the paper's stated consequence of unsupervised PSD learning.Source: §1 · p. 2
- PBNS yields standard LBS blend shapes, making its output compatible with graphic engines and their existing optimization.The paper contrasts this with complex formulations in related deep-learning approaches, which hinder training and deployment.Source: §1 · p. 2
- PBNS predictions are collision-free even under extreme unseen poses and preserve edge and surface quality, avoiding collision-solving post-processing.The method enforces edge and bending constraints to prevent distorted edges and produce smooth surfaces.Source: §1 · p. 2
- PBNS explicitly handles interactions between overlapping cloth layers, allowing animation of complete, complex outfits.The authors describe this as the first learning-based approach to explicitly handle cloth-to-cloth interactions.Source: §1 · p. 2
- PBNS models use a few megabytes of memory and run at over 14,000 samples per second, while deep-based alternatives can require substantial computational resources.The comparison concerns applicability: when resources are available, the paper says real-time GPU cloth simulation is preferable.Source: Table 1 · p. 3, §1 · p. 3
2. State-of-the-art
Garment animation research spans physically based simulation, efficient LBS/PSD methods, and data-driven learning approaches. The section describes their trade-offs and the data and per-garment training demands that motivate PBNS.
2.1. Computer Graphics
- PBS can deliver highly realistic cloth dynamics, but high realism comes at substantial computational expense, limiting its use when resources are unavailable or real-time performance is required.The section also notes that simulation parameters require expert fine-tuning; LBS is used when efficiency matters, at the cost of realism.Source: §2.1 · p. 3
2.2. Learning-Based Approaches
- LBS is efficient but can miss cloth's nonlinear motion and produce skinning artifacts; PSD adds pose-dependent corrective deformations and high-frequency detail.Hand-crafted PSD becomes intractable for complex models and is commonly learned from data.Source: §2.2 · p. 3
- Deep garment models learn from PBS data or scans, but require large datasets and often repeat simulation and learning for each garment or outfit.Body-model garment encodings can represent multiple garment types, but still require large training volumes; the section describes PBNS as skipping simulation and learning PSD for a garment or outfit in minutes.Source: §2.2 · p. 4
3. Neural Cloth Simulation
PBNS learns garment-specific pose deformations without PBS or scan training data, using physical consistency to produce cloth-like configurations. Its formulation combines a neural pose mapping with an LBS-compatible PSD model and potential-energy training.
3. Neural Cloth Simulation
- PBNS learns garment- or outfit-specific PSD by enforcing physical laws, addressing prior methods’ dependence on costly simulated data for each garment, body shape, or fabric.The approach is framed as an implicit physically based simulation.Source: §3 · p. 4
3.1. PBS Data and Physical Consistency
- Supervised PBS learning treats a multivalued pose-to-outfit mapping as single-valued, so similar poses with different valid garments can average toward physically inconsistent predictions.Minimizing vertex error against PBS data does not guarantee physical consistency; some prior methods require penetration post-processing.Source: §3.1 · p. 4
3.2. Formulation
- The formulation seeks blend weights and pose-dependent vertex deformations that, after skinning, align garments with the body without collisions and produce realistic cloth behavior.Blend weights begin from the closest body vertex in rest pose; for skirts, optimizing them with network weights can improve convergence speed.Source: §3.2 · pp. 4–5
3.3. Architecture
- A neural pose embedding multiplied by a learned PSD matrix supplies garment deformations, allowing nonlinear pose-to-deformation mappings while controlling matrix capacity.The architecture uses a four-layer, 32-dimensional MLP with ReLU activations; the MLP and PSD matrix are learned together.Source: §3.2 · p. 5, §3.3 · p. 5
3.4. Training
- Training minimizes potential energy so the model predicts consistent, low-energy configurations using cloth elasticity and bending, body-collision, gravity, and pinning terms.Cloth losses discourage stretching or compression and smooth neighboring face normals; collision gradients push outfit vertices outside the body.Source: §3.4 · pp. 5–6
- For layered outfits, PBNS orders garments from inner to outer and iteratively applies collision handling against the body and preceding layers.The authors describe this as the first learning-based approach to explicitly handle cloth-to-cloth interactions.Source: §3.4 · p. 6
- Gravity minimizes garment vertices’ vertical coordinate, while pinning regularizes selected vertices’ deformations to keep them near their original positions.The pinning loss allows slight movement rather than imposing a hard constraint that could cause body collisions; cloth and gravity weights relate to fabric elasticity and mass.Source: §3.4 · p. 6
4. Experiments
The experiments use SMPL and public pose data, with fixed body shape during simulation. Garment templates are prepared in rest pose, and a filtered, subject-split pose database supplies training and validation examples.
4.1. Body model
- SMPL, an LBS model with 24 joints, is used because its model and pose data are public, though the method is compatible with any skeleton-rigged 3D model.Body shape is fixed during neural simulation.Source: §4.1 · pp. 6–7
4.2. Template outfit
- Garment or outfit templates are designed on the body in rest pose at approximately 1 cm resolution, smoothed before simulation, and initialized with blend weights from body proximity.Source: §4.2 · p. 7
4.3. Pose database
- The pose database excludes body poses with self-collisions and uses 3,000 distance-balanced CMU MoCap poses, split by subject into 2,550 training and 450 validation poses.The full dataset contains 2,667 sequences and about 4.3 million poses; train/test sequences are split 85/15% by subject.Source: §4.3 · p. 7
5. Results
PBNS is evaluated against supervised alternatives and tested across unseen poses, layered garments, resizing, and custom avatars. The results report physically consistent, efficient garment deformations across these settings.
5.1. Multi-Layer Perceptron
- Without an MLP, blend shapes are tied linearly to pose parameters, and both no-MLP variants show leg-merging artifacts even after hundreds of epochs.Using θ directly also produces unrealistic V-shaped hip wrinkles; this does not appear when rotation matrices Rθ are used instead.Source: §5.1 · p. 7, Figure 3 · p. 8
5.2. Supervised learning and quantitative evaluation
- PBNS has higher Euclidean error than supervised alternatives but yields cloth-consistent, nearly collision-free predictions quickly, while supervised approaches trade physical consistency for lower error.The hybrid approach generalizes poorly to extreme poses, and PBNS is reported as more robust to simulation failures than classic PBS.Source: Figure 4 · p. 8, Table 2 · p. 9, §5.2 · p. 9
5.3. Qualitative
- Across unseen test poses, learned deformations produce folds and wrinkles, including for a different body shape, after only a few minutes of training.Templates are smoothed, and each sample pairs the dressed human with its deformed outfit in rest pose.Source: §5.3 · p. 9, Figure 5 · p. 10
5.5. Comparison
- PBNS produces pose-dependent wrinkles for unseen poses while maintaining physical consistency, with model size of a few megabytes and hundreds or thousands of executions per second.TailorNet is reported at 3–5 FPS without post-processing and 0.6–0.9 FPS with it; PBNS complexity and computation time are several orders of magnitude lower.Source: §5.5 · p. 11, Figure 8 · p. 13
5.4. Multiple Layers and Controllable Parameters
- PBNS handles overlapping cloth layers with minimal spacing and can vary wrinkle count and size through per-vertex weights for cloth-related losses.The reported examples include up to three overlapping cloth layers near the waist, and the method also supports gloves and boots through parameter control.Source: §5.4 · pp. 9, 11, Figure 7 · p. 12
5.6. Garment Resizing
- For resizing, PBNS takes body shape β and garment tightness γ as input and produces blend shapes that resize outfits across body shapes and tightness settings.The paper reports that this standard blend-shape format requires minimal engineering effort to integrate into existing 3D software.Source: §5.6 · pp. 13–14, Figure 9 · p. 14
5.7. Custom Avatars
- PBNS applies to outfits on any rigged animated model, producing enhanced avatar garments in minutes without requiring PBS data.The reported experiment uses three avatars and outfits designed or reused for them.Source: §5.7 · p. 14
6. Performance
PBNS trains quickly on modest hardware and runs efficiently at test time. The reported speed reflects effective deployment performance because no post-processing is required.
6. Performance
- Over 14k samples per second were generated for an outfit with 12k vertices and 23.7k triangles.The reported speed is effective application speed because PBNS requires no post-processing.Source: §6 · p. 14
- Training takes around 1–2 minutes per epoch on a GTX1080Ti or 2–4 minutes per epoch without a GPU.Convergence takes 10–50 epochs depending on outfit complexity and body shape.Source: §6 · p. 14
- The model was trained on 2,550 poses with batch size 16, and validation predictions were assessed qualitatively because no quantitative error was available.Training used Adam; epoch time varies with the amount of body collisions.Source: §6 · p. 14
- The small MLP does not grow with vertex count, while the skinned PSD models use a standard, highly optimized 3D-animation representation.The authors note this makes the approach suitable when computing resources are needed for other tasks, including games and virtual reality.Source: §6 · p. 14
7. Conclusions, Limitations and Future work
PBNS presents an unsupervised deep-learning approach that produces outfit animation as PSD blend shapes for LBS models. The authors report efficient training, multi-layer handling, avatar adaptation, and unsupervised outfit resizing, while identifying the absence of temporal modeling as a limitation.
7. Conclusions, Limitations and Future work
- PBNS is presented as the first unsupervised deep-learning approach for outfit animation, producing Pose Space Deformations as blend shapes for Linear Blend Skinning models.The authors say this representation integrates into current 3D animation pipelines and can run on low-computing or portable devices.Source: §7 · pp. 14–15
- PBNS handles multiple cloth layers and can be adapted to custom 3D avatars; the authors also present unsupervised outfit resizing.The custom-avatar examples include different garments and outfits.Source: §7 · p. 15, Figure 10 · p. 15
- The authors report training in minutes, including without a GPU, and say this reduces the time from outfit design to deployment compared with previous approaches.They connect the approach to more efficient character design for CGI artists and adding animated models to videogame and virtual try-on outfit databases.Source: §7 · p. 15
- The pose-to-mesh mapping does not account for time, so it cannot represent different garment meshes for the same pose reached through different motion speeds.The authors single out very loose garments, such as dresses and long skirts, as especially dynamic cases static approaches cannot reproduce.Source: §7 · p. 15
- The authors suggest extending unsupervised prediction of stable, physically consistent systems to other soft-tissue bodies, including hair and human-body self-collisions.Source: §7 · p. 15