Source-linked AI summary
GarNet: A Two-Stream Network for Fast and Accurate 3D Cloth Draping
Erhan Gundogdu, Victor Constantin, Amrollah Seifoddini, Minh Dang, Mathieu Salzmann, Pascal Fua
TL;DR
PBS can accurately drape garments but is too costly for real-time use. GarNet uses a two-stream network that fuses multiscale garment and body features under physics-inspired training, achieving PBS-like draping at much higher speed and supporting varied garment patterns.
Problem
Physics-Based Simulation accurately drapes garments but is too computationally costly for real-time applications such as virtual try-on.
Method
GarNet uses parallel body and garment streams to fuse body features with point-wise, patch-wise, and global garment features, trained with physics-inspired loss terms.
Results
GarNet produces garment shapes within 1 cm average distance of PBS results while achieving a 100× speedup.
Takeaways & Limitations
GarNet can drape different garment styles across varied bodies and poses while incorporating garment sewing patterns as input.
Takeaways & Limitations
The model tends to remove high-frequency details because regression produces smoothing.
Abstract
from arXiv · showhide
While Physics-Based Simulation (PBS) can accurately drape a 3D garment on a 3D body, it remains too costly for real-time applications, such as virtual try-on. By contrast, inference in a deep network, requiring a single forward pass, is much faster. Taking advantage of this, we propose a novel architecture to fit a 3D garment template to a 3D body. Specifically, we build upon the recent progress in 3D point cloud processing with deep networks to extract garment features at varying levels of detail, including point-wise, patch-wise and global features. We fuse these features with those extracted in parallel from the 3D body, so as to model the cloth-body interactions. The resulting two-stream architecture, which we call as GarNet, is trained using a loss function inspired by physics-based modeling, and delivers visually plausible garment shapes whose 3D points are, on average, less than 1 cm away from those of a PBS method, while running 100 times faster. Moreover, the proposed method can model various garment types with different cutting patterns when parameters of those patterns are given as input to the network.
1. Introduction
GarNet replaces costly physics-based cloth simulation with a two-stream deep network that models body–garment interactions and produces plausible draping results in real time. Its training objective incorporates terms designed to reduce cloth–body interpenetration and undue tightness.
- Motivation: PBS produces realistic cloth draping but is too computationally expensive for real-time and web-based applications.Garment simulation is relevant to virtual try-on, online shopping, gaming, and virtual reality.
- Architecture: GarNet combines garment and body streams to extract point-wise, patch-wise, and global garment features alongside body features.The resulting representations are fused to predict the fitted garment shape.
- Training objective: The training objective includes loss terms that reduce cloth–body interpenetration and undue tightness without requiring extra post-processing at test time.The approach also scales naturally to point clouds with arbitrary resolution through convolution and pooling operations.
- Capabilities: GarNet handles multiple body poses and shapes, incorporates garment cutting-pattern information, and outperforms the state-of-the-art method of [41] on its dataset.The training dataset contains jeans, a T-shirt, and a sweater worn by 600 SMPL bodies in various poses.
- Efficiency: PBS takes more than 10 seconds to predict a garment shape, whereas GarNet takes less than 70 ms.This runtime difference makes the method practical for real-time applications.
2. Related Work
Related work contrasts computationally expensive PBS with faster data-driven alternatives and develops the point-cloud and mesh-processing foundations used by GarNet. These foundations motivate combining PointNet-style global processing with mesh-based local features for garment fitting.
- Physics-based simulation: PBS provides realistic cloth deformations but is computationally expensive and often requires manual parameter tuning.These limitations motivate data-driven approaches for practical garment simulation.
- Data-driven approaches: Data-driven methods reduce runtime and memory demands by predicting garment shapes from databases, physical parameters, motion graphs, or learned models.Their approaches include interpolation, material-parameter estimation, and stored wrinkle effects.
- Cloth reconstruction: Image- and scan-based methods reconstruct or retarget clothing, while conditional GANs can add fine wrinkles but may require computationally demanding registration.GarNet instead aims to emulate PBS for cloth fitting while producing visually plausible results.
- Point-cloud processing: PointNet uses point-wise MLPs and max-pooling to represent unordered point clouds, but point-wise operations alone are insufficient for visually plausible garment fitting.GarNet therefore also relies on local mesh features.
- Mesh processing: Graph and mesh convolutions provide local vertex features by exploiting triangulated topology and learned neighbor contributions.GarNet adopts this local-feature perspective alongside global point-cloud processing.
3. 3D Garment Fitting
GarNet refines a pose-dependent, skinned garment by combining body and garment streams, then predicts per-vertex translations through a fusion network. Its training objective combines vertex accuracy with physical regularizers that discourage interpenetration, normal mismatch, and bending deviations.
- Problem formulation: GarNet starts from a dual-quaternion-skinned garment and predicts per-vertex translations to obtain the final fitted shape.The network maps the skinned garment and body to predicted translations that are added to the skinned vertex positions.
- Network architecture: The two-stream architecture processes the body as a 3D point cloud and the garment as a triangulated mesh before fusing their features to predict garment translations.The fusion network uses shared MLP blocks and outputs the predicted translations from the warped garment.
- Network architecture: The body stream extracts point-wise and global features, while the garment stream uses body context and mesh convolutions to derive point-wise, patch-wise, and global garment features.Skip connections forward lower-level features into later stages, preserving local information while higher-level representations are formed.
- Network architecture: GarNet-Global uses global body features, whereas GarNet-Local additionally pools local body features from nearest body vertices for each garment vertex.The local variant explicitly incorporates point-wise body features without requiring direct body-garment correspondences.
- Loss function: The loss combines average vertex distance with normal, interpenetration, and bending terms weighted by separate coefficients.The interpenetration term activates near the ground-truth position and also penalizes body-garment triangle intersections; the bending term preserves two-edge neighborhood distances.
- Loss function: The bending regularizer emulates the PBS bending constraint by penalizing deviations in distances between garment vertices connected through two-edge shortest paths.Larger neighborhoods are possible in principle, but their pair count would grow exponentially.
4. Experiments
Experiments evaluate GarNet on simulated garments using vertex-distance and facet-normal metrics, runtime comparisons, unseen poses, qualitative results, and ablations. The results show that local and global variants closely approach PBS quality, while patch-wise features and physics-inspired loss terms improve fidelity and both variants provide a 100× speedup.
- Evaluation Metrics: Edist measures average vertex-to-vertex distance, while Enorm measures average angular deviation between predicted and ground-truth facet normals.Enorm matters because facet normals influence rendered garment appearance.
- Quantitative Results: GarNet-Naive performs worse than GarNet-Global and GarNet-Local, highlighting the importance of patch-wise garment features.GarNet-Global and GarNet-Local achieve comparable results, with an overall advantage for GarNet-Local.
- Quantitative Results: 100× speedup is achieved by both GarNet variants compared with the employed PBS software.The computation-time comparison uses a single Nvidia TITAN X GPU, with network forward propagation at batch size 16.
- Tests on unseen poses: On unseen poses, distance and angle errors increase to 1.16 (1.68) cm and 9.71 (11.88)°, respectively, across the reported pose splits.The splits use 50% (25%) of poses for training and the remainder for testing; the authors interpret performance as indicating generalization ability.
- Qualitative Results: GarNet-Naive produces visible shoulder artifacts, whereas GarNet-Local, GarNet-Global, and PBS yield similar T-shirt results.GarNet-Global can show a gap between body and garment, while GarNet-Local is more similar to PBS.
- Ablation study: Removing normal and bending loss terms reduces angle accuracy, while removing the penetration term causes more severe interpenetration despite limited quantitative impact.The ablation study evaluates the individual penetration, bending, and normal terms and their combination.
5. Conclusion
GarNet drapes garments across bodies and poses while running 100 times faster than a physics-based simulator. Its physical-constraint loss and pattern conditioning support plausible, varied garment fits, though high-frequency details remain limited.
- 100 times faster than a physics-based simulator, GarNet drapes 3D garments on different bodies across many poses.
- Using the full loss produces results very similar to PBS, whereas removing loss terms can cause interpenetration or different back wrinkles.
- Different garment sewing patterns allow the method to generalize to accurately draping different garment styles.
- Less than 1 cm average distance from PBS results is achieved while limiting interpenetrations and other artifacts.
- The method still tends to remove high-frequency details because regression tends to smooth.