Source-linked AI summary
DeepWrinkles: Accurate and Realistic Clothing Modeling
Zorah Laehner, Daniel Cremers, Tony Tung
TL;DR
Realistic cloth reconstruction remains difficult because physics-based simulation is labor-intensive and visual methods often miss fine geometric details. DeepWrinkles combines a data-learned subspace model for global deformation with a conditional adversarial network that adds temporally consistent wrinkle details to normal maps. The authors report high-quality rendering of clothing deformation, including recovery of global shape and fine wrinkles, with body-shape and pose retargeting applications.
Problem
Realistic clothing reconstruction must recover fine wrinkles that existing physics-based and visual approaches handle inefficiently or incompletely.
Method
DeepWrinkles learns global clothing deformation from 3D scans in a body- and pose-factored subspace and generates fine normal-map details with a temporally consistent conditional adversarial network.
Results
The authors report high-quality rendering that recovers global clothing shape and fine wrinkles from real high-resolution observations using an entirely data-driven approach.
Takeaways & Limitations
The complementary modules provide a flexible data-driven framework for accurate clothing rendering and body-shape or pose retargeting.
Takeaways & Limitations
Pose retargeting works best for poses similar to the training data, while unseen joint-parameter sequences can make global-shape reconstruction challenging.
Abstract
from arXiv · showhide
We present a novel method to generate accurate and realistic clothing deformation from real data capture. Previous methods for realistic cloth modeling mainly rely on intensive computation of physics-based simulation (with numerous heuristic parameters), while models reconstructed from visual observations typically suffer from lack of geometric details. Here, we propose an original framework consisting of two modules that work jointly to represent global shape deformation as well as surface details with high fidelity. Global shape deformations are recovered from a subspace model learned from 3D data of clothed people in motion, while high frequency details are added to normal maps created using a conditional Generative Adversarial Network whose architecture is designed to enforce realism and temporal consistency. This leads to unprecedented high-quality rendering of clothing deformation sequences, where fine wrinkles from (real) high resolution observations can be recovered. In addition, as the model is learned independently from body shape and pose, the framework is suitable for applications that require retargeting (e.g., body animation). Our experiments show original high quality results with a flexible model. We claim an entirely data-driven approach to realistic cloth wrinkle generation is possible.
1 Introduction
DeepWrinkles addresses realistic clothing reconstruction by combining data-driven global deformation modeling with neural generation of fine wrinkle details. The framework supports high-quality rendering and body-shape or pose retargeting.
- Realistic garment reconstruction matters for clothed-human estimation, AR/VR, movies, games, and virtual try-on.
- Physics-based simulations require substantial expert labor, while visual reconstruction methods struggle to recover fine cloth wrinkles.
- DeepWrinkles uses real data capture to generate accurate and realistic clothing deformation.
- A statistical subspace model represents clothing shape after factoring out body shape and pose, enabling body retargeting.
- A conditional adversarial network adds fine geometric details to normal maps while enforcing realism and temporal consistency.
- The framework recovers global shape and fine wrinkles from high-resolution observations for high-quality clothing-deformation rendering.
2 Related Work
Related work spans physics-based simulation and real-data reconstruction. Existing approaches trade computational effort against reconstruction detail, motivating DeepWrinkles’ combination of learned shape modeling and neural detail generation.
- Cloth modeling traditionally uses physics-based simulation or captures and reconstructs details from real-world data.
- Physics-based methods model material properties and external forces but can require weeks of computation, retouching, and expert parameter tuning.
- DeepWrinkles’ outline combines a statistical model and pose-to-blend-shape mapping with a cGAN that generates fine normal-map details.
- Vision-based methods reconstruct garment surfaces from videos or scans, but insufficient priors can produce crude geometry.
3D capture and reconstruction.
Prior reconstruction methods improve coarse garment geometry through templates, dimensionality reduction, and learned visual models, but fine-detail recovery and generalization remain constrained. DeepWrinkles instead uses normal mapping and a deep neural network to refine clothing geometry from coarse inputs.
- Templates, super-resolution, photometric stereo, lighting, and depth information can improve reconstruction quality from visual observations.
- Template registration from 4D scans supports retargeting to different body shapes but remains limited in recovering fine wrinkles.
- Linear subspace and coarse-to-fine methods represent detailed clothing with dimension reduction, while DRAPE factors out body shape and pose.
- Some learned models generalize wrinkle patterns or reconstruct global shape from images, but DeepGarment does not support retargeting.
- GANs and related neural methods enhance low-dimensional inputs with realistic details, motivating conditional generation for this problem.
- DeepWrinkles revisits the 3D rendering pipeline by using computer vision and normal mapping to represent high runtime detail.
- Existing displacement-map wrinkle methods require high-resolution meshes and do not guarantee temporal consistency.
- The paper claims the first entirely data-driven method using a deep neural network on normal maps to leverage clothing’s 3D geometry.
3 Deformation Subspace Model
DeepWrinkles learns a compact clothing-deformation model from real 4D scans, separating body pose and shape from garment deformation. Non-rigid registration, PCA, and pose-to-shape prediction support detailed, flexible retargeting, although the subspace model alone misses high-frequency wrinkles.
- Data preparation: A clothing template is non-rigidly registered to each scan using grid deformation and an objective combining data, rigidity, smoothness, and boundary terms.Boundary alignment is explicitly included because nearest-neighbor matching can miss distant fold details.
- Data preparation: Boundary matching selects a farther scan point among nearby candidates for each template boundary point, improving consistency and capture of high-frequency details.This strategy addresses the failure of simple closest-neighbor pairing to represent distant boundary details.
- Statistical model: Pose-normalized registrations are decomposed with PCA into a mean shape and principal deformation directions, while body pose and shape are factored out.The resulting representation uses k shape parameters instead of full vertex coordinates.
- Statistical model: 500 principal components represent all registrations with less than 5 mm error, yielding a compact linear subspace model.The principal directions are ordered by singular value, prioritizing dominant deformations.
- Pose-to-shape prediction: Pose-to-shape prediction uses pose-derived control vectors and can incorporate velocity, acceleration, and shape history through linear regression or recurrent sequence models.Linear regression provides limited pose retargeting but loses reconstruction details because identical poses can produce different folds during motion.
- Pose-to-shape prediction: Registration and subspace modeling recover global shape but do not recover all visible high-frequency details, even when template resolution increases.This limitation motivates adding a separate surface-detail module.
4 Fine Wrinkle Generation
DeepWrinkles generates fine clothing details by enhancing low-resolution normal maps with a conditional GAN, while incorporating temporal consistency for smoother animation.
- Template- and subspace-based methods cannot recover fine cloth wrinkles because of resolution and data scaling limitations.
- The method encodes high-resolution scan details in normal maps and uses a generative adversarial network to add them to reconstructed clothing.
- Training pairs combine low-resolution normal maps from blend-shape reconstruction with high-resolution normal maps projected from 4D scans.
- The cGAN uses U-Net skip connections to preserve image structure and a patch-based discriminator to enforce realistic wrinkle patterns.
- A temporal consistency loss penalizes discrepancies across consecutive frames so generated clothing details transition without sudden jumps.
5 Experiments
Experiments compare DeepWrinkles with simulation, subspace, and registration methods, test input-detail requirements, and demonstrate retargeting across body shapes and motion.
- 5.1 Comparison of approaches: The comparison uses 5,048-vertex meshes for the three data-driven approaches, while the physics-based simulation uses 53,518 vertices.
- 5.1 Comparison of approaches: DeepWrinkles adds high-frequency scan details to a reconstructed surface, whereas subspace reconstruction is mostly flat and registration captures only major folds.
- 5.2 Importance of reconstruction details in input: 500 blend-shape basis functions provide sufficient input detail for a reasonable wrinkle result, while regression inputs produce many high-frequency wrinkles instead of long defined folds.
- 5.2 Importance of reconstruction details in input: The input structure matters because the network redefines visible structures rather than reconstructing important folds absent from its input.
- 5.3 Retargeting: An encoder-decoder with LSTM units produced promising reconstructed sequences with MSErnn = 1.892.
- 5.3 Retargeting: Replacing the clothing mean shape and using tangent-space normal maps makes the model applicable to different body shapes.
6 Conclusion
DeepWrinkles reconstructs clothing in motion from 4D scans through complementary data-driven modules for global shape and fine details. The method achieves high-detail rendering while retaining practical limitations around occlusion and pose retargeting.
- 6 Conclusion: The framework reconstructs clothing in motion from 4D scan sequences using complementary modules for global shape and surface details.The approach combines a clothing subspace model with neural normal-map refinement.
- 6 Conclusion: A conditional adversarial neural network adds high-frequency details to low-resolution normal maps, while an additional temporal loss preserves consistency across time.Registration fidelity is crucial for training the conditional GAN.
- 6 Conclusion: The two modules jointly produce accurate and realistic rendering of both global clothing shape and fine details.The authors report qualitative and quantitative evaluations on different datasets.
- 6 Conclusion: Pose retargeting works best for new poses similar to those represented in the training dataset.The network generalizes to some unseen poses, but reconstructing global shape from new joint sequences can remain challenging.
- 6 Conclusion: Unseen camera regions, including armpits, can cause missing normal-map information and visually disruptive artifacts.The clothing template can fix most issues, for example through smoothing.