PaperlayerView exact revision on arXiv

Source-linked AI summary

TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style

Chaitanya Patel, Zhouyingcheng Liao, Gerard Pons-Moll

arXiv:2003.04583v2cs.CVcs.GR

TL;DR

Existing clothing models do not jointly represent pose, body shape, and garment style while retaining wrinkle detail. TailorNet decomposes deformation into low- and high-frequency components, predicting the latter with a narrow-kernel mixture of style-shape-specific pose models. It generalizes to novel AMASS poses, produces detailed and temporally smooth animations, and runs more than 1000 times faster than PBS while remaining differentiable.

  • Problem

    Existing methods do not jointly model pose, body shape, and garment style while retaining fine wrinkle detail.

  • Method

    TailorNet combines an MLP for low-frequency deformation with a narrow-bandwidth mixture of style-shape-specific pose models for high-frequency deformation.

  • Results

    TailorNet generalizes to novel AMASS poses, produces detailed smooth animations, and runs more than 1000 times faster than PBS.

  • Takeaways & Limitations

    The model provides controllable, differentiable 3D clothing deformation across pose, shape, and style, with wrinkle detail retained for computer-vision and animation applications.

  • Takeaways & Limitations

    The style-shape similarity kernel is an approximation because directly measuring deformation similarity across all pose-shape-style combinations would require extensive simulation.

Abstract

from arXiv · show

In this paper, we present TailorNet, a neural model which predicts clothing deformation in 3D as a function of three factors: pose, shape and style (garment geometry), while retaining wrinkle detail. This goes beyond prior models, which are either specific to one style and shape, or generalize to different shapes producing smooth results, despite being style specific. Our hypothesis is that (even non-linear) combinations of examples smooth out high frequency components such as fine-wrinkles, which makes learning the three factors jointly hard. At the heart of our technique is a decomposition of deformation into a high frequency and a low frequency component. While the low-frequency component is predicted from pose, shape and style parameters with an MLP, the high-frequency component is predicted with a mixture of shape-style specific pose models. The weights of the mixture are computed with a narrow bandwidth kernel to guarantee that only predictions with similar high-frequency patterns are combined. The style variation is obtained by computing, in a canonical pose, a subspace of deformation, which satisfies physical constraints such as inter-penetration, and draping on the body. TailorNet delivers 3D garments which retain the wrinkles from the physics based simulations (PBS) it is learned from, while running more than 1000 times faster. In contrast to PBS, TailorNet is easy to use and fully differentiable, which is crucial for computer vision algorithms. Several experiments demonstrate TailorNet produces more realistic results than prior work, and even generates temporally coherent deformations on sequences of the AMASS dataset, despite being trained on static poses from a different dataset. To stimulate further research in this direction, we will make a dataset consisting of 55800 frames, as well as our model publicly available at https://virtualhumans.mpi-inf.mpg.de/tailornet.

1. Introduction

TailorNet addresses the lack of a unified clothing model that handles pose, body shape, and garment style while retaining wrinkle detail. It separates low- and high-frequency deformation and uses a mixture of style-shape-specific pose models to preserve fine structure.

  • Motivation: Existing methods model only subsets of pose, shape, and style, while joint models often produce overly smooth clothing without fine wrinkles.The paper identifies jointly modeling these intertwined factors while retaining high-frequency detail as the central gap.
  • Results: For fixed garment style and body shape, an MLP performs as well as or better than Graph Neural Networks and UV-space image-decoder models for pose-dependent deformation.This comparison motivates using a simple MLP within the broader model.
  • Method: TailorNet predicts low-frequency deformation with an MLP and high-frequency deformation with a mixture of style-shape-specific pose models.A narrow-bandwidth kernel combines predictions with similar style-shape deformation patterns, reducing smoothing of fine wrinkles.
  • Contribution: TailorNet is presented as the first joint model of clothing style, pose, and shape variation that is easy to deploy and fully differentiable.Differentiability supports integration with computer-vision and deep-learning systems.
  • Resources: 55800 frames cover 20 aligned real static garments simulated in 1782 poses for 9 body shapes.The dataset is released to stimulate further research.
  • Results: TailorNet generalizes to unseen AMASS poses, produces pose-, shape-, and style-dependent variation, and generates smooth continuous animations despite training on static poses.The model is reported to be more detailed than previous style-specific models.

2. Related Work

Prior clothing-animation methods rely on physics simulation, real captures, or data-driven approximations, each with practical or generalization limitations. TailorNet is positioned against these approaches as a model that jointly handles garment style, pose, and body shape.

  • Physics Based Simulation: Physics-based simulation can be highly realistic but is computationally expensive, complex to control, and laborious to design and tune.PBS workflows may require 2D garment design, 3D adjustment, and parameter fine-tuning taking hours or weeks.
  • Data-driven cloth models: Real-capture methods are limited in retargeting motion to novel shapes, while pose-dependent capture models generally demonstrate only motion patterns similar to training data.Accurately capturing sufficient real clothing data remains a major challenge.
  • Data-driven cloth models: PBS-generated data can train efficient pose or pose-shape models, but prior methods do not jointly model garment style, pose, and shape.The paper compares MLPs, Graph-NNs, and UV-space image translation for fixed style and shape.
  • Pixel based models: Pixel-based approaches can produce photorealistic images in controlled settings but do not capture the 3D shape of wrinkles or easily control motion, viewpoint, and illumination.Their output is therefore not equivalent to controllable 3D cloth deformation.

3. Garment Model Aligned with SMPL

The garment model aligns clothing with the SMPL body representation, encoding garment style as displacements over an unposed body shape and applying articulated skinning. TailorNet learns these non-rigid clothing variations as functions of pose, shape, and style rather than keeping displacement fixed.

  • SMPL alignment: SMPL represents the body parametrically through pose and shape, with pose- and shape-dependent functions followed by learned skinning.The garment template is aligned with the SMPL template through a submesh and vertex associations.
  • Garment representation: A garment style is represented by displacement D over the unposed body shape, while the articulated garment motion is produced by the SMPL skinning function.The model separates non-rigid deformation from articulated deformation.
  • Garment representation: Keeping D fixed assumes clothing deforms like the body, which the paper identifies as a practical but over-simplifying assumption.TailorNet instead learns non-rigid deformation as a function of pose, style, and shape.
  • Model overview: The model overview adds low- and high-frequency deformation predictions before applying standard skinning to obtain the posed garment.The high-frequency component combines pose predictions for prototype shape-style pairs with an RBF kernel.
  • Comparison: Table 1 presents TailorNet as the first method modeling garment deformation as a function of pose, shape, and style and notes public release of the model and dataset.The table caption frames the comparison around joint factor modeling and research availability.

4. Method

TailorNet separates clothing deformation into low- and high-frequency components, combining pose, shape, and style while preserving wrinkle detail. It represents style variation in a canonical pose and uses style-shape-specific pose models blended according to similarity.

  • 4.1. Un-posing Garment Deformation: TailorNet disentangles articulated motion from non-rigid garment deformation by un-posing simulated garments and representing the result as body-relative displacements.The un-posed deformation depends on pose θ, body shape β, and garment style γ.
  • 4.2. Style Space: The method constructs a physically constrained garment-style subspace by alternating PCA sampling with physics-based simulation in a canonical A-pose.This process addresses flying garments, pose-inconsistent wrinkles, and registration distortions.
  • 4.3. Single Style-Shape Model: For each fixed style-shape pair, an MLP predicts pose-dependent displacements that reproduce garment fit, fine-scale wrinkles, and unseen-pose behavior.These models remain specific to one body shape and garment style.
  • 4.4. TailorNet: Directly regressing deformation from pose, shape, and style produces overly smooth results because combining samples with different wrinkle patterns suppresses high-frequency detail.The paper motivates separate treatment of low- and high-frequency geometry from this observation.
  • 4.2. Style Space: The style space’s first two components control overall T-shirt size and sleeve length, and sampling them generates varied garment styles.The displayed samples use −1.5σ, 0, and 1.5σ for the first two components.
  • 4.4. TailorNet: Low-frequency geometry is predicted with an MLP, while high-frequency geometry is produced by a mixture of style-shape-specific pose models.The mixture uses a narrow-bandwidth kernel to combine only nearby style-shape or wrinkle patterns, enabling generalization beyond prototypes.
  • 4.4. TailorNet: The kernel compares style-shape pairs through canonical-pose displacement similarity, an approximation intended to measure clothing-fit similarity without simulating every combination.The paper reports that this simplifying assumption works well in practice.
  • 4.4. TailorNet: The prototype-selection procedure greedily adds style-shape pairs with the largest canonical-pose approximation error, and reports good coverage when K ⩾20.The selection targets convex-combination approximation of static displacements.

5. Dataset

The dataset is generated from real-world digital-wardrobe garments and physics-based simulations across garment styles, body shapes, and poses. Iterative style-space construction and systematic train/test splits provide data for evaluating pose and style-shape generalization.

  • Dataset generation: TailorNet’s simulated training data starts from publicly available digital-wardrobe garments containing real-world variation in styles, shapes, and poses.The garments are simulated with Marvelous Designer rather than manually designed and positioned.
  • Style-space construction: Two rounds of PCA sampling and physics-based simulation expand 43 canonical garments into consistent style variations, with the first two PCA components representing γ.The expanded style space includes variation in sleeve length, size, and fit.
  • Shape-style sampling: The dataset uses 9 body shapes and 25 styles, producing 225 canonical shape-style instances before selecting prototype and test pairs.Twenty prototypes and 20 additional pairs are selected for testing.
  • Pose simulation: Pose variation is generated from 1782 static SMPL poses, including extreme poses, with interpolated transitions and relaxation frames to target quasi-static deformation.Each style-shape pair is simulated through a sequence of poses.
  • Evaluation splits: Four splits separately test seen and unseen poses and style-shape pairs: train-train, train-test, test-train, and test-test.The splits distinguish pose generalization from generalization to new style-shape combinations.

6. Experiments

Experiments evaluate TailorNet against MLP baselines and prior models on pose, shape, and style generalization. The mixture model preserves garment details qualitatively, runs far faster than PBS, and produces temporally coherent results on unseen AMASS poses.

  • Single style-shape model: 8.04 mm average error across single style-shape models, with 14.50 mm maximum and 6.56 mm minimum errors.The maximum error occurs for a loose-fitting garment and the minimum for a fit-fitting garment.
  • Single style-shape model: A simple MLP performs as well as or better than UV decoding and Graph CNNs for pose-dependent deformation at fixed style and shape.The comparison uses mean per vertex error on two style-shape pairs.
  • TailorNet results: The mixture model slightly outperforms the baseline quantitatively on three testing splits and significantly outperforms it qualitatively.Qualitative comparisons report more meaningful folds, garment details, and wrinkles than baseline and prior work.
  • TailorNet results: Decomposing high- and low-frequency deformation retains fine wrinkles while preserving overall garment fit, unlike direct prototype mixing.TailorNet retains as many wrinkles as PBS ground truth in the qualitative comparison.
  • Generalization: TailorNet generalizes to unseen AMASS poses, supports multiple garments and styles, and produces temporally coherent animations despite training on static poses.The evaluated garments include Shirt, Pants, Skirt, and T-Shirt.
  • Efficiency: 1–2 ms per frame makes TailorNet 1000 times faster than PBS on a laptop GPU, and 100 times faster on CPU.The reported hardware is an NVIDIA GeForce GTX 1060 GPU with an Intel i7 CPU.

7. Discussion and Conclusion

The discussion identifies joint pose, shape, and style modeling as a difficult source of oversmoothing and presents frequency decomposition with a narrow-bandwidth mixture as the remedy. TailorNet combines detail preservation with fast, differentiable, manually controllable clothing synthesis, while future work targets real scans, video, cloth, and soft-tissue properties.

  • Discussion: 11 mm accuracy from a simple MLP matches graph-CNN and UV-space displacement methods for clothing deformation.The model shares the limitation that training across body shapes and styles can produce overly smooth, unrealistic results.
  • Discussion: Jointly modeling pose, shape, and style can oversmooth clothing, while narrow-bandwidth high-frequency mixtures preserve significantly more detail.TailorNet predicts low- and high-frequency components separately to address this limitation.
  • Future work: Planned extensions include fitting scans, images, and videos, refining with video or real 3D cloth captures, and modeling fabric and human soft-tissue properties.These are identified as future work rather than capabilities established by the current experiments.
  • Conclusion: TailorNet is 1000 times faster than PBS, enables manual-edit-free control of style, shape, and pose, and is differentiable for computer vision applications.The paper positions realistic clothing as a missing component of body models such as SMPL and ADAM.
Loading 2003.04583v2…