PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Igor Santesteban, Nils Thuerey, Miguel A. Otaduy, Dan Casas

arXiv:2105.06462v1cs.CV

TL;DR

Data-driven virtual try-on methods can produce garment-body interpenetrations at test time, while physics-based solutions are computationally expensive. This paper combines a diffused body model, a canonical garment space, and self-supervised generative collision learning. It reports collision-free garment configurations across shape and pose settings without postprocessing, with detailed garments inferred quickly.

  • Problem

    Data-driven garment models commonly produce body interpenetrations at test time despite training penalties, while physics-based methods are computationally expensive for real-time virtual try-on.

    Source: §1 · p. 1

  • Method

    The method diffuses body-model parameters to 3D points, defines a canonical garment space, and trains a generative deformation model with self-supervised collision testing.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    The method reports garments that do not interpenetrate the body across shape and pose settings, with high-quality detailed garments inferred extremely quickly.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The paper concludes that a learned garment representation can avoid collisions while reducing inference complexity and retaining spatial and temporal detail.

    Source: §7 · p. 8

Abstract

from arXiv · show

We propose a new generative model for 3D garment deformations that enables us to learn, for the first time, a data-driven method for virtual try-on that effectively addresses garment-body collisions. In contrast to existing methods that require an undesirable postprocessing step to fix garment-body interpenetrations at test time, our approach directly outputs 3D garment configurations that do not collide with the underlying body. Key to our success is a new canonical space for garments that removes pose-and-shape deformations already captured by a new diffused human body model, which extrapolates body surface properties such as skinning weights and blendshapes to any 3D point. We leverage this representation to train a generative model with a novel self-supervised collision term that learns to reliably solve garment-body interpenetrations. We extensively evaluate and compare our results with recently proposed data-driven methods, and show that our method is the first to successfully address garment-body contact in unseen body shapes and motions, without compromising realism and detail.

1. Introduction

Physics-based methods handle garment-body contact but are too costly for real-time virtual try-on, while data-driven methods can still produce test-time interpenetrations. The paper introduces a diffused body model, a canonical garment space, and self-supervised generative learning to produce collision-free garments without postprocessing.

1. Introduction

  • Physics-based methods solve contact and generalize across garments, but their computationally expensive runtime evaluations do not meet virtual try-on's combined robustness and performance needs.They are also not easily differentiable for computer-vision pipelines.Source: §1 · p. 1
  • Existing data-driven methods can produce test-time garment-body interpenetrations despite training penalties, and commonly require postprocessing to push garments outside the body.Residual prediction errors can create artifacts even when the predicted mesh closely matches the ground truth.Source: §1 · pp. 1–2
  • The expanded human body model smoothly extends surface parameters to any 3D point, enabling deformation modeling at garment vertices.The extension leverages existing body-model deformation capabilities.Source: §1 · p. 2
  • The garment model uses an unposed, deshaped canonical space that removes deformations captured by the expanded body model, rather than relying on simplified inverse-LBS transformations.The paper says those simplified transformations cannot represent complex garment deformations correctly and can introduce artifacts.Source: §1 · p. 2
  • The method learns a generative deformation subspace with self-supervised collision handling, producing garments that do not interpenetrate the body across shape and pose settings.A regressor predicts deformations in the subspace, and the garment model projects them to the final state.Source: §1 · p. 2

2. Related Work

Prior work models cloth with physics-based simulation or data-driven methods, each with distinct strengths and limitations. The paper positions its contribution as implicit-function-based collision handling and diffused body parameters for data-driven garment deformation.

Physics-Based Cloth Simulation

  • Physics-based cloth simulation models internal forces, collision detection, and collision response, producing realistic results that generalize across garments but miss real-time virtual try-on needs.Approaches to reduce its computational bottleneck include approximations, adaptive remeshing, detail upsampling, and GPU-based solvers.Source: §2 · p. 2

Data-Driven Cloth Models

  • Data-driven cloth models span garment design, animation, and virtual try-on, using 3D meshes from simulation, reconstruction, or hybrid methods.Representations include positions or offsets, while garment deformations may be parameterized by body-model spaces, geometric features, or sketches.Source: §2 · p. 2
  • Data-driven methods model pose, shape, style, or dynamics, but commonly require exhaustive test-time postprocessing to fix body-garment collisions.Most methods use training losses that penalize geometric penetrations, yet the cited approaches still require collision-fixing steps in test sequences.Source: §2 · p. 3

Implicit Function Learning

  • The paper uses implicit-function learning for two purposes: a differentiable garment-body collision loss and smooth diffusion of shape correctives and rigging weights to arbitrary 3D points.Figure 2 depicts preprocessing above and runtime below, with a collision-trained decoder used by a regressor to reproduce states at runtime.Source: §2 · p. 3, Figure 2 · p. 3

3. Method Overview

The method predicts dynamic garment deformations from body shape and motion while aiming to avoid interpenetrations without postprocessing. It combines a diffused body model, a canonical garment representation, and self-supervised collision learning.

3. Method Overview

  • The target is dynamic garment geometry conditioned on body pose and shape that does not interpenetrate the body after inference or require postprocessing.The stated goal retains regressed garment details such as wrinkles and dynamics.Source: §3 · p. 3
  • Diffused skinning parameters define an unposed, deshaped garment space that removes body-modelled pose and shape deformations while preserving pose- and shape-dependent wrinkle details.These parameters also support representation of garment-body sliding and loose clothing.Source: §3 · p. 3
  • Projected physics-based simulations provide training data for a generative deformation space, where random garment states are tested for collisions against a constant body mesh.A neural regressor then outputs dynamic garments from body shape and motion without body interpenetrations.Source: §3 · p. 4
  • Network architectures, training data, parameters, and other implementation details are provided in the supplementary material.The main text directs readers there for implementation specifics.Source: §3 · p. 4

4. Canonical Space of Garment Deformations

The section builds a canonical garment-deformation pipeline by extending body-model parameters beyond the surface, removing body-captured pose and shape changes, projecting training garments with collision constraints, and learning a generative deformation subspace.

4.1. Diffused Human Model

  • Nearest-body-vertex assignments fail when garments slide and can create medial-axis discontinuities, so the method smoothly diffuses skinning weights and blendshape offsets to any 3D point.Averaging neighboring surface values mitigates medial-axis discontinuities.Source: §4.1 · p. 4

4.2. Garment Model

  • The garment canonical space removes pose and shape deformations already captured by the body model, while X-dependent diffused skinning articulates garments of any topology without garment-specific retraining.Unlike fixed weight assignments, the method defines skinning parameters from the unposed, deshaped garment.Source: §4.2 · p. 5

4.3. Projecting the Ground-truth Data

  • Figure 3 contrasts colliding constant-weight unposing, artifact-prone and temporally unstable nearest-vertex weights, and the authors’ optimization-based projection.Earlier inverse-skinning projections can collide when garments deform substantially or slide tangentially over the body.Source: Figure 3 · p. 5, §4.3 · p. 5
  • The canonical projection optimization combines a simulated-garment data term, a Green-Lagrange strain penalty, and a term preventing garment-body interpenetration.The collision term uses a precomputed static body SDF represented by a differentiable network.Source: §4.3 · pp. 5–6

4.4. Generative Garment Deformation Subspace

  • A VAE learns a garment-deformation subspace with self-supervised collision checks against a constant body, enabling collision-free outputs in unseen sequences.The sampled-latent collision term needs no ground-truth garments and generalizes to arbitrary bodies.Source: §4.4 · p. 6

5. Regressing Garment Deformations

The regressor predicts garment deformations in canonical space from body shape and motion inputs, using recurrent units to capture dynamic effects. Its motion descriptor combines a learned pose representation with movement information.

5. Regressing Garment Deformations

  • A learned 10-dimensional pose descriptor replaces joint rotations in the motion representation because it generalizes better.The collision-count training figure identifies the self-supervised term as key to reducing collisions in unseen sequences.Source: §5 · p. 7, Figure 4 · p. 7
  • The 42-dimensional motion descriptor combines pose information with pose velocities and accelerations, global rotation, and translation.The regressor also receives 10 shape coefficients and predicts a 25-dimensional encoded deformation.Source: §5 · p. 7
  • A recurrent regressor predicts encoded canonical garment deformations from body shape and motion, using GRUs to model effects that depend on previous frames.It is trained on canonical positions, velocities, and accelerations.Source: §5 · pp. 6–7

6. Evaluation

Evaluation reports fewer garment-body collisions than competing data-driven methods, including on unseen sequences and shapes. Qualitative results cover unseen motions, while runtime evaluation reports detailed meshes at high frame rates.

6.1. Quantitative Evaluation

  • Adding self-supervision drives test-set collisions close to zero across four unseen sequences and 17 shapes, unlike using only supervised collision loss.The comparison tracks collisions during training of the generative subspace.Source: §6.1 · p. 7
  • Our method produces almost no collisions in test sequence 86 07, while Santesteban et al. and TailorNet consistently collide without postprocessing.The remaining collisions primarily come from self-intersections already present in the AMASS body mesh.Source: §6.1 · p. 7, Figure 5 · p. 7
  • 0.09% residual collisions remain with the full model, while competitive methods have significantly higher counts across 105 AMASS test motions.The ablation compares configurations without the full collision loss, without self-supervision, and the full model.Source: §6.1 · p. 7, Table 1 · p. 7

6.2. Qualitative Evaluation

  • On body shapes far beyond the training range, our method shows no visible garment-body collision, unlike TailorNet and Santesteban et al.The evaluation interpolates between two extremely different unseen AMASS shapes.Source: §6.2 · p. 8, Figure 6 · p. 8
  • The method is evaluated on unseen motions, including a challenging dress sequence with dynamics, against a physics-based simulator.The figure caption describes realistic dynamics and wrinkles in the synthesized results.Source: §6.2 · p. 8, Figure 8 · p. 8

6.3. Runtime Performance

  • The method produces detailed garment meshes at high frame rates, including for garments with many triangles.Runtime is reported in Table 2 for a desktop PC.Source: §6.3 · p. 8, Table 2 · p. 8

7. Conclusions and Future Work

The paper presents a collision-avoidant garment deformation algorithm based on a diffused volumetric body representation and a differentiable canonical garment subspace. The authors identify handling collisions between multiple clothing layers as future work.

7. Conclusions and Future Work

  • The diffused volumetric body representation and differentiable canonical garment subspace support collision-avoidant deformation, lower inference complexity, and detailed spatial and temporal garment outputs.The conclusion reports that the learned representation yields higher quality than previously achievable and can be inferred extremely quickly.Source: §7 · p. 8
  • Handling collisions between multiple clothing layers remains a proposed direction for future research.Source: §7 · p. 8

Related papers

Loading 2105.06462v1…