Source-linked AI summary
CLOTH3D: Clothed 3D Humans
Hugo Bertiche, Meysam Madadi, Sergio Escalera
TL;DR
CLOTH3D targets the lack of large, varied 3D clothed-human data needed for garment learning. It builds a synthetic dataset with simulated garment dynamics and introduces a graph-convolutional CVAE for conditioned garment generation. The resulting GCVAE produces realistic garments and meaningful static and dynamic latent spaces across poses, shapes, and garment variations.
Problem
Existing 3D cloth research lacks sufficient data, while garments exhibit substantial variability in geometry, topology, fabric, and dynamics.
Method
The paper constructs CLOTH3D and trains a graph-convolutional conditional variational auto-encoder on body-to-cloth offsets conditioned on pose, shape, and tightness.
Results
The proposed GCVAE generates realistic garments and learns meaningful latent spaces that organize garment and wrinkle variation.
Takeaways & Limitations
CLOTH3D provides a challenging corpus and a baseline for 3D garment generation on SMPL bodies across varied poses and shapes.
Abstract
from arXiv · showhide
This work presents CLOTH3D, the first big scale synthetic dataset of 3D clothed human sequences. CLOTH3D contains a large variability on garment type, topology, shape, size, tightness and fabric. Clothes are simulated on top of thousands of different pose sequences and body shapes, generating realistic cloth dynamics. We provide the dataset with a generative model for cloth generation. We propose a Conditional Variational Auto-Encoder (CVAE) based on graph convolutions (GCVAE) to learn garment latent spaces. This allows for realistic generation of 3D garments on top of SMPL model for any pose and shape.
1 Introduction
CLOTH3D addresses the shortage and complexity of 3D cloth data with a large synthetic dataset and a graph-convolutional generative baseline. Its varied, simulated garments support learning realistic dressed-human generation on SMPL bodies.
- 3D clothing research has largely used 2D data because deep learning needs more data and garments vary widely in shape, size, topology, fabric, and texture.
- Synthetic generation offers scalable, ground-truth-error-free data, whereas scans are costly and RGB-based reconstruction is inaccurate for cloth dynamics.
- CLOTH3D provides thousands of high-resolution dressed-human sequences with more than 2 million 3D samples and broad garment, shape, and pose variability.
- The dataset generation pipeline creates a unique outfit per sequence by varying garment type, topology, shape, size, tightness, and fabric.
- GCVAE encodes garments as body-to-cloth offsets with predicted body masks, learning latent spaces that generate 3D garments on SMPL for arbitrary pose and shape.
2 Related Work
Prior 3D garment datasets are limited by small scale, few garments or poses, low resolution, missing geometry, absent cloth dynamics, and scarce metadata. Existing generation methods also face representation and mesh-consistency challenges.
- 3D scans provide high-resolution data but usually cover few subjects, poses, garments, or samples and can contain occlusions, missing vertices, corrupted shapes, or unlayered models.
- 3DPW emphasizes in-the-wild pose and shape rather than garments, with 18 clothed models, low resolution, rigid posing, and no cloth dynamics.
- Existing datasets often provide static poses, short sequences, low garment variability, or no fabric metadata, limiting coverage of cloth behavior and dynamics.
- Prior data-driven garment generation factorizes shape and pose deformations but can produce inconsistent meshes requiring reconstruction.
3 Dataset
CLOTH3D constructs layered animated humans wearing procedurally varied garments, simulates their dynamics across diverse actions and fabrics, and filters body self-collisions that would corrupt cloth simulation.
- 3 Dataset: CLOTH3D is a large-scale dataset of animated human 3D sequences wearing different garments.
- 3.2 Garment Generation: Garments are generated from upper- and lower-body templates through shaping, cutting, resizing, and optional sewing into full-body outfits.
- 3 Dataset: The dataset contains over 7000 non-overlapping sequences of 300 frames each at 30fps, yielding 2.1M samples.
- 3.1 Human 3D Sequences: Human sequences are generated with SMPL from pose data covering around 2600 sequences, 23 actions, and over 100 subjects.
- 3.1 Human 3D Sequences: Body self-collisions are automatically corrected when separable or otherwise rejected because intersection volumes can corrupt global cloth dynamics.
- 3.3 Simulation: Blender simulates cloth with a mass-spring model, varying fabric parameters and optional elastics to produce different dynamics and wrinkles.
4 Dressed Human Generation
CLOTH3D represents garments as body offsets and uses a Graph Conditional Variational Auto-Encoder to generate clothed humans while modeling garment structure and dynamics. Its preprocessing and mesh hierarchy address topology differences, registration, and body-part-aware graph pooling.
- 4 Dressed Human Generation: CLOTH3D encodes garments as body offsets and uses masks to separate garment vertices from the SMPL body.This fixes data dimensionality and supports different garment types, including skirt-like garments.
- 4.1 Data Pre-processing: Non-rigid ICP registers garments to SMPL bodies in rest pose, with mesh subdivision producing a final 14,475-vertex representation.Head, hands, and feet are excluded from correspondence finding, and garment masks are extracted by nearest-neighbor matching.
- 4.1 Data Pre-processing: A connected inner-leg topology and Laplacian smoothing enable registration of skirt- and dress-like garments with topology differing from SMPL.The additional topology is also used for graph convolutions, whereas smoothing is used only for skirt/dress registration.
- 4.3 Network: The GCVAE learns garment latent spaces from body-and-offset inputs, with conditional variables encoded separately and decoder outputs comprising offsets and a garment mask.Reconstruction adds predicted offsets to the body and removes body vertices according to the predicted mask.
- 4.3 Network: Static and dynamic garment variation is disentangled into garment and wrinkle codes, conditioned on body shape, tightness, pose, and static garment information.Separate SVAE and DVAE networks model static garments and dynamic garments, respectively.
- 4.3 Network: A six-level, 21-segment mesh hierarchy allocates smaller segments to high-offset-variability regions and uses max-pooling with correspondence-based unpooling.The hierarchy reduces dimensionality from 14,475 to 21 vertices, leaving one node per segment at the final level.
5 Experiments
The experiments evaluate reconstruction quality, latent-space factorization, pooling, garment-category behavior, and dynamic sequence conditioning. Ablations show distinct trade-offs among surface, normal, mask, collision, and KL losses, while the learned latent space organizes garment variation meaningfully.
- Metrics: The evaluation measures surface error in millimeters, normals error in radians, garment-mask IoU, and KL loss for latent-code quality.Surface and normals compare predictions with ground truth; mask uses intersection over union.
- Ablation Study: Adding normal consistency reduces surface and normals errors but worsens KL loss relative to the baseline.The normal loss encourages geometry consistent with the input while trading off latent-space quality.
- Ablation Study: Mask prediction improves surface and normals errors, while collision loss improves collided-vertex locations with non-significant increases in other losses.The mask ablation reports higher surface and normals errors without mask prediction; collision loss pushes collided vertices toward correct positions.
- Ablation Study: Conditional-variable regression improves KL loss without significant changes in surface or normals error.The conditional variables are regressed from the encoder’s first fully connected layer to improve latent-space factorization.
- Ablation Study: The proposed pooling modification marginally changes surface and normals errors but benefits mask prediction.It constrains mesh simplification across segmented body regions to improve correspondence distribution between pooling layers.
- Learned Latent Space: The GCVAE groups static garments meaningfully in t-SNE space, including shared latent regions for dresses and jumpsuits and transitions across topologies, genders, and shapes.The visualization uses 5K random static samples and linearly graduated variables for transitions.
- Dynamic Dataset: Training DVAE on sequences of frames leads to better results than conditioning on a single frame.The sequence setting uses four frames sampled every three frames to form a 12-frame clip.
- Dynamic Dataset: The dynamic model achieves a 3mm average improvement across all garment categories without temporal information in the encoder or a sequence-specific prediction loss.Garment reconstruction is shown for both single frames and sequences, although temporal consistency was not an explicit training objective.
6 Conclusions
The paper concludes that CLOTH3D is a large-scale synthetic dataset with broad body, pose, garment, and fabric variability. Its graph-convolutional variational autoencoder produces realistic garment generation on the dataset.
- Conclusions: CLOTH3D provides large variability in body shape, pose, garment type, topology, shape, tightness, and fabric, with complex garment dynamics.The dataset is presented as a challenging corpus for 3D garment generation.
- Conclusions: The baseline uses a graph convolutional network trained as a variational autoencoder and introduces a new pooling grid.Evaluation on CLOTH3D showed realistic garment generation.
CLOTH3D: Supplementary Material
The supplementary material expands the dataset-generation description, dataset variability and size, data format, video content, graph-CNN implementation, and ablation analyses.
- Supplementary Material: The supplementary material provides additional details on CLOTH3D generation, dataset variability and size, data format, the supplementary video, graph CNN implementation, and ablations.It organizes these details across Sections 7.1–7.3, Section 8, Section 9, and related analyses.
7 CLOTH3D
CLOTH3D is generated by combining SMPL human sequences, randomized outfit construction, and cloth simulation. The released data includes static and dynamic garment representations, with approximately 2 million simulated frames and compact PC16 animation storage.
- Generation Pipeline: The generation algorithm samples SMPL humans, constructs randomized outfits, and simulates them with fabric settings, body-shape transitions, and pose transitions.The summarized stages are HUMAN, OUTFIT, and SIMULATION.
- Human Sequence Generation: Self-collision handling separates vertices in problematic body regions or rejects samples when collisions are unsolvable, enabling more valid cloth simulations.Problematic regions include the armpit and crotch; a 4mm separation allows folded cloth to fit in solvable cases.
- Outfit Generation: Garments are resized using SMPL shape parameters plus an offset for tightness, with a gender-dependent sign correction preserving semantic consistency.The first SMPL shape parameter has opposite meaning for male and female bodies, motivating the correction.
- Simulation: A body transition changes the body from β + γ, representing shape plus tightness, to β before transitioning from rest pose to the sequence’s initial pose.This reduces body-to-cloth penetration caused by resizing to a different shape.
- Outfit Generation: Template garments undergo linear shaping, randomized cuts, resizing, and optional sewing into jumpsuits or dresses before simulation.Uniform sampling supports varied sleeve, shirt, leg, and waist dimensions.
- Dataset Scale and Format: Around 2M frames result from simulating each source subsequence with a different outfit.Samples include static rest-pose information and dynamic garment animation data.
- Data Format: Each sample uses 300 frames at 30fps and stores static garments as OBJ files plus dynamic animation in PC16 format.PC16 converts 32-bit floats to 16-bit values, halving storage requirements with minimal precision loss in the stated ranges.
- Data Modalities: CLOTH3D supports multiple derived modalities, including depth maps, surface normals, 3D velocities, and segmentation masks.These modalities are illustrated through random samples from different sequences.
8 Video
The supplementary video visualizes CLOTH3D’s static and dynamic clothed-human samples alongside multiple data representations, including RGB, depth, normals, velocities, and segmentation masks.
- CLOTH3D static samples are shown as representative garment and human configurations.
- CLOTH3D dynamic samples visualize garments changing across animated sequences.
- The dataset represents samples using RGB, depth, normals, velocities, and segmentation masks.
9 Dressed human generation
The dressed-human generation system uses graph-convolutional variational models with conditioning and separate static and dynamic garment codes. Qualitative results show controllable wrinkle generation, while post-processing reduces noisy garment boundaries.
- Model architecture: The model applies graph-convolutional VAEs to learn latent codes for generating dressed humans conditioned on garment and body variables.The pipeline uses graph structure and conditions generation on garment type, tightness, body shape, and pose.
- Latent-space factorization: Separate SVAE and DVAE networks encode static garment shape and dynamic wrinkles into garment code zs and wrinkle code zd.Both codes have dimensionality R128, with static garments modeled separately from dynamic garment behavior.
- Latent-space factorization: The system uses pose, garment code, and wrinkle code as conditioning variables to generate garments in different scenarios.Qualitative latent-space results evaluate how these variables control generated garments.
- Latent-space factorization: Conditioning on pose or garment code accurately retargets fixed wrinkle codes to new scenarios.
- Reconstruction error: Reconstruction error is higher near the feet because skirts and dresses exhibit stronger dynamics there.The error distribution is analyzed per vertex for both static and dynamic samples.
- Post-processing: Ten smoothing iterations improve noisy garment boundaries, while more iterations may destroy high-frequency details.The boundary operation is implemented through sparse matrix multiplication and produces more natural-looking garments.
10 Applications of the Dataset
CLOTH3D supports applications beyond garment generation, including depth-based human analysis, garment motion analysis, scan completion, design assistance, and RGB-to-3D cloth estimation.
- CLOTH3D can support human pose and action recognition using rendered depth images.
- The dataset can be used for garment motion analysis through vertex-velocity representations.
- CLOTH3D can help fill missing vertices in scanned bodies using additional metadata such as garment segments.
- The dataset may support designers and animators and enable estimation of 3D garments from RGB images.