Source-linked AI summary
Deep Fashion3D: A Dataset and Benchmark for 3D Garment Reconstruction from Single Images
Heming Zhu, Yu Cao, Hang Jin, Weikai Chen, Dong Du, Zhangye Wang, Shuguang Cui, Xiaoguang Han
TL;DR
3D garment reconstruction is limited by scarce datasets despite the importance of clothing for photorealistic human digitization. The paper introduces Deep Fashion3D and a single-view reconstruction baseline using adaptable templates and combined mesh–implicit representations. The dataset contains 2078 real-garment models with extensive annotations, and the baseline achieves state-of-the-art single-view reconstruction performance on the benchmark.
Problem
3D garment reconstruction lacks large-scale, richly annotated clothing data, although clothing is important for photorealistic human digitization.
Method
The paper constructs Deep Fashion3D and proposes a single-view baseline combining mesh and implicit representations with an adaptable template for varied clothing topologies.
Results
Deep Fashion3D contains 2078 real-garment models across 10 categories and 563 instances, and the baseline sets a new state of the art for single-view garment reconstruction.
Takeaways & Limitations
Deep Fashion3D provides a benchmark and annotations for evaluating image-based garment reconstruction, including feature lines, body pose, and multi-view real images.
Takeaways & Limitations
The authors report that graph convolutional networks have difficulty learning geometric details, while deeper analysis is left for future work.
Abstract
from arXiv · showhide
High-fidelity clothing reconstruction is the key to achieving photorealism in a wide range of applications including human digitization, virtual try-on, etc. Recent advances in learning-based approaches have accomplished unprecedented accuracy in recovering unclothed human shape and pose from single images, thanks to the availability of powerful statistical models, e.g. SMPL, learned from a large number of body scans. In contrast, modeling and recovering clothed human and 3D garments remains notoriously difficult, mostly due to the lack of large-scale clothing models available for the research community. We propose to fill this gap by introducing Deep Fashion3D, the largest collection to date of 3D garment models, with the goal of establishing a novel benchmark and dataset for the evaluation of image-based garment reconstruction systems. Deep Fashion3D contains 2078 models reconstructed from real garments, which covers 10 different categories and 563 garment instances. It provides rich annotations including 3D feature lines, 3D body pose and the corresponded multi-view real images. In addition, each garment is randomly posed to enhance the variety of real clothing deformations. To demonstrate the advantage of Deep Fashion3D, we propose a novel baseline approach for single-view garment reconstruction, which leverages the merits of both mesh and implicit representations. A novel adaptable template is proposed to enable the learning of all types of clothing in a single network. Extensive experiments have been conducted on the proposed dataset to verify its significance and usefulness. We will make Deep Fashion3D publicly available upon publication.
1 Introduction
Deep Fashion3D addresses the scarcity and limited diversity of 3D garment data by providing a large, richly annotated repository reconstructed from real garments. It also introduces a single-view reconstruction baseline designed to handle diverse clothing topologies using combined mesh and implicit representations.
- Motivation: Clothing reconstruction remains difficult because 3D garment datasets are scarce, while garments exhibit complex deformations and diverse topologies.These limitations have produced inferior clothing-reconstruction performance relative to unclothed human-body reconstruction.
- Dataset: Deep Fashion3D contains 2078 garment models from 563 instances across 10 clothing categories, reconstructed from real garments.The dataset is described as an order of magnitude larger than MGN and includes diverse garment styles.
- Dataset: The dataset provides 3D feature lines, calibrated multi-view real images, 3D body pose, and randomly posed garments for modeling dynamic wrinkles.Feature lines mark prominent garment-surface geometry such as necklines, cuff contours, and hemlines, providing priors for reconstruction.
- Baseline approach: The proposed adaptable template enables one network to learn clothing with varying topologies instead of training separate category-specific models.The baseline combines mesh and implicit representations to capture complementary garment structure and local detail.
- Benchmark: The paper establishes a benchmark through extensive evaluation of state-of-the-art single-view garment-reconstruction approaches on Deep Fashion3D.The authors present the benchmark as a basis for evaluating image-based garment reconstruction systems.
2 Related Work
Prior 3D garment work is limited by scarce, incomplete, or synthetic data, while single-view reconstruction remains highly ill-posed and difficult.
- 3D Garment Datasets: Existing 3D garment datasets are scarce, and available scans may provide limited quantities or fail to separate clothing from the body.BUFF offers high-resolution 4D clothed-human scans but in limited amount, while garment segmentation can be laborious and produce corrupted surfaces.
- Performance capture: Vision-based performance-capture methods use RGB cues, silhouettes, templates, stereo, photometric constraints, or fusion to estimate cloth deformation.
- Performance capture: Many performance-capture systems estimate body and clothing jointly, preventing them from obtaining a separated cloth surface from the output.
- Single-view garment reconstruction: Single-view garment reconstruction is highly challenging because one image provides scarce input while the search space is enormous, motivating statistical priors for plausible predictions.
3 Dataset Construction
Deep Fashion3D addresses the scarcity of comprehensive 3D garment data by collecting diverse real garments, augmenting their poses, and providing multi-view imagery, feature lines, pose labels, and dataset comparisons.
- Dataset motivation: Deep Fashion3D was created to provide a more comprehensive 3D garment dataset with richer annotations and greater garment-style variation than prior collections.
- Cloth Capture: 563 diverse garment items span 10 clothing categories, and pose augmentation produces 2078 reconstructed 3D garment models.
- Cloth Capture: The collection uses multi-view studio imagery and image-based geometry reconstruction to generate high-resolution garment reconstructions as dense point clouds.
- Annotations: Feature lines mark prominent garment structures such as open boundaries, necklines, cuffs, and waistlines, providing priors for faithful reconstruction.These labels supervise 3D key-line prediction and provide explicit constraints for mesh generation.
- Annotations: Each reconstructed model includes SMPL-based 3D pose labels intended to support inference of global garment shape and pose-dependent deformations.
- Data Statistics: Compared with other 3D garment datasets, Deep Fashion3D provides broad category coverage, real multi-view images, and the richest annotations, including 3D feature lines.
4 A Baseline Approach for Single-view Reconstruction
The baseline reconstructs a garment from one image through a hybrid mesh–implicit pipeline: an adaptable template is posed, feature-line deformed, and refined with implicit surface details.
- The method combines mesh flexibility for open garment surfaces with implicit representations for fine geometric details.
- Template Mesh Generation: The adaptable template activates semantic regions according to the estimated clothing topology, allowing one network to generate varied garment types.It is built from SMPL after removing the head, hands, and feet regions.
- Cloth Classification: Synthetic images augment cloth classification by rendering each garment under varied global illuminations and five random views.Around 10,000 synthetic images are generated, with 90% used for training and the remainder for testing.
- Feature line-guided Mesh Generation: The pipeline estimates body pose, deforms the template, regresses feature-line displacements with an image-guided GCN, and uses those lines as deformation handles.Activated feature lines are selected according to the estimated cloth category before regression.
- Detail Transfer: OccNet supplies fine surface details, which are adaptively registered onto the feature-line-deformed mesh while avoiding incompatible closed-surface artifacts.Registration transfers valid high-frequency details under additional geometric constraints.
- Loss functions: Feature-line regression minimizes a weighted sum of Chamfer distance and edge-length regularization to smooth the predicted lines.
5 Experimental Results
Experiments compare the baseline with reconstruction methods using diverse 3D representations and report both quantitative and qualitative advantages, while ablations identify limitations of direct GCN refinement.
- Benchmarking on Single-view Reconstruction: The benchmark compares seven methods, including point-set, voxel, depth-map, mesh, template, multi-garment, and implicit representations.AtlasNet is evaluated with both sphere and 25-square-patch templates.
- Quantitative Results: Our approach achieves the highest reconstruction accuracy on the testing dataset using Chamfer distance and Earth Mover’s distance.Outputs are converted to point clouds before computing both metrics for fair comparison.
- Qualitative Results: Qualitatively, the proposed method produces reconstructions closer to ground truth across randomly selected garment categories and poses.
- Qualitative Results: The hybrid design addresses limitations of competing representations by combining open-surface handling with detailed geometric synthesis.Implicit methods provide details but generate closed surfaces, whereas fixed templates struggle with diverse clothing complexity.
- Ablation Analysis: Ablations evaluate direct GCN fitting, pose-deformed templates, feature-line-guided deformation, registration, and the full approach.The baseline approach produces the best results among the tested settings.
- Ablation Analysis: GCNs have difficulty learning geometric details, whereas feature lines are easier to learn and explicit handle-based deformation performs effectively.The authors attribute the difficulty partly to high-frequency synthesis and the sparse graph approximation of surfaces.
6 Conclusions and Discussions
Deep Fashion3D provides a large, richly annotated real-garment dataset and a single-network baseline for single-view reconstruction across varied clothing types.
- Deep Fashion3D contains over 2,000 garment models from 563 distinct items spanning 10 clothing categories.The models are reconstructed from real clothing images.
- Each model includes 3D body pose, 3D feature lines, and multi-view real images as annotations.
- The baseline uses an adaptable template to learn varied clothing types within a single network.
- Extensive benchmarking shows that single-view garment reconstruction remains an extremely challenging problem with substantial room for improved methods.
Appendix
The appendix provides additional implementation details, adaptable-template experiments, and reconstructions from in-the-wild images.
- The appendix expands training settings for cloth classification, pose estimation, feature-line regression, and adaptable-template design.
- It includes further experiments on adaptable-template effectiveness and additional in-the-wild reconstruction results.
A Implementation Details
This section details the training settings, loss functions, and hyperparameters used across the approach's stages and feature line regression module.
- The implementation section specifies training settings for cloth classification, pose estimation, implicit reconstruction, and feature line regression.It also covers the associated loss functions and feature line regression hyperparameters.
A.1 Cloth Classification
The approach uses dedicated image-based modules for cloth classification, partial pose estimation, and implicit reconstruction, with pose training combining parameter accuracy and regularization.
- Cloth classification uses a VGG16 feature extractor and fully connected layer trained with cross-entropy on synthetic and real clothing images.The hybrid source combines rendered dataset images with images from DeepFashion and DeepFashion2.
- Partial clothing pose is estimated from images using a VGG16 feature extractor followed by a fully connected layer.The pose representation uses a subset of SMPL parameters, while deformation-irrelevant parameters are set to zero.
- Pose training minimizes a loss combining cloth pose-parameter error with pose regularization.The objective is defined as Lpose = Lparam + λreg Lreg.
- The pose-parameter loss is MSE, while squared pose parameters regularize unexpected rotations with λreg = 1e-5.The regularization term is designed to eliminate unexpected rotations.
- Implicit reconstruction adopts OccNet conditioned on image features, using a finetuned ResNet-18 and a fully connected occupancy predictor.The predictor determines whether an input 3D point lies inside the surface from its coordinates and image latent code.
A.4 Feature Line Regression
Feature line regression combines a novel feature line loss with edge-length regularization to synthesize feature lines while reducing zigzag artifacts.
- The regression module introduces feature line and edge-length losses to guide feature line synthesis and reduce zigzag artifacts.The combined objective is Lfitting = Lline + λedge Ledge.
- The feature line loss uses Chamfer distance between predicted vertices and corresponding annotated ground-truth feature lines.The adaptable template represents feature lines as closed polygonal curves, whereas ground truth consists of points around intended landmarks.
- The adaptable template activates semantic regions and red feature-line curves according to garment type.These activated regions and curves are illustrated for different garment scenarios.
B Evaluation of Adaptable Template
The evaluation compares an adaptable template trained across all clothing categories with type-specific templates trained separately by category. The adaptable approach produces more plausible and accurate feature lines on novel images, indicating stronger generalization.
- The comparison evaluates an adaptable template against a method using separate type-specific templates for different clothing categories.The alternative trains each model on its category-specific template and corresponding 3D data and images.
- Long and short skirts share the same activated semantic patterns in the adaptable template.
- The adaptable-template approach is trained on the entire training set, whereas the comparison uses category-specific subsets under otherwise identical settings.The network structure and losses are shared between the two approaches except for template and training-data differences.
- The adaptable approach generates more plausible feature lines that remain close to ground truth despite differing data modalities.Its stronger generalization performance supports the advantage of using an adaptable template.
C More Results on In-the-wild Images
The baseline reconstructs garments from unseen in-the-wild images, capturing varied clothing topologies and geometric details. Results are visualized as reconstructed meshes from two viewpoints.
- The method reconstructs garments from in-the-wild Internet images that were not seen during training.
- It captures clothing topologies across a variety of input styles.
- The reconstructions faithfully recover geometric details and are shown from two different views.