Source-linked AI summary
Multi-Garment Net: Learning to Dress 3D People from Images
Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, Gerard Pons-Moll
TL;DR
Existing image-based human reconstruction methods often represent body and clothing as one surface, limiting separation and transfer to new body shapes. MGN predicts layered garments and body shape from a few frames using registered real-scan garments and SMPL associations. It enables garment re-dressing and transfer of geometry and texture, while its posing and PCA formulation impose practical scope boundaries.
Problem
Single-surface human representations cannot separately capture clothing and body or map clothing to novel body shapes.
Method
MGN predicts body shape and layered garment geometry from images using a digital wardrobe of registered real garments associated with SMPL.
Results
MGN enables dressing novel body shapes with reconstructed garments and transferring garment texture and geometry.
Takeaways & Limitations
The layered representation provides control over body shape, clothing texture, and clothing geometry.
Takeaways & Limitations
Reposing across subjects with very different poses can produce undesired artifacts, while PCA garment models are biased toward smooth results.
Abstract
from arXiv · showhide
We present Multi-Garment Network (MGN), a method to predict body shape and clothing, layered on top of the SMPL model from a few frames (1-8) of a video. Several experiments demonstrate that this representation allows higher level of control when compared to single mesh or voxel representations of shape. Our model allows to predict garment geometry, relate it to the body shape, and transfer it to new body shapes and poses. To train MGN, we leverage a digital wardrobe containing 712 digital garments in correspondence, obtained with a novel method to register a set of clothing templates to a dataset of real 3D scans of people in different clothing and poses. Garments from the digital wardrobe, or predicted by MGN, can be used to dress any body shape in arbitrary poses. We will make publicly available the digital wardrobe, the MGN model, and code to dress SMPL with the garments.
1. Introduction
MGN reconstructs body shape and layered garments as separate meshes from a few images, enabling garment transfer and greater control than single-surface representations. It combines registered real-scan garments with image-based prediction and segmentation refinement.
- Single-surface mesh or voxel methods cannot separate clothing from the body or map clothing to a novel body shape.
- MGN infers human body and layered garments as separate meshes directly from one or more images.
- The model supports dressing a second subject with garments reconstructed from another subject, including transfer of garment texture and geometry.
- A registered digital wardrobe and per-garment PCA models relate garment geometry to SMPL bodies, enabling transfer across body shapes and poses.
- Given segmented images and 2D joints, MGN predicts garments and body shape in a canonical pose, then reposes them using per-frame pose estimates.
- The approach introduces automatic garment scan segmentation and registration, a top-down segmentation-fitting objective, and publicly released models, wardrobe data, and dressing code.
2. Related Work
Related work covers performance-capture methods requiring specialized viewpoints, depth sensing, or constrained capture, alongside data-driven clothing models. MGN differs by separating garments and body and transferring clothing to new subjects from image input.
- Classical performance-capture methods deform meshes or volumes to silhouettes, while fusion methods incrementally build geometry and appearance.
- Many prior capture systems require multi-view imagery, depth cameras, or subjects standing still while cameras move around them.
- Alldieck et al. reconstruct clothing and hair as displacements on SMPL but cannot separate garments from the body or transfer clothing to new subjects.
- Data-driven clothing work includes simulation-trained models, real-data models, two-layer models for dressing novel shapes, and methods estimating body shape under clothing.
3. Method
MGN builds a SMPL-compatible representation that separates body shape from layered garments, using registered scan data and garment-specific priors to reconstruct and retarget clothing. Its pipeline supports garment transfer across subjects and poses, while acknowledging artefacts from highly nonlinear reposing.
- Data preprocessing: 356 scans are processed with SMPL registration, body-aware segmentation, and template registration to create a digital wardrobe of five garment categories.Registered garments remain associated with SMPL bodies, enabling reposing to arbitrary poses.
- Garment registration: Garment templates are defined on SMPL and registered across scans despite variation in styles, geometries, body shapes, and poses.The workflow segments scans into skin, upper-clothes, and pants before non-rigidly registering body and garment templates.
- Scan segmentation: MRF segmentation combines garment priors based on geodesic distance with CNN semantic terms and appearance models in CIELAB color space.The prior penalizes labels outside likely garment regions and labels inconsistent with those regions.
- Garment registration: A Laplacian least-squares initialization preserves local garment structure while matching scan clothing boundaries before nonlinear registration.The linear solve globally stretches or compresses templates to fit boundaries, reducing reliance on iterative nonlinear alignment.
- Dressing and retargeting: The SMPL dressing function poses and skins garment vertices, allowing inferred garments to be transferred to different SMPL bodies and poses.Re-targeting in unposed space preserves garment-to-body distances to reduce distortion, but highly nonlinear posing can create artefacts for very different poses.
- Image reconstruction: MGN predicts body pose and shape, garment PCA coefficients, and detail displacements from one or more segmented images and joint estimates.A differentiable renderer projects the separately predicted body and garments into per-garment segmentation masks for fine-grained image matching.
4. Dataset and Experiments
MGN is evaluated on real scans through qualitative reconstruction, texture-transfer, and garment re-targeting experiments. It achieves 5.78 mm garment vertex-to-surface error with eight input frames, while providing explicit garment control and comparable performance to a retrained single-mesh baseline.
- Dataset: The dataset contains 356 scans spanning varied body shapes, poses, and clothing, with 70 scans held out for testing.Registered scans support re-posing and re-shaping for data augmentation.
- Applications: Texture transfer preserves source garment geometry while mapping novel texture across registered instances of the same garment class.Fixed topology and surface parameterization enable transfer between corresponding garments.
- Qualitative comparison: MGN explicitly models garment geometry, producing sharper garment details and minimal distortions across five qualitative test subjects compared with Alldieck et al.The comparison visualizes the prior method on the left and MGN on the right.
- Quantitative evaluation: 5.78 mm is MGN's mean garment vertex-to-surface error with 8 frames, compared with 5.72mm for retrained octopus [3].The criterion excludes error on skin parts, differing slightly from [3].
- Limitations: The PCA garment formulation is tractable but biased toward smooth results, motivating future models for geometry variation over fixed topology.This contrasts with the GraphCNN used by [3].
- Re-targeting: MGN re-targets garment texture and geometry from source to target subjects using images, including cases with different poses.The procedure unposes source garments and bodies before transferring them to the target body.
5. Conclusion and Future Works
The conclusion presents MGN as a layered-mesh model for reconstructing body shape and garment geometry from few images. The representation offers control for re-dressing novel shapes, while the accompanying registered-garment dataset supports further research.
- MGN jointly reconstructs body shape and garment geometry as layered meshes from few images.
- The layered representation provides control to re-dress novel body shapes with reconstructed clothing and better matches real-world clothing layers than a single mesh.
- The paper introduces a dataset of registered real garments from real scans produced with a robust registration pipeline.
- Compared with classical single-mesh representations, MGN allows more control while producing qualitatively similar results.
- The authors describe MGN as a first step and plan to release the model and digital wardrobe to stimulate research.