PaperlayerView exact revision on arXiv

Source-linked AI summary

BCNet: Learning Body and Cloth Shape from A Single Image

Boyi Jiang, Juyong Zhang, Yang Hong, Jinhao Luo, Ligang Liu, Hujun Bao

arXiv:2004.00214v2cs.CVcs.GR

TL;DR

BCNet addresses reconstruction of body and garment shapes from a single near-front RGB image. It uses layered garments on SMPL with body-independent skinning weights and a supervised network trained on constructed clothed-body data. The method supports loose garments and flexible downstream controls such as garment transfer.

  • Problem

    Reconstructing accurate body and garment shapes from a single near-front RGB image remains challenging across varied poses, body shapes, and garment categories.

  • Method

    BCNet separately reconstructs the body and layered garment meshes, using a garment-type-independent skinning-weight network and supervised training on constructed clothed-body data.

  • Results

    BCNet infers body and garment shapes from single images with different poses, including loose garments such as skirts, and produces comparable or better reconstruction results than state-of-the-art methods.

  • Takeaways & Limitations

    Separate garment topology and skinning weights provide flexible controls such as garment and pose transfer between images.

  • Takeaways & Limitations

    The method currently supports six garment types.

Abstract

from arXiv · show

In this paper, we consider the problem to automatically reconstruct garment and body shapes from a single near-front view RGB image. To this end, we propose a layered garment representation on top of SMPL and novelly make the skinning weight of garment independent of the body mesh, which significantly improves the expression ability of our garment model. Compared with existing methods, our method can support more garment categories and recover more accurate geometry. To train our model, we construct two large scale datasets with ground truth body and garment geometries as well as paired color images. Compared with single mesh or non-parametric representation, our method can achieve more flexible control with separate meshes, makes applications like re-pose, garment transfer, and garment texture mapping possible. Code and some data is available at https://github.com/jby1993/BCNet.

1 Introduction

BCNet reconstructs body and layered garments from one near-front RGB image, addressing limitations of SMPL-bound and non-parametric representations through independent garment topology and skinning weights.

  • Motivation: Single-image reconstruction must handle varied body shapes, poses, and garment categories for applications including virtual try-on, VR/AR, and entertainment.Simulation-based alternatives are described as dedicated and time-consuming.
  • Limitations of Existing Representations: SMPL-based methods robustly infer body shape and pose but constrain reconstructed geometry to the model space, limiting complex cloth shapes.
  • Limitations of Existing Representations: Voxel and implicit representations recover arbitrary shapes but face high-resolution memory costs, broken-arm artifacts, or limited semantic information for garment transfer.
  • Limitations of Existing Representations: SMPL displacement representations capture tight garments but miss garment edges and can produce artifacts when loose garments inherit body connectivity and skinning weights.
  • BCNet: BCNet separately reconstructs body and layered garment meshes from one RGB image without semantic input or a rough A-pose constraint, supporting six garment categories including loose skirts.
  • Dataset: The authors construct clothed-body data by dressing varied garments on different SMPL neutral bodies and reposing them with a physics engine.The constructed dataset includes synthetic images and clothed body shapes and is intended for public release.
  • Applications: The method infers body and garment shapes from single images with different poses and supports garment and pose transfer between images.
  • BCNet: A generic skinning-weight network supports garments with different topologies, while graph convolutions recover category-specific garment details.

2 Related Work

Related work spans template-free clothed-human estimation and garment-deformation representations, while BCNet uses a staged network architecture to reconstruct and re-pose body and garments.

  • Template-Free Clothed Human Estimation: Template-free methods use voxel or implicit representations to model complex garment topology, but voxel memory costs can miss high-frequency details.
  • BCNet Architecture: BCNet encodes an image into latent features, predicts SMPL parameters and garment features, then separately predicts garment displacements and skinning weights before reposing the meshes.
  • Garment Deformation Representation: Garment-deformation methods model clothing through precomputed deformations, body-shape regression, or pose-sensitive rigging schemes.

3 Algorithm

BCNet reconstructs body and layered garment meshes from a single near-front RGB image using SMPL, garment-specific shape bases, independent learned skinning weights, and displacement refinement. Supervised losses constrain parameters, geometry, projections, garment classification, interpenetration, and skinning-weight distributions.

  • Garment representation: Garments use category-specific templates and PCA shape coefficients, share SMPL pose and skeleton parameters, but receive independently predicted skinning weights.A neural network estimates garment weights instead of copying SMPL vertex weights, making the garment representation more flexible and garment-type independent.
  • Image-to-dressed-body pipeline: The model estimates body shape, pose, translation, garment shape, and displacement parameters from a single near-front RGB image.Its pipeline contains an image encoder, classification module, skinning weight network, and displacement network.
  • Garment representation: The displacement network adds high-frequency garment details beyond the PCA representation using garment, body, latent, geometric, and skinning-transformation features.The PCA model captures category-specific size variation, while learned displacements target details such as pose-dependent folds.
  • Training losses: Training uses ground-truth supervision for shape parameters, reconstructed geometry and joints, vertex and Laplacian displacements, 2D projections, and garment classifications.Projection losses use a shared camera intrinsic matrix, and classification uses softmax loss for upper- and lower-garment types.
  • Training losses: Additional losses penalize body–garment interpenetration and measure divergence between predicted and ground-truth garment skinning-weight distributions.The skinning-weight network is trained first, after which the remaining network components are trained while its weights are fixed.

4 Dataset Construction

BCNet constructs paired body, garment, and image data through physics-based garment synthesis, rendering, and rigged-avatar processing. The resulting datasets support training and evaluating reconstruction, registration, and posed garment models.

  • The skinning-weight dataset contains 48,000 neutral garments for training and 6,467 for testing across garment types.
  • The physics-based synthesis pipeline comprises sewing-pattern design, neutral garment synthesis, posed garment simulation, and rendering.Pant synthesis stitches a generated pattern around a neutral skeleton, then deforms and simulates the garment in a target pose with gravity.
  • Synthetic images combine randomized body, garment, background, and illumination assets to produce varied near-front-view dressed-body images.Textures are sampled from SURREAL, Fabrics, and DTD; backgrounds come from Places365-Standard, with illumination from HDR images.
  • The synthetic dataset contains 168,602 posed dressed bodies for training and 8,874 for testing, generated from SPRING body shapes and CMU Mocap motions.
  • A second dataset uses 285 rigged avatars with high-definition textures, yielding about 89,425 posed training meshes and 4,386 test meshes.The avatars are sourced from RenderPeople and Axyz and driven with Mixamo.
  • Rigged-avatar registration optimizes garment, displacement, body, pose, and translation parameters using geometric, interpenetration, silhouette, and regularization losses.The pipeline reconstructs garments and body shapes and extracts garment textures for the registered avatars.

5 Experiments

BCNet is evaluated through internal ablations, test-set measurements, quantitative comparisons, and qualitative results. The experiments show accurate reconstruction, benefits from displacement and interpenetration terms, and flexible separate body-garment control.

  • 5.1 Analysis of BCNet: The skinning weight network achieves an average ℓ1 reconstruction error of 6.5 × 10^-4 and a 0.43mm average MED for reposed meshes.The reposing evaluation samples 20 Mocap poses across all garment types.
  • 5.1 Analysis of BCNet: The interpenetration loss alleviates the unreal interpenetration problem between separately reconstructed body and garment meshes.The ablation study in Fig. 5 compares results with and without this term.
  • 5.1 Analysis of BCNet: The displacement network greatly improves reconstruction accuracy over the garment representation without displacement.It captures detailed geometry and large-scale deformations, including tie and suit boundary details and skirt deformation from leg bending.
  • 5.2 Quantitative Comparison.: On BUFF, BCNet uses one image, outperforms compared methods without post-optimization, and achieves an equivalent result to Octopus with optimization.Octopus and MGN post-optimization takes several seconds and several minutes, respectively; MGN optimization can introduce unnatural folds.
  • 5.2 Quantitative Comparison.: On Digital Wardrobe, BCNet achieves similar reconstruction accuracy to single-image PIFu while retaining separate semantic body and garment meshes.DeepHuman tends to bend the leg, introducing large errors on this dataset.
  • 5.3 Qualitative Results.: Qualitative results show plausible loose-garment shapes, more accurate body and garment sizes than MGN, and garment transfer between input images.MGN without post-optimization produces similar shapes across inputs and lacks garment details.

6 Conclusion

BCNet reconstructs body and garment shapes from a single RGB image using layered garments with topology and skinning weights independent of the body mesh. Experiments report comparable or better reconstruction results while supporting flexible controls such as garment transfer.

  • 6 Conclusion: BCNet produces layered garments with different topology and skinning weights, enabling joint reconstruction of bodies and loose garments such as skirts.The paper contrasts this representation with prior SMPL-bound garment models.
  • 6 Conclusion: The authors construct a complete pipeline for generating clothed-body data to train BCNet.
  • 6 Conclusion: Experiments show comparable or better reconstruction results than state-of-the-art methods while allowing flexible controls such as garment transfer.

Appendix

The appendix introduces the supplementary material and presents the architecture of the skinning-weight and displacement networks. BCNet takes a near-front color image and outputs separate garment and body geometries.

  • Appendix: BCNet takes a near-front view color image of a clothed human body as input and outputs garment and body 3D geometries separately.
  • Appendix: The supplementary material describes implementation details, presents additional qualitative results, and discusses limitations of the approach.
  • Appendix: The skinning-weight and displacement networks use similar architectures with Multi-Layer transforms and residual blocks.The Multi-Layer transform changes feature dimensions for each vertex using a shared MLP.

A Implementation Details

The implementation uses point-to-point feature calculation in the skinning-weight and displacement networks, with shared architectural components and optional feature aggregation. Avatar and dataset meshes require garment-skin segmentation before registration and optimization.

  • A Implementation Details: Both networks calculate point-to-point features without upsampling or downsampling, using Multi-Layer transforms and ResBlocks.
  • A Implementation Details: Max-pooling middle-layer features provides global information before the final Multi-Layer transform.Shallow features can optionally be concatenated to improve high-frequency detail inference.
  • A Implementation Details: The two networks differ in graph convolution, input features, outputs, module configuration, and computing flow.
  • A Implementation Details: Rigged avatars and BUFF data are segmented into garment and skin regions before optimization, using available labels or manual Blender segmentation.Manual segmentation uses texture or color information when accurate or available segmentation is absent.

B More Results

The paper evaluates its skinning-weight network and reconstruction quality across datasets, compares against MGN, and demonstrates garment transfer and switching applications.

  • Dataset and evaluation: The constructed datasets contain varied postures and garments, with geometry paired to corresponding color images.
  • Dataset and evaluation: The skinning-weight network is evaluated using weight ℓ1 error and deformation Euclidean distance against ground-truth weights.The evaluation deforms neutral garments in twenty random Mocap postures and visualizes error maps for different garment types.
  • Skinning-weight ablation: Vertex normals help distinguish nearby garment vertices with different skinning weights; removing normals produces significant errors and artifacts.The ambiguity is illustrated at regions such as opposite sides of the crotch, which belong to different legs.
  • Comparison and qualitative results: Compared with MGN using eight views without postprocessing, the method produces body shapes that match input images more closely and garments that capture size variation.MGN garments tend to maintain similar style and size for the same garment type because they are bound with SMPL.
  • Comparison and qualitative results: Results on the test set, Digital Wardrobe, and real images show recovery of body and garment shapes, while transfer and switching results exchange garment geometry and texture between images.These applications are presented across reconstructed clothed-body images and different garment types.

C Limitations

The method has scope and generalization limitations: it supports six garment types, omits several clothing-related elements, and can produce over-smooth results on images unlike the training data.

  • Supported garment scope: The trained model currently supports six garment types and cannot correctly predict garment types outside those categories.The paper states that new garment types could be added using the same strategy.
  • Supported garment scope: The reconstruction does not consider hair, shoes, hats, or multi-layered clothing.Multi-layered clothing is treated as outside the current reconstruction scope.
  • Generalization: Images with cloth, body, lighting, background, or camera styles very different from the training data may produce over-smooth results.The paper suggests synthesizing more realistic data and using more real data to alleviate this issue.
Loading 2004.00214v2…