PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Transfer Texture from Clothing Images to 3D Humans

Aymen Mir, Thiemo Alldieck, Gerard Pons-Moll

arXiv:2003.02050v2cs.CV

TL;DR

Automatic transfer of clothing textures to 3D garments has received little attention, while scanned garments provide limited texture variety. Pix2Surf learns silhouette-based correspondences from clothing images to 3D garment surfaces, outperforming classical warping and image-to-image translation while running in real time.

  • Problem

    Automatic transfer of clothing textures to 3D garments has received little attention, and scanning garments is time-consuming and limits texture variety.

    Source: §1 · pp. 1–2

  • Method

    Pix2Surf learns a direct mapping from clothing-image pixels to 3D garment UV surfaces using silhouette shape, trained on image and aligned-mesh pairs computed through non-rigid optimization.

    Source: §5 · pp. 7–8

  • Results

    Pix2Surf performs significantly better than 2D TPS warping and direct image-to-image translation while being orders of magnitude faster.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Pix2Surf enables direct, real-time virtual try-on from clothing images in 3D.

    Source: §1 · p. 2

    The method focuses on texture and assumes garment geometry is given; inferring geometry from images remains future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

In this paper, we present a simple yet effective method to automatically transfer textures of clothing images (front and back) to 3D garments worn on top SMPL, in real time. We first automatically compute training pairs of images with aligned 3D garments using a custom non-rigid 3D to 2D registration method, which is accurate but slow. Using these pairs, we learn a mapping from pixels to the 3D garment surface. Our idea is to learn dense correspondences from garment image silhouettes to a 2D-UV map of a 3D garment surface using shape information alone, completely ignoring texture, which allows us to generalize to the wide range of web images. Several experiments demonstrate that our model is more accurate than widely used baselines such as thin-plate-spline warping and image-to-image translation networks while being orders of magnitude faster. Our model opens the door for applications such as virtual try-on, and allows for generation of 3D humans with varied textures which is necessary for learning.

1. Introduction

The paper targets texture transfer from front-and-back garment photos onto 3D garment templates, addressing limited attention to 3D texture transfer and shortcomings of image-space methods. It learns silhouette-based correspondences from automatically aligned training pairs, enabling real-time transfer and supporting applications including virtual try-on and synthetic-human generation.

1. Introduction

  • Image-space reposing and try-on methods do not reason explicitly in 3D, struggle with difficult poses, and may lack temporal consistency.Source: §1 · p. 1
  • The method transfers retail-image textures to predefined garment templates for dressing SMPL in 3D.Source: Figure 2 · p. 2
  • Pix2Surf learns silhouette-based correspondences from clothing images to a garment UV surface, ignoring appearance to generalize across varied clothing textures.Training pairs are produced by non-rigidly fitting garment templates to image silhouettes; the fitting takes 5–15 minutes per image and fails in 5% of cases, while the learned mapping runs in milliseconds.Source: §1 · p. 2
  • After mapping texture to 3D, transformations of the 3D geometry support novel views, shapes, and poses.Source: §1 · p. 2
  • The authors plan to make Pix2Surf publicly available for research use, including adding texture to SMPL+Garments, generating synthetic humans, and visualizing garment images in 3D.Source: §1 · p. 2

2. Related Work

Prior work spans image synthesis and 3D reconstruction, but often focuses on image-space outputs or garment geometry rather than garment appearance. Pix2Surf instead creates textured 3D garments automatically without test-time alignment, enabling real-time virtual try-on and novel-view synthesis.

2. Related Work

  • Image-space virtual try-on methods exchange or alter outfits but do not explicitly reason about 3D, so consistent output under changing poses and camera angles is not guaranteed.Source: §2 · pp. 2–3
  • 3D reconstruction and garment-modeling methods primarily target geometry, while automatic texturing from photos requires an expensive, error-prone alignment step.Source: §2 · p. 3
  • Unlike prior methods, Pix2Surf creates textured 3D garments fully automatically without test-time alignment, supporting real-time virtual try-on and novel-view synthesis.The related work [16] targets synthetic training data rather than photo-realistic mapping, and its texture-mapping code is unavailable for direct comparison.Source: §2 · p. 3

3. Method

The method learns garment-image-to-UV correspondences from silhouettes, using automatically fitted garments to create training pairs. Parametric fitting and a later free-form deformation stage align garment meshes to retail-image silhouettes.

3. Method

  • Pix2Surf learns correspondences between garment-image pixels and garment UV surfaces from shape information alone, using fitted retail-image pairs for training.The approach ignores texture when learning the mapping.Source: §3 · p. 3, Figure 3 · p. 4

3.1. Preliminary: Parametric Garment Model

  • Garments are represented as SMPL sub-meshes with displacements, and their shape and pose parameters articulate the template while displacement can still change after texture transfer.The model keeps displacement fixed during fitting.Source: §3.1 · p. 3

3.2.1 Automatic segmentation

  • Retail garments are automatically segmented with GrabCut initialized by threshold-derived foreground and background priors, avoiding manual silhouette annotation.Morphological closing, erosion, and dilation create the priors.Source: §3.2.1 · p. 4

3.2.2 Garment fitting

  • Parametric fitting minimizes a weighted objective combining silhouette overlap, shape regularization, and a pose prior, starting from an A-pose with category-specific optimization scheduling.The silhouette term penalizes leakage into the background; shirts optimize shape and translation before joint optimization, while pants and shorts reverse that order.Source: §3.2.2 · p. 4
  • A second stage frees mesh vertices to refine silhouette boundaries while coupling them to the parametric garment and adding edge, Laplacian, and boundary-smoothing constraints.For shorts and pants, the edge constraint aligns the top ring with the image's horizontal waistline.Source: §3.2.2 · p. 5

3.3. Learning Automatic Texture Transfer

Pix2Surf learns to transfer clothing textures by mapping garment-image silhouettes to UV correspondences, using curated image-to-garment fits as supervision. It represents inputs and targets in image-coordinate and UV spaces, then samples image pixels to produce texture maps.

3.3. Learning Automatic Texture Transfer

  • Pix2Surf learns a silhouette-based mapping from garment images to 3D garment surfaces, using curated fits to train an efficient neural model.The model predicts correspondences from image pixels to a garment's UV parameterization rather than mapping directly from image texture.Source: §3.3 · p. 5, §3.3.2 · p. 5

3.3.1 Pix2Surf: Input and Output Representation

  • Curated fits yield RGB texture targets and UV correspondence maps, paired with coordinate masks that store foreground pixel coordinates and zero-valued background.A segmentation network predicts the foreground mask at test time.Source: §3.3.1 · p. 5

3.3.2 Pix2Surf: Learning

  • Training combines correspondence regression against fitting-derived pseudo-ground truth with reconstruction and perceptual losses on sampled texture maps.The predicted correspondences provide sampling locations for comparing the reconstructed texture with projective-texturing targets.Source: §3.3.2 · pp. 5–6

3.3.3 Implementation Details

  • Pix2Surf uses a six-block ResNet, and category-specific UV maps split garment surfaces into front and back islands to keep the mapping continuous.A separate segmentation network uses a UNet with instance normalization.Source: §3.3.3 · p. 6

4. Experiments

The experiments evaluate Pix2Surf qualitatively and through a user study, comparing it with shape-context/TPS, image-to-image translation, and image-based virtual try-on methods. Participants preferred Pix2Surf over the baselines, while the authors note that the virtual try-on comparison is not directly fair.

4. Experiments

  • Because no ground-truth data exists, the authors evaluate qualitatively and with a user study; examples cover three garment types and show textures without background, holes, or seams.The garment types are T-shirts, shorts, and long pants.Source: §4 · p. 6, Figure 4 · p. 7

4.1. Dataset

  • Training sets include 2,267 T-shirt fronts, 2,277 shorts, and 3,410 pants fronts; scarce back views are supplemented with front views or horizontally flipped images.The back-view T-shirt set has 964 back images and the remainder are front images; the pants back set has 3,211 items.Source: §4.1 · p. 6

4.2. Shape Context Matching and Image-to-Image Translation Baselines

  • Shape-context matching plus TPS leaves artifacts along garment sides, while Pix2Pix produces blurry results that do not preserve input texture details.The Pix2Pix baseline is trained using the same texture maps as Pix2Surf.Source: §4.2 · p. 6
  • 100% of comparisons favored Pix2Surf-textured avatars over baseline-textured avatars in a user study with 30 participants.The study used 20 videos showing reference clothing textures and rotating 3D avatars.Source: §4.2 · pp. 6–7

4.3. Towards Photo-Realistic Rendering

  • VITON and CP-VTON work reasonably well for some poses but fail badly on extreme poses, whereas Pix2Surf does not; the authors caution that the comparison is not directly fair.The methods are conceptually different and the image-based approaches are not explicitly trained for such poses.Source: §4.3 · p. 7

5. Conclusions and Future Work

Pix2Surf transfers clothing textures to 3D garments using learned UV correspondences and runs in real time after a slower fitting stage. The authors report better performance than TPS warping and image-to-image translation, while assuming garment geometry is given.

5. Conclusions and Future Work

  • The method learns image-pixel-to-UV correspondences from silhouette shape, using accurate but slow non-linear fitting to create aligned training examples.Figure 6 depicts automatically textured garments on an avatar.Source: §5 · pp. 7–8, Figure 6 · p. 8
  • Pix2Surf performs significantly better than 2D TPS warping and image-to-image translation while being orders of magnitude faster; Figure 5 shows SC artifacts and Pix2Pix blur.The fitting optimization takes ten minutes to converge, while Pix2Surf runs in real time.Source: §5 · p. 8, Figure 5 · p. 8
  • The method focuses on texture and assumes garment geometry is given; inferring geometry from flat clothing photographs remains future work requiring modeling garment deformation.The authors identify geometry inference as a possible extension.Source: §5 · p. 8

A. Non-Rigid Garment Fitting

The non-rigid garment fitting appendix specifies pose variables and registration weights by garment category and view. It uses a neutral-gendered SMPL model and different joint subsets for shirts, shorts, and pants.

A. Non-Rigid Garment Fitting

  • Registration uses neutral-gendered SMPL and varies pose joints by garment: shoulders 13, 14, 16, 17 for shirts, hips 1 and 2 for shorts, and joints 1–4 for pants.The joint numbers use the canonical ordering.Source: A. Non-Rigid Garment Fitting · p. 12
  • For T-shirts, front registration uses wβ=3.0, ws=100, wθ=1.0, w′e=0; back registration uses wβ=5.0, ws=100, wθ=0, w′e=0.Source: A. Non-Rigid Garment Fitting · p. 12
  • For the front view of shorts, wβ = 4.0 and ws = 100; wθ = 30.0 in the first stage without shape optimization, and wθ = 20 when shape is optimized.Source: A. Non-Rigid Garment Fitting · p. 12
  • For both pants views, wβ=4.0 and ws=100; wθ is 40.0 when optimizing pose alone and 5 when optimizing shape and pose.Source: A. Non-Rigid Garment Fitting · p. 12

B. Learning Automatic Texture Transfer

The experiments use fixed optimizer settings and training durations, with color jittering during segmentation-network training.

B. Learning Automatic Texture Transfer

C. Baseline Implementation

The baseline setup uses default settings for SC-matching TPS and sets pix2pix’s adversarial-loss weight to 1; increasing that weight worsens performance.

C. Baseline Implementation

D. Custom UV Maps

The UV maps are created in Blender, and performance is best when their shapes resemble the input garments.

D. Custom UV Maps

E. Results

The results section displays Pix2Surf-mapped textures rendered on SMPL and shows that the mapped textures can be rendered across different body shapes and poses.

E. Results

Related papers

Loading 2003.02050v2…