PaperlayerView exact revision on arXiv

Source-linked AI summary

360-Degree Textures of People in Clothing from a Single Image

Verica Lazova, Eldar Insafutdinov, Gerard Pons-Moll

arXiv:1908.07117v1cs.CVcs.GR

TL;DR

Recovering a complete 3D person, especially texture, from one image remains a challenge despite the need for personalized human models. This paper predicts texture, geometry, and clothing layout in SMPL UV-space using scan registrations and image-to-image translation, and reports plausible single-image avatars with editable clothing.

  • Problem

    Although single-image pose and shape recovery is extensively studied, predicting a person’s complete texture from one image has received almost no attention.

    Source: §1 · p. 1

  • Method

    The model translates partial texture and clothing-layout maps into complete SMPL UV-space maps, learning correspondence from non-rigid registrations of 3D scans.

    Source: §1 · p. 2

  • Results

    Trained on 4,541 registered scans, the method generalizes to real images and reconstructs plausible single-image avatars with texture, geometry, and segmentation layout.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The predicted factors support pose and shape changes, garment swapping, and editing clothing geometry and texture on 3D avatars.

    Source: §5 · p. 8

    The model cannot predict clothing with topology substantially different from the human body and struggles with high-frequency detail and complex texture patterns.

    Source: §5 · p. 8

Abstract

from arXiv · show

In this paper we predict a full 3D avatar of a person from a single image. We infer texture and geometry in the UV-space of the SMPL model using an image-to-image translation method. Given partial texture and segmentation layout maps derived from the input view, our model predicts the complete segmentation map, the complete texture map, and a displacement map. The predicted maps can be applied to the SMPL model in order to naturally generalize to novel poses, shapes, and even new clothing. In order to learn our model in a common UV-space, we non-rigidly register the SMPL model to thousands of 3D scans, effectively encoding textures and geometries as images in correspondence. This turns a difficult 3D inference task into a simpler image-to-image translation one. Results on rendered scans of people and images from the DeepFashion dataset demonstrate that our method can reconstruct plausible 3D avatars from a single image. We further use our model to digitally change pose, shape, swap garments between people and edit clothing. To encourage research in this direction we will make the source code available for research purpose.

1. Introduction

The paper addresses the lack of methods for predicting a person’s complete texture from one image, despite the need for personalized 3D human models. It learns to complete texture and clothing layout in SMPL UV-space, producing an animatable avatar with controllable clothing geometry.

1. Introduction

  • The paper targets the gap in single-image 3D human recovery: while pose and shape recovery are extensively studied, complete person-texture prediction has received almost no attention.The introduction cites repetitive garment patterns, relatively homogeneous skin and hair, and similar left and right shoes as cross-view regularities.Source: §1 · p. 1
  • The model learns appearance correlations from real people and predicts a full 360° texture map from the visible portion of an input image in SMPL UV-space.Registering SMPL non-rigidly to 4,541 scans brings their appearances into a common UV-map representation for learning.Source: §1 · pp. 1–2
  • The model also predicts a full clothing segmentation layout, from which it predicts plausible garment geometry that users can control by editing the layout.The paper describes changing sleeve length and interpolating between shorts and pants.Source: §1 · p. 2
  • The method predicts full texture, geometry, and segmentation layout for a 3D animatable avatar from one image, trained on 4,541 scans registered to a common SMPL template.The authors report generalization to real images, often reproducing clothing boundaries and completing garment textures and skin.Source: §1 · p. 2

2. Related work

Related image-generation methods can synthesize posed images, but disocclusions and out-of-plane rotations remain difficult, and image-based models do not provide the full 3D representation some applications require. The paper distinguishes its approach through aligned, complete scan-derived UV maps and predictions of a textured avatar with geometry and segmentation.

2. Related work

  • DensePose-based texture completion can be limited by parsing errors and misalignment, whereas this work trains on complete, aligned SMPL UV maps from registered scans that include clothing and hair.The cited approaches [49] [26] map multiple views to a custom UV map; this paper uses DensePose only to create the partial input texture map.Source: §2 · p. 2
  • Image-based generation can produce plausible posed images, but disocclusions and out-of-plane rotations are problematic, and such models do not provide temporally coherent pose sequences or full 3D representations.Video-generation methods often target a particular subject; complex scenes with objects or multiple people are also a challenge for image-based approaches.Source: §2 · p. 2
  • Multi-view texture methods combine partial views through blending, mosaicing, graph cuts, or flow-based registration, while this work learns from complete texture maps obtained through 3D registration.Figure 2 shows example scan registrations and appearances encoded as texture maps in a common UV-space.Source: §2 · p. 3, Figure 2 · p. 3
  • Prior work includes clothing-conditioned image synthesis and appearance swapping, while 3D person-reconstruction methods may require video or scans, lack pose, shape, and clothing control, or focus on faces or garments.The cited clothing-synthesis approaches condition generation on an item [30] [66] or description [78]; other works swap appearances between subjects.Source: §2 · pp. 2–3

3. Method

The method learns texture and geometry completion in SMPL UV space, using scans registered into correspondence and separate image-to-image stages to build a 3D avatar. Its segmentation output also supports editing garment properties.

3.1. Synthetic data generation

  • 4541 static scans are non-rigidly registered to SMPL to create complete textures aligned in the model’s UV space.Source: §3.1 · p. 3

3.1.1 SMPL Body Model with Clothing

  • Clothing-aware registration adds vertex offsets beyond SMPL’s undressed shape space, initializes pose and shape from 2D joints, then balances scan fit with closeness to the model.The robust objective pulls registrations to scans while keeping them human-shaped, bringing scans into correspondence.Source: §3.1.1 · pp. 3–4, Figure 3 · p. 4

3.2. 3D Model Generation

  • The pipeline completes texture and segmentation maps, predicts clothing displacement, and applies the generated maps to SMPL to form a full 3D avatar.The complete segmentation can also be edited to change garment properties or redress the person.Source: §3.2 · p. 4, §3.2.3 · p. 5

3.2.1 Texture Completion

  • DensePose maps visible image pixels to a partial SMPL UV texture, which an inpainting network completes despite extraction distortions and imperfect input-output alignment.Training uses adversarial, reconstruction, perceptual, and dissimilarity terms.Source: §3.2.1 · p. 4

3.2.2 Segmentation completion

  • A translation model inpaints the missing regions of a partial clothing segmentation, then assigns each pixel its nearest-neighbour label.Source: §3.2.2 · p. 5

3.2.3 Displacement map prediction

  • The displacement model predicts clothing-related vertex offsets from segmentation, yielding plausible geometry that may not match image details such as a tight or loose T-shirt.Conditioning on segmentation makes the resulting 3D model straightforward to edit.Source: §3.2.3 · p. 5

4. Experiments and Results

The experiments use registered scans for training and evaluate qualitative avatar completion on real images. Results show completed textures and segmentation, with clothing-editing applications and a single-view comparison to a monocular-video method.

4. Experiments and Results

  • Evaluation is qualitative because the tested GAN metrics did not correspond to human-perceived texture quality.The real-image tests use fully visible, roughly frontal people without overly complex clothing or poses.Source: §4 · p. 5, §4.1 · p. 5

4.1. Dataset

  • Models were trained for texture inpainting on 4,541 scans plus 242 SURREAL texture maps [64]; testing used 10 scans, People Snapshot [8], and 526 preselected DeepFashion images.For geometry prediction, the selected subset contained 2,056 scans, with 30 held out for validation.Source: §4.1 · p. 5
  • The segmentation ground truth is stitched from maps projected from 20 rendered views, while partial input maps are extracted with DensePose.The scans are also rendered from 10 roughly frontal views for the texture-inpainting dataset.Source: §4.1 · p. 5

4.2. Results

  • Real-image examples from People Snapshot and DeepFashion show recovered texture, segmentation, predicted geometry, and textured avatars.Predicting body shape is outside the contribution.Source: §4.2 · p. 5, Figure 4 · p. 6
  • The single-view results look comparable and visually pleasing to the monocular-video method [6], which produces higher-resolution reconstructions.The single-view method requires just one view.Source: §4.2 · p. 7, Figure 5 · p. 7
  • Segmentation supports changing a garment’s texture, transferring garments between subjects, and editing sleeve or pants length.For texture edits, the network hallucinates wrinkles to make the added texture look more cloth-like.Source: §4.2 · pp. 7–8, Figure 6 · p. 7, Figure 7 · p. 7

5. Conclusions and Future Work

The method represents texture, geometry, and clothing layout as controllable factors in a common SMPL UV-space, supporting full-avatar reconstruction and garment editing. The authors highlight control and novel-pose generalization while noting limits in clothing topology and texture detail.

5. Conclusions and Future Work

  • Texture, displacement, and segmentation maps separate appearance, geometry, and clothing control in the reconstructed 3D avatars.The authors demonstrate garment swapping and editing of garment geometry and texture.Source: §5 · p. 8
  • The model cannot predict clothing with topology substantially different from the human body, such as skirts or dresses.The authors suggest implicit-function representations as a possible direction, while noting they do not allow control.Source: §5 · p. 8
  • The full model can be reposed in challenging target poses outside the training set, unlike the image-based reposing method shown.The authors identify control and generalization to novel poses and views as advantages of full 3D completion.Source: Figure 9 · p. 8, §5 · p. 8
  • The model struggles with high-frequency detail and complex texture patterns.The authors report that style-based approaches they tested were not photorealistic, lacked control, or required retraining per texture.Source: §5 · p. 8
  • The authors plan to explore hybrid methods combining 3D representations with neural rendering.They describe this as future work.Source: §5 · p. 8
Loading 1908.07117v1…