PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Garment DensePose for Robust Warping in Virtual Try-On

Aiyu Cui, Sen He, Tao Xiang, Antoine Toisoul

arXiv:2303.17688v1cs.CV

TL;DR

Virtual try-on methods rely on garment warping that can fail on hard poses. This work learns garment DensePose weakly supervised and uses its correspondence with person DensePose for warping. It reports state-of-the-art-equivalent benchmark performance and robust warping on hard-pose images, while noting a domain-shift limitation in training from masked garments.

  • Problem

    Pose-conditioned garment warping can be inaccurate on hard poses, and VITON benchmarks lack garment DensePose annotations.

    Source: §3.2 · p. 3

  • Method

    The method learns garment DensePose weakly supervised from garment images and warps through UV-space correspondence with person DensePose.

    Source: §1 · pp. 1–2

  • Results

    The method achieves performance equivalent to state-of-the-art virtual try-on methods on VITON-HD benchmarks and demonstrates robust warping on people in hard poses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Garment-only DensePose prediction supports warping to new person poses without the prediction being affected by human pose diversity.

    Source: §1 · p. 2

    Training on masked garments from person images does not perform well on real garment images because of their distribution shift.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Virtual try-on, i.e making people virtually try new garments, is an active research area in computer vision with great commercial applications. Current virtual try-on methods usually work in a two-stage pipeline. First, the garment image is warped on the person's pose using a flow estimation network. Then in the second stage, the warped garment is fused with the person image to render a new try-on image. Unfortunately, such methods are heavily dependent on the quality of the garment warping which often fails when dealing with hard poses (e.g., a person lifting or crossing arms). In this work, we propose a robust warping method for virtual try-on based on a learned garment DensePose which has a direct correspondence with the person's DensePose. Due to the lack of annotated data, we show how to leverage an off-the-shelf person DensePose model and a pretrained flow model to learn the garment DensePose in a weakly supervised manner. The garment DensePose allows a robust warping to any person's pose without any additional computation. Our method achieves the state-of-the-art equivalent on virtual try-on benchmarks and shows warping robustness on in-the-wild person images with hard poses, making it more suited for real-world virtual try-on applications.

1. Introduction

Virtual try-on generates an image of a person wearing a new garment, but pose-conditioned warping can fail on hard poses. This work learns garment DensePose from garment images and uses its correspondence with person DensePose for robust warping.

1. Introduction

  • Virtual try-on generates a photorealistic image of a person wearing a new garment from a person image and a garment image.The paper motivates the task by uncertainty about online fashion purchases and resulting returns.Source: §1 · p. 1
  • Pose-conditioned flow warping can be inaccurate for hard poses and complex limb gestures, limiting robustness on in-the-wild images.The passage attributes this to dependence on the person image and difficulty predicting different offsets for connected garment regions.Source: §1 · p. 1
  • The method learns garment DensePose from garment images and uses UV-space correspondence with person DensePose to warp garments onto new poses.Because prediction comes directly from the garment image, human pose diversity does not affect its accuracy.Source: §1 · p. 1
  • Garment DensePose is learned weakly supervised, without manually annotated garment DensePose data, which is unavailable and tedious to acquire.The contribution is intended to support virtual try-on in more complex and realistic cases.Source: §1 · p. 2
  • The method achieves performance equivalent to state-of-the-art methods on VITON-HD benchmarks and demonstrates more robust warping on people in hard poses.The authors also apply the model learned on VITON-HD to hard cases.Source: §1 · p. 2

2. Related Work

Prior image-based try-on methods warp garments using TPS or appearance flow, while 3D approaches can require scans or still use TPS warping. This work instead uses garment-to-person correspondence in DensePose's fixed IUV space.

2. Related Work

  • TPS methods predict non-rigid deformation grids, whereas appearance-flow methods predict per-pixel garment offsets and offer more warping freedom.Appearance flow is used by most recent state-of-the-art try-on works.Source: §2 · p. 2
  • The proposed garment DensePose uses fixed IUV-space correspondence and avoids direct dependence on the person image during warping.Prior work [7] applied a similar idea to garment swap, but this setting requires predicting ghost-mannequin garment DensePose without annotations.Source: §2 · p. 2
  • Traditional 3D try-on methods use SMPL garment templates requiring 3D body scans, while a recent 2.5D approach still requires TPS warping.DensePose represents the body in 24 parts, assigning each surface location a body-part label and 2D UV location.Source: §2 · p. 2

3. Methodology

The method predicts garment DensePose from garment-only images and uses its UV correspondence with person DensePose to warp garments; it then blends the warped garment with the person image. Its design addresses sparse UV matches and the resolution trade-off in DensePose warping.

3.1. Preliminary: DensePose

  • DensePose assigns each pixel a body-part label and UV coordinates, with matching body parts sharing UV coordinates across poses, body shapes, and viewpoints.The body surface is divided into 24 parts, each with its own UV space.Source: §3.1 · p. 3

3.2. Weakly Supervised Garment DensePose Prediction

  • Garment DensePose is predicted from garment-only images and aligned to person DensePose using a pretrained flow model during training, not inference.The approach uses masked person DensePose as weak supervision because garment DensePose annotations are unavailable; losses include classification, L1, and total-variation terms.Source: §3.2 · p. 3

3.3. Warping via Garment DensePose

  • Naive UV-space alignment can produce poor warps because DensePose is sparse and its texture map has limited resolution.Small coordinate shifts can leave query UV locations without source values.Source: §3.3 · p. 3, §3.3.1 · pp. 3–4

3.3.1 Inpainting and Nearest Neighbor Sampling for UV Sparsity Issue

  • To fill UV gaps without expanding the garment shape, the method refines a coarse warped mask and uses nearest valid source values only inside that mask.The coarse mask has holes and noisy boundaries; unrestricted UV inpainting can distort garment shape, such as turning short sleeves into long sleeves.Source: §3.3.1 · p. 4

3.3.2 Coordinate Warping for Resolution Issue

  • Warping RGB values through a low-resolution UV map trades reduced sparsity for blur, so the method warps source-image coordinates and samples RGB directly.This reduces information loss from UV-map downsampling and upsampling.Source: §3.3.2 · p. 4

3.4. Try-On Image Generation

  • A generator takes the processed person image, warped garment, and arm-and-hand DensePose mask to predict a coarse try-on image and blending mask.The final image blends the coarse result with the warped garment to restore garment-region high-frequency detail; training also uses L1, perceptual, style, and blending-mask losses.Source: §3.4 · p. 4

4. Experiments

The experiments compare virtual try-on quality on VITON-HD, test warping robustness on hard poses, and ablate the training and warping design. The method matches leading benchmark methods and performs better than the compared methods on out-of-distribution and complex-pose cases.

4.1. Implementation Details

  • NM-SSIM measures garment-region SSIM normalized by the union area of the warped and ground-truth garment masks, alongside mean IoU for warping robustness.MPV-hard contains 1,670 full-body test pairs with varied arm poses.Source: §4.1 · p. 5

4.2. Comparisons

  • The method outperforms most VITON-HD methods and matches StyleFlow and ClothFlow.StyleFlow was retrained in a parser-based manner for a fair comparison.Source: §4.2 · p. 5, Table 1 · p. 5
  • On out-of-distribution and complex-pose cases, the method significantly outperforms all compared methods.The methods were trained on simple-pose VITON-HD images and tested on MPV-hard.Source: §4.2 · p. 5, Table 2 · p. 6
  • The user study found the method slightly less favorable than StyleFlow and ClothFlow, with nonsmooth textures attributed to flow interference, but the difference was not significant.The reported user-study outcome remains similar to state-of-the-art performance on VITON-HD alongside SSIM and FID.Source: §4.2 · p. 6

4.3. Ablation Study

  • The cropped-person DensePose baseline warps inaccurately at garment-only test time because of the domain gap between those inputs.The baseline learns from cropped garments in person images, whereas the test input is a garment-only image.Source: §4.3 · p. 6

B. Mask-guided UV inpainting, nearest neighbor sampling and coordinate warping

The ablations identify mask-guided UV inpainting and grid sampling as important for warping quality, and report limitations from person DensePose errors and two-stage sampling.

B. Mask-guided UV inpainting, nearest neighbor sampling and coordinate warping

  • Mask-guided UV inpainting fills holes created by DensePose warping, while grid sampling preserves texture sharpness.Removing either component degrades the corresponding property.Source: B. Mask-guided UV inpainting, nearest… · p. 6
  • Adding a style loss and a training-time inpainting mechanism to the inherited fusion model is supported by ablations showing both changes are necessary.The fusion architecture and other losses are inherited from [18].Source: B. Mask-guided UV inpainting, nearest… · p. 6

4.4. Failure Cases

  • Inaccurate person DensePose, especially in garment regions, causes warp artifacts, while two-stage UV sampling distorts complex garment patterns.The user study was less favorable for striped and checkered garments.Source: §4.4 · p. 6

5. Conclusion

The conclusion presents garment DensePose as a weakly supervised warping approach with benchmark-equivalent performance and greater robustness on complex-pose images.

5. Conclusion

  • The paper proposes garment DensePose for warping garments in virtual try-on.The garment DensePose is learned from weakly supervised labels without manual annotation.Source: §5 · p. 6
  • The method achieves state-of-the-art-equivalent results and is more robust to real-world person images with complex poses.The paper describes this robustness as making the method suited to real-world try-on applications.Source: §5 · p. 6
  • The garment DensePose is trained without manual annotation.The conclusion describes the labels as weakly supervised.Source: §5 · p. 6

Related papers

Loading 2303.17688v1…