PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Not Mask What You Do Not Need to Mask: a Parser-Free Virtual Try-On

Thibaut Issenhuth, Jérémie Mary, Clément Calauzènes

arXiv:2007.02721v2cs.CVcs.LGeess.IV

TL;DR

Parser- and pose-based reconstruction for virtual try-on can fail, mask details that should remain, and add inference cost. The paper distills a standard pipeline into a student trained with an adversarial loss, and reports faster inference with preserved person attributes. The conclusion is a parser-free approach that refocuses try-on on changing the cloth.

  • Problem

    Standard virtual try-on uses parser- and pose-based masking to train reconstruction, but parsers can fail and masking can remove person attributes that should be preserved.

    Source: §1 · pp. 2–3

  • Method

    A student learns from a pretrained parser-, pose-, and synthesis-based pipeline and an adversarial loss that encourages use of information masked to the teacher.

    Source: §1 · p. 3, §3.3 · p. 9

  • Results

    Removing parsing and pose estimation at inference raises speed from 6FPS to 77FPS, while S-WUTON preserves attributes such as fingers or handbags.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The teacher-student approach refocuses virtual try-on on changing the cloth of a non-masked person and preserves attributes such as hands or accessories.

    Source: §5 · p. 13

    S-WUTON can be surpassed by VTNFP when models cross their arms, although the paper notes this is not a general failure case.

    Source: §4.2 · p. 11

Abstract

from arXiv · show

The 2D virtual try-on task has recently attracted a great interest from the research community, for its direct potential applications in online shopping as well as for its inherent and non-addressed scientific challenges. This task requires fitting an in-shop cloth image on the image of a person, which is highly challenging because it involves cloth warping, image compositing, and synthesizing. Casting virtual try-on into a supervised task faces a difficulty: available datasets are composed of pairs of pictures (cloth, person wearing the cloth). Thus, we have no access to ground-truth when the cloth on the person changes. State-of-the-art models solve this by masking the cloth information on the person with both a human parser and a pose estimator. Then, image synthesis modules are trained to reconstruct the person image from the masked person image and the cloth image. This procedure has several caveats: firstly, human parsers are prone to errors; secondly, it is a costly pre-processing step, which also has to be applied at inference time; finally, it makes the task harder than it is since the mask covers information that should be kept such as hands or accessories. In this paper, we propose a novel student-teacher paradigm where the teacher is trained in the standard way (reconstruction) before guiding the student to focus on the initial task (changing the cloth). The student additionally learns from an adversarial loss, which pushes it to follow the distribution of the real images. Consequently, the student exploits information that is masked to the teacher. A student trained without the adversarial loss would not use this information. Also, getting rid of both human parser and pose estimator at inference time allows obtaining a real-time virtual try-on.

1 Introduction

The introduction frames virtual try-on as an online-shopping opportunity whose standard parser- and pose-dependent reconstruction pipeline can fail, mask useful person details, and slow inference. It proposes a distilled student and a Warping U-Net architecture, with experiments used to demonstrate the approach.

1 Introduction

  • Earlier 3D-measurement and model-based approaches are computationally intensive and require expensive materials, limiting their suitability at scale.The introduction also identifies photo-realistic try-on as a potential improvement for online shopping.Source: §1 · p. 1
  • Parser- and pose-based masking can fail, hide attributes such as handbags or jewels, and create a wall-clock bottleneck.Fashion-domain retraining would require segmentation labels or unsupervised domain adaptation, and parsers would remain imperfect.Source: §1 · pp. 2–3
  • Removing human parsing and pose estimation at inference raises speed from 6FPS to 77FPS while S-WUTON preserves attributes such as fingers and handbags.The student is distilled from the standard pipeline and trained with an adversarial loss to use information masked to its teacher.Source: §1 · p. 3
  • WUTON combines a convolutional geometric matcher with a siamese-encoder U-net generator, warps the generator’s feature maps, and is trained end-to-end.The authors propose it as an efficient teacher architecture for high-quality synthesis.Source: §1 · p. 3
  • The paper evaluates its method on a virtual try-on dataset using quantitative and visual results and a user study.Source: §1 · p. 3

2 Problem statement and related work

The paper defines 2D try-on as aligning an in-shop garment to a person and synthesizing the dressed image. It reviews parser-based supervised approaches and proposes teacher-generated training targets so a student can operate without the parser at inference.

2 Problem statement and related work

  • Try-on separates into garment deformation and refinement, with a spatial transformer predicting deformation parameters and a conditional generator producing the dressed image.The target is an image of the person wearing the in-shop garment.Source: §2 · p. 3
  • Prior supervised methods use parser- and pose-derived agnostic person representations to train reconstruction from a masked person and cloth image.The representation hides clothing while preserving identity and body shape.Source: §2 · p. 3
  • The student is trained on original-person and cloth inputs paired with teacher-generated images, removing the need for the agnostic representation at inference.The student uses its own geometric-transform and generation modules.Source: §2 · p. 4
  • CAGAN requires a reference person, the worn garment, and the target garment, while VITON and CP-VTON use supervised masking and garment warping.CP-VTON’s L1 alignment loss can be overwhelmed by white backgrounds and solid colors, making patterns and inner structure difficult to align or preserve.Source: §2 · pp. 4–5
  • Other related approaches add parsing-generation modules or dense flow, while appearance-transfer methods rely on pose matching that is not feasible for try-on.The cited 3D-based work also requires expensive scans, which the paper describes as unsuitable for online users.Source: §2 · pp. 4–5

3 Our approach

The approach distills a parser- and pose-estimator-based teacher into a student that takes person and cloth images directly. WUTON aligns cloth features within a U-net, while teacher supervision and an adversarial loss train the student to preserve person attributes.

3 Our approach

  • The overall method distills a standard parser-, pose-estimator-, and synthesis-based try-on pipeline so the student can perform inference without the parser or pose estimator.The teacher-student process trains the student on person and cloth inputs using teacher-generated synthetic targets.Source: §3 · p. 5

3.1 WUTON architecture

  • WUTON predicts a TPS transform with a convolutional geometric matcher, then warps cloth-encoder feature maps through U-net skip connections to synthesize the try-on image.The same transform parameters are used across U-net scales.Source: §3.1 · pp. 6–7

3.2 Training procedure of the teacher model

  • The teacher is trained from RGB person images masked over the arms, upper-body clothing, and neck, using worn-cloth, pixel-level, perceptual, and adversarial supervision.The worn cloth guides the geometric matcher; the person image supervises reconstruction, and adversarial training supports fitting another in-shop cloth without a paired ground truth.Source: §3.2 · pp. 7–8

3.3 Training procedure of the student model

  • The student learns from synthetic triplets produced by the parser and pretrained teacher, while an independent adversarial loss encourages matching real-person images and using attributes masked from the teacher.The paper identifies hands as an example of information the teacher masks.Source: §3 · p. 5, Figure 3 · p. 8, §3.3 · p. 9

4 Experiments and analysis

Across a held-out virtual try-on test set, S-WUTON was preferred to CP-VTON in user judgments, while the student-teacher method also reduced runtime. Visual comparisons examine cloth rendering and preserved person details, alongside a student ablation of the adversarial loss.

4.1 Dataset

  • The Image-based Multi-pose Virtual Try-on dataset contains person and cloth images at 256x192 resolution, with 4,175 test pairs whose cloth was unseen during training.Source: §4.1 · p. 9

4.2 Visual results

  • Figure 4 compares the method with CP-VTON using the same dataset and agnostic person representation, and also presents a comparison with VTNFP.Source: Figure 4 · p. 10
  • T-WUTON preserves complex cloth patterns and produces sharper edges than CP-VTON and VTNFP, while S-WUTON preserves body details such as hands.Source: §4.2 · pp. 10–11
  • VTNFP can surpass S-WUTON when models cross their arms, although the paper notes other crossed-arm examples where S-WUTON composes the arms with occluded cloth.Source: §4.2 · p. 11

4.3 Quantitative results

  • Evaluation uses paired LPIPS and SSIM metrics and unpaired inception score and Fréchet Inception Distance; paired metrics cannot evaluate S-WUTON because its input is unmasked.Source: §4.3 · p. 11

4.4 User study

  • 88% of user votes preferred S-WUTON to CP-VTON; the study supports p > 0.85 with 98.7% confidence.Source: §4.4 · p. 12

4.5 Runtime analysis

  • Runtime decreases from 6FPS to 77FPS with S-WUTON, despite T-WUTON and S-WUTON’s try-on architecture being slightly slower than CP-VTON.Source: §4.5 · p. 12

4.6 The impact of adversarial loss in the teacher-student setting

  • The student’s adversarial loss encourages it to match the real-data distribution beyond the teacher’s distribution, and without it the student does not preserve unmasked person attributes.Source: §4.6 · p. 13, Figure 5 · p. 13

5 Conclusion

The paper proposes distilling the standard virtual try-on pipeline into a student focused on changing clothes on an unmasked person. It reports improved image quality and computational speed, with preservation of hands and accessories.

5 Conclusion

  • Table 3 compares IS and FID scores for S-WUTON with and without the adversarial loss.The supplied caption identifies the models and metrics but does not provide score values.Source: Table 3 · p. 13
  • The teacher-student approach distills the standard virtual try-on pipeline to change clothes on a non-masked person.The conclusion frames this as refocusing on the initial task.Source: §5 · p. 13
  • The approach largely improves image quality and significantly speeds up computation.The conclusion also identifies preserving hands and accessories as necessary for virtual try-on.Source: §5 · p. 13

6 Appendix

The appendix specifies the model implementation and reports visual results for S-WUTON, including its preservation of person details and comparisons with VTNFP. The comparisons also identify cases where VTNFP handles difficult crossing-arm poses better.

6.1 Implementation details

  • The implementation uses a geometric matcher and siamese U-net generator, with a fully convolutional discriminator and relativistic adversarial loss.The matcher warps the generator's feature maps; the discriminator uses five downsampling layers.Source: §6.1 · p. 18
  • Training uses Adam with batch size 8, and S-WUTON inference runs at approximately 77 frames per second on a Tesla V100.The reported training times are around two days for T-WUTON and three days for S-WUTON.Source: §6.1 · p. 18

6.2 More results from S-WUTON

  • S-WUTON preserves cloth patterns and body details across a wide range of poses, with oversized source-person sleeves identified as a common failure case.The examples include stripes, text, and textures.Source: §6.2 · p. 18

6.3 More visual examples on the importance of distillation

  • The visual examples attribute robustness to parsing errors and preservation of masked attributes, including hands, skin, and handbags, to the student-teacher approach.The comparison images are described as visual evidence of the approach's soundness.Source: §6.3 · p. 20, Figure 8 · p. 21

6.4 Comparisons with VTNFP [42]

  • S-WUTON is reported to produce sharper images with better-preserved body details, especially hands, than VTNFP, while VTNFP handles difficult crossing-arm poses better.S-WUTON is also reported to be at least 13 times faster at inference.Source: §6.4 · p. 22

6.5 Comparisons with images from ClothFlow [12] paper

The appendix compares S-WUTON with CP-VTON and ClothFlow using visual examples. It reports that S-WUTON preserves hand shape and handles complex geometric deformations, while ClothFlow has difficulty keeping stripes straight.

6.5 Comparisons with images from ClothFlow [12] paper

  • The visual comparisons include CP-VTON, ClothFlow, T-WUTON, and S-WUTON, with prior-work images taken from their papers.The authors note their models were trained on the MG-VTON dataset rather than the original VITON dataset.Source: §6.5 · p. 24
  • S-WUTON preserves the person's hand shape better than the compared methods in rows 6, 7, and 12.Source: §6.5 · p. 24
  • S-WUTON handles complex geometric deformations as well as ClothFlow and seems slightly better on stripes, whose straightness dense-flow warping can disrupt.The stripe comparison is based on rows 4 and 11.Source: §6.5 · p. 24

6.6 Ablation studies on T-WUTON

The ablations examine adversarial loss and end-to-end geometric learning in T-WUTON. Unpaired adversarial loss handles large cloth-shape variation in the intended use case, while end-to-end learning helps preserve cloth structure.

6.6 Ablation studies on T-WUTON

  • Without adversarial loss, images are blurrier and less sharp; adversarial loss improves sharpness and contrast.The figure also reports that end-to-end training is key to realistic geometric deformations.Source: Figure 10 · p. 27, §6.6 · p. 28
  • Back-propagating synthesized-image losses to the geometric matcher alleviates the failure to preserve the cloth's inner structure when training uses only Lwarp.The supplied results identify end-to-end learning as important and say IS and SSIM are less suited to this task than LPIPS for the not-end-to-end variant.Source: §6.6 · pp. 27–28, Table 4 · p. 28
  • Unpaired adversarial loss better handles large source-to-target cloth-shape variation in the real use case, although paired-setting LPIPS and SSIM favor paired adversarial loss.The unpaired loss is also associated with sharper, higher-contrast images; the figure notes sleeve-shape artifacts in a significant shape-change case.Source: Figure 10 · p. 27, §6.6 · p. 28, Table 4 · p. 28
Loading 2007.02721v2…