PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangled Cycle Consistency for Highly-realistic Virtual Try-On

Chongjian Ge, Yibing Song, Yuying Ge, Han Yang, Wei Liu, Ping Luo

arXiv:2103.09479v2cs.CV

TL;DR

Virtual try-on must match person images with in-shop clothes despite unpaired data, while existing inpainting and cycle-consistency approaches have difficulty preserving both clothing and non-clothing content. DCTON separates clothes warping, skin synthesis, and image composition in an end-to-end cycle-consistent framework. Experiments report favorable performance against state-of-the-art virtual try-on approaches.

  • Problem

    Person and in-shop clothing images are unpaired, and existing inpainting and straightforward cycle-consistency methods struggle with clothing correspondence or coupled clothing and body generation.

    Source: §1 · pp. 1–2

  • Method

    DCTON disentangles clothes warping, skin synthesis, and image composition within an end-to-end cycle-consistent training framework.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    Experiments show DCTON achieves favorable performance compared with state-of-the-art virtual try-on approaches.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The paper concludes that the three disentangled modules can be integrated in one framework for end-to-end cycle-consistent training.

    Source: §5 · p. 8

Abstract

from arXiv · show

Image virtual try-on replaces the clothes on a person image with a desired in-shop clothes image. It is challenging because the person and the in-shop clothes are unpaired. Existing methods formulate virtual try-on as either in-painting or cycle consistency. Both of these two formulations encourage the generation networks to reconstruct the input image in a self-supervised manner. However, existing methods do not differentiate clothing and non-clothing regions. A straight-forward generation impedes virtual try-on quality because of the heavily coupled image contents. In this paper, we propose a Disentangled Cycle-consistency Try-On Network (DCTON). The DCTON is able to produce highly-realistic try-on images by disentangling important components of virtual try-on including clothes warping, skin synthesis, and image composition. To this end, DCTON can be naturally trained in a self-supervised manner following cycle consistency learning. Extensive experiments on challenging benchmarks show that DCTON outperforms state-of-the-art approaches favorably.

1. Introduction

Virtual try-on is difficult because person and in-shop clothing images are unpaired, and existing one-way and cycle-consistency approaches struggle to preserve clothing and non-clothing content together. DCTON separates clothes warping, skin synthesis, and image composition within cycle-consistency training, and the paper reports favorable benchmark performance.

1. Introduction

  • Person and in-shop clothing images are unpaired because collecting multiple pixel-aligned outfits for each model is infeasible.This limits the direct use of paired person-and-clothing examples for 2D try-on.Source: §1 · p. 1
  • DCTON separates clothes warping from non-clothes generation, extending cycle consistency for self-supervised virtual try-on.Its three modules are clothes warping, skin synthesis for occluded body parts, and image composition.Source: Figure 1 · p. 1, §1 · p. 2
  • One-way inpainting methods struggle when target clothes differ substantially from the input, with inaccurate arm generation and skirt artifacts among the examples.The approaches also use separate modules, limiting end-to-end training.Source: §1 · pp. 1–2
  • Vanilla cycle consistency can generate coupled clothing, skin, and pose content, with artifacts reported around arms and logos.The paper argues that straightforward cycle-consistent generation is insufficient for high-quality try-on.Source: Figure 1 · p. 1, §1 · p. 2
  • DCTON performs favorably against state-of-the-art virtual try-on approaches in experiments on benchmark datasets.The network separates clothes warping, skin synthesis, and image composition within a self-supervised try-on cycle.Source: §1 · p. 2

2. Related Work

Related work frames 2D virtual try-on as a response to practical image availability but notes the challenge of missing paired training examples. Prior one-way reconstruction and cycle-consistency approaches each have limitations that motivate disentangling try-on components.

2. Related Work

  • The related-work overview distinguishes 3D-based and 2D-based try-on, noting 2D methods’ suitability for real-world scenarios because image data are lightweight to collect.It situates virtual try-on within fashion editing and learning-based methods.Source: §2.1 · p. 2, §2 · p. 2

2.1. Virtual Try-on

  • Prior 2D try-on methods use one-way reconstruction or vanilla cycle consistency to address the lack of paired triplet training data.The triplet comprises a reference person, target in-shop clothes, and the person wearing those clothes.Source: §2.1 · p. 2
  • One-way try-on methods struggle with substantially different target clothes and lack of end-to-end training, while CA-GAN improves clothes-characteristic preservation but introduces texture and body-generation artifacts.DCTON responds by separating clothes warping, skin synthesis, and image composition within cycle consistency.Source: §2.1 · p. 3

2.2. Cycle Consistency for Image Generation

  • Cycle consistency provides pixel-wise supervision for unpaired image generation, but straightforward cycle consistency is challenging for try-on because clothes texture and occluded body parts must both be generated.The paper reframes try-on as conditional unpaired image-to-image generation from the reference person and target clothes.Source: §2.2 · p. 3

3. Proposed Method

DCTON separates virtual try-on into clothes warping, skin synthesis, and image composition, then fuses their features to generate the output. Cycle consistency trains the system with self-supervision and losses for reconstruction, preserving non-clothing content, and improving warped-clothes correspondence.

3. Proposed Method

  • DCTON disentangles clothes warping, skin synthesis, and image composition, then fuses their feature representations to generate the try-on result.The framework uses three encoders for the disentanglement.Source: §3 · p. 3, §3.1.3 · p. 4

3.1.1 Clothes Warping

  • Densepose-guided mask prediction and a spatial transformer network warp target clothes, with homography-based regularization addressing unstable transformations under large pose variation.The mask prediction network uses parsing-label supervision; the STN is pretrained and kept fixed during cycle-consistency training.Source: §3.1.1 · pp. 3–4

3.1.2 Skin Synthesis

  • Skin synthesis extracts the input skin region using the Densepose descriptor and encodes it for feature-level fusion into the output.The skin features are concatenated with other encoded features at each feature level.Source: §3.1.2 · p. 4

3.1.3 Image Composition

  • Image composition encodes the input image for global representation and concatenates its features with warped-clothes and skin features before decoding the try-on image.The features are concatenated sequentially and sent into the decoders.Source: §3.1.3 · p. 4

3.2. Cycle Consistency Training

  • Cycle consistency sends a generated try-on image through a second same-architecture CNN to form self-supervision for end-to-end learning.The cycle loss uses ℓ1 terms on synthesized try-on results and corresponding skin regions; the objective also includes adversarial, content-preserving, and perceptual losses.Source: §3.2 · pp. 4–5

4. Experiments

Experiments evaluate DCTON on VITON and VITON-HD through qualitative comparisons, quantitative metrics, a user study, and ablations. The reported results include favorable metric comparisons and visual performance, plus faster single-GPU inference than ACGPN.

4.1. Implementation Details

  • VITON has 14,221 training and 2,032 test groups; VITON-HD uses 512×384 images, and DCTON is trained end-to-end for 100 epochs after STN pretraining.The reported setup uses random in-shop clothes during DCTON training.Source: §4 · p. 5, §4.1 · p. 6

4.2. Qualitative Evaluations

  • DCTON preserves target garment characteristics and subtle textures, while producing more convincing occluded body parts than the compared methods in VITON examples.The comparisons describe collar and sleeve limitations, texture distortions, and errors around occluded bodies in competing methods.Source: Figure 4 · p. 6, §4.2 · p. 6
  • On VITON-HD, DCTON preserves target-clothing characteristics and generates occluded body parts in higher-resolution examples whose competing results are upsampled.The dataset is 512×384, where artifacts are more obvious.Source: §4 · p. 5, Figure 5 · p. 7, §4.2 · p. 7

4.3. Quantitative Evaluations

  • DCTON exceeds CA-GAN/VITON/CP-VTON/CP-VTON+/ACGPN by IS 0.29/0.56/0.26/0.10/0.16 and SSIM 0.09/0.09/0.11/0.08/0.02, respectively.Source: §4.3 · p. 7
  • DCTON’s online inference is nearly twice as fast as ACGPN’s on one V100 GPU, reaching 19 FPS, while using more parameters and fewer FLOPs.Training time is reported as similar under the same eight-GPU configuration.Source: §4.3 · pp. 7–8

4.4. User Study

  • In the VITON user study, subjects preferred DCTON for both overall perceptual quality and fine-grained texture details.More than 50 subjects assessed 1,000 image groups, with each subject assigned 100 groups.Source: Table 2 · p. 8, §4.4 · p. 8

4.5. Ablation Study

  • Removing the skin synthesis encoder degrades skin appearance, while removing STN regularization distorts detailed clothing textures.The ablation reports peculiar colors or blurring for generated skin without the encoder, and improved target-clothing warping with regularization.Source: §4.5 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

5. Concluding Remarks

The conclusion identifies insufficient training in one-way reconstruction and weak texture preservation in vanilla cycle consistency as limitations. DCTON separates try-on into three modules within an end-to-end cycle-consistent framework and performs favorably against state-of-the-art approaches.

  • One-way reconstruction hinders sufficient training, while vanilla cycle consistency lacks texture preservation.Source: §5 · p. 8
  • DCTON disentangles virtual try-on into clothes warping, skin synthesis, and image composition within one end-to-end cycle-consistent framework.Source: §5 · p. 8
  • Experiments validate that DCTON performs favorably compared to state-of-the-art virtual try-on approaches.Source: §5 · p. 8
Loading 2103.09479v2…