PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Diffusion Models for Authentic Virtual Try-on in the Wild

Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, Jinwoo Shin

arXiv:2403.05139v3cs.CV

TL;DR

Prior virtual try-on methods improve naturalness but fail to preserve fine-grained garment identity and details. IDM-VTON fuses high-level garment semantics and low-level garment features into a diffusion model, while using detailed captions and pair-based customization. IDM-VTON outperforms prior diffusion- and GAN-based methods in preserving garment details and generating authentic virtual try-on images, including in-the-wild settings.

  • Problem

    Prior virtual try-on methods improve naturalness but fail to preserve fine-grained garment identity and details.

    Source: §1 · p. 2

  • Method

    IDM-VTON fuses high-level garment semantics and low-level garment features into a diffusion model, while using detailed captions and pair-based customization.

    Source: §1 · pp. 2–3, §3.2 · p. 6, GarmentNet · p. 7

  • Results

    IDM-VTON outperforms prior diffusion- and GAN-based methods in preserving garment details and generating authentic virtual try-on images, including in-the-wild settings.

    Source: Qualitative results · p. 11, Quantitative results · p. 11, §4.3 · p. 12

  • Takeaways & Limitations

    IDM-VTON with customization outperforms other methods on all reported metrics and significantly improves generalization.

    Source: §4.3 · p. 12

    IDM-VTON struggles to preserve human attributes such as tattoos or skin moles in masked regions, and broader text-controlled garment generation remains future work.

    Source: Limitation · p. 14

Abstract

from arXiv · show

This paper considers image-based virtual try-on, which renders an image of a person wearing a curated garment, given a pair of images depicting the person and the garment, respectively. Previous works adapt existing exemplar-based inpainting diffusion models for virtual try-on to improve the naturalness of the generated visuals compared to other methods (e.g., GAN-based), but they fail to preserve the identity of the garments. To overcome this limitation, we propose a novel diffusion model that improves garment fidelity and generates authentic virtual try-on images. Our method, coined IDM-VTON, uses two different modules to encode the semantics of garment image; given the base UNet of the diffusion model, 1) the high-level semantics extracted from a visual encoder are fused to the cross-attention layer, and then 2) the low-level features extracted from parallel UNet are fused to the self-attention layer. In addition, we provide detailed textual prompts for both garment and person images to enhance the authenticity of the generated visuals. Finally, we present a customization method using a pair of person-garment images, which significantly improves fidelity and authenticity. Our experimental results show that our method outperforms previous approaches (both diffusion-based and GAN-based) in preserving garment details and generating authentic virtual try-on images, both qualitatively and quantitatively. Furthermore, the proposed customization method demonstrates its effectiveness in a real-world scenario. More visualizations are available in our project page: https://idm-vton.github.io

1 Introduction

Image-based virtual try-on aims to render a person wearing a specified garment, but existing methods often sacrifice garment detail or image authenticity. IDM–VTON addresses this with dual garment encoding, detailed captions, and customization for improved fidelity, authenticity, and wild-scene generalization.

Motivation

  • Existing GAN- and diffusion-based methods struggle to preserve fine-grained garment patterns, textures, shapes, and colors while generating natural images.GAN-based approaches also generalize poorly to varied human images, whereas diffusion-based methods improve naturalness but still lose garment details.Source: §1 · p. 2

Method

  • IDM–VTON combines an image prompt adapter for high-level garment semantics with GarmentNet for low-level features that preserve fine-grained details.The two modules condition the diffusion model through complementary attention mechanisms.Source: §1 · p. 2

Customization

  • Customization from a single garment-person pair further improves visual quality, particularly for in-the-wild virtual try-on.Source: §1 · pp. 2–3

Method

  • Detailed garment captions help retain the prior knowledge of text-to-image diffusion models and improve authentic virtual try-on generation.Source: §1 · p. 3

Results

  • Experiments on VITON-HD, DressCode, and a collected In-the-Wild dataset show superior qualitative and quantitative performance, with customization notably improving garment identity retention.The In-the-Wild dataset contains intricate garment patterns and varied poses or gestures unlike the training data.Source: §1 · p. 3

2 Related Works

Prior virtual try-on methods use garment warping, diffusion-based inpainting, or conditional controls, but retaining meticulous garment details and generalizing to arbitrary people remain challenges. Diffusion customization offers a route toward adapting models to real-world virtual try-on scenarios.

Image-based virtual try-on

  • GAN-based methods deform garments before blending them onto person images, but they lack generalization to complex backgrounds and intricate poses.Source: Image-based virtual try-on · p. 3

Diffusion-based methods

  • Diffusion-based virtual try-on improves image quality but still struggles to preserve meticulous garment details needed for real-world applications.Source: Image-based virtual try-on · p. 3

Conditional control

Customization

  • Few-image customization adapts diffusion models to unseen examples, and this paper applies customization to virtual try-on to improve adaptation to real-world scenarios.Source: Customizing diffusion models · p. 4

3 Method

IDM–VTON adapts latent diffusion for virtual try-on by combining person, pose, mask, garment-semantic, and garment-detail conditioning. It evaluates this design on public datasets and uses customization to address inputs outside the training distribution.

3.2 Proposed Method

  • IDM–VTON combines a TryonNet, IP-Adapter, and GarmentNet to condition virtual try-on on person information and both high- and low-level garment features.GarmentNet features enter TryonNet through self-attention, while text and IP-Adapter features are fused through cross-attention.Source: §3.2 · p. 6
  • TryonNet receives person latents, a garment-removal mask, masked-person latents, and Densepose latents, using an expanded 13-channel input layer.The model uses the SDXL inpainting model as its latent-diffusion base.Source: TryonNet · p. 6
  • IP-Adapter encodes high-level garment semantics with a frozen OpenCLIP image encoder, while its projection and cross-attention layers are fine-tuned.Textual garment prompts are also passed into the model's conditioning pathway.Source: Image prompt adapter · p. 6
  • GarmentNet uses a frozen pretrained UNet encoder to extract garment details and fuses its intermediate features with TryonNet through self-attention.This complements IP-Adapter because CLIP image features can miss fine-grained patterns and graphic prints.Source: GarmentNet · pp. 6–7
  • IDM–VTON is evaluated on VITON-HD and DressCode, while customization fine-tunes only TryonNet up-block attention layers using a person–garment image pair.When only a person image is available, the garment can be segmented and the background whited out to obtain the garment image.Source: Figure 3 · p. 7, Customization of IDM–VTON · pp. 7–8
  • Detailed garment captions are supplied to both GarmentNet and TryonNet to encode garment semantics with natural language alongside image conditioning.The captions describe attributes such as garment shape and texture, using annotators and manual templates.Source: GarmentNet · p. 7

4 Experiment

Across public and In-the-Wild datasets, IDM–VTON is evaluated against GAN- and diffusion-based baselines, with customization and component ablations testing garment fidelity and generalization.

4.1 Experimental Setup

  • The evaluation uses VITON-HD, DressCode, and a more challenging In-the-Wild dataset containing varied poses, complex backgrounds, and garments with patterns and logos.The In-the-Wild dataset contains 62 upper garments and 312 person images, including multiple people per garment for customization experiments.Source: Implementation details · pp. 8–9, Evaluation dataset · p. 9
  • The experiments compare GAN-based and diffusion-based baselines using LPIPS, SSIM, CLIP image similarity, and FID where applicable.The baselines include HR-VITON, GP-VTON, LaDI-VTON, DCI-VTON, and StableVITON.Source: Baselines · p. 9, Table 1 · p. 9, Evaluation metrics · p. 10

4.2 Results on Public Dataset

  • IDM–VTON preserves both low-level garment details and high-level semantics better than competing methods on VITON-HD and DressCode.GAN-based methods capture some fine details but produce less natural images, while prior diffusion methods produce natural images with lower garment consistency.Source: Qualitative results · p. 11, Quantitative results · p. 11
  • IDM–VTON consistently outperforms prior diffusion-based methods on VITON-HD and DressCode, while customized IDM–VTON performs best on In-the-Wild metrics.The public-dataset comparison reports advantages in garment detail, naturalness, and DressCode generalization; Table 2 identifies customized IDM–VTON as the best In-the-Wild variant.Source: Quantitative results · p. 11, Qualitative results · p. 11, Table 2 · p. 11

4.3 Results on In-the-Wild Dataset

  • On In-the-Wild images, IDM–VTON generates more authentic results than diffusion baselines, while customization improves consistency with logos and text renderings.Customization notably improves garment fidelity, whereas customization applied to StableVITON is reported as marginal or ineffective.Source: Figure 5 · p. 10, §4.3 · p. 12

4.4 Ablative Studies

  • Removing GarmentNet reduces preservation of fine-grained garment identity, while adding it improves reconstruction and image-similarity scores.The qualitative and quantitative comparisons specifically identify graphics such as those on t-shirts as better retained with GarmentNet.Source: Effect of GarmentNet · pp. 12–13, Figure 6 · p. 12, Table 3 · p. 13
  • Detailed garment captions produce images more consistent with the garment than naïve captions, including improved rendering of meshes around necks and arms.The captions complement image conditioning by exploiting the generative prior of the text-to-image diffusion model.Source: Effect of detailed caption · p. 13, Figure 7 · p. 13
  • All customization strategies improve LPIPS and SSIM over default IDM–VTON, with decoder-layer fine-tuning performing best and LoRA underperforming it.Customization methods have comparable CLIP image-similarity scores; fine-tuning all UNet layers is comparable to decoder-layer fine-tuning.Source: Table 4 · p. 13, Ablation on customization · p. 14

5 Conclusion

IDM–VTON combines separate high- and low-level garment encoders, detailed garment descriptions, and customization to support authentic virtual try-on, especially in the wild. The paper also reports ethical responsibilities and limitations involving human attributes and future textual control.

5 Conclusion

  • IDM–VTON uses a visual encoder and parallel UNet to provide high-level semantics and low-level garment features to the base UNet.Detailed natural-language garment descriptions further support authentic virtual try-on images.Source: §5 · p. 14
  • The method outperforms prior works in preserving garment details and generating high-fidelity images across various datasets, including in-the-wild settings.Customization is proposed by fine-tuning UNet decoder layers using a garment-person image pair.Source: §5 · p. 14
  • Virtual try-on users must protect ownership and avoid malicious use.Source: Potential negative impact · p. 14
  • IDM–VTON struggles to preserve human attributes such as tattoos or skin moles in masked regions.The paper suggests conditioning these attributes during generation as a possible remedy.Source: Limitation · p. 14
  • Using textual prompts to control garment generation beyond comprehensive captions remains future work.Source: Limitation · p. 14

A.1 In-the-Wild Dataset

The In-the-Wild dataset pairs garments with human images collected from online shopping and social media sources. It includes diverse backgrounds and standardized image dimensions for processing.

A.1 In-the-Wild Dataset

  • The dataset examples consist of paired garment images and humans wearing those garments.Source: Figure 8 · p. 19
  • The dataset contains multiple human images wearing each target garment, with garments from an online shopping mall and human images from social media.Source: §A.1 · p. 19
  • Human images span diverse backgrounds, including parks, buildings, and snowy landscapes.Source: §A.1 · p. 19
  • Garment and human images are center-cropped or resized to compatible 1024 × 768 dimensions.Source: §A.1 · pp. 19–20

A.2 Training and Inference

The model is trained with Adam and standard augmentations, then inferred with DDPM denoising conditioned on low- and high-level garment features. Classifier-free guidance combines these conditions using a fixed scale.

A.2 Training and Inference

  • Training uses Adam with learning rate 1e-5 for 130 epochs, while customization uses learning rate 1e-6 for 100 steps.Training takes around 40 hours on four A800 GPUs, whereas customization takes around 2 minutes on one A800 GPU.Source: §A.2 · p. 20
  • Inference generates images with a DDPM scheduler for 30 steps, starting denoising from random Gaussian noise at strength 1.0.This setting ignores the masked portion of the input image.Source: §A.2 · p. 20
  • Classifier-free guidance merges GarmentNet low-level features Lg and IP-Adapter high-level semantics Hg from the same garment image.Source: §A.2 · p. 20
  • The guidance prediction interpolates conditioned and unconditional noise estimates using guidance scale s=2.0.The conditioned estimate uses Lg and Hg, while the unconditional estimate omits them.Source: §A.2 · p. 20

A.3 Detailed captioning of garments

The method generates detailed garment captions from automatically extracted fashion attributes. These captions encode features such as sleeve length, neckline type, and item name for diffusion conditioning.

A.3 Detailed captioning of garments

  • A commercial fashion attribute annotator extracts garment information to generate detailed captions.The annotator was trained on more than 1,000 fashion attributes.Source: §A.3 · p. 20
  • Caption generation uses sleeve length, neckline type, and item name, producing descriptions such as “short sleeve off shoulder t-shirts”.Source: §A.3 · p. 21
  • Figure 10 compares IDM–VTON with OOTDiffusion [55] on VITON-HD and DressCode, with both methods trained on VITON-HD.Source: Figure 10 · p. 21

B Comparison with Concurrent Work

IDM–VTON is compared with OOTDiffusion across standard and In-the-Wild virtual try-on datasets using qualitative and quantitative evaluations. The comparisons emphasize garment semantics, fine details, image authenticity, and generalization.

Qualitative comparison

  • IDM–VTON captures both high-level semantics and low-level garment details more effectively than OOTDiffusion while generating more authentic images.The qualitative comparisons include VITON-HD, DressCode, and In-the-Wild datasets.Source: B Comparison with Concurrent Work · p. 22

In-the-Wild comparison

  • On the In-the-Wild dataset, IDM–VTON shows notable qualitative improvements over OOTDiffusion, demonstrating stronger generalization.Both models are compared using training on VITON-HD data for the In-the-Wild visual comparison.Source: B Comparison with Concurrent Work · p. 22, Figure 11 · p. 23

Quantitative comparison

Comparison with prior methods

  • Qualitative comparisons show GAN-based methods introducing unrealistic body and arm distortions, while diffusion baselines miss low-level features or high-level semantics.The comparison covers VITON-HD and DressCode and reports that IDM–VTON preserves clothing details while producing realistic images on In-the-Wild data.Source: Figure 12 · p. 24, Figure 13 · p. 25

Related papers

Loading 2403.05139v3…