Source-linked AI summary
OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on
Yuhao Xu, Tao Gu, Weifeng Chen, Chengcai Chen
TL;DR
Image-based virtual try-on must generate realistic outfitted images while retaining garment details and controllability. OOTDiffusion uses an outfitting UNet with self-attention outfitting fusion and outfitting dropout for classifier-free garment guidance, and experiments on VITON-HD and Dress Code report superiority over other VTON methods in realism and controllability. The method remains limited on some cross-category try-ons and can alter details in the original human image.
Problem
Image-based VTON must improve realism while preserving garment details such as complicated text, textures, colors, patterns, and lines.
Method
OOTDiffusion uses an outfitting UNet, self-attention outfitting fusion without redundant warping, and outfitting dropout to enable classifier-free control of garment-feature strength.
Results
OOTDiffusion outperforms other VTON methods in realism and controllability on the VITON-HD and Dress Code datasets.
Takeaways & Limitations
The authors conclude that OOTDiffusion has broad application prospects for image-based virtual try-on.
Takeaways & Limitations
Because training uses paired human and garment images, OOTDiffusion may fail on some cross-category try-ons and may alter original human details such as muscles, watches, or tattoos.
Abstract
from arXiv · showhide
We present OOTDiffusion, a novel network architecture for realistic and controllable image-based virtual try-on (VTON). We leverage the power of pretrained latent diffusion models, designing an outfitting UNet to learn the garment detail features. Without a redundant warping process, the garment features are precisely aligned with the target human body via the proposed outfitting fusion in the self-attention layers of the denoising UNet. In order to further enhance the controllability, we introduce outfitting dropout to the training process, which enables us to adjust the strength of the garment features through classifier-free guidance. Our comprehensive experiments on the VITON-HD and Dress Code datasets demonstrate that OOTDiffusion efficiently generates high-quality try-on results for arbitrary human and garment images, which outperforms other VTON methods in both realism and controllability, indicating an impressive breakthrough in virtual try-on. Our source code is available at https://github.com/levihsu/OOTDiffusion.
1 Introduction
Image-based virtual try-on seeks realistic images of target people wearing specified garments while preserving fine garment details. OOTDiffusion addresses these challenges with latent-diffusion architecture, outfitting fusion, and outfitting dropout, then evaluates on two high-resolution benchmarks.
- Motivation: VTON generates an outfitted image of a target human wearing a given garment for e-commerce applications.The technology is described as improving consumers’ shopping experience and reducing clothing merchants’ advertising costs.
- Challenges: Realism and preservation of garment details are the two main challenges in image-based VTON.Important details include folds, lighting, textures, colors, patterns, lines, and complicated text.
- Method: OOTDiffusion uses an outfitting UNet to learn garment detail features in latent space and outfitting fusion to align them with noisy human bodies without redundant warping.The fusion occurs in the denoising UNet’s self-attention layers and is intended to avoid information loss or feature distortion from independent warping.
- Method: Outfitting dropout enables classifier-free guidance, allowing garment-control strength to be adjusted during inference.The training process randomly drops garment latents so the model learns conditional and unconditional denoising.
- Evaluation: OOTDiffusion is trained and evaluated on the VITON-HD and Dress Code high-resolution benchmarks.The authors report extensive qualitative and quantitative evaluations for realism and controllability across target humans and garments.
2 Related Work
Prior VTON research uses GANs or latent diffusion models and often relies on warping or feature fusion to align garments. These approaches can struggle with realistic appearance or preservation of fine garment details.
- VTON approaches: Recent VTON methods primarily use GANs or latent diffusion models to generate more natural and accurate results.The related work discusses both model families as major approaches to image-based virtual try-on.
- GAN-based methods: GAN-based methods address alignment and high-resolution synthesis but may involve separate warping and segmentation operations.VITON-HD uses ALIAS normalization and a generator, while HR-VITON performs warping and segmentation simultaneously.
- Latent-diffusion methods: Some latent-diffusion VTON methods require explicit warping before conditioning or refinement.LaDI-VTON warps garment features before conditioning diffusion, while DCI-VTON combines warped clothes with a masked person image before refinement.
- Limitations of prior work: Warping-based latent-diffusion methods have not fully preserved complicated garment patterns and text.The cited discussion attributes this shortcoming to information loss during the process.
3 Method
OOTDiffusion extends latent diffusion with an outfitting UNet that learns garment details and integrates them into denoising through outfitting fusion. Outfitting dropout enables classifier-free control over garment-feature strength during inference.
- Architecture: OOTDiffusion extends Stable Diffusion with separate outfitting and denoising UNets for garment-conditioned virtual try-on.The garment is encoded into latent space, while the masked human latent is combined with Gaussian noise for denoising.
- Outfitting UNet: The outfitting UNet learns garment detail features from the encoded garment latent in a single step.Its features are later supplied to the denoising UNet alongside auxiliary CLIP-based conditioning.
- Outfitting Fusion: Outfitting fusion concatenates garment and human feature maps before self-attention, then retains the human-feature half of the output.This implicitly warps garment features and correlates them with the target body without redundant warping.
- Outfitting Dropout: Outfitting dropout randomly replaces garment latents with an all-zero latent during training, enabling conditional and unconditional denoising.Classifier-free guidance then adjusts garment-control strength using guidance scale s_g.
- Outfitting Dropout: The authors use a 10% outfitting-dropout ratio and report an optimal guidance scale usually around 1.5–2.0.These settings are reported from the ablation study and illustrated in Fig. 4 and Table 1.
4 Experiments
Experiments evaluate OOTDiffusion on VITON-HD and Dress Code using high-resolution virtual try-on datasets with different garment and body coverage. Comparisons are selected according to each dataset’s supported scope.
- Datasets: Experiments use the VITON-HD and Dress Code datasets, both containing high-resolution 1024 × 768 virtual try-on imagery.VITON-HD contains frontal half-body upper-body garments, while Dress Code covers full-body upper-, lower-body garments and dresses.
- Baselines: VITON-HD comparisons include GAN-based VITON-HD, HR-VITON, GP-VTON and LDM-based LaDI-VTON and StableVITON.The comparison spans multiple state-of-the-art methods from both major VTON model families.
- Baselines: Dress Code comparisons include GP-VTON, LaDI-VTON and Paint-by-Example because several baselines do not support the entire dataset.The evaluation covers upper-body garments, lower-body garments and dresses rather than only upper-body cases.
- Evaluation Settings: Evaluation includes paired reconstruction and unpaired virtual try-on settings.The paired setting uses corresponding human–garment images, whereas the unpaired setting uses different garments for the target human.
Evaluation Metrics.
The evaluation uses standardized lower-resolution experiments for fair comparison, with separate metrics for paired reconstruction and unpaired realism and fidelity. Ablations examine outfitting dropout and guidance-scale choices.
- Resolution: All quantitative comparisons run at 512 × 384, although OOTDiffusion also supports 1024 × 768 virtual try-on.This resolution choice is made for fair comparison with previous VTON methods.
- Paired Metrics: LPIPS and SSIM measure generated-image quality in the paired reconstruction setting.These metrics assess restoration of the original target image.
- Unpaired Metrics: FID and KID measure realism and fidelity in the unpaired setting.The metrics are applied when the target human and garment images differ.
- Implementation: Models are initialized from Stable Diffusion v1.5 and fine-tuned with AdamW at a fixed learning rate of 5e-5.Training uses 36,000 iterations on a single NVIDIA A100 GPU, while inference uses 20 UniPC sampling steps on an RTX 4090.
4.3 Ablation Study
Outfitting dropout enables classifier-free guidance over garment features, with guidance improving detail until color distortion emerges at high scales. The study identifies approximately 1.5–2.0 as the optimal guidance range in most cases.
- Outfitting Dropout: Outfitting dropout enables classifier-free guidance with respect to garment features.Without dropout, classifier-free guidance is unsupported; with dropout, garment-feature strength can be adjusted during inference.
- Guidance Scale: Garment details become clearer as the guidance scale increases above 1.0.The effect is observed for models trained with outfitting dropout.
- Guidance Scale: Color distortion begins at guidance scales of 2.5 or higher and becomes extremely significant at 5.0.The flower patterns exhibit especially visible distortion at the highest tested scale.
- Guidance Scale: The optimal guidance scale is approximately 1.5–2.0 in most cases.The quantitative ablation supports consistently using outfitting dropout in OOTDiffusion.
4.4 Experimental Results
On VITON-HD, OOTDiffusion achieves the strongest qualitative try-on effects across varied upper-body garments, improving both human realism and garment-detail preservation over the compared methods.
- VITON-HD: OOTDiffusion consistently achieves the best try-on effects for varied upper-body garments on VITON-HD.The comparison uses test-set examples shown in Fig. 5.
- VITON-HD: GP-VTON often produces unrealistic human bodies or unnatural garment folds.The failures occur across multiple qualitative comparison rows.
- VITON-HD: OOTDiffusion provides the strongest qualitative results among the compared VITON-HD methods.The reported comparison covers upper-body garments and emphasizes overall try-on quality.
Qualitative Results.
Qualitative and supporting evaluation results show that OOTDiffusion preserves garment details and remains stable across garment categories and cross-dataset testing. The VITON-HD comparison also links competing methods’ weaknesses to realism or detail-fidelity failures.
- VITON-HD: LaDI-VTON and StableVITON tend to lose complicated text or garment patterns on VITON-HD.OOTDiffusion preserves most fine-grained garment details while generating realistic images.
- Dress Code: OOTDiffusion remains stable across upper-body garments, lower-body garments, and dresses on Dress Code.Paint-by-Example and LaDI-VTON fail to preserve garment features, while GP-VTON causes body and background distortion.
- Cross-Dataset Evaluation: In cross-dataset testing, OOTDiffusion generates more realistic images and preserves more garment details than the other models.The models are trained on VITON-HD and tested on Dress Code.
- Quantitative Context: On VITON-HD, GAN-based methods retain structural information but lack detail fidelity, while LDM-based methods generate more realistic images.The comparison is organized around SSIM, LPIPS, FID, and KID metrics.
Quantitative Results.
Across Dress Code and cross-dataset evaluation, OOTDiffusion achieves state-of-the-art quantitative performance across garment categories and out-of-distribution conditions, outperforming prior VTON methods in realism and controllability.
- Dress Code: OOTDiffusion outperforms other methods on every metric across all Dress Code garment categories.The categories are upper-body garments, lower-body garments, and dresses.
- Dress Code: OOTDiffusion achieves the best results on the original, more challenging Dress Code test dataset.This holds despite GP-VTON using extra data modifications and only reporting part of its test data.
- Cross-Dataset Evaluation: In cross-dataset evaluation, OOTDiffusion leads on all metrics for the out-of-distribution test dataset.The paper reports this as quantitative evidence of generalization capability.
- Overall Results: Overall, OOTDiffusion significantly outperforms previous VTON methods in realism and controllability across scenarios and conditions.The conclusion synthesizes the quantitative results from Tables 2–4.
4.5 Limitations
OOTDiffusion has two reported limitations: weaker performance for some cross-category try-ons and alteration of details in the original human image.
- Paired human–garment training may fail to produce perfect results for cross-category try-on.Examples include putting a T-shirt on a woman in a long dress or letting a man in pants wear a skirt.
- Collecting datasets with each person wearing different clothes in the same pose is proposed as a partial future remedy.
- Virtual try-on can alter original human-image details such as muscles, watches, or tattoos.The relevant body area is masked and repainted by the diffusion model.
- More practical pre- and post-processing methods are needed to address these altered details.
5 Conclusion
The paper presents OOTDiffusion, an LDM-based architecture that learns garment features and integrates them into denoising through outfitting fusion. Outfitting dropout enables classifier-free garment-feature guidance, while experiments show stronger realism and controllability than other VTON methods.
- OOTDiffusion is an LDM-based architecture for image-based virtual try-on.
- The outfitting UNet learns garment features and incorporates them into the denoising UNet through outfitting fusion with negligible information loss.
- Outfitting dropout enables classifier-free guidance for garment features and enhances controllability.
- Experiments on high-resolution datasets report superiority over other VTON methods in both realism and controllability.
- The reported results indicate broad application prospects for virtual try-on.