Source-linked AI summary
Taming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow
Junhong Gou, Siyu Sun, Jianfu Zhang, Jianlou Si, Chen Qian, Liqing Zhang
TL;DR
Virtual try-on requires preserving garment details while maintaining the target person’s appearance and pose, yet naive inpainting and prior GAN-based systems have limitations. DCI-VTON warps clothes before diffusion-based inpainting and uses the warped garment as a local denoising condition. The method demonstrates superior virtual try-on results on VITON-HD, while small and complex clothing patterns remain difficult to reproduce exactly.
Problem
Virtual try-on methods must preserve garment details and person attributes, but naive diffusion inpainting and prior GAN-based approaches have limited clothing fidelity and high-resolution realism.
Method
DCI-VTON predicts appearance flow to warp clothes, combines them with a clothes-agnostic person image, and refines the result through diffusion with local and global conditions.
Results
Experiments on VITON-HD demonstrate the superiority of the proposed method.
Takeaways & Limitations
Warping before diffusion inpainting helps retain clothing characteristics while producing realistic virtual try-on images.
Takeaways & Limitations
The method cannot accurately preserve every detail in tiny, complex clothing patterns, partly because latent-space inpainting causes information loss.
Abstract
from arXiv · showhide
Virtual try-on is a critical image synthesis task that aims to transfer clothes from one image to another while preserving the details of both humans and clothes. While many existing methods rely on Generative Adversarial Networks (GANs) to achieve this, flaws can still occur, particularly at high resolutions. Recently, the diffusion model has emerged as a promising alternative for generating high-quality images in various applications. However, simply using clothes as a condition for guiding the diffusion model to inpaint is insufficient to maintain the details of the clothes. To overcome this challenge, we propose an exemplar-based inpainting approach that leverages a warping module to guide the diffusion model's generation effectively. The warping module performs initial processing on the clothes, which helps to preserve the local details of the clothes. We then combine the warped clothes with clothes-agnostic person image and add noise as the input of diffusion model. Additionally, the warped clothes is used as local conditions for each denoising process to ensure that the resulting output retains as much detail as possible. Our approach, namely Diffusion-based Conditional Inpainting for Virtual Try-ON (DCI-VTON), effectively utilizes the power of the diffusion model, and the incorporation of the warping module helps to produce high-quality and realistic virtual try-on results. Experimental results on VITON-HD demonstrate the effectiveness and superiority of our method.
1 INTRODUCTION
Virtual try-on must preserve both garment details and the target person’s attributes, but existing GAN-based and naive diffusion-inpainting approaches struggle with realism and clothing fidelity. The proposed framework combines appearance-flow warping with diffusion refinement to improve controllability and synthesis quality.
- Virtual try-on transfers clothing to a target person while preserving garment texture, appearance, and pose.
- Prior methods commonly use GANs and explicit warping, but high-resolution synthesis remains difficult.
- Naive diffusion inpainting can produce high-quality composites but changes garment colors, patterns, and other fine details.
- The proposed framework predicts appearance flow to align clothes, forms a coarse composite, and refines it through diffusion denoising.
- Global conditioning from the original clothes image and local conditioning from warped clothes guide generation toward detailed, controllable try-on results.
2 RELATED WORK
Earlier virtual try-on systems use geometric warping followed by image synthesis, while diffusion models offer an alternative with stronger high-resolution generation capabilities. This work applies diffusion through an inpainting framework that combines coarse warped results with local and global conditioning.
- 2.1 Virtual Try-On: 2D virtual try-on is lighter-weight and more prevalent than 3D approaches, although it is less flexible in views and poses.
- 2.1 Virtual Try-On: Flow-based warping improves garment-body alignment beyond simple TPS deformation, but high-resolution synthesis still challenges GAN-based pipelines.
- 2.2 Diffusion Models: Diffusion models generate realistic and diverse images through iterative denoising, while DDIM accelerates sampling with a deterministic non-Markovian process.
- 2.2 Diffusion Models: Text conditioning is unsuitable for depicting precise garment appearances, motivating spatially guided diffusion for virtual try-on.
- 2.2 Diffusion Models: The method combines a clothes-agnostic person representation with roughly warped clothes before diffusion denoising produces the final image.
3 OUR METHOD
The method combines appearance-flow warping with diffusion-based reconstruction and refinement to preserve garment details while producing realistic person images. Its training uses complementary branches, spatial and global garment conditions, and losses that encourage smooth, pose-aligned warping.
- Problem formulation: The method treats virtual try-on as inpainting while preserving the target person’s attributes and incorporating the clothing image into the masked region.The mask is applied to the upper body and arms, while the unmasked region remains associated with the original person image.
- Warping Network: Appearance-flow warping predicts dense clothes-to-person correspondences and iteratively refines them to handle flexible transformations and substantial misalignment.The final flow provides 2D coordinates indicating which clothing pixels fill each person-image pixel.
- Warping Network: Total-variation and second-order smoothness constraints regularize the high-degree-of-freedom appearance flow, while perceptual and L1 losses encourage pose-fitting warped clothes and masks.The warping objective also uses hyperparameters to control the relative importance of its component losses.
- Diffusion Model: The diffusion model uses reconstruction and refinement branches that are optimized simultaneously with different inputs and objectives.The reconstruction branch generates a person image from global and local conditions, while the refinement branch further processes a rough synthesis.
- Diffusion Model: The warped clothes provide both an initial noisy condition and local denoising guidance, while global clothing features enter the diffusion UNet through cross-attention.The final inpainting redraws the mask rather than exactly copying the initial warp, helping connect clothing and body regions and reduce adverse effects from poor warping.
- Diffusion Model: The refinement branch addresses missing spatial guidance by using the rough synthesis to restore clothing pattern layout and improve transitions where clothes meet the body.This is motivated by cases such as striped garments, where global and local conditions alone may not fully recover spatial arrangement.
4 EXPERIMENTS
Experiments evaluate DCI-VTON on VITON-HD across resolutions, compare it with virtual try-on and diffusion baselines, and ablate its components. The results show strong quantitative and qualitative performance, while the warping and condition branches contribute to clothing-detail preservation.
- 4.1 Experiments Setting: Experiments use VITON-HD, with additional evaluation on DeepFashion and DressCode described for supplementary material.VITON-HD contains 13,679 frontal-view women and top-clothes image pairs, split into 11,647 training and 2,032 test pairs.
- 4.2 Quantitative Evaluation: The method is compared with CP-VTON, PF-AFN, VITON-HD, HR-VTON, and Paint-by-Example across three resolutions.HR-VTON performs strongly across resolutions, while fine-tuned Paint-by-Example is competitive but has difficulty preserving clothes details in paired settings.
- 4.2 Quantitative Evaluation: Our method achieves the best results on various metrics at all three resolutions while retaining original clothing details.The authors attribute this to combining diffusion generation with three conditions guiding the generation process.
- 4.3 Ablation Study: The warping module facilitates synthesis, especially when posture changes substantially and clothes are difficult to place without pre-warping.The ablation also indicates that the method can cope with some poor warping results.
- 4.3 Ablation Study: Removing local or initial conditions reduces performance, with the initial condition having the larger impact; the global condition has the least effect.Visualizations show that the full model better restores plaid textures, colors, stripe arrangements, and meaningful clothing patterns.
- 4.4 Qualitative Evaluation: At 512 × 384, the method produces more realistic results for crossed hands, neckline-hand interactions, transparent materials, and hollow clothing styles.Previous methods struggle with these interactions and often produce texture or pattern differences from the given clothes.
5 CONCLUSION
The paper treats virtual try-on as diffusion-based inpainting and adds appearance-flow warping and multiple conditions to preserve clothing characteristics. Experiments on VITON-HD demonstrate the method’s superiority.
- 5 CONCLUSION: The method uses diffusion-based inpainting with a warping network that predicts appearance flow before synthesis.Warped clothes provide a local condition, while an additional branch helps use coarse synthesis results to guide later generation.
- 5 CONCLUSION: Experiments on VITON-HD demonstrate the superiority of the proposed method.
Supplementary for Taming the Power of Diffusion Models for
The supplementary document adds qualitative comparisons, experiments on additional datasets, comparisons with text-to-image inpainting, and failure-case analysis. It also discusses method limitations.
- Supplementary Scope: The supplement provides additional qualitative comparisons on VITON-HD and reports experiments on DressCode and DeepFashion.
- Supplementary Scope: It compares the proposed approach with a text-to-image-based inpainting method.
- Supplementary Scope: The supplement presents failure cases and discusses the method’s limitations.
A.1 Results on VITON-HD
Supplementary VITON-HD comparisons indicate that the method better restores clothing characteristics and produces more authentic synthesized images than prior methods. The qualitative examples cover stripes, transparent materials, spots, poses, wrinkles, and occlusions.
- A.1 Results on VITON-HD: The method outperforms previous methods in clothing-characteristic restoration and synthesized-image authenticity.Other methods commonly show insufficient clothing restoration and blurry, unrealistic results.
- A.1 Results on VITON-HD: The method preserves stripe layouts near hair, restores transparent and spotted materials, and handles sideways poses more reasonably.
- A.1 Results on VITON-HD: The method more effectively generates stacking wrinkles and maintains clothing patterns when arms occlude the garments.
A.2 Results on DressCode
On DressCode, the method is evaluated across dresses, upper-body, and lower-body subsets using paired and unpaired LPIPS and FID measurements. It achieves the best reported performance across all three subsets and produces realistic, natural try-on results.
- Dataset and setup: DressCode contains 53,795 high-quality image pairs across dresses, upper-body, and lower-body subsets.The subsets contain 29,478 dress pairs, 15,366 upper-body pairs, and 8,951 lower-body pairs; experiments use 512 × 384 resolution.
- Quantitative evaluation: The evaluation compares PF-AFN, HR-VTON, and Ours using LPIPS and FID under paired and unpaired settings.
- Results: Our method achieves the best performance on all three DressCode sub-datasets.
- Qualitative evaluation: Visualizations show realistic and natural try-on results across dresses, upper-body, and lower-body subsets.
A.3 Results on DeepFashion
On DeepFashion, the method is trained at 512 resolution using pose-varied image pairs and evaluated through qualitative visualizations. The results indicate effective clothing-characteristic preservation and realistic composites despite transfers between different poses.
- Task: DeepFashion transfers clothing worn by one person to another person in a different image, making the task more challenging than using template clothes.
- Training setup: Training uses 51,644 retained pairs after filtering 101,966 pose-transfer training pairs for insufficient clothing area.The pairs use two images of the same person wearing the same dress in different poses.
- Training setup: The DeepFashion model is trained at 512 resolution using the same training process as on VITON-HD.
- Qualitative results: Visualizations show that clothing characteristics are effectively preserved when garments transfer between people in different poses.
- Qualitative results: The resulting composites are described as realistic on DeepFashion.
A.4 Comparisons to Text-to-Image Approach
The text-to-image inpainting comparison tests whether a clothing description alone can guide garment transfer. The reported results show that text-only conditioning does not recover the required clothing qualities consistently.
- Comparison setup: The comparison uses a pretrained Stable Diffusion inpainting model conditioned on the clothing’s text description.The upper half of the human body is masked before generation.
- Comparison result: Text-only conditioning cannot recover the required clothing qualities in the reported comparison.
- Comparison result: Generated clothing color, material, and pattern details vary when text is used as the sole condition.
B DISCUSSIONS ON LIMITATIONS
The method does not accurately preserve every detail in relatively tiny and complex clothing patterns. The authors attribute this limitation partly to information loss during latent-space inpainting.
- Pattern fidelity: The method sometimes fails to accurately reproduce relatively tiny and complex clothing patterns.
- Pattern fidelity: Small writing on clothing can be difficult to replicate exactly, although less strict patterns may remain fairly consistent.
- Possible cause: Latent-space inpainting may cause information loss, especially for small and precise targets.