Source-linked AI summary
StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-On
Jeongho Kim, Gyojung Gu, Minho Park, Sunghyun Park, Jaegul Choo
TL;DR
Virtual try-on must preserve clothing details while generating natural images, but prior approaches face limited background generalizability and diffusion-based adaptations have correspondence or utilization gaps. StableVITON addresses this by learning clothing–body correspondence in a pre-trained diffusion model’s latent space with zero cross-attention, attention total variation loss, and augmentation. The method is reported to achieve state-of-the-art qualitative and quantitative performance and promising quality across datasets.
Problem
Virtual try-on needs to preserve clothing details while using pre-trained diffusion models’ generative capability, despite prior limits in background generalizability and semantic-correspondence learning.
Method
StableVITON learns clothing–body semantic correspondence end-to-end in latent space using zero cross-attention blocks, with attention total variation loss and augmentation.
Results
StableVITON shows state-of-the-art performance over existing virtual try-on methods in qualitative and quantitative evaluations, including cross-dataset evaluation.
Takeaways & Limitations
The approach produces high-fidelity try-on images while preserving clothing details and maintaining quality across multiple datasets and arbitrary person images.
Takeaways & Limitations
The model fails to preserve objects occluding the person and accessories such as bracelets and watches because sampling cannot incorporate additional information beyond clothing.
Abstract
from arXiv · showhide
Given a clothing image and a person image, an image-based virtual try-on aims to generate a customized image that appears natural and accurately reflects the characteristics of the clothing image. In this work, we aim to expand the applicability of the pre-trained diffusion model so that it can be utilized independently for the virtual try-on task.The main challenge is to preserve the clothing details while effectively utilizing the robust generative capability of the pre-trained model. In order to tackle these issues, we propose StableVITON, learning the semantic correspondence between the clothing and the human body within the latent space of the pre-trained diffusion model in an end-to-end manner. Our proposed zero cross-attention blocks not only preserve the clothing details by learning the semantic correspondence but also generate high-fidelity images by utilizing the inherent knowledge of the pre-trained model in the warping process. Through our proposed novel attention total variation loss and applying augmentation, we achieve the sharp attention map, resulting in a more precise representation of clothing details. StableVITON outperforms the baselines in qualitative and quantitative evaluation, showing promising quality in arbitrary person images. Our code is available at https://github.com/rlawjdghek/StableVITON.
1. Introduction
StableVITON adapts a pre-trained diffusion model for standalone virtual try-on by learning clothing–body correspondence in latent space. Its attention design and training objectives target clothing-detail preservation and high-fidelity generation, including for complex backgrounds.
- Previous virtual try-on methods commonly combine a clothing warping network with a generator that fuses warped clothing and the person image.
- Limited training diversity constrains prior methods’ ability to preserve complex backgrounds in arbitrary person images.Matching clothing and individuals makes diverse-environment data difficult to collect.
- StableVITON learns clothing–body semantic correspondence within the latent space of a pre-trained diffusion model using zero cross-attention blocks.A clothing encoder conditions the U-Net with intermediate encoder features.
- Latent-space attention performs patch-wise warping by activating tokens corresponding to clothing alignment within the generation region.
- An attention total variation loss and augmentation sharpen attention maps and improve preservation of clothing details.
- StableVITON is reported to achieve state-of-the-art qualitative and quantitative performance, including promising quality across multiple datasets.
2. Related Work
GAN-based virtual try-on methods generally use a two-stage pipeline that first deforms clothing and then fuses it with the target person. Dense-flow estimation is commonly used to obtain precise clothing deformation.
- GAN-based approaches deform clothing to a proposal region before fusing it with the person image using a try-on generator.
- Many prior methods estimate a dense flow map with a trainable network to deform clothing onto the human body.
3. Preliminary
Stable Diffusion is a large-scale latent diffusion model that denoises image representations in an autoencoder’s latent space. Its training objective uses a U-Net denoising network conditioned by a CLIP text encoder.
- Stable Diffusion transforms an input image into a latent feature with a fixed autoencoder encoder before denoising in latent space.
- Stable Diffusion trains with the simplified LDM objective of predicting the noise added during the diffusion process.
- The denoising network is implemented as a U-Net, while a CLIP text encoder conditions it on the text prompt.
4. Method
StableVITON treats virtual try-on as exemplar-based latent inpainting, conditioning a pre-trained U-Net on person and clothing representations. Zero cross-attention learns clothing–body correspondence, while augmentation and attention total variation sharpen alignment.
- Model overview: StableVITON concatenates noisy image, latent agnostic map, agnostic mask, and dense pose as U-Net inputs, while CLIP encodes the clothing exemplar.The U-Net input is expanded to 13 channels, and the original blocks are not updated.
- Zero cross-attention: A spatial encoder conditions U-Net intermediate features through zero cross-attention blocks using latent clothing features.The decoder feature map supplies queries, while spatial-encoder features provide keys and values; a zero-initialized linear layer removes harmful noise.
- Semantic correspondence: Cross-attention aligns clothing key tokens with human-body query tokens, enabling patch-wise warping in latent space.For example, a right-shoulder query should attend more strongly to the corresponding right-shoulder clothing region.
- Semantic correspondence: Augmentation with random shifts forces fine-grained correspondence rather than merely injecting clothing at similar positions.The resulting attention maps show high scores for clothing tokens corresponding to clothing-agnostic regions.
- Attention refinement: Attention total variation loss computes center coordinates from attention-weighted spatial grids and enforces their uniform distribution within the clothing region.The ground-truth clothing mask restricts the loss, producing sharper attention maps that more accurately reflect clothing color.
5. Experiment
StableVITON is evaluated against GAN-, diffusion-, and inpainting-based baselines in single- and cross-dataset settings, with qualitative, quantitative, and human evaluations. It achieves strong clothing-detail preservation, background consistency, and overall performance, while RePaint improves paired-setting results.
- Experimental setup: Experiments compare StableVITON with six virtual try-on baselines and Paint-by-Example across VITON-HD, DressCode, and SHHQ-1.0.The evaluation includes paired, unpaired, single-dataset, and cross-dataset settings.
- Qualitative results: StableVITON generates realistic images while preserving text and clothing textures better than the baselines in single-dataset qualitative comparisons.Compared methods include GAN-based artifacts, lost text, and clothing-person overlap, whereas StableVITON produces high-fidelity results.
- Qualitative results: StableVITON generates high-fidelity cross-dataset images while preserving clothing details, whereas competing methods show artifacts, background failures, or clothing-shape errors.This pattern is reported for evaluations on DressCode and SHHQ-1.0, including comparison with augmented DCI-VTON.
- Quantitative results: StableVITON outperforms all baselines on unpaired FID and KID in single-dataset evaluation.Paired-setting degradation is attributed to autoencoder reconstruction error of the agnostic map; RePaint restores known regions during denoising and yields superiority across all metrics.
- Quantitative results: StableVITON achieves state-of-the-art performance with a large margin on all cross-dataset evaluation metrics.GAN methods struggle with background consistency, while diffusion methods produce plausible images but preserve clothing details less effectively.
- User study: In a 40-participant user study, StableVITON is preferred on most criteria, especially in the cross-dataset setting.LADI-VTON is preferred for fidelity and person attributes on VITON-HD, but receives 35% preference for clothing identity.
6. Conclusion
The paper concludes that StableVITON performs virtual try-on in the latent feature space of a pre-trained diffusion model. Its zero cross-attention block, attention total variation loss, and augmentation support clothing-detail preservation and state-of-the-art performance across cross-dataset evaluations.
- Conclusion: StableVITON uses a pre-trained diffusion model and zero cross-attention to learn clothing-body semantic correspondence in latent feature space.The method is presented as an end-to-end virtual try-on approach.
- Conclusion: Attention total variation loss and augmentation are designed to preserve clothing details more effectively.The conclusion links these components to more precise semantic correspondence learning.
- Conclusion: Extensive experiments, including cross-dataset evaluation, demonstrate state-of-the-art performance and promising quality in real-world settings.The claim is made relative to existing virtual try-on methods.
A. Implementation details
The implementation adopts Stable Diffusion components and evaluates outputs through a user study with criteria covering fidelity, person attributes, clothing identity, and background quality.
- Architecture details: StableVITON adopts the Stable Diffusion v1.4 autoencoder and denoising U-Net, initializing the U-Net with Paint-by-Example weights.The U-Net encoder and decoder each contain 12 blocks with three downsampling and upsampling steps.
- User study: The user study asks participants to compare baseline and StableVITON images using fidelity, person attributes, clothing identity, and cross-dataset background quality.The criteria assess realism, preservation of person characteristics, clothing characteristics, and input-background preservation.
C. Additional Qualitative Results
In a cross-dataset setting, StableVITON is compared with GAN-based and diffusion-based baselines on VITON-HD using models trained on DressCode upper-body data. The baselines exhibit either artifacts around the target person or inadequate clothing-detail preservation.
- GAN-based baselines show significant artifacts around the target person, corresponding to high FID and KID scores.The comparison includes HR-VITON and GP-VTON; the quantitative results are reported in Table 2 of the main paper.
- Diffusion-based baselines produce plausible appearances but fail to preserve clothing details.
D. StableVITON at High Resolution
StableVITON was additionally trained at 1024 × 768 resolution to synthesize higher-fidelity images. Progressive training at this resolution converged faster and yielded clearer facial and clothing details across several datasets.
- StableVITON was further trained at 1024 × 768 resolution for high-fidelity image synthesis.The model received an additional 85k training iterations using the same training settings.
- Progressive training with 1024 × 768 images led to faster convergence than training from scratch.
- At 1024 × 768 resolution, generated images showed clearer preservation of facial or clothing details across VITON-HD, DressCode, SHHQ-1.0, and web-crawled datasets.
E. Limitations & Discussion
StableVITON has limitations in preserving fine facial or clothing details and non-clothing objects, especially occlusions and accessories. Higher resolution improves fine-detail preservation, but occluding objects and accessories remain unresolved.
- Limitations: Fine facial and clothing details remain challenging to preserve even after fine-tuning the autoencoder decoder.The paper notes subtle facial-feature variations, such as differences around the eyes.
- Limitations: StableVITON fails to preserve objects occluding the person and accessories such as bracelets and watches.The authors attribute this to the sampling process lacking additional information beyond clothing for filling masked regions of the agnostic map, leaving preservation as future work.