PaperlayerView exact revision on arXiv

Source-linked AI summary

LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-On

Davide Morelli, Alberto Baldrati, Giuseppe Cartella, Marcella Cornia, Marco Bertini, Rita Cucchiara

arXiv:2305.13501v3cs.CVcs.AIcs.MM

TL;DR

Image-based virtual try-on has not fully used diffusion models despite their image-generation quality and stable training. LaDI-VTON combines latent diffusion with EMASC skip connections and garment textual inversion, achieving superior realism and state-of-the-art results on Dress Code and VITON-HD, while remaining limited in faithfully rendering garment logos and text.

  • Problem

    Image-based virtual try-on needs to generate a target model wearing an in-shop garment, but existing methods rely on GANs and have not exploited latent diffusion models.

  • Method

    LaDI-VTON uses latent diffusion conditioned on garment and pose information, enhanced with EMASC skip connections and textual inversion of garment features into CLIP token embeddings.

  • Results

    LaDI-VTON achieves state-of-the-art results and outperforms competitors in realism on both Dress Code and VITON-HD.

  • Takeaways & Limitations

    The results support latent diffusion as an effective direction for image-based virtual try-on and show benefits from preserving garment texture and high-frequency body details.

  • Takeaways & Limitations

    LaDI-VTON cannot always synthesize logos and words on try-on garments with precise and highly comprehensible letters or numbers.

Abstract

from arXiv · show

The rapidly evolving fields of e-commerce and metaverse continue to seek innovative approaches to enhance the consumer experience. At the same time, recent advancements in the development of diffusion models have enabled generative networks to create remarkably realistic images. In this context, image-based virtual try-on, which consists in generating a novel image of a target model wearing a given in-shop garment, has yet to capitalize on the potential of these powerful generative solutions. This work introduces LaDI-VTON, the first Latent Diffusion textual Inversion-enhanced model for the Virtual Try-ON task. The proposed architecture relies on a latent diffusion model extended with a novel additional autoencoder module that exploits learnable skip connections to enhance the generation process preserving the model's characteristics. To effectively maintain the texture and details of the in-shop garment, we propose a textual inversion component that can map the visual features of the garment to the CLIP token embedding space and thus generate a set of pseudo-word token embeddings capable of conditioning the generation process. Experimental results on Dress Code and VITON-HD datasets demonstrate that our approach outperforms the competitors by a consistent margin, achieving a significant milestone for the task. Source code and trained models are publicly available at: https://github.com/miccunifi/ladi-vton.

1 INTRODUCTION

LaDI-VTON brings latent diffusion models to image-based virtual try-on, combining garment- and pose-conditioning with textual inversion and enhanced skip connections. The architecture targets texture preservation, body-detail reconstruction, and realism, and is validated on two benchmarks.

  • Motivation: Existing virtual try-on methods rely on GANs, while latent diffusion offers a more computationally efficient image-generation alternative.Latent diffusion operates in the latent space of a pretrained autoencoder, balancing computational load and image quality.
  • Method: LaDI-VTON conditions latent diffusion on the target garment and human keypoints to preserve the model’s body pose.The method uses cross-attention to condition generation on predicted garment token embeddings.
  • Method: Textual inversion maps visual garment features into CLIP token-embedding space, producing pseudo-word embeddings that retain garment texture during generation.The generated embeddings condition the latent diffusion process through cross-attention.
  • Method: EMASC uses learnable skip connections to transfer encoding details into decoding, addressing losses in high-frequency body details caused by latent compression.The module is designed to improve autoencoder reconstruction capabilities, especially for hands, feet, and faces.
  • Evaluation: LaDI-VTON achieves superior quantitative and qualitative performance over state-of-the-art methods on Dress Code and VITON-HD.The authors report higher realism than GAN-based counterparts across the two virtual try-on benchmarks.

2 RELATED WORK

Prior image-based virtual try-on methods use encoder-decoder, warping, distillation, and refinement strategies, but their generation stages rely on GANs. LaDI-VTON introduces latent diffusion to this task, building on broader diffusion-model and textual-inversion research.

  • Image-Based Virtual Try-On: Image-based virtual try-on transfers a desired garment onto a target subject while preserving human pose and identity.Earlier systems include encoder-decoder generation, refinement networks, and warped clothing obtained through thin-plate-spline transformations.
  • Image-Based Virtual Try-On: Prior work improves try-on generation through warping, semantic layouts, teacher-student distillation, tutor knowledge, and result refinement.These methods differ in how they represent garment geometry, semantics, and generation quality.
  • Diffusion Models: Current virtual try-on methods share GAN-based generation, whereas LaDI-VTON is presented as the first architecture entirely relying on diffusion models.The paper positions this shift as an application of diffusion models to image-based virtual try-on.
  • Diffusion Models: Diffusion models generate images by destroying data structure through forward diffusion and reconstructing it with a learned reverse process.Latent diffusion performs this process in the latent space of a pretrained autoencoder for greater computational efficiency.
  • Diffusion Models: Diffusion models have been applied to synthesis, translation, editing, inpainting, and pose-guided human generation with texture-aware conditioning.Related human-generation methods condition on pose, texture patterns, garment sketches, or textual garment descriptions.
  • Textual Inversion: Textual inversion learns pseudo-words in a text-encoder embedding space from visual concepts for personalized image generation and editing.Related methods extend this idea through fine-tuning, multi-concept composition, prior preservation, or CLIP-space decomposition.

3 PROPOSED METHOD

LaDI-VTON extends Stable Diffusion for virtual try-on by conditioning generation on garment, pose, and spatial inputs. Textual inversion preserves garment appearance, while EMASC skip connections reduce compression-related detail loss during reconstruction.

  • Stable Diffusion adaptation: LaDI-VTON adapts Stable Diffusion’s inpainting pipeline to replace a model’s garment while retaining physical characteristics, pose, and identity.The spatial inputs include an encoded masked image, inpainting information, and garment-related inputs.
  • Textual-Inversion Enhanced Virtual Try-On: Textual inversion maps visual features of the in-shop garment into CLIP token embeddings that condition the diffusion process.The adapter predicts Pseudo-word Token Embeddings in the CLIP token embedding space, rather than linguistically meaningful tokens.
  • Stable Diffusion adaptation: The model adds pose maps and warped garments so generation preserves the model’s pose and fits the target clothing to the body.The pose map represents human keypoints, while the warped garment is aligned with the model’s body pose.
  • Textual-Inversion Enhanced Virtual Try-On: The textual inversion adapter generates garment embeddings in a single forward pass and combines them with prompt embeddings for Stable Diffusion conditioning.The resulting concatenation is passed through the CLIP text encoder and then used through textual cross-attention.
  • Enhanced Mask-Aware Skip Connections: EMASC adds learnable, mask-aware skip connections from masked-image encoder features to corresponding decoder layers to reduce reconstruction information loss.Masking prevents features from the inpainting region from propagating through the skip connections, keeping the process cloth agnostic.
  • Enhanced Mask-Aware Skip Connections: EMASC modules can be added plug-and-play to the standard Stable Diffusion pipeline after training and improve high-frequency human-related reconstructed details.The motivation is that 48× latent compression can lose small details such as hands, feet, and faces.

4 EXPERIMENTAL EVALUATION

Experiments evaluate LaDI-VTON on Dress Code and VITON-HD using paired and unpaired settings, coherence and realism metrics, and comparisons with prior methods.

  • Datasets: Dress Code contains over 53,000 high-resolution clothing-model image pairs across upper-body, lower-body, and dress categories.Its test split contains 5,400 pairs, with 1,800 per category.
  • Datasets: VITON-HD contains 13,679 frontal-view women paired with upper-body garments at 1024 × 768 resolution.The dataset uses 11,647 training pairs and 2,032 test pairs.
  • Evaluation Metrics: LPIPS and SSIM measure coherence against ground truth in paired settings, while FID and KID measure realism in both paired and unpaired settings.The study computes FIDp, KIDp, FIDu, and KIDu for realism.
  • Implementation Details: Training separately covers EMASC modules, textual inversion, and warping before optimization of the enhanced Stable Diffusion pipeline.All experiments generate images at 512 × 384 resolution.
  • Comparison Protocol: The evaluation compares LaDI-VTON with CP-VTON, CP-VTON†, PSAD, VITON-HD, and HR-VITON using common resolutions and metric implementations where possible.Dress Code competitors are retrained at 512 × 384, while VITON-HD comparisons use released code and checkpoints.

CP-VTON† PSAD LaDI-VTON VITON-HD HR-VITON LaDI-VTON

LaDI-VTON is compared with competing virtual try-on systems through qualitative samples, quantitative benchmarks, human preferences, and configuration analyses. Across these evaluations, it shows stronger realism and preserves garment details, while textual inversion and EMASC contribute in specific settings.

  • Quantitative Comparison: 4.14 and 6.48 are LaDI-VTON’s Dress Code FID scores for paired and unpaired settings, respectively, both lower than the best-performing competitor.The method has comparable LPIPS and SSIM coherence to PSAD while outperforming competitors on realism.
  • Quantitative Comparison: LaDI-VTON surpasses all VITON-HD competitors by a large margin on FID and KID.The reported comparison evaluates realism on the VITON-HD benchmark.
  • Qualitative Results: LaDI-VTON produces realistic images while preserving original garment texture, garment details, and target-model physical characteristics.These properties are assessed qualitatively against competitor outputs in Figure 4.
  • Human Evaluation: LaDI-VTON is preferred over each competitor more than 60% of the time in the user study.The study collects around 2,000 evaluations per test from more than 50 users.
  • Configuration Analysis: The textual inversion adapter outperforms empty-text and CLIP-retrieved textual alternatives in the Dress Code configuration analysis.Removing textual inversion helps paired results but does not appreciably affect unpaired results.
  • Configuration Analysis: 16 pseudo-word token embeddings provide the best unpaired FID and KID trade-off between computational load and performance.Using 32 embeddings is slightly better for paired metrics, but increases memory usage.

ORIGINAL w/o EMASC w/ EMASC

The analysis compares Stable Diffusion autoencoder reconstructions with and without EMASC modules. Masked nonlinear EMASC modules provide the strongest reconstruction results across both evaluated datasets.

  • Autoencoder Reconstruction: Figure 5 compares Stable Diffusion autoencoder image reconstructions with and without EMASC modules.The comparison isolates the reconstruction effect of the proposed modules.
  • EMASC Variants: The qualitative reconstruction analysis complements the quantitative comparison of EMASC configurations.Figure 5 presents sample reconstruction results alongside the metric-based analysis.
  • EMASC Variants: Masked nonlinear EMASC modules achieve better results than two EMASC variants on all metrics across Dress Code and VITON-HD.The variants remove final-layer feature masking or use one convolutional layer without nonlinear activation.

5 CONCLUSION

LaDI-VTON combines latent diffusion, textual inversion, and EMASC modules for virtual try-on. The reported results show large realism gains on Dress Code and VITON-HD, especially while retaining garment and human-detail fidelity.

  • Contributions: LaDI-VTON is presented as the first latent diffusion-based approach for virtual try-on.The paper applies textual inversion to condition generation on the in-shop garment.
  • Contributions: Textual inversion increases retention of the input garment’s details and texture during generation.The method uses textual inversion to condition the generation process on garment information.
  • Contributions: EMASC modules reduce autoencoder compression loss and improve perceived high-frequency details in hands, faces, and feet.The modules enhance the inpainting output image quality.
  • Conclusion: LaDI-VTON outperforms competitors by a large margin in realism on both Dress Code and VITON-HD.These are the paper’s two widely used virtual try-on benchmarks.

A CLOTHES WARPING PROCEDURE

The clothes-warping procedure first estimates a geometric transformation to align the in-shop garment with the target body, then refines the warped result with a U-Net.

  • The refinement U-Net implements the mapping C_W = U-Net(Ĉ, P, I_M).
  • A geometric matching module computes a correlation map between garment and cloth-agnostic person representations, then predicts thin-plate spline transformation parameters.The person representation combines the pose map and masked model image.
  • The coarse warped garment, pose map, and masked model image are concatenated as inputs to a U-Net that predicts the refined warped garment.
  • The geometric matching module is trained for 50 epochs with L1 loss, while U-Net refinement uses combined L1 and VGG losses for another 50 epochs.Both phases use learning rate 1e-4 and Adam optimization.

B ADDITIONAL RESULTS

Additional experiments report strong quantitative and qualitative performance across Dress Code categories and datasets, while EMASC improves reconstruction of high-frequency body details.

  • LaDI-VTON outperforms competitors across all three Dress Code categories on FID and KID in paired and unpaired settings.It also performs better than CP-VTON and CP-VTON† on input-adherence metrics such as LPIPS and SSIM.
  • EMASC improves reconstruction of high-frequency details including hands, feet, faces, toes, and eyes on Dress Code and VITON-HD samples.The VITON-HD results also show fewer artifacts and better preservation of eye color and shape.
  • Table 7 presents quantitative results separately for each category of the Dress Code dataset.

C LIMITATIONS

LaDI-VTON does not always faithfully synthesize readable logos and words on try-on garments, despite preserving their general shapes and overall pattern structure.

  • LaDI-VTON struggles to render precise, highly comprehensible letters or numbers in garment logos and text.The authors attribute this limitation to reliance on Stable Diffusion and suggest non-latent diffusion as a possible remedy, with higher computational demands.

ORIGINAL w/o EMASC w/ LINEAR EMASC w/ NON-LINEAR EMASC

The qualitative figures compare original images, Stable Diffusion autoencoder reconstructions with different EMASC variants, and virtual try-on outputs against competitors.

  • AUTOENCODER RECONSTRUCTIONS: Figures 7 and 8 order reconstruction examples as original, no EMASC, linear EMASC, and non-linear EMASC for Dress Code and VITON-HD.
  • The figures are designed to compare reconstruction variants and generated try-on results across the two datasets.
  • VIRTUAL TRY-ON RESULTS: Figures 9 and 10 order virtual try-on examples as original image, in-shop garment, and outputs from competing methods and LaDI-VTON.
Loading 2305.13501v3…