Source-linked AI summary
Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On
Xu Yang, Changxing Ding, Zhibin Hong, Junhao Huang, Jin Tao, Xiangmin Xu
TL;DR
Image-based virtual try-on must preserve garment texture while fitting garments to a person, but efficient high-fidelity generation remains underexplored. TPD combines texture transfer through diffusion-model self-attention with precise mask prediction, and achieves state-of-the-art performance on VITON and VITON-HD.
Problem
Efficiently generating high-fidelity try-on images remains underexplored, with fine-grained texture preservation and accurate removal of the original garment among the challenges.
Source: §1 · p. 2Method
Results
TPD achieves state-of-the-art performance on VITON and VITON-HD and the best FID, SSIM, and LPIPS scores on both databases.
Source: §1 · p. 3, §4.2 · p. 7, §5 · p. 8Takeaways & Limitations
TPD applies to both garment-to-person and person-to-person virtual try-on tasks.
Source: §5 · p. 8The authors report no obvious negative social impacts from this work.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Image-based virtual try-on is an increasingly important task for online shopping. It aims to synthesize images of a specific person wearing a specified garment. Diffusion model-based approaches have recently become popular, as they are excellent at image synthesis tasks. However, these approaches usually employ additional image encoders and rely on the cross-attention mechanism for texture transfer from the garment to the person image, which affects the try-on's efficiency and fidelity. To address these issues, we propose an Texture-Preserving Diffusion (TPD) model for virtual try-on, which enhances the fidelity of the results and introduces no additional image encoders. Accordingly, we make contributions from two aspects. First, we propose to concatenate the masked person and reference garment images along the spatial dimension and utilize the resulting image as the input for the diffusion model's denoising UNet. This enables the original self-attention layers contained in the diffusion model to achieve efficient and accurate texture transfer. Second, we propose a novel diffusion-based method that predicts a precise inpainting mask based on the person and reference garment images, further enhancing the reliability of the try-on results. In addition, we integrate mask prediction and image synthesis into a single compact model. The experimental results show that our approach can be applied to various try-on tasks, e.g., garment-to-person and person-to-person try-ons, and significantly outperforms state-of-the-art methods on popular VITON, VITON-HD databases.
1. Introduction
Virtual try-on must fit a reference garment to a person while preserving its patterns and texture, but existing approaches face fidelity and efficiency challenges. TPD uses self-attention for texture transfer and predicts a precise inpainting mask; it reports strong performance on VITON and VITON-HD.
1. Introduction
- Virtual try-on must naturally fit a reference garment to a person's body shape and pose while realistically preserving garment patterns and texture.The task aims to replace the clothes in a person image photorealistically.Source: §1 · p. 2
- Existing methods may lose fine-grained garment textures through image encoders, incur extra computational costs, or use rough masks that remove body details.The cited examples of garment-irrelevant details include tattoos and muscle structures.Source: §1 · p. 2
- TPD concatenates the masked person and reference garment spatially so the diffusion UNet's original self-attention transfers garment texture without specialized garment encoders.The combined image lets self-attention capture long-range pixel correlations across multiple feature scales.Source: §1 · p. 2
- Decoupled Mask Prediction iteratively denoises a mask conditioned on the reference garment and unions it with the original garment area to set the inpainting region.The mask adapts to the garment-person pair and is reported to preserve fingers, arms, and tattoos compared with existing methods.Source: §1 · pp. 2–3
- TPD consistently outperforms state-of-the-art methods in realism and coherence on VITON and VITON-HD.The reported results also identify leading FID, SSIM, and LPIPS performance on both databases.Source: §1 · p. 3
2. Related Work
Prior virtual try-on work uses either garment warping or warping-free diffusion methods, which transfer garment texture using image encoders and cross-attention. TPD instead uses the diffusion model's self-attention and a pair-specific inpainting area to address fidelity issues.
2. Related Work
- Warping-based methods warp garments before synthesis, but the resulting artifacts can be difficult to correct in the synthesis stage.Approaches include Thin Plate Splines, flow maps, and landmarks.Source: §2 · p. 3
- Warping-free diffusion methods avoid garment warping but typically transfer textures through image encoders and cross-attention, with encoder choices involving texture loss or added computational cost.CLIP-derived features can be coarse, while specialized encoders or additional U-Nets add model costs.Source: §2 · p. 3
- TPD uses the diffusion model's inherent self-attention for efficient garment-texture transfer and predicts an inpainting area for each person-garment pair.The paper states that it generates high-fidelity try-on images without specialized image encoders and can manage significant pose differences.Source: §2 · p. 3
- Other diffusion approaches condition generation on image features or inject image conditions with an additional network, but ControlNet may struggle with virtual try-on's significant pose differences.The passage describes ControlNet as adequate when input and output structures are aligned.Source: §2 · p. 3
3. Method
TPD uses a diffusion model for virtual try-on, with self-attention-based texture transfer and pair-specific inpainting-mask prediction. Its method also addresses the paired-data setting used by existing databases.
3.1. Preliminary: Diffusion Models
- Stable Diffusion encodes images into latents, denoises them with a UNet, and decodes the recovered latent; inpainting also supplies a mask and masked-image background.The stated training objective predicts the noise added at each diffusion step.Source: §3.1 · p. 4
3.2. Overview
- Because collecting person, target-garment, and target-image triplets is challenging, the overview describes databases using paired person and garment images, with the garment matching the one worn by the person.The passage denotes the person as S and the target try-on image as I*; it describes the paired-data convention as <S, C>.Source: §3.2 · p. 4
3.3. Self-Attention-based Texture Transfer
- TPD concatenates the masked person and reference garment along the spatial dimension, letting the UNet’s self-attention transfer garment texture through pixel correlations in the combined image.The authors report that channel-wise concatenation performs significantly worse because the garment and masked-person pixels are not spatially aligned.Source: §3.3 · p. 5
3.4. Decoupled Mask Prediction
- DMP predicts the new garment area for each person–garment pair and unions it with the original garment area from human parsing to form the inpainting mask.At test time, the method first predicts a coarse try-on image and new-garment area from a bounding-box mask, then unions that area with the original garment area; the resulting mask preserves unrelated background and body-part pixels.Source: §3.4 · p. 5
- Pose and dense-pose inputs provide body information in masked areas, while mask augmentation trains robustness to the coarse and accurate masks used at inference.The training mask is randomly interpolated between the parsed original garment area and a bounding box.Source: §3.4 · p. 5
4. Experiments
Experiments report state-of-the-art quantitative performance on VITON and VITON-HD, qualitative comparisons, and ablations of TPD’s texture-transfer and mask components.
4. Experiments
- Experiments use VITON, VITON-HD, and DeepFashion, with paired and unpaired evaluations using SSIM and LPIPS for paired settings and FID for unpaired settings.DeepFashion experiments address person-to-person try-on; VITON-HD images are resized to 512 × 384 pixels for comparison.Source: §4 · p. 6
4.1. Qualitative Comparisons
- TPD produces higher-quality, higher-fidelity images than other methods in qualitative comparisons on VITON-HD.The authors attribute competing methods’ artifacts on complex textures and challenging poses to warping, texture loss in image encoders, and inaccurate masks.Source: §4.1 · p. 6, Figure 4 · p. 7
4.2. Quantitative Results
- TPD achieves the best FID, SSIM, and LPIPS scores on both VITON and VITON-HD among the compared methods.The authors associate FID with image quality and SSIM and LPIPS with semantic correctness.Source: §4.2 · p. 7
4.3. Ablation Study
- Removing SATT and using CLIP garment features with UNet cross-attention yields notably poorer performance than SATT.The authors link this difference to CLIP’s difficulty capturing fine-grained garment textures.Source: §4.3 · p. 7
- Channel-dimension concatenation produces lower-fidelity results than spatial concatenation, which makes the garment part of the masked person’s context.The authors state that spatial concatenation enables easier and more accurate texture transfer to the masked area.Source: §4.3 · p. 7
- Compared with traditional masks, DMP-predicted masks improve try-on results by preserving body details such as arms and tattoos.The authors attribute this to minimizing the loss of textures irrelevant to the try-on task.Source: §4.3 · p. 8
- Without mask augmentation, artifacts appear because training uses coarse masks while inference uses accurate masks; augmentation removes these artifacts.The comparison is shown in Figure 7.Source: §4.3 · p. 8, Figure 7 · p. 8
5. Conclusion and Limitations
TPD combines spatially concatenated person and garment images with native Stable Diffusion self-attention and predicts precise inpainting masks. It achieves state-of-the-art performance on VITON and VITON-HD, while performance on complex backgrounds remains to be explored.
- TPD uses native self-attention for garment-to-person feature transfer and predicts a precise inpainting mask to preserve background and body-part details.The approach is applicable to garment-to-person and person-to-person try-on tasks.Source: §5 · p. 8
- TPD achieves state-of-the-art performance on VITON and VITON-HD.Source: §5 · p. 8
- Performance on images with more complex backgrounds remains to be explored, since nearly all task databases contain single-color backgrounds.Source: §5 · p. 8