Source-linked AI summary
CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models
Zheng Chong, Xiao Dong, Haoxiang Li, Shiyue Zhang, Wenqing Zhang, Xujie Zhang, Hanqing Zhao, Dongmei Jiang, Xiaodan Liang
TL;DR
Diffusion-based virtual try-on can require extra encoders, many training parameters, and complex preprocessing. CatVTON concatenates garment and person images for a streamlined diffusion model; the authors report improved quality and efficiency, including reduced memory use.
Problem
Diffusion-based try-on methods often require additional encoding modules, many training parameters, and complex preprocessing, increasing training and inference burdens.
Source: §1 · p. 2Method
CatVTON spatially concatenates garment and person images for a VAE and simplified denoising UNet, training only self-attention modules and omitting text-related components.
Source: §1 · pp. 2–3, §3.1 · p. 5, §3.2 · p. 5Results
CatVTON reports superior qualitative and quantitative results to state-of-the-art methods, with 899.06M total parameters and 49.57M trainable parameters.
Source: §1 · p. 3, §5 · pp. 9–10Takeaways & Limitations
The model requires no pose estimation, human parsing, or text-based inputs and reduces memory requirements while maintaining a compact architecture.
Source: §5 · p. 10VAE decoding may lose detail or shift colors, and try-on quality depends on mask accuracy; retained Stable Diffusion v1.5 weights may also inherit NSFW biases.
Source: §A.5 · p. 16
Abstract
from arXiv · showhide
Virtual try-on methods based on diffusion models achieve realistic effects but often require additional encoding modules, a large number of training parameters, and complex preprocessing, which increases the burden on training and inference. In this work, we re-evaluate the necessity of additional modules and analyze how to improve training efficiency and reduce redundant steps in the inference process. Based on these insights, we propose CatVTON, a simple and efficient virtual try-on diffusion model that transfers in-shop or worn garments of arbitrary categories to target individuals by concatenating them along spatial dimensions as inputs of the diffusion model. The efficiency of CatVTON is reflected in three aspects: (1) Lightweight network. CatVTON consists only of a VAE and a simplified denoising UNet, removing redundant image and text encoders as well as cross-attentions, and includes just 899.06M parameters. (2) Parameter-efficient training. Through experimental analysis, we identify self-attention modules as crucial for adapting pre-trained diffusion models to the virtual try-on task, enabling high-quality results with only 49.57M training parameters. (3) Simplified inference. CatVTON eliminates unnecessary preprocessing, such as pose estimation, human parsing, and captioning, requiring only a person image and garment reference to guide the virtual try-on process, reducing over 49% memory usage compared to other diffusion-based methods. Extensive experiments demonstrate that CatVTON achieves superior qualitative and quantitative results compared to baseline methods and demonstrates strong generalization performance in in-the-wild scenarios, despite being trained solely on public datasets with 73K samples.
1 INTRODUCTION
Diffusion-based try-on can produce natural results, but existing designs often add computationally costly modules. CatVTON instead integrates garment and person images in a streamlined diffusion model and reports stronger quality and efficiency than baseline methods.
1 INTRODUCTION
- CatVTON achieves lower FID on VITON-HD with fewer total and trainable parameters and less memory, while outperforming state-of-the-art baselines qualitatively and quantitatively.The paper also reports strong performance in in-the-wild scenarios.Source: Figure 2 · p. 2, §1 · p. 3
- Diffusion-based try-on methods achieve natural results but often add ReferenceNet or Dual-UNet structures and image encoders, increasing training and inference burdens.Earlier warping-based methods can produce unnatural fits and struggle with complex poses.Source: §1 · p. 2
- CatVTON asks whether garment and person images can be processed together by one UNet in a shared latent space, avoiding redundant encoders.Pre-trained image encoders may not preserve the fine details important for try-on.Source: §1 · p. 2
- The model uses a VAE and simplified denoising UNet, removes text encoders and cross-attention, and trains only self-attention modules with 49.57M parameters.Its total model size is 899.06M parameters; inference needs no pose estimation, human parsing, or captioning.Source: §1 · p. 3
2 RELATED WORK
Related work spans subject-driven generation and image-based virtual try-on, with prior methods often trading detail preservation or architectural simplicity. CatVTON is presented against this backdrop as a streamlined diffusion-based approach.
2.1 SUBJECT-DRIVEN IMAGE GENERATION
- Subject-driven generation integrates a target subject into new scenes or perspectives while maintaining consistency, but prior methods can require complex architectures or many trainable parameters.Encoder-based approaches may fall short in preserving details, while other methods add specialized modules or progressive models.Source: §2.1 · p. 3
2.2 IMAGE-BASED VIRTUAL TRY-ON
- Image-based try-on includes warping-and-fusion pipelines and diffusion methods; diffusion avoids warping limits, but many approaches still use complex networks, extra trainable modules, or inference conditions.CatVTON's overview instead uses spatial concatenation and omits pose and parsing conditions.Source: §2.2 · pp. 3–4, Figure 3 · p. 4
3 METHODS
CatVTON streamlines diffusion-based virtual try-on by removing redundant components, focusing training on key modules, and simplifying inference. Its method combines spatially concatenated garment and person representations with a lightweight UNet and a mask-free training pipeline.
3 METHODS
- The methods section frames CatVTON's goal as streamlining diffusion try-on by eliminating redundant components, focusing on key modules, and simplifying preprocessing.Source: §3 · p. 4
3.1 LIGHTWEIGHT NETWORK
- CatVTON concatenates person and garment images spatially as inputs to the original denoising UNet, avoiding additional image encoders and ReferenceNet modules.The method retains a VAE and simplified UNet; removing the text encoder and cross-attention yields a model with 899.06M parameters, over 44% fewer than other diffusion-based methods.Source: §3.1 · pp. 4–5
3.2 PARAMETER-EFFICIENT TRAINING
- Training only self-attention layers uses 49.57M trainable parameters, while full-UNet, transformer-block, and self-attention variants all produce satisfactory results without substantial visual-quality or metric differences.For mask-free training, the trained mask-based model generates synthetic person images from random person-garment pairs, which are combined with original person and garment images as training data.Source: §3.2 · p. 5, Figure 4 · p. 5
3.3 SIMPLIFIED INFERENCE
- Inference requires only a person image and garment reference for the mask-free model, while the mask-based model additionally takes a binary cloth-agnostic mask.The input images are VAE-encoded and concatenated spatially; the denoising UNet iteratively predicts a latent that is split to extract the person part and decoded into the output image.Source: §3.3 · pp. 5–6
4 EXPERIMENTS
CatVTON is evaluated on three public datasets using paired and unpaired metrics, qualitative comparisons, efficiency measurements, and ablations. It outperforms compared methods across metrics while using less memory and faster inference, and the ablations examine training modules, classifier-free guidance, and DREAM strength.
4.1 DATASETS
- The experiments use VITON-HD, DressCode, and DeepFashion, including upper, lower, dress, and garment-transfer data.The selected training/testing pairs are 11,647/2,032 for VITON-HD, 48,392/5,400 for DressCode, and 13,098/1,896 for DeepFashion garment transfer.Source: §4.1 · pp. 6–7
4.2 IMPLEMENTATION DETAILS
- Training uses Stable Diffusion v1.5 variants, with separate 16,000-step models and multi-task models trained for 48,000 steps.The multi-task setup uses approximately 73K image pairs; experiments use 512×384 or 1024×768 resolution.Source: §4.2 · p. 7
4.3 METRICS
- Paired evaluation uses SSIM, LPIPS, FID, and KID, while unpaired evaluation uses FID and KID.The metrics compare synthesized images with authentic images or compare synthesized and real-sample distributions, depending on the setting.Source: §4.3 · p. 7
4.4 QUALITATIVE COMPARISON
- Qualitative comparisons show stronger handling of complex garment patterns, occlusions, varied garment types, and in-the-wild scenes.Examples include preserving garment text and patterns and reproducing wrinkles, lighting, and shadows.Source: §4.4 · pp. 7–8, Figure 6 · p. 7
4.5 QUANTITATIVE COMPARISON
- CatVTON outperformed all compared methods across paired and unpaired metrics, with lower memory use and faster inference than other models.The comparisons cover VITON-HD and DressCode; inference efficiency is evaluated at 512×384 and 1024×768 resolutions.Source: §4.5 · p. 8, Table 1 · p. 8, Table 2 · p. 8
4.6 ABLATION STUDIES
- Training only self-attention balances performance and efficiency; adding text conditions decreases performance, while more trainable weights raise memory use and slow training.The Table 4 caption reports that extra text conditions harm performance and that more trainable modules slightly affect performance.Source: §4.6 · pp. 8–9, Table 4 · p. 9
- CFG strengths from 2.5 to 3.5 produce the most natural results; strengths above 3.5 cause color distortion and high-frequency noise.The other experiments use CFG strength 2.5.Source: §4.6 · p. 9
- DREAM strength λ trades perceptual quality against distortion, and λ = 10 is selected to balance naturalness and detail fidelity.The ablation varies λ on VITON-HD; increasing λ raises SSIM while FID and LPIPS first improve and then degrade.Source: §4.6 · p. 9
5 CONCLUSION
CatVTON combines a lightweight architecture and self-attention-focused training with streamlined inference. The paper reports superior qualitative and quantitative results, while its DREAM ablation shows a trade-off between perceptual quality and distortion.
5 CONCLUSION
- On VITON-HD, increasing DREAM λ trades reduced distortion against perceptual quality, and extreme λ values can reduce naturalness.The reported visual ablation associates small λ with overly smooth images and large λ with excessive high-frequency noise.Source: Table 5 · p. 9, Figure 7 · p. 10
- CatVTON uses 899.06M total parameters and 49.57M trainable parameters by focusing training on self-attention layers.The model removes unnecessary text-related modules to reduce model complexity while preserving high-quality try-on performance.Source: §5 · p. 9
- CatVTON removes the need for pose estimation, human parsing, and text-based inputs, reducing memory requirements and enhancing inference speed.The conclusion also reports superior qualitative and quantitative results compared with state-of-the-art methods.Source: §5 · p. 10
A.1 PRELIMINARY
Latent Diffusion Models reduce computation by operating in a VAE-defined latent space, while DREAM adjusts the training target using the model’s noise prediction.
A.1 PRELIMINARY
- LDMs map images into a VAE-defined lower-dimensional latent space, reducing computation while retaining high-quality image generation.A denoising UNet operates on the latents, which the VAE decoder maps back to images.Source: §A.1 · p. 14
- DREAM combines predicted noise with the original noise to obtain an adjusted noise estimate used to compute the adjusted latent.Its strength is controlled by λ, and the training objective is expressed in Equation 8.Source: §A.1 · p. 14
- DREAM is designed to balance distortion minimization and image quality, but its additional forward pass slightly slows training.Source: §A.1 · p. 14
A.2.1 MASK-FREE DATASET
The mask-free training data is constructed from public datasets using generated pseudo-data and checks intended to prevent information leakage.
A.2.1 MASK-FREE DATASET
- Mask-free paired data is generated from VITON-HD, DressCode, and DeepFashion using pseudo-data and cross-validated masks for uppers, lowers, and dresses.Parsing models and DensePose cross-validate mask regions; convex-hull and pooling operations help prevent information leakage outside them.Source: §A.2.1 · p. 14
A.2.2 HARDWARE ENVIRONMENT
Experiments ran on a Linux x86 server equipped with eight NVIDIA A800 GPUs.
A.2.2 HARDWARE ENVIRONMENT
- Experiments used a Linux x86 server with an Intel Xeon CPU and 8 NVIDIA A800 GPUs, each with 80GB VRAM.Source: §A.2.2 · p. 14
A.2.3 SOFTWARE ENVIRONMENT
The appendix reports the software environment, tests spatial concatenation directions and visual results on two datasets, evaluates transferability, and discusses limitations.
A.2.3 SOFTWARE ENVIRONMENT
- The implementation used PyTorch 2.1.2 and a diffusion model modified from Hugging Face’s Diffusers library.Source: §A.2.3 · p. 14
A.2.4 CONCATENATION ALONG X/Y-AXIS
- Training with either x-axis or y-axis spatial concatenation produced normal results, and a model trained with x-axis concatenation also worked with y-axis concatenation at inference.Source: §A.2.4 · p. 14
A.3.1 VTION-HD DATASET
- On VITON-HD, CatVTON preserved garment text, patterns, and logos, while producing natural sleeves and collars and maintaining reasonable scale.Figure 8 provides additional visual comparisons.Source: Figure 8 · p. 15, §A.3.1 · p. 15
A.4 TRANSFERABILITY
- CatVTON’s DiT-based version on VITON-HD outperformed most existing methods, although DiT converged more slowly than UNet and had not fully fit the data.Source: Table 6 · p. 16, §A.4 · p. 16
A.3.2 DRESSCODE DATASET
- On DressCode, CatVTON better matched clothing lengths and generated more coherent patterns in cases such as arm occlusion.Figure 9 presents the visual comparisons.Source: Figure 9 · p. 16, §A.3.2 · p. 16
A.5 LIMITATIONS & SOCIAL IMPACTS
- The method is limited by VAE decoding artifacts, mask accuracy, and possible inherited biases from Stable Diffusion v1.5’s NSFW training data.The authors note detail loss and color discrepancies particularly at 512×384 resolution; inaccurate masks can significantly degrade results.Source: §A.5 · p. 16