Source-linked AI summary
TryOnDiffusion: A Tale of Two UNets
Luyang Zhu, Dawei Yang, Tyler Zhu, Fitsum Reda, William Chan, Chitwan Saharia, Mohammad Norouzi, Ira Kemelmacher-Shlizerman
TL;DR
Virtual try-on must preserve garment details while adapting garments to major pose, body-shape, and occlusion changes. TryOnDiffusion uses a diffusion-based Parallel-UNet with implicit cross-attention warping and joint warp-and-blend processing. In user studies, it was selected as best for 92.72% of inputs overall and 95.8% of challenging inputs, while limitations remain in preprocessing errors, identity representation, background complexity, and full-body coverage.
Problem
Virtual try-on lacks a reliable way to preserve garment details while warping garments across substantial pose and body-shape changes.
Method
TryOnDiffusion uses Parallel-UNet, whose two communicating UNets perform implicit cross-attention warping and combine garment warping with person blending in one pass.
Results
TryOnDiffusion was selected as best for 92.72% of inputs overall and 95.8% of inputs with more challenging poses in user studies.
Takeaways & Limitations
The method preserves fine garment details and blends garments realistically under heavy occlusions, difficult poses, and complex materials.
Takeaways & Limitations
Performance can be affected by segmentation and pose-estimation errors, incomplete identity representation, complex backgrounds, and the lack of full-body try-on experiments.
Abstract
from arXiv · showhide
Given two images depicting a person and a garment worn by another person, our goal is to generate a visualization of how the garment might look on the input person. A key challenge is to synthesize a photorealistic detail-preserving visualization of the garment, while warping the garment to accommodate a significant body pose and shape change across the subjects. Previous methods either focus on garment detail preservation without effective pose and shape variation, or allow try-on with the desired shape and pose but lack garment details. In this paper, we propose a diffusion-based architecture that unifies two UNets (referred to as Parallel-UNet), which allows us to preserve garment details and warp the garment for significant pose and body change in a single network. The key ideas behind Parallel-UNet include: 1) garment is warped implicitly via a cross attention mechanism, 2) garment warp and person blend happen as part of a unified process as opposed to a sequence of two separate tasks. Experimental results indicate that TryOnDiffusion achieves state-of-the-art performance both qualitatively and quantitatively.
1. Introduction
Virtual try-on must preserve garment detail while accommodating substantial changes in pose, body shape, occlusion, and deformation. TryOnDiffusion addresses this with Parallel-UNet, which implicitly warps garments and combines warping and blending in one diffusion process.
- Virtual try-on seeks to visualize a source garment on a target person from separate person and garment images.
- Large pose and body-shape changes require non-rigid garment warping that creates or flattens wrinkles while preserving patterns and textures.
- TryOnDiffusion handles large occlusions, pose changes, and body-shape changes while preserving garment details at 1024×1024 resolution.The model is trained on 4 Million image pairs showing the same person and garment in different poses.
- Parallel-UNet uses two communicating sub-UNets, cross attention for implicit warping, and a single pass for garment warping and person blending.Cross attention operates over features at multiple pyramid levels to establish long-range correspondence, including under heavy occlusion and extreme pose differences.
- Cascaded diffusion models operate at 128×128 and 256×256 resolutions before super-resolution produces the final 1024×1024 image.
- A user study with 15 non-experts found TryOnDiffusion was selected as best 92.72% of the time against three recent state-of-the-art methods.The study ranked more than 2K distinct random samples.
2. Related Work
Prior virtual try-on methods commonly separate garment warping from blending, while alternative latent-space approaches can lose fine garment details. TryOnDiffusion instead performs implicit warping and blending in one network pass using cross attention.
- Image-based virtual try-on methods generate a target person wearing a source garment and commonly decompose the task into warping and blending stages.Representative approaches use thin-plate-spline warping or neural flow fields.
- Explicit flow estimation and pixel warping can introduce misalignment, especially when occlusions and shape deformations are difficult to model.
- TryOnGAN avoids explicit warping through latent-space optimization but can lose garment details that are poorly represented in the latent space.The loss is evident for patterns, pockets, and special sleeves.
- TryOnDiffusion performs implicit warping without computing flow and combines warping and blending in a single network pass.Experiments report preservation of garment details under heavy occlusions and varied body poses and shapes.
- Standard channel-wise concatenation is not directly suitable for try-on because garment warping involves highly non-linear transformations.Parallel-UNet addresses this by warping the garment implicitly through cross attention.
3. Method
The method preprocesses person and garment images into clothing, segmentation, and pose conditions, then uses cascaded diffusion models with Parallel-UNets to synthesize the try-on result. Cross attention connects garment and person features for implicit correspondence, while the final stage performs super-resolution.
- The model takes person and garment images, with training pairs showing the same person and garment in different poses, while inference uses different people and garments.
- Preprocessing predicts human parsing maps and 2D pose keypoints, segments the garment, and creates a clothing-agnostic RGB image for the person.The clothing-agnostic image removes original clothing while retaining person identity.
- 3.1. Cascaded Diffusion Models for Try-On: The cascaded system uses a 128×128 Parallel-UNet base model, a 256×256 Parallel-UNet super-resolution model, and a final 1024×1024 Efficient-UNet super-resolution model.The final stage has no try-on conditioning and operates as a pure super-resolution model.
- 3.1. Cascaded Diffusion Models for Try-On: Conditional inputs include the clothing-agnostic person image, segmented garment image, pose information, and noise-conditioning augmentation levels.Noise is added to person and garment conditional images before processing, and the augmentation levels are supplied as conditions.
- 3.2. Parallel-UNet: The diffusion network predicts corrupted-image noise from the noisy image, timestep, try-on conditions, and conditional-image noise levels.The predicted noise can be used to recover the ground-truth image from the noisy input.
- 3.2. Parallel-UNet: Parallel-UNet uses cross attention between noisy-person features and garment features to learn long-range correspondence for implicit garment warping.The attention similarity is computed from person queries and garment keys and values across multiple representation subspaces.
- 3.2. Parallel-UNet: Two parallel UNets exchange feature information so garment warping and blending with the target person occur in one pass rather than sequentially.The person-UNet processes aligned clothing-agnostic and noisy images, while the garment-UNet supplies features through cross attention.
- 3.2. Parallel-UNet: Person and garment pose embeddings guide the warp-and-blend process through attention layers in the person-UNet.
4. Experiments
TryOnDiffusion is evaluated on paired and unpaired try-on data using quantitative metrics, user studies, qualitative comparisons, and ablations. The results show stronger garment-detail preservation, fewer warping and blending artifacts, and robust performance under challenging pose and shape changes, while failures remain tied to preprocessing and identity representation.
- Datasets: The authors collect 4 Million paired training samples and evaluate on 6K unpaired test samples plus VITON-HD.Training pairs show the same person wearing the same garment in different poses; test samples use different people, garments, and poses.
- Quantitative comparison: TryOnDiffusion achieves significantly better FID and KID performance than the compared methods on both test datasets.The comparison uses Frechet Inception Distance and Kernel Inception Distance because the test dataset is unpaired.
- User study: 92.72% of random inputs and 95.8% of challenging inputs were judged best for TryOnDiffusion by 15 non-expert raters.The random study covered 2804 pairs, while the challenging study covered 2K pairs selected for more difficult poses.
- Qualitative comparison: Qualitatively, TryOnDiffusion preserves fine garment details and produces realistic wrinkles under difficult poses, while baselines lose texture or introduce warping artifacts.The comparisons include TryOnGAN, SDAFN, and HR-VITON on challenging body poses, shapes, and garment materials.
- Ablation studies: Cross attention preserves garment details better than concatenation under substantial pose and shape changes.This is the first ablation comparison in Figure 5.
- Ablation studies: Sequencing warping and blending creates garment-boundary artifacts, whereas a single network pass blends the person and garment more cleanly.The ablation compares two separate diffusion models with the unified warp-and-blend design.
5. Summary and Future Work
TryOnDiffusion synthesizes try-on images from a person and garment image, using Parallel-UNet to achieve state-of-the-art warping and garment preservation. The architecture trains two UNets in parallel, with cross attention transferring information between them.
- Parallel-UNet trains two UNets in parallel, with one sending information to the other through cross attention.
- The method achieves state-of-the-art results in warping garments to new body shapes and poses while preserving garment details.
- The authors propose extending Parallel-UNet to general image editing and videos in future work.
A.1. Parallel-UNet
The 256×256 Parallel-UNet adds an upsampled 128×128 try-on result, concentrates attention at 16×16 resolution, and uses additional high-resolution blocks. Its implementation uses GroupNorm, swish activations, residual blocks, FiLM conditioning, and over one billion parameters.
- Self attention and cross attention occur only at 16×16 resolution, while extra UNet blocks operate at 256×256 resolution.
- Both Parallel-UNet variants use Group Normalization and swish activations with residual blocks and FiLM conditioning from diffusion and noise-augmentation levels.
- The 128×128 and 256×256 Parallel-UNets contain 1.13B and 1.06B parameters, respectively.
A.2. Training and Inference
TryOnDiffusion uses JAX and trains three diffusion models on 32 TPU-v4 chips for 500K iterations each. Full-pipeline inference runs on four TPU-v4 chips with batch size 4, taking about 18 seconds per batch.
- Three diffusion models are trained for 500K iterations on 32 TPU-v4 chips, requiring around three days per model.
- Full-pipeline inference uses four TPU-v4 chips with batch size 4 and takes around 18 seconds per batch.
- The system is implemented in JAX.
B. Additional Results
Additional experiments compare TryOnDiffusion with prior methods, study ablations and training-set size, and document both broad qualitative performance and failure cases. Results emphasize garment detail preservation, blending, realistic pose-dependent folds, and performance under challenging pose, shape, and occlusion changes.
- Comparisons: Even on simple cases, TryOnDiffusion is reported to preserve garment details and blend the person and garment better than baseline methods.
- Ablations: Ablation studies compare combining warp and blend in one network with sequencing the two tasks, and compare cross attention with concatenation for implicit warping.
- Training-set size: Training-set-size experiments retrain the method on 10K and 100K random pairs from the 4M-pair dataset and evaluate FID and KID on two test sets.
- Failure cases: Failure cases arise from erroneous garment segmentation and clothing-agnostic images that remove identity cues such as tattoos, muscle structure, fine hair, and accessories.
- Qualitative results: The method generates realistic garment wrinkles according to the new body pose rather than copying folds from the garment input.
- Comparisons: On challenging cases, TryOnDiffusion produces more realistic results than the compared baselines under heavy occlusions and extreme pose and shape differences.