Source-linked AI summary
AnyFit: Controllable Virtual Try-on for Any Combination of Attire Across Any Scenario
Yuhan Li, Hao Zhou, Wenxiang Shang, Ran Lin, Xuanhong Chen, Bingbing Ni
TL;DR
Virtual try-on methods still face limitations in fidelity, robustness, and support for garment combinations. AnyFit combines multi-condition garment encoding with model-weight evolution and adaptive mask boosting, and reports surpassing baselines across benchmarks and real-world data. The paper also notes remaining limitations in complex hand generation and text control.
Problem
Existing virtual try-on approaches fall short in high-fidelity, robust fitting across scenarios and do not support freely combining multiple garments.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
AnyFit demonstrates high-fidelity single- and multi-garment try-ons across varied scenes, including harmonized upper and lower styles.
Source: §1 · p. 3, §4.3 · p. 9AnyFit can be unstable when generating complex hands, and its text-control capabilities are initial rather than fully mature.
Source: §5 · p. 10
Abstract
from arXiv · showhide
While image-based virtual try-on has made significant strides, emerging approaches still fall short of delivering high-fidelity and robust fitting images across various scenarios, as their models suffer from issues of ill-fitted garment styles and quality degrading during the training process, not to mention the lack of support for various combinations of attire. Therefore, we first propose a lightweight, scalable, operator known as Hydra Block for attire combinations. This is achieved through a parallel attention mechanism that facilitates the feature injection of multiple garments from conditionally encoded branches into the main network. Secondly, to significantly enhance the model's robustness and expressiveness in real-world scenarios, we evolve its potential across diverse settings by synthesizing the residuals of multiple models, as well as implementing a mask region boost strategy to overcome the instability caused by information leakage in existing models. Equipped with the above design, AnyFit surpasses all baselines on high-resolution benchmarks and real-world data by a large gap, excelling in producing well-fitting garments replete with photorealistic and rich details. Furthermore, AnyFit's impressive performance on high-fidelity virtual try-ons in any scenario from any image, paves a new path for future research within the fashion community.
1 Introduction
AnyFit targets high-fidelity, robust virtual try-on across real-world scenarios and combinations of garments. It combines scalable multi-garment conditioning with model evolution and adaptive mask boosting, and reports strong benchmark and real-world results.
Motivation
- Prior VTON methods struggle with high-fidelity details, robustness across scenes, and freely combining tops and bottoms.The paper describes overly smooth warped garments, difficulties with poses and occlusions, and artifacts in cross-category try-ons.Source: §1 · p. 2
- The desired VTON workflow should scale to customizable combinations of garments and robustly reproduce clothing details, textures, and natural lighting.Examples include mixing tops and bottoms or layering inner and outer garments.Source: §1 · p. 2
Contributions
- AnyFit combines HydraNet garment encoding with MainNet try-on generation, using parallelized attention and Hydra Fusion to inject multiple garment conditions.Each added Hydra Encoding branch increases parameters by 8%; mask regions are adapted to garment aspect ratio during inference.Source: §1 · pp. 2–3
Results
- AnyFit reports outperforming current methods on benchmarks and in-the-wild sets, with multi-garment try-ons showing harmonized upper and lower styles.The reported evaluation spans garment fidelity and robustness in street-captured scenarios.Source: §1 · p. 3
2 Related works
Prior virtual try-on work includes GAN pipelines that warp and fuse garments and diffusion-based approaches that use pretrained models. The paper identifies limitations in detail, fitting, robustness, and support for multiple garments.
GAN-based virtual try-on
- GAN-based methods typically warp clothing to a target shape, then fuse it with a GAN-based try-on generator.HR-VITON jointly handles warping and segmentation, while GP-VTON uses localized warping and global parsing.Source: §2 · p. 3
- Explicit-warping approaches struggle to supplement clothing sides and reproduce natural lighting and shadows.The passage describes these limits across the existing GAN-based approaches it discusses.Source: §2 · p. 3
Diffusion-based virtual try-on
- Diffusion-based methods use pretrained models, but the cited approaches lack multi-garment support and can show artifacts or unstable fits on complex, out-of-distribution images.The passage discusses methods that deform garments before diffusion refinement, inject spatial features, or extract garment features with a parallel U-Net.Source: §2 · p. 3
3 Method
AnyFit treats try-on as image inpainting, with a MainNet generating the try-on and HydraNet encoding garment features. It adds scalable multi-condition feature injection, Prior Model Evolution, and Adaptive Mask Boost to address fitting and robustness challenges.
3.1 Model overview
- AnyFit uses an inpainting MainNet that combines noisy image, encoded masked-person image, and agnostic mask inputs, with a Pose Guider aligning pose features.The backbone is SDXL; the method also uses a HydraNet to encode clothing information.Source: §3.1 · pp. 3–4
3.2 HydraNet for multi-condition VTON
- HydraNet shares a U-Net structure while parallelizing self-attention modules across garment conditions, preserving fine details without replicating full encoding networks.It encodes clothing in one forward pass before MainNet denoising; added-condition overhead is described as minimal.Source: §3.2 · p. 4
- Hydra Fusion injects HydraNet key and value features into MainNet by concatenating them with learnable positional embeddings that distinguish source conditions.The resulting features are combined along the sequence dimension for subsequent attention calculations.Source: §3.2 · p. 4
3.3 Model evolution and mask boost for robust VTON
- Prior Model Evolution initializes the model by merging weights from SDXL-base, SDXL-inpainting, and DreamshaperXL.The search uses CLIP scores on 20 fixed inpainting image-text pairs and selects coefficients α = 1.0 and β = 1.1.Source: §3.3 · p. 5
- Adaptive Mask Boost uses OpenPose-derived masks without human parsing and randomly elongates the mask during training to reduce clothing-shape leakage.During inference, the agnostic area is proportionally extended when the laid-out garment’s bounding-box aspect ratio σ exceeds 1.2.Source: §3.3 · pp. 5–6
- The method’s mask strategy aims to let the model determine garment length autonomously and adapt the mask to garment style at inference.The reported experiments validate appropriate length determination and robust try-on results across clothing categories.Source: §3.3 · p. 6
4 Experiments
Across single- and multi-garment benchmarks, AnyFit significantly surpasses all baselines. The experiments examine its visual results and ablate Hydra Blocks, Prior Model Evolution, and Adaptive Mask Boost.
4.1 Experimental setup
- Experiments use VITON-HD, DressCode, a proprietary e-commerce dataset, and 24,314 DressCode-multiple upper/lower triplets, with paired LPIPS/SSIM and unpaired FID/KID evaluation.Models are trained at 1024 × 768 for 150 epochs; evaluation is conducted at 512 × 384.Source: §4.1 · pp. 6–7
4.2 Qualitative results
- On single-garment try-ons, AnyFit retains intricate patterns and clothing silhouettes; proprietary-dataset training also supports filling inner garments or unzipping clothing based on posture.The latter capability is absent from the VITON-HD-trained version, which lacks such training data.Source: §4.2 · p. 8
- For multi-garment try-ons, AnyFit preserves the upper/lower boundary, while VTON-concat distorts garments and IDM-2Stage produces junction artifacts.AnyFit also handles inference with both garments as flat lays despite training with one flat lay and one person-cropped garment.Source: Figure 5 · p. 7, §4.2 · p. 8
4.3 Quantitative results
- AnyFit significantly surpasses all baselines across VITON-HD, DressCode, the proprietary dataset, and DressCode-multiple, covering single- and multi-garment try-on.The reported comparisons include paired and unpaired settings.Source: Table 1 · p. 8, Table 2 · p. 8, Table 3 · p. 8
4.4 Ablation study
- Hydra Blocks reduce artifacts and garment-feature interference at the junction of upper and lower garments compared with encoding both conditions through one HydraNet.The ablation compares AnyFit with a no-Hydra-Block variant that concatenates both garment conditions into MainNet.Source: §4.4 · p. 9, Table 4 · p. 9
- Prior Model Evolution reduces artifacts and oversaturated colors, improves lighting and shadows, and strengthens outfitting capacity and logo fidelity.The evolved model’s capabilities improve gradually, and the strategy is evaluated qualitatively and quantitatively.Source: Figure 6 · p. 7, §4.4 · p. 9
- Adaptive Mask Boost improves robustness across clothing categories and supports autonomous garment-length determination rather than reliance on masks.The ablation also examines adaptive elongation by manually varying the aspect ratio at inference.Source: Figure 6 · p. 7, §4.4 · p. 9
5 Conclusion
AnyFit combines multi-garment conditioning with model evolution and mask-region boosting, and its experiments on high-resolution benchmarks and real-world data surpass all baselines by a large gap.
5 Conclusion
- AnyFit significantly surpasses all baselines on high-resolution benchmarks and real-world data.The conclusion presents it as a robust VTON pipeline for combinations of attire across scenarios.Source: §5 · p. 9
- The paper identifies risks that synthesized images could be used to breach intellectual-property rights or privacy norms and advocates conscientious use.Source: §5 · p. 9
A.1 The role of text prompt in AnyFit
- Adjusting prompts can vary overall try-on apparel style, but this form of text control is unstable.Source: §A.1 · p. 14
A.2 More ablation study about Adaptive Mask Boost strategy
- Mask augmentation during training removes clothing cues from the mask, while inference adapts mask shape to garment aspect ratio for better-fitting results.AnyFit autonomously determines garment length, and the visual ablation discusses parsing-free mask augmentation.Source: §A.2 · pp. 14–15, Figure 9 · p. 15
A.3 More ablation study about Hydra Block
The Hydra Block improves the stability of multi-garment try-ons, especially where upper and lower garments meet.
A.3 More ablation study about Hydra Block
- The ablation’s no-block version removes the Hydra Encoding Block and Hydra Fusion positional embeddings, using one conditioned HydraNet to encode both garments.In this setup, the single conditioned HydraNet degenerates to ReferenceNet before injecting features into MainNet.Source: §A.3 · pp. 15–16
- Without the Hydra Block, outputs more often show artifacts where top and bottom garments meet; AnyFit with the block produces more stable results.Source: §A.3 · p. 16, Figure 10 · p. 16
- Removing the Hydra Block also lets bottom-garment features affect the top, producing overly long tops or incorrect stripe patterns.Source: §A.3 · p. 16
A.4 More ablation study about Prior Model Evolution
The Prior Model Evolution comparison examines untrained models across CFG settings, while the dataset section describes DressCode-multiple as built from DressCode.
A.4 More ablation study about Prior Model Evolution
- The comparison uses models initialized with synthetic weights without training and explores CFG values beyond the CFG 3.0 used for the main-body comparison.Source: §A.4 · p. 17
- Across a reasonable range of CFG values, Prior Model Evolution yields more realistic fabric textures and more plausible inpainting than outputs without it.Without the strategy, outputs typically have oversaturated colors, lack detailed wrinkles, and show lighting and shadows that do not harmonize with the background.Source: §A.4 · p. 17
B Datasets
- DressCode-multiple is a multi-garment VTON dataset constructed using the publicly available DressCode dataset.Source: B Datasets · p. 17
B.1 The proprietary dataset
The proprietary dataset contains paired garment and person images, while DressCode-multiple provides upper-lower garment triplets assembled from DressCode data.
B.1 The proprietary dataset
- The proprietary dataset has 50,602 training pairs and 2,500 test pairs across 12 clothing categories, with images standardized to 1024x768.Pairs contain a flat-laid garment and a frontal upper-body model image; the collection spans menswear and womenswear and multiple seasons.Source: §B.1 · p. 17
B.2 The DressCode-multiple dataset
- DressCode-multiple is constructed by pairing a flat-lay garment with the opposite garment cropped from a model image wearing the corresponding item.This produces 24,314 publicly available upper-lower triplets from 15,363 upper-body pairs and 8,951 lower-body pairs for training.Source: §B.2 · p. 19
- Testing combines shuffled flat-lay upper and lower garments with test models to create unpaired, real-world multi-garment try-on inputs.Paired testing is not conducted because ground-truth triplets are unavailable.Source: §B.2 · p. 20
- Although the model was not trained on flat-lay upper-and-lower garment triplets, it demonstrates robustness on that test configuration.The authors suggest real triplets for training could improve performance further.Source: §B.2 · p. 20
C Method supplement
The supplement describes a discrete search for model-weight coefficients and reviews the latent-diffusion setup underlying AnyFit.
C.1 Discrete greedy algorithm
- Because the evaluation function is non-differentiable, the method discretizes the coefficient domain [0, 2]^2 and searches for balancing coefficients α and β.The function is empirically monotonic or convex in most regions.Source: §C.1 · p. 20
- The discrete greedy search evaluates neighboring coefficient pairs on a grid and stops when no neighbor improves the evaluation function.It starts at (0.5, 0.5); under a convex or monotonic evaluation function, it is guaranteed to converge to the global optimum in the discrete parameter space.Source: §C.1 · p. 20
C.2 Preliminary
- Stable Diffusion trains a denoising U-Net on noisy encoded images, conditioned on text through cross-attention.The setup uses a VAE encoder and decoder, and the loss minimizes the difference between sampled Gaussian noise and the U-Net's prediction.Source: §C.2 · pp. 20–21
- Table 5 presents full quantitative comparisons on the proprietary dataset.Source: Table 5 · p. 21
D Experimental details
Experimental details describe paired image augmentation and garment-latent dropout; the proprietary-dataset comparison reports a considerable performance lead for AnyFit.
D Experimental details
- Training augmentation independently applies horizontal flips, padding-based resizing, hue shifts from -5 to +5, and contrast changes from 0.8 to 1.2 times the original, each with 50% probability.Each operation is applied to both garment and model images.Source: D Experimental details · p. 21
- Outfitting Dropout randomly replaces the MainNet garment latent with an all-zero latent during training, teaching conditional and unconditional denoising.At inference, a guidance scale controls conditional influence; the default dropout ratio is 10% and guidance scale is 1.2, with dropout limited to the Hydra Fusion Block because applying it elsewhere produced pixel-level artifacts.Source: D Experimental details · p. 21
E.1 Full comparison on the proprietary dataset
- AnyFit exhibits a considerable performance lead in proprietary-dataset comparisons, with paired quantitative results reported in Table 5 and additional visual comparisons in Figure 14.Source: §E.1 · p. 21
E.2 More visual results
The supplement provides additional visual try-on results across public and proprietary datasets, including multi-garment examples.
E.2 More visual results
- Additional visual results cover VITON-HD, DressCode, and proprietary data, with further multi-garment try-ons shown in Figure 10.Source: §E.2 · p. 21
- Figure 14 presents proprietary-dataset comparisons, where AnyFit is described as showing superior garment details and outfit styling.Source: Figure 14 · p. 22
- Figures 15, 16, and 17 show results on VITON-HD, proprietary, and DressCode test data, respectively, using models trained on the corresponding dataset.Source: Figure 15 · p. 23, Figure 16 · p. 24, Figure 17 · p. 25