Source-linked AI summary
Toward Characteristic-Preserving Image-based Virtual Try-On Network
Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, Meng Yang
TL;DR
Image-based virtual try-on must fit in-shop clothes to a person while preserving clothing identity despite large spatial misalignment. CP-VTON uses a fully learnable geometric matching module and a composition-based try-on module, and achieves state-of-the-art performance on the evaluated fashion dataset. The paper also reports failure cases involving old-clothes shape preservation, rare poses, and indistinguishable clothing sides.
Problem
Image-based virtual try-on struggles to handle large spatial misalignment while preserving clothing characteristics such as texture, logos, and text.
Method
CP-VTON learns a thin-plate spline alignment through a Geometric Matching Module and combines warped clothes with a rendered person using a learned composition mask.
Results
CP-VTON achieves state-of-the-art virtual try-on performance qualitatively and quantitatively on the dataset collected by Han et al.
Takeaways & Limitations
The overall pipeline produces high-fidelity try-on results that retain key characteristics of the in-shop clothes.
Takeaways & Limitations
Reported failures involve improperly preserved old-clothes shape information, rare poses, and inner and outer clothing sides that are indistinguishable.
Abstract
from arXiv · showhide
Image-based virtual try-on systems for fitting new in-shop clothes into a person image have attracted increasing research attention, yet is still challenging. A desirable pipeline should not only transform the target clothes into the most fitting shape seamlessly but also preserve well the clothes identity in the generated image, that is, the key characteristics (e.g. texture, logo, embroidery) that depict the original clothes. However, previous image-conditioned generation works fail to meet these critical requirements towards the plausible virtual try-on performance since they fail to handle large spatial misalignment between the input image and target clothes. Prior work explicitly tackled spatial deformation using shape context matching, but failed to preserve clothing details due to its coarse-to-fine strategy. In this work, we propose a new fully-learnable Characteristic-Preserving Virtual Try-On Network(CP-VTON) for addressing all real-world challenges in this task. First, CP-VTON learns a thin-plate spline transformation for transforming the in-shop clothes into fitting the body shape of the target person via a new Geometric Matching Module (GMM) rather than computing correspondences of interest points as prior works did. Second, to alleviate boundary artifacts of warped clothes and make the results more realistic, we employ a Try-On Module that learns a composition mask to integrate the warped clothes and the rendered image to ensure smoothness. Extensive experiments on a fashion dataset demonstrate our CP-VTON achieves the state-of-the-art virtual try-on performance both qualitatively and quantitatively.
1 Introduction
CP-VTON addresses image-based virtual try-on challenges by learning to align clothes with a target person under large spatial deformations while preserving clothing characteristics. Its geometric matching and try-on modules support realistic synthesis, with experiments reporting superior performance.
- Image-based virtual try-on must retain the person’s body shape and pose while smoothly fitting clothes and preserving texture, logos, and text.
- Existing systems struggle with large geometric changes because shape-context matching and appearance merging inadequately preserve clothing details.
- CP-VTON introduces a fully learnable thin-plate spline transformation through a Geometric Matching Module instead of handcrafted shape-context matching.
- The method uses a Try-On Module to dynamically merge rendered and warped results, balancing clothing-detail preservation with image smoothness.
- Experiments on the dataset collected by Han et al. demonstrate significant superior performance for the proposed image-based virtual try-on task.
2 Related Work
Related image-synthesis methods often assume aligned inputs, whereas virtual try-on requires handling substantial spatial deformation between clothes and people. Prior systems use coarse-to-fine generation, pose correspondences, or handcrafted shape-context transformations, motivating CP-VTON’s learnable alignment approach.
- Many image-to-image translation methods assume that conditioned inputs and targets are roughly aligned, limiting their handling of large spatial deformations.
- Methods for unaligned inputs commonly use coarse-to-fine processing, while other approaches rely on deformable skip connections and pose-point correspondences.
- VITON computes a shape-context thin-plate spline transformation between in-shop-clothing and predicted foreground masks within a coarse-to-fine framework.
- Graphics-based virtual try-on methods provide geometric and physical control but require manual labor, specialized devices, or substantial computation.
3 Characteristic-Preserving Virtual Try-On Network
CP-VTON formulates virtual try-on as conditional image generation while preserving the target garment’s characteristics and the person’s body shape and pose. Its learnable geometric alignment and masked fusion address spatial misalignment and boundary artifacts.
- CP-VTON synthesizes a wearer in target clothes while retaining the reference person’s body shape and pose and removing effects of the old clothes.
- The cloth-agnostic person representation preserves pose, body shape, face, and hair while excluding source-clothing effects.It concatenates an 18-channel pose heatmap, one body-shape channel, and three reserved-region channels into a 22-channel map.
- 3.2 Geometric Matching Module: The Geometric Matching Module predicts thin-plate spline parameters from person and clothing features, then warps the target clothes through an end-to-end learnable pipeline.A correlation layer feeds a regressor, and pixel-wise L1 loss supervises the warped clothes against the ground-truth garment.
- 3.3 Try-on Module: The Try-On Module uses UNet to render a person image and predict a composition mask that fuses the rendered image with warped clothes.This combines the detail preservation of direct pasting with the smoother appearance of image translation.
- 3.3 Try-on Module: The Try-On Module is trained with L1 and VGG perceptual losses while regularizing the mask toward selecting warped clothes.The VGG feature maps come from a VGG19 network pretrained on ImageNet.
4 Experiments and Analysis
Experiments use paired fashion images with unpaired validation combinations for testing and evaluate characteristic preservation through human perceptual judgments. Clothing detail richness is stratified with total variation into LARGE and SMALL subsets.
- The dataset contains 16,253 cleaned front-view woman and top-clothing pairs, split into 14,221 training and 2,032 validation pairs.Validation images are rearranged into unpaired pairs for testing.
- Total variation norm measures clothing-image detail richness, focusing evaluation on garments with rich details.The 50 pairs with largest TV norm form LARGE, while the 50 with smallest TV norm form SMALL.
- Figure 3 orders clothing examples from lower to higher TV norm, with each line containing clothes at the same level.
- Pairwise A/B tests ask four Amazon Mechanical Turk workers per job to choose the more realistic and detail-preserving virtual try-on result.The comparison is made between outputs from different methods given a person image and target clothing image.
4.3 Implementation Details
The implementation trains the Geometric Matching Module and Try-On Module with specified optimization settings and convolutional architectures.
- Training Setup: Both modules are trained for 200K steps with batch size 4 using Adam and a learning-rate schedule that decays after 100K steps.The learning rate starts at 0.0001 and linearly decays to zero during the remaining steps.
- Training Setup: Inputs and outputs are resized to 256 × 192, while λL1 and λvgg are set to 1 and λmask is 1 when composition masking is used.
- Geometric Matching Module: The Geometric Matching Module uses convolutional feature extractors and predicts 50 TPS anchor-point offsets through its fully connected regression layer.The 50 outputs represent x- and y-coordinate offsets for a 5×5 grid of TPS anchor points.
- Try-On Module: The Try-On Module uses a 12-layer UNet with six down-sampling and six up-sampling convolutional layers.
4.4 Comparison of Warping Results
CP-VTON improves clothing-detail preservation and uses learned matching and composition to address the weaknesses of prior virtual try-on warping and synthesis.
- Comparison of Warping Results: GMM takes learned person and clothing representations to estimate TPS parameters and is more robust to large shape differences than SCMM.SCMM relies on boundary shape-context correspondences and can overly shrink long sleeves into a thin band.
- Comparison of Warping Results: 0.06s on GPU and 0.52s on CPU are GMM processing times per sample pair, compared with 2.01s on CPU for SCMM.
- Comparison of Warping Results: 49.5% for LARGE and 42.0% for SMALL are the user-study realism ratings for GMM-synthesized warps, indicating comparability with SCMM.
- Comparison of Try-on Results: CP-VTON preserves sharp, intact texture, logo, and embroidery details on richly patterned clothes, while VITON produces blurry results.For simpler clothing patterns, the two pipelines perform roughly the same qualitatively.
- Comparison of Try-on Results: VITON’s coarse-to-fine strategy can bias its composition mask toward the rendered person instead of warped clothes despite mask regularization.The resulting ragged masks are presented as evidence of this behavior.
- Comparison of Try-on Results: CP-VTON simultaneously learns a UNet-rendered person image and composition mask, allowing warped clothes to dominate early and rendered regions to be exposed adaptively later.The process supports seamless fusion around regions such as hair and arms.
4.6 Discussion and Ablation Studies
The ablation study compares CP-VTON variants that remove composition masking or its L1 regularization.
- Discussion and Ablation Studies: The study evaluates CP-VTON without mask composition and CP-VTON without mask L1 regularization.
Effects of composition mask
Composition masking and mask L1 regularization are both necessary for handling spatial misalignment and preserving realistic clothing details.
- Effects of composition mask: Without mask composition, CP-VTON loses characteristic details and produces blurry results even when warped clothes are roughly aligned.
- Effects of composition mask: Without mask L1 regularization, the composition mask tends to select the rendered person and produces blurry results.This variant performs as poorly as the version without mask composition.
- Effects of composition mask: Minor misalignment creates large perceptual disagreement between warped clothes and the ground truth, explaining the need for mask regularization.
Robustness against minor misalignment
Under simulated minor misalignment, CP-VTON remains more robust than VITON stage II and CP-VTON(w/o mask), preserving detailed clothing characteristics as perturbations increase.
- Experimental setup: The controlled experiment perturbs otherwise perfectly aligned clothes within a random range of N pixels to simulate imperfect alignment.The compared systems are VITON stage II, CP-VTON(w/o mask), and the full CP-VTON pipeline.
- Results: CP-VTON decays more slowly than the other methods under increasing perturbation.Figure 8 compares robustness against random shifts within radius N.
- Failure analysis: Figure 9 presents failure cases of CP-VTON alongside the robustness comparison.The cited figure passage does not specify the individual failure causes.
- Results: VITON stage II and CP-VTON(w/o mask) both experience rapid performance degradation as perturbation increases.The comparison uses increasing perturbation magnitudes after training with perfectly aligned inputs.
- Results: The full CP-VTON pipeline remains robust to perturbation while preserving detailed clothing characteristics.This robustness is contrasted directly with the faster decay of VITON stage II and CP-VTON(w/o mask).
Failure cases
CP-VTON has failure cases associated with incorrect old-clothing shape preservation, rare poses, and ambiguity between a garment’s inner and outer sides.
- Failure cases: Three reported failure causes are improperly preserved old-clothing shape information, rare poses, and indistinguishable garment sides.These cases are shown in Figure 9.
5 Conclusions
CP-VTON is a fully learnable virtual try-on pipeline combining geometric alignment with a learned merging strategy to preserve clothing characteristics. Experiments report high-fidelity results and state-of-the-art qualitative and quantitative performance on the cited fashion dataset.
- Conclusion: CP-VTON combines a geometric matching module with a try-on module using a new merging strategy.The pipeline is fully learnable and targets characteristic-preserving image generation.
- Conclusion: The geometric matching module aligns in-shop clothes with the target person’s body despite large spatial displacement.The try-on module then operates on aligned clothes to preserve detailed characteristics.
- Conclusion: The overall pipeline produces high-fidelity virtual try-on results that retain key characteristics of the in-shop clothes.The reported retained characteristics include the clothing identity conveyed by its details.
- Conclusion: CP-VTON achieves state-of-the-art qualitative and quantitative performance on the dataset collected by Han et al.The conclusion reports this result at the dataset level without providing specific metric values here.