Source-linked AI summary
GP-VTON: Towards General Purpose Virtual Try-on via Collaborative Local-Flow Global-Parsing Learning
Zhenyu Xie, Zaiyu Huang, Xin Dong, Fuwei Zhao, Haoye Dong, Xijin Zhang, Feida Zhu, Xiaodan Liang
TL;DR
Image-based virtual try-on methods can fail to preserve garment-part semantics on challenging inputs and can distort textures when fitting preserved-region boundaries. GP-VTON addresses these issues with local-flow global-parsing warping and dynamic gradient truncation, and reports superiority over existing methods on two high-resolution benchmarks.
Problem
Existing VTON methods struggle with semantic-correct garment deformation for challenging inputs and texture distortion caused by preserved-region boundary constraints.
Method
GP-VTON combines the Local-Flow Global-Parsing module, which warps garment parts individually, with Dynamic Gradient Truncation for the warping network.
Results
GP-VTON reports superiority over existing state-of-the-art methods on two challenging high-resolution virtual try-on benchmarks.
Takeaways & Limitations
The framework generates semantic-correct, photo-realistic try-on results for challenging scenarios and can be extended to multi-category try-on.
Takeaways & Limitations
Gradient truncation is inappropriate for tucking-out training data because inaccurate preserved-region warps are not penalized, producing stretched results.
Abstract
from arXiv · showhide
Image-based Virtual Try-ON aims to transfer an in-shop garment onto a specific person. Existing methods employ a global warping module to model the anisotropic deformation for different garment parts, which fails to preserve the semantic information of different parts when receiving challenging inputs (e.g, intricate human poses, difficult garments). Moreover, most of them directly warp the input garment to align with the boundary of the preserved region, which usually requires texture squeezing to meet the boundary shape constraint and thus leads to texture distortion. The above inferior performance hinders existing methods from real-world applications. To address these problems and take a step towards real-world virtual try-on, we propose a General-Purpose Virtual Try-ON framework, named GP-VTON, by developing an innovative Local-Flow Global-Parsing (LFGP) warping module and a Dynamic Gradient Truncation (DGT) training strategy. Specifically, compared with the previous global warping mechanism, LFGP employs local flows to warp garments parts individually, and assembles the local warped results via the global garment parsing, resulting in reasonable warped parts and a semantic-correct intact garment even with challenging inputs.On the other hand, our DGT training strategy dynamically truncates the gradient in the overlap area and the warped garment is no more required to meet the boundary constraint, which effectively avoids the texture squeezing problem. Furthermore, our GP-VTON can be easily extended to multi-category scenario and jointly trained by using data from different garment categories. Extensive experiments on two high-resolution benchmarks demonstrate our superiority over the existing state-of-the-art methods.
1. Introduction
GP-VTON targets real-world image-based virtual try-on by addressing semantic failures on challenging inputs and texture distortion at preserved-region boundaries. It combines local-part warping with global parsing and dynamic gradient truncation to produce realistic, high-fidelity garments across diverse scenarios.
- Motivation: Existing image-based VTON methods struggle with intricate poses, difficult garments, preserved-region boundary constraints, and limited garment-category coverage.These issues can cause indistinguishable or adhesive garment parts, distorted textures, and restricted application scenarios.
- Results and scope: GP-VTON is a unified framework for photo-realistic try-on in diverse scenarios and is supported by experiments on two challenging high-resolution benchmarks.The authors report superiority over existing state-of-the-art methods and describe extension to multi-category try-on.
- Proposed framework: LFGP estimates local flows and assembles the warped parts using global garment parsing to form an intact, semantic-correct garment.The framework then supplies the intact warped garment and person-related conditions to a generator for the final try-on image.
- Motivation: Global deformation fields cannot represent complicated non-rigid transformations that differ across garment parts.GP-VTON instead learns diverse local deformation fields so parts can be warped individually and semantic information preserved.
- Proposed framework: DGT truncates gradients in the preserved-region overlap so the warped garment need not strictly match the preserved boundary, reducing texture squeezing and distortion.The strategy addresses the boundary constraint during warping-network training rather than forcing the garment to fit the boundary shape.
2. Related Work
Image-based virtual try-on methods commonly deform an in-shop garment before synthesizing the try-on image. Related approaches use sparse control points with TPS transformations or dense appearance-flow correspondence.
- Image-based Virtual Try-on: Most image-based VTON methods use a two-stage framework that deforms the garment and then synthesizes the try-on result with the reference person.Garment deformation quality directly affects the realism of the generated result.
- Image-based Virtual Try-on: TPS-based methods regress sparse garment control points and fit a Thin Plate Splines transformation for deformation.Their deformation is based on sparse correspondence between control points.
- Image-based Virtual Try-on: Flow-based methods estimate an appearance-flow map that predicts the source-image location corresponding to each target-image pixel.This directly models dense correspondence for non-rigid garment deformation.
3. Methodology
GP-VTON combines Local-Flow Global-Parsing warping with Dynamic Gradient Truncation to produce semantic-correct, distortion-free garments across challenging and multi-category try-on scenarios.
- Local-Flow Global-Parsing: GP-VTON first deforms local garment parts with separate flows, then assembles them into an intact warped garment using global garment parsing.The partition separates left/right sleeves and torso, while parsing assigns each pixel to a particular warped part to eliminate overlap artifacts.
- Local-Flow Global-Parsing: LFGP uses cascaded Coarse/Fine Flow Blocks and Garment Parsing Blocks to estimate local flows and global garment parsing.The module extracts multi-scale person and garment features before progressively refining local deformation and parsing.
- Local-Flow Global-Parsing: Global garment parsing guides assembly because directly combining overlapping warped parts produces artifacts, whereas pixel-wise part selection eliminates them.The parsing labels include background, sleeves, torso, arms, and neck, and its guidance keeps garment shape consistent with local warped parts.
- Dynamic Gradient Truncation: DGT dynamically truncates gradients according to wearing style, avoiding boundary-driven texture squeezing while retaining penalties where truncation would cause stretched results.It truncates gradients for tucking-in samples, removes truncation for tucking-out samples, and randomly applies it when Rstyle is between 0.9 and 0.95.
- Multi-category extension: GP-VTON extends to multiple garment categories through a unified partition mechanism for upper garments, dresses, and lower garments.Upper garments and dresses share a left/right-part plus torso partition, while the framework is jointly trained with data from various categories.
4. Experiments
Experiments on VITON-HD and DressCode evaluate GP-VTON qualitatively, quantitatively, and through ablations. Results show stronger semantic correctness, visual quality, and texture preservation, while local flows, global parsing, and DGT each address distinct warping problems.
- Experimental Setup: Experiments use 512 × 384 images from VITON-HD and DressCode, including upper, lower, and dress garment categories.VITON-HD provides 13,679 image pairs, while DressCode includes 48,392 training and 5,400 testing pairs across categories.
- Qualitative Results: GP-VTON produces more semantic-correct and visually realistic try-on results than baselines on intricate poses and complex garments.Baselines show damaged sleeves, blended pant legs, indistinguishable sleeves, adhesive artifacts, and distorted textures around preserved regions.
- Quantitative Results: GP-VTON surpasses baselines on all reported VITON-HD metrics and achieves the finest score on all DressCode-Upper metrics.On DressCode-Lower and DressCode-Dresses, it wins most metrics and remains comparable to SDAFN on FID.
- Ablation Study: The ablation evaluates warped results with SSIM, LPIPS, mIoU, and Rdiff, where lower Rdiff indicates better preservation of the garment height-width ratio.Rdiff measures the height-width-ratio difference between warped and in-shop garments.
- Ablation Study: Local flows improve SSIM, LPIPS, and mIoU over global-flow warping, while global parsing eliminates overlap artifacts during part assembly.The comparisons support more realistic and semantically correct warped garments.
- Ablation Study: DGT achieves the lowest Rdiff and preserves texture details, avoiding the squeezing or stretching observed with alternative training strategies.The qualitative comparisons show squeezing without gradient truncation and stretching with the normal gradient-truncation strategy.
5. Conclusion
GP-VTON targets general-purpose virtual try-on with robust garment warping for challenging self-occlusion scenarios and multi-category extension. It combines LFGP and DGT, and experiments on two high-resolution benchmarks demonstrate superiority over existing methods.
- Conclusion: GP-VTON generates semantic-correct, photo-realistic try-on results in challenging self-occlusion scenarios and extends to multi-category scenarios.The framework is designed as a general-purpose virtual try-on system.
- Conclusion: LFGP warps garment parts individually with local flows and assembles them using estimated global garment parsing.This design targets robustness to intricate inputs.
- Conclusion: DGT is used in the warping network to alleviate texture distortion.The strategy addresses distortion associated with garment warping around preserved regions.
- Conclusion: Experiments on two high-resolution virtual try-on benchmarks demonstrate GP-VTON’s superiority over existing methods.