Source-linked AI summary
Style-Based Global Appearance Flow for Virtual Try-On
Sen He, Yi-Zhe Song, Tao Xiang
TL;DR
Image-based virtual try-on requires garment-person alignment, but local appearance-flow methods are vulnerable to large misalignment, difficult poses, and occlusions. The paper introduces a StyleGAN-based global flow estimator with local refinement, achieving state-of-the-art performance on VITON and robustness on augmented, highly misaligned examples.
Problem
Existing appearance-flow methods lack global context, limiting accurate garment warping under large misalignment, difficult poses, and occlusions.
Method
The method uses StyleGAN style modulation to estimate global appearance flow, then applies local refinement to model fine-grained deformation.
Results
The method achieves state-of-the-art performance on VITON and outperforms previous state-of-the-art parser-free methods on all evaluation metrics.
Takeaways & Limitations
The method is more robust to large person-garment misalignment and difficult poses or occlusions, including augmented examples with large misalignments.
Abstract
from arXiv · showhide
Image-based virtual try-on aims to fit an in-shop garment into a clothed person image. To achieve this, a key step is garment warping which spatially aligns the target garment with the corresponding body parts in the person image. Prior methods typically adopt a local appearance flow estimation model. They are thus intrinsically susceptible to difficult body poses/occlusions and large mis-alignments between person and garment images (see Fig.~\ref{fig:fig1}). To overcome this limitation, a novel global appearance flow estimation model is proposed in this work. For the first time, a StyleGAN based architecture is adopted for appearance flow estimation. This enables us to take advantage of a global style vector to encode a whole-image context to cope with the aforementioned challenges. To guide the StyleGAN flow generator to pay more attention to local garment deformation, a flow refinement module is introduced to add local context. Experiment results on a popular virtual try-on benchmark show that our method achieves new state-of-the-art performance. It is particularly effective in a `in-the-wild' application scenario where the reference image is full-body resulting in a large mis-alignment with the garment image (Fig.~\ref{fig:fig1} Top). Code is available at: \url{https://github.com/SenHe/Flow-Style-VTON}.
1. Introduction
Image-based virtual try-on must spatially align garments with corresponding body parts, but local appearance-flow methods struggle with large misalignment, difficult poses, and occlusions. The paper proposes a StyleGAN-based global flow estimator with local refinement to address these challenges.
- Virtual try-on aligns an in-shop garment with corresponding body parts before image fusion, because misalignment produces unrealistic results in occluded regions.
- Prior TPS-based warping methods exploit feature correlations but have limitations in handling complex warping where different garment regions require different transformations.
- Existing appearance-flow methods rely on local feature correspondence and therefore deteriorate under large garment-person misalignment or difficult poses and occlusions.
- The proposed model uses StyleGAN and a global style vector extracted from whole person and garment images to capture global context.
- A local flow-refinement module complements StyleGAN’s global modeling because vanilla StyleGAN is weaker at local deformation modeling.
- The method is designed to improve robustness to large misalignment and support in-the-wild try-on with full-body person images and natural poses.
2. Related Work
Related image-based virtual try-on methods differ in their use of human parsing and garment-warping strategies. This work is parser-free and uses StyleGAN-inspired appearance flow to preserve garment details while improving deformation handling.
- Parser-based methods use human segmentation and related representations to estimate warping, but inaccurate parsing can produce inaccurate warping and try-on results.
- Parser-free methods use only person and garment images at inference time and are designed to avoid negative effects from parsing errors.
- The proposed method is parser-free and focuses on a global appearance-flow garment-warping module.
- The garment-warping network is inspired by StyleGAN’s shape-deformation capability but predicts appearance flow that samples the garment rather than directly generating it.
- Appearance flow acts as a sampling grid for garment warping, making the process information lossless and detail-preserving.
3. Methodology
The method uses a parser-based teacher during training but performs try-on with a parser-free model whose StyleGAN-based warping module combines global context with local flow refinement. The resulting flow warps the garment before a generator synthesizes the target image.
- Virtual try-on generates an image in which the garment from g fits the corresponding body parts in p while preserving garment details and non-garment regions.
- The parser-based model F_PB produces training inputs and supervision for the parser-free model F, and is used only during F's training.F_PB takes semantic representations and an unpaired garment to generate p, while knowledge distillation guides F.
- The parser-free model extracts person and garment feature maps with two convolutional encoders, which the warping module uses to predict appearance flow.The encoders use stacked residual blocks and provide feature maps at multiple resolutions.
- The central contribution is a StyleGAN-based global appearance-flow estimator that first predicts coarse flow from a global style vector and then refines it with local feature correspondence.The style vector is formed from the lowest-resolution person and garment features; each warping block contains style-based prediction and local refinement components.
- Global style modulation gives the coarse flow a global receptive field for large mis-alignments, while local refinement estimates fine-grained flow after coarse warping aligns corresponding features.The final flow in each block is the sum of coarse and refinement flows, and the last flow warps the garment.
- The warped garment is concatenated with the person image and passed to an encoder-decoder generator with skip connections for target try-on synthesis.The generator design follows prior texture-detail-preserving architectures.
4. Experiments
Experiments on VITON and augmented VITON evaluate image quality, human preference, robustness to misalignment, and the contributions of global and local flow estimation. The proposed method achieves strong benchmark performance and remains robust under shifted person images.
- Robustness evaluation: The augmented VITON dataset tests robustness to randomly positioned person images with larger garment-person misalignments than standard VITON.Standard VITON images are usually well pre-aligned, so the augmented dataset targets a harder setting.
- Evaluation setup: The model is evaluated on VITON using SSIM, FID, and human preference studies against parser-based and parser-free baselines.The compared methods include VTON, CP-VTON, Cloth-flow, CP-VTON++, ACGPN, DCTON, ZFlow, and PF-AFN.
- Main results: The proposed model achieves new state-of-the-art performance on the VITON testing dataset and lowers PF-AFN’s FID of 10.09 by 11.9%.It outperforms previous state-of-the-art parser-free methods on all evaluation metrics and exceeds all compared models by more than 10% preference rate in human evaluation.
- Robustness evaluation: On augmented VITON, the model performs best while other models’ performance drops dramatically, and its SSIM remains comparable to standard VITON.Qualitative results report consistent, high-quality try-on images under large misalignments.
- Ablation study: Combining global style modulation with local refinement improves over either flow-estimation component alone.Global modulation handles large misalignment, while local refinement addresses fine-grained regions such as sleeves.
5. Conclusion
The paper proposes style-based global appearance flow estimation for garment warping, followed by local refinement. It reports state-of-the-art VITON performance and improved robustness to large misalignment and difficult poses or occlusions.
- Conclusion: The method estimates garment-warping flow globally through style modulation and then refines it locally.The conclusion identifies this two-stage flow estimation as the central design.
- Conclusion: The method achieves state-of-the-art performance on the VITON benchmark.The authors report extensive experiments validating the method and its architecture.
- Conclusion: The method is more robust to large person-garment misalignment and difficult poses or occlusions.The conclusion states this robustness as a key outcome of the proposed approach.