Source-linked AI summary
Towards Photo-Realistic Virtual Try-On by Adaptively Generating$\leftrightarrow$Preserving Image Content
Han Yang, Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, Ping Luo
TL;DR
Photo-realistic virtual try-on remains challenging under large occlusions and complex human poses, despite prior methods’ ability to preserve clothing character. ACGPN predicts post-try-on semantic layouts and adaptively generates or preserves content through layout generation, clothes warping, and content fusion. Experiments report better perceptual quality, richer fine details, and stronger quantitative and user-study results than state-of-the-art methods.
Problem
Photo-realistic virtual try-on is difficult when large occlusions and complex poses accompany the reference person.
Method
ACGPN predicts post-try-on semantic layouts, warps clothes with a second-order difference constraint, and fuses generated and preserved content adaptively.
Results
ACGPN outperforms state-of-the-art methods qualitatively and quantitatively, producing photo-realistic images with richer fine details across three VITON difficulty levels.
Takeaways & Limitations
Adaptive generation and preservation allow ACGPN to retain clothing character and human-identity details while adapting to difficult semantic layouts.
Abstract
from arXiv · showhide
Image visual try-on aims at transferring a target clothing image onto a reference person, and has become a hot topic in recent years. Prior arts usually focus on preserving the character of a clothing image (e.g. texture, logo, embroidery) when warping it to arbitrary human pose. However, it remains a big challenge to generate photo-realistic try-on images when large occlusions and human poses are presented in the reference person. To address this issue, we propose a novel visual try-on network, namely Adaptive Content Generating and Preserving Network (ACGPN). In particular, ACGPN first predicts semantic layout of the reference image that will be changed after try-on (e.g. long sleeve shirt$\rightarrow$arm, arm$\rightarrow$jacket), and then determines whether its image content needs to be generated or preserved according to the predicted semantic layout, leading to photo-realistic try-on and rich clothing details. ACGPN generally involves three major modules. First, a semantic layout generation module utilizes semantic segmentation of the reference image to progressively predict the desired semantic layout after try-on. Second, a clothes warping module warps clothing images according to the generated semantic layout, where a second-order difference constraint is introduced to stabilize the warping process during training. Third, an inpainting module for content fusion integrates all information (e.g. reference image, semantic layout, warped clothes) to adaptively produce each semantic part of human body. In comparison to the state-of-the-art methods, ACGPN can generate photo-realistic images with much better perceptual quality and richer fine-details.
1. Introduction
Image-based virtual try-on remains difficult when clothing and body regions differ geometrically or are occluded by complex poses. ACGPN addresses this by predicting the post-try-on semantic layout, adaptively generating or preserving content, and producing photo-realistic results.
- Semantic and geometric differences, together with torso–limb occlusions, make photo-realistic virtual try-on challenging in real-world scenarios.
- ACGPN predicts the reference image’s post-try-on semantic layout before deciding which content should be generated or preserved.Examples include changing long sleeve shirt regions into arms and arm regions into jacket regions.
- The Semantic Generation Module progressively generates exposed-body and warped-clothing masks, making the original reference-clothing shape agnostic to the network.It generates body-part masks first and synthesizes the clothing mask progressively.
- The Clothes Warping Module uses a second-order difference constraint to stabilize warping, especially for complex clothing textures.
- The Content Fusion Module combines synthesized body masks, warped clothing, and original body information to adaptively produce the final semantic parts.
- Experiments on VITON show photo-realistic results across easy, medium, and hard try-on cases, including cases with intersecting semantic regions.
2. Related Work
Related work spans GAN-based image synthesis, fashion analysis, and virtual try-on methods based on 3D models or 2D images. Existing 2D methods improve clothing transfer but differ in their ability to preserve source information and handle non-target body parts.
- GANs use generators and discriminators to synthesize realistic images and support tasks including style transfer, inpainting, and image editing.
- Fashion-analysis research covers clothing compatibility, landmark detection, image analysis, and virtual try-on, with virtual try-on identified as especially challenging.
- Deep-learning virtual try-on methods include 3D model-based and 2D image-based approaches, with 2D methods further categorized by posture preservation.
- CAGAN preserves color and coarse shape, while VITON and CP-VTON use coarse shape and pose information with different warping strategies.
- Table 1 compares ACGPN with CAGAN, VITON, CP-VTON, and VTNFP by input representations, source-information preservation, and problems addressed.
- VITON and CP-VTON retain clothing character but overlook non-target body parts and bottom clothes; VTNFP adds weak supervision to preserve more details but still exhibits artifacts.
3. Adaptive Content Generating and Preserving Network
ACGPN predicts post-try-on semantic layouts, warps clothing accordingly, and adaptively preserves or generates body content for photo-realistic results with fine details.
- Semantic Generation Module: The Semantic Generation Module progressively predicts body-part and clothing masks from fused body masks, pose maps, and target clothing to establish semantic alignment.Its two-stage strategy generates body parts first and synthesizes clothing masks progressively.
- Clothes Warping Module: The Clothes Warping Module uses the synthesized clothing region to deform target garments while retaining their visual character.A second-order difference constraint is applied to TPS warping to stabilize transformations and preserve local geometric properties.
- Content Fusion Module: The model combines reference information, semantic layouts, and warped clothing through fusion networks to produce the final try-on image.The composited body mask and body-part image are formed from information generated in previous steps.
- Clothes Warping Module: The second-order warping constraint minimizes differences between neighboring intervals and slopes, maintaining collinearity, parallelism, and affine-transformation properties.The warping loss additionally measures the discrepancy between the warped clothing image and its ground truth.
- Content Fusion Module: The Content Fusion Module preserves untargeted body parts, adaptively preserves changeable parts, and inpaints generated regions using masks and images from earlier stages.This design addresses the loss of fine body-part details caused by relying only on coarse body shape.
4. Experiments
Experiments evaluate ACGPN on VITON across easy, medium, and hard try-on cases using qualitative comparisons, SSIM, IS, and ablations. ACGPN produces more photo-realistic results, preserves clothing and body details, and benefits from semantic layout prediction, body-part composition, and the second-order difference constraint.
- Experimental Setup: ACGPN is evaluated on 16,253 valid VITON image pairs, split into 14,221 training pairs and 2,032 testing pairs.The experiments compare ACGPN with VITON, CP-VTON, and VTNFP at three pose-based difficulty levels.
- Experimental Setup: The dataset partitions try-on images into easy, medium, and hard cases based on posture, torso twisting, hand overlap, and occlusion complexity.Hard cases contain torso twisting and two hands blocking in front of the body, while easy cases generally show a forward-facing person with hands down.
- Qualitative Results: ACGPN produces more photo-realistic visual results than VITON, CP-VTON, and VTNFP by preserving clothing texture, logos, embroideries, and non-target body parts.Competing methods show artifacts such as color mixture, boundary blurring, broken arms, stretched logos, and blurry body details.
- Quantitative Results: ACGPN outperforms competing methods on SSIM across all difficulty levels and gains 0.179, 0.072, and 0.045 in IS against VITON, CP-VTON, and VTNFP, respectively.For SSIM, the gains over VITON, CP-VTON, and VTNFP are 0.067, 0.101, and 0.044 on easy cases; 0.062, 0.099, and 0.040 on medium cases; and 0.049, 0.099, and 0.040 on hard cases.
- Ablation Study: Ablations show that non-target body-part composition improves visual results by clarifying body layout and preserving details that otherwise may be generated incorrectly.Without composition, correct body-part shapes can still contain wrong details; the visual comparison includes blurry, deformed, and distorted body parts.
- Ablation Study: The second-order difference constraint stabilizes warping and is especially important for complex clothing textures and embroideries.Without the constraint, warped clothing can become distorted; pure-color or simply embroidered garments are less vulnerable to this degeneration.
5. Conclusion
ACGPN is an adaptive content generating and preserving network designed to produce photo-realistic try-on images while retaining clothing character and human identity details.
- ACGPN targets photo-realistic try-on results while preserving clothing character and human identity details, including posture, body parts, and bottom clothes.
- The method is evaluated on the VITON dataset across three try-on difficulty levels.
- Experiments show superiority over state-of-the-art methods in quantitative metrics, visual quality, and user study.
6. The Reconstruction loss
The reconstruction losses combine L1 and perceptual losses to train warped-clothes refinement and inpainting-based content fusion.
- A U-Net refines warped clothes to fit the clothing mask, producing a coarse result and composition mask for image composition.
- The composition equation combines warped clothes and reference clothing content using a composition mask through element-wise multiplication.
- The refinement loss combines L1 loss with perceptual loss, using VGG19 features to measure perceptual differences.
- The inpainting-based fusion GAN uses L1 and perceptual losses between the synthesized image and ground-truth image.
- The stated loss weights are Lo = Li = 1 and Lpc = Lpi = 10.
7. More Try-on Results
Additional results show ACGPN handling varying try-on difficulty, diverse poses, occlusions, and cross-arm configurations while reducing artifacts.
- The paper provides more try-on results in Figures 9 and 10 and an accompanying example video.
- ACGPN performs equally well when transferring long-sleeve clothes to short-sleeve references and short-sleeve clothes to long-sleeve references.
- Across four reference people, ACGPN robustly handles varied poses including occlusions and crossed arms, with artifacts reduced to a minimum.