PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Multi-pose Guided Virtual Try-on Network

Haoye Dong, Xiaodan Liang, Bochao Wang, Hanjiang Lai, Jia Zhu, Jian Yin

arXiv:1902.11026v1cs.CV

TL;DR

Existing virtual try-on methods largely target fixed poses and can lose identity and clothing details, while multi-pose try-on remains less explored. MG-VTON combines pose-clothes-guided parsing, clothes warping with Warp-GAN, and refinement to generate people in desired clothes and poses, and experiments report strong qualitative and quantitative performance across benchmarks.

  • Problem

    Multi-pose virtual try-on is less explored than fixed-pose try-on, although arbitrary poses create self-occlusion, misalignment, and clothing-texture challenges.

  • Method

    MG-VTON synthesizes a target parsing map, warps desired clothes with Warp-GAN, and refines the result using pose-conditioned composition information.

  • Results

    MG-VTON significantly outperforms state-of-the-art methods qualitatively and quantitatively on well-known datasets and the newly collected multi-pose benchmark.

  • Takeaways & Limitations

    The work establishes a multi-pose guided virtual try-on task, supported by a dataset covering more poses and clothes diversity.

  • Takeaways & Limitations

    3D-based alternatives require expert knowledge, 3D-annotated data, substantial labor, and massive computation, limiting practical application.

Abstract

from arXiv · show

Virtual try-on system under arbitrary human poses has huge application potential, yet raises quite a lot of challenges, e.g. self-occlusions, heavy misalignment among diverse poses, and diverse clothes textures. Existing methods aim at fitting new clothes into a person can only transfer clothes on the fixed human pose, but still show unsatisfactory performances which often fail to preserve the identity, lose the texture details, and decrease the diversity of poses. In this paper, we make the first attempt towards multi-pose guided virtual try-on system, which enables transfer clothes on a person image under diverse poses. Given an input person image, a desired clothes image, and a desired pose, the proposed Multi-pose Guided Virtual Try-on Network (MG-VTON) can generate a new person image after fitting the desired clothes into the input image and manipulating human poses. Our MG-VTON is constructed in three stages: 1) a desired human parsing map of the target image is synthesized to match both the desired pose and the desired clothes shape; 2) a deep Warping Generative Adversarial Network (Warp-GAN) warps the desired clothes appearance into the synthesized human parsing map and alleviates the misalignment problem between the input human pose and desired human pose; 3) a refinement render utilizing multi-pose composition masks recovers the texture details of clothes and removes some artifacts. Extensive experiments on well-known datasets and our newly collected largest virtual try-on benchmark demonstrate that our MG-VTON significantly outperforms all state-of-the-art methods both qualitatively and quantitatively with promising multi-pose virtual try-on performances.

1. Introduction

MG-VTON addresses the underexplored problem of virtual try-on across diverse poses while preserving person identity and clothing appearance. It introduces a multi-stage network and a new diverse-pose dataset, with experiments reporting competitive qualitative and quantitative results.

  • Existing virtual try-on methods mainly assume fixed poses and often lose clothing, hair, identity, and texture details under varied poses.They can produce blurry or unreasonable images because they overlook human parsing features.
  • 3D-based approaches provide detailed body-shape information but require expert knowledge, annotated 3D data, substantial labor, and heavy computation.These costs and complexity limit practical virtual try-on applications.
  • MG-VTON generates a new person image conditioned on an input person, desired clothes, and desired pose.Its design combines pose-clothes-guided parsing, clothes warping, Warp-GAN synthesis, and refinement.
  • The authors collect the MPV dataset with greater pose and clothes diversity and report competitive quantitative and qualitative results on two datasets.Experiments include DeepFashion evaluation and a human subjective study on Amazon Mechanical Turk.
  • Multi-pose virtual try-on restructures a person image by manipulating both diverse clothes and poses.

2. Related Work

Related work spans GAN-based image synthesis, pose-guided person generation, and image-based virtual try-on. Prior try-on systems use warping but largely target fixed-pose clothing transfer, motivating the multi-pose setting addressed by MG-VTON.

  • GANs use a generator to synthesize realistic images and a discriminator to distinguish synthesized images from real images.
  • Person image synthesis methods condition generation on inputs such as person images and target skeletons, often using coarse-to-fine or decomposition strategies.
  • The proposed framework emphasizes human parsing because parsing maps guide synthesis at precise body-part regions and support structural coherence.
  • VITON and CP-VTON transfer desired clothes using warping, with VITON using shape-context TPS and CP-VTON learning transformation parameters.

3. MG-VTON

MG-VTON uses an outline-coarse-fine strategy that separates conditional parsing, warped-clothes synthesis, and refinement. Its inputs include person structure, clothing, and target-pose information to preserve coherence during pose and clothing manipulation.

  • The framework takes an input person image, desired clothes, and target pose to synthesize a new person image.
  • MG-VTON divides multi-pose try-on into conditional parsing learning, Warp-GAN synthesis, and refinement rendering.
  • The parsing stage predicts a target human parsing map from clothes, target pose, body shape, face mask, and hair mask.Pose is represented with heatmaps, while masks are extracted from human parsing outputs.
  • The network conditions parsing on clothing and pose to preserve structural coherence while manipulating both.This design targets failures where trousers’ color or hair style are not preserved.

(b). Conditional Parsing

MG-VTON combines conditional parsing, warping, refinement, and geometric matching to synthesize person images with desired clothes and poses. These modules respectively model human structure, address pose-induced misalignment, recover details, and estimate clothing transformations.

  • Conditional Parsing: The conditional parsing stage predicts target human parsing from pose, clothes, and person-structure cues to guide image synthesis.Its formulation uses a conditional generator with L1 and pixel-wise softmax losses for smoother, higher-quality parsing maps.
  • Warp-GAN: Warp-GAN takes warped clothes, a clothes-free reference, the target pose, and synthesized parsing as inputs to generate the coarse result.It uses warping features to alleviate misalignment caused by different input and target poses, with adversarial, perceptual, feature, and L1 losses.
  • Refinement Render: The refinement render predicts a multi-pose composition mask that combines warped clothes with the coarse result to recover texture details and reduce artifacts.The generator uses perceptual and mask losses, with element-wise multiplication controlling the composition.
  • Geometric Matching: Geometric matching estimates a transformation from the original clothes mask to the synthesized clothes mask under the target body shape, producing warped clothes for later stages.The warped clothing is defined as Cw = Tθ(C), supporting misalignment handling and composition-mask learning.

4. Experiments

The experiments evaluate MG-VTON through visual comparisons, quantitative benchmarks, perceptual study, ablations, and cross-dataset testing. The newly collected MPV dataset supplies diverse-pose person and clothing examples for this evaluation.

  • Experimental Protocol: The evaluation includes visual and quantitative comparisons, human perceptual study, ablation experiments, and testing a model trained on MPV against DeepFashion.The quantitative evaluation uses SSIM and IS, while the perceptual study follows the object evaluation protocol on Amazon Mechanical Turk.
  • Visual Comparison: Figure 4 compares methods on MPV, including MG-VTON variants without refinement rendering or without the multi-pose composition mask.MG-VTON (w/o Render) removes refinement render, while MG-VTON (w/o Mask) removes the multi-pose composition mask.
  • Dataset: MPV contains 35,687 person images and 13,524 clothes images, including 62,780 same-person, same-clothes tuples with diverse poses.The images have 256 × 192 resolution, and the tuples are divided into 52,236 training and 10,544 test examples.

4.2. Evaluation Metrics

MG-VTON is evaluated with human perceptual judgments and objective image-quality measures, while implementation uses separately trained modules and a specified optimization setup.

  • Evaluation measures: Human perceptual quality is measured through pairwise A/B tests on Amazon Mechanical Turk.Workers compare synthesized images according to realism and preservation of target clothes, pose, identity, and appearance.
  • Evaluation measures: SSIM measures similarity between synthesized images and ground-truth target images.The target image depicts the same person wearing the same clothes.
  • Evaluation measures: Inception Score measures the quality of generated images as a standard image-generation evaluation.
  • Implementation: The conditional parsing network, Warp-GAN, refinement render, and geometric matching network are trained for 200, 15, 5, and 35 epochs, respectively.Training uses ADAM with batch size 40 and learning rate 0.0002.
  • Implementation: MG-VTON generators use ResNet-like architectures, while discriminators use a pix2pixHD-style multiscale architecture.Generators contain downsampling, upsampling, and residual blocks; discriminators process feature maps at different scales.

4.4. Baselines

The evaluation compares MG-VTON with fixed-pose virtual try-on baselines on MPV and DeepFashion using SSIM and IS, with higher scores preferred.

  • Compared methods: VITON and CP-VTON are state-of-the-art image-based virtual try-on baselines that assume a fixed person pose.Both use warped clothes images but cannot generate images under arbitrary poses.
  • Evaluation setup: MG-VTON is compared with two related methods on MPV and DeepFashion using SSIM and IS.The comparison is summarized in Table 1, where higher scores are better.
  • Quantitative results: MG-VTON consistently outperforms all baselines on both datasets according to the reported SSIM and IS comparisons.The authors attribute the result to cooperation among conditional parsing, Warp-GAN, and refinement render.
  • Ablations: MG-VTON without rendering achieves the best SSIM, while MG-VTON without masking achieves the best IS, but both have worse visual quality and AMT scores than full MG-VTON.

4.6. Qualitative Results

Qualitative comparisons show that MG-VTON preserves identity, clothing details, hairstyle, and face more convincingly than fixed-pose baselines, while cross-dataset examples demonstrate pose-and-clothes capture.

  • Visual comparison: MG-VTON produces reasonable results with convincing details compared with VITON, CP-VTON, and its ablated variants.
  • Baseline limitations: Baseline methods fail to preserve lower-body clothing, identity, hairstyle, and facial appearance under multi-pose try-on.They also produce blurry results and artifacts.
  • MG-VTON design: Clothes- and pose-guided human parsing helps address failures in preserving lower-body clothing and hair.
  • MG-VTON design: The deep warping network addresses identity preservation, while the multi-pose refinement network learns to remove noise and artifacts.
  • Cross-dataset results: Models trained on MPV and tested on DeepFashion synthesize realistic images that capture the desired pose and clothes.

4.7. Human Perceptual Study

The human perceptual study uses pairwise AMT comparisons on MPV and DeepFashion, while ablations examine refinement components and parsing quality.

  • Human study: The AMT study compares 1,600 synthesized images using 8,000 pairwise judgments from 100 unique workers.Workers assess realism, target clothes and pose capture, and preservation of identity and appearance.
  • Human study: MG-VTON receives higher human evaluation scores than the baseline methods on MPV and DeepFashion.
  • Parsing quality: Higher-quality human parsing is positively correlated with higher-quality synthesized results.

4.8. Ablation Study

The ablations identify key components that improve MG-VTON, while human-parsing experiments show that parsing quality is positively associated with output quality and supports effective pose-and-clothes manipulation.

  • Component ablations: The multi-pose composition mask loss, perceptual loss, refinement-stage pose, and Warp-GAN warping module all enhance performance.The ablation study identifies each component as important to the model's performance.
  • Human parsing: Human-parsing quality is positively correlated with the quality of the synthesized result.This experiment evaluates the contribution of human parsing within MG-VTON.
  • Human parsing: Manipulating synthesized human parsing enables easier and more effective synthesis of person images than directly manipulating the person image.Figure 8 examines human parsing under changes to the desired pose and clothes.
  • Generalization: The model trained on MPV and tested on DeepFashion captures the target pose and clothes well.This experiment evaluates generalization across the collected MPV dataset and DeepFashion.

5. Conclusions

The paper presents MG-VTON as a first attempt at multi-pose guided virtual try-on, combining parsing, warping, and refinement to transfer clothes under diverse poses. A new diverse-pose dataset and extensive experiments support its reported qualitative and quantitative superiority over state-of-the-art methods.

  • Conclusion: MG-VTON is presented as a first system for transferring clothes onto person images under diverse poses.It generates a new person image by fitting desired clothes into an input image while manipulating human poses.
  • Conclusion: The framework decomposes virtual try-on into human-parsing guidance, Warp-GAN warping, and refinement rendering.The stages respectively guide synthesis, alleviate pose-induced misalignment, and recover clothing texture details.
  • Conclusion: The authors construct a new dataset covering person images with greater pose and clothing diversity.
  • Conclusion: Extensive experiments report that MG-VTON significantly outperforms state-of-the-art methods qualitatively and quantitatively.
Loading 1902.11026v1…