PaperlayerView exact revision on arXiv

Source-linked AI summary

VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization

Seunghwan Choi, Sunghyun Park, Minsoo Lee, Jaegul Choo

arXiv:2103.16874v2cs.CV

TL;DR

Image-based virtual try-on has struggled to produce high-resolution results because warped-clothing misalignment causes artifacts and existing architectures inadequately preserve body and garment details. VITON-HD combines clothing-agnostic representations with ALIAS normalization and an ALIAS generator, successfully synthesizing 1024×768 images and surpassing existing methods in image-quality evaluations.

  • Problem

    Virtual try-on must preserve identity and body structure while fitting and detailing a separate garment, but prior outputs were often limited to 256×192 and suffered from misalignment artifacts and weak high-resolution detail preservation.

  • Method

    VITON-HD uses a clothing-agnostic person representation, a segmentation-guided warped garment, and ALIAS normalization with multi-scale feature refinement.

  • Results

    1024×768 images: VITON-HD successfully synthesizes high-resolution virtual try-on results and surpasses existing methods in qualitative and quantitative image-quality comparisons.

  • Takeaways & Limitations

    The method provides photo-realistic high-resolution virtual try-on images while handling misaligned regions and preserving clothing details.

  • Takeaways & Limitations

    The clothing-agnostic representation assumes the original garment can be removed while retaining pose, body shape, and identity-preserving regions such as the face and hands.

Abstract

from arXiv · show

The task of image-based virtual try-on aims to transfer a target clothing item onto the corresponding region of a person, which is commonly tackled by fitting the item to the desired body part and fusing the warped item with the person. While an increasing number of studies have been conducted, the resolution of synthesized images is still limited to low (e.g., 256x192), which acts as the critical limitation against satisfying online consumers. We argue that the limitation stems from several challenges: as the resolution increases, the artifacts in the misaligned areas between the warped clothes and the desired clothing regions become noticeable in the final results; the architectures used in existing methods have low performance in generating high-quality body parts and maintaining the texture sharpness of the clothes. To address the challenges, we propose a novel virtual try-on method called VITON-HD that successfully synthesizes 1024x768 virtual try-on images. Specifically, we first prepare the segmentation map to guide our virtual try-on synthesis, and then roughly fit the target clothing item to a given person's body. Next, we propose ALIgnment-Aware Segment (ALIAS) normalization and ALIAS generator to handle the misaligned areas and preserve the details of 1024x768 inputs. Through rigorous comparison with existing methods, we demonstrate that VITON-HD highly surpasses the baselines in terms of synthesized image quality both qualitatively and quantitatively. Code is available at https://github.com/shadow2496/VITON-HD.

1. Introduction

Image-based virtual try-on must preserve a person’s identity and body while naturally fitting and detailing a separate clothing product. VITON-HD targets high-resolution synthesis by addressing misalignment artifacts and detail preservation with clothing-agnostic representations and ALIAS-based generation.

  • Virtual try-on changes a person’s clothing using a separate product image while preserving pose, body shape, identity, garment deformation, details, and previously occluded body parts.
  • Existing methods commonly warp clothing to the body and fuse it with the person, sometimes using segmentation maps to determine the target layout.
  • 256×192 synthetic images remain limited because warping misalignment creates increasingly visible artifacts, while high-resolution architectures and one-time pixel refinement struggle with quality and clothing details.
  • VITON-HD introduces a clothing-agnostic person representation using pose and segmentation information to remove the original clothing while retaining relevant person information.
  • ALIAS normalization and the ALIAS generator address misaligned regions by removing irrelevant information, propagating semantic information, and preserving clothing details through multi-scale feature refinement.
  • The method is presented as successfully synthesizing 1024×768 images and outperforming baselines in qualitative and quantitative image-quality comparisons.

2. Related Work

Related work spans conditional image synthesis, normalization layers, and virtual try-on systems. Prior 2D methods improve deformation or layout guidance, but the cited approaches do not produce photorealistic high-resolution outputs.

  • Conditional Image Synthesis: Conditional generative models use labels, text, attributes, or images to steer generation, but large spatial deformations can produce blurry outputs.
  • Normalization Layers: Conditional normalization methods estimate affine parameters from external data, while SPADE and SEAN use segmentation maps for spatially varying transformations.
  • Virtual Try-On Approaches: Virtual try-on methods fall into 3D model-based and 2D image-based categories, with 3D approaches limited in applicability by their dependence on measurement data.
  • Virtual Try-On Approaches: 2D methods are computationally efficient and practical, using approaches such as TPS-based deformation, geometric matching, and predicted human-parsing maps.
  • Virtual Try-On Approaches: Despite deformation and layout improvements, the listed prior methods do not generate photorealistic virtual try-on images at high resolution.

3. Proposed Method

VITON-HD builds a staged high-resolution try-on pipeline that removes the original clothing, predicts the target layout, deforms the target garment, and synthesizes the result with alignment-aware normalization.

  • Clothing-Agnostic Person Representation: The clothing-agnostic representation uses segmentation and pose maps to remove the original garment while preserving pose, body shape, hands, face, and identity-relevant regions.The segmentation map removes the replacement clothing region, while the pose map removes arms but preserves hands; RGB pose information is concatenated with the representation.
  • Segmentation Generation: The segmentation generator predicts the person’s target-clothing layout from the clothing-agnostic representation and target clothing item.It is trained against the reference segmentation using pixel-wise cross-entropy and conditional adversarial losses.
  • Clothing Image Deformation: The geometric matching module warps the target clothing item to align with the predicted clothing region using a thin-plate spline transformation.Its training objective combines an L1 reconstruction term with a second-order difference constraint.
  • Try-On Synthesis via ALIAS Normalization: The ALIAS generator fuses the clothing-agnostic person representation with the warped garment while injecting these inputs at multiple resolutions.It uses residual blocks and upsampling layers, with predicted segmentation and misalignment masks supplied to ALIAS normalization.
  • Try-On Synthesis via ALIAS Normalization: ALIAS normalization separately standardizes misaligned and other regions, then uses the divided segmentation map to generate affine modulation parameters.The misalignment mask is derived from the difference between the predicted clothing segmentation and the aligned warped clothing mask.

4. Experiments

Experiments evaluate VITON-HD on a 1024×768 dataset using qualitative, paired, unpaired, resolution-based, and misalignment-focused comparisons. VITON-HD preserves clothing details, handles clothing-agnostic segmentation, and generally outperforms the baselines, especially at high resolution and under greater misalignment.

  • Experiment Setup: The dataset contains 13,679 frontal-view woman–top clothing pairs, split into 11,647 training and 2,032 test pairs.Paired evaluation reconstructs original clothing, while unpaired evaluation shuffles clothing images.
  • Experiment Setup: VITON-HD is compared with CP-VTON and ACGPN, whose segmentation and geometric matching modules operate at 256×192 before inference upscaling.The baselines’ outputs are upscaled to 1024×768 during inference.
  • Qualitative Analysis: VITON-HD generates more perceptually convincing 1024×768 images while preserving target-clothing logos and textures through multi-scale feature-level refinement.Its synthesized body shape also remains natural regardless of the reference clothing item.
  • Qualitative Analysis: The VITON-HD segmentation generator predicts clothing-independent layouts, whereas ACGPN retains the original clothing shape in its synthetic segmentation maps.This difference is attributed to VITON-HD’s clothing-agnostic person representation.
  • ALIAS Normalization: ALIAS normalization fills misaligned areas with target-clothing texture by removing misleading warped-clothing information; replacing it with channel-wise standardization produces artifacts.The comparison uses VITON-HD* as the variant without ALIAS standardization.
  • Quantitative Analysis: Across 256×192, 512×384, and 1024×768, VITON-HD outperforms the baselines on SSIM and LPIPS and achieves a lower FID than ACGPN at 1024×768.At 256×192, ACGPN’s FID is slightly lower, while VITON-HD’s advantage appears with a large margin at 1024×768.
  • Quantitative Analysis: Performance worsens as misaligned regions widen, but VITON-HD consistently outperforms the baselines and degrades less as misalignment increases.LPIPS is measured in paired experiments using shared segmentation maps and warped clothes.

5. Conclusions

VITON-HD synthesizes photo-realistic 1024×768 virtual try-on images by handling misaligned regions and preserving clothing details. Qualitative and quantitative experiments report that it surpasses existing virtual try-on methods with a large margin.

  • VITON-HD synthesizes photo-realistic 1024×768 virtual try-on images.
  • ALIAS normalization handles misaligned areas and propagates semantic information throughout the ALIAS generator.Multi-scale refinement preserves clothing details.
  • Qualitative and quantitative experiments demonstrate that VITON-HD surpasses existing virtual try-on methods with a large margin.

A. Implementation Details

The clothing-agnostic person representation removes original clothing information while preserving body parts that must be reproduced. It uses pose and segmentation information to construct the representation.

  • The clothing-agnostic image and segmentation map remove the original clothing shape while preserving body parts that need reproduction.

A.2. Model Architectures

The implementation uses separate generators and matching modules, with ALIAS ResBlks forming the high-resolution synthesis architecture. These components combine convolutional processing, geometric matching, normalization, and multi-scale discrimination.

  • Segmentation Generator: The segmentation generator uses a U-Net with convolutional, downsampling, and upsampling layers.Two multi-scale discriminators provide conditional adversarial loss.
  • Geometric Matching Module: The geometric matching module extracts two feature representations, computes their correlation matrix, and predicts TPS parameters θ.Its regression network ends with a fully connected layer.
  • ALIAS Generator: The ALIAS generator consists of ALIAS ResBlks with nearest-neighbor upsampling layers and two multi-scale discriminators.Spectral normalization is applied to all convolutional layers.
  • ALIAS Generator: ALIAS normalization separately standardizes activations using the misalignment mask in the first five layers.

A.3. Training Details

Training details cover the segmentation, geometric matching, and ALIAS generators, along with their losses, optimization settings, and evaluation comparisons. The ALIAS generator receives segmentation and misalignment information to address alignment while preserving high-resolution clothing details.

  • Segmentation Generator: The segmentation generator predicts a target-person segmentation map from a clothing-agnostic segmentation map, pose map, and target clothing item.It is trained with cross-entropy and conditional adversarial losses.
  • Geometric Matching Module: The geometric matching module uses clothing, pose, clothing-agnostic appearance, and predicted clothing-region segmentation to estimate TPS transformation parameters.The deformation objective combines an L1 reconstruction term with a second-order difference constraint.
  • Optimization: The segmentation generator is trained for 200,000 iterations with batch size 8, while the ALIAS generator uses batch size 4 for 200,000 iterations.The segmentation-generator learning rate is 0.0004, and the ALIAS-generator rates are 0.0001 for the generator and 0.0004 for the discriminator.
  • ALIAS Generator: The ALIAS generator passes the segmentation map and misalignment mask through ALIAS residual blocks to synthesize the final image.Its loss follows SPADE and pix2pixHD, combining conditional adversarial, feature matching, and perceptual losses.
  • Comparison with ClothFlow: At 1024×768, ClothFlow achieves better warping accuracy, but VITON-HD obtains higher Mask-SSIM and therefore more effective misalignment handling.ClothFlow’s warping module requires 130.03G MACs at 1024×768, while its cost can be reduced by predicting flow at 256×192.
  • User Study: In an unpaired user study, VITON-HD receives rank-1 votes above 88% for both photo-realism and preservation of target-clothing details.The study compares VITON-HD with CP-VTON and ACGPN using 30 reference-and-clothing sets.

B.3. Qualitative Results

This section presents additional qualitative results for VITON-HD, including comparisons with baselines across resolutions and 1024×768 examples.

  • Figures 16–19 qualitatively compare baseline methods across different resolutions.These results are presented to demonstrate the model’s capability for high-quality image synthesis.
  • Figures 20–23 show additional VITON-HD results at 1024×768 resolution.
  • The qualitative results cover both cross-resolution baseline comparisons and high-resolution VITON-HD examples.

C. Failure Cases and Limitations

The paper reports segmentation and collar-related failures, fading clothing-texture boundaries, and broader limitations tied to clothing presentation, dataset diversity, and in-the-wild images.

  • Failure Cases: Failure cases arise from inaccurately predicted segmentation maps and inner collar regions that are indistinguishable from other clothing regions.These errors are shown in Figure 15.
  • Failure Cases: Clothing-texture boundaries occasionally fade away in generated results.This is identified as an additional failure mode in Figure 15.
  • Model Limitations: Because VITON-HD preserves bottom clothing items, it limits presentation of whether target clothes are tucked in.The authors identify generating multiple outputs from one input pair as a future direction.
  • Model Limitations: The dataset mostly contains slim women and top-clothing images, limiting inference to a restricted range of body shapes and clothing.The authors expect broader coverage when images of diverse body shapes and clothing types are provided.
  • Model Limitations: VITON-HD, like existing virtual try-on methods, does not provide robust performance for in-the-wild images.The authors identify realistic in-the-wild try-on synthesis as future work.
Loading 2103.16874v2…