Source-linked AI summary
VITON: An Image-based Virtual Try-on Network
Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, Larry S. Davis
TL;DR
VITON tackles realistic virtual try-on without 3D information, using only RGB images to transfer a target garment onto a person. It combines clothing-agnostic person features, coarse synthesis, mask-guided warping, and learned refinement, achieving promising qualitative and quantitative results and outperforming state-of-the-art generative models in user evaluation. The method remains constrained by artifacts near neck and collar regions caused by occlusion handling and human-parser segmentation.
Problem
Existing virtual try-on systems rely on costly 3D measurements or annotated data, motivating realistic clothing transfer from plain RGB images.
Method
VITON uses a clothing-agnostic person representation, a multi-task encoder-decoder for coarse image and mask generation, and a refinement network for mask-guided clothing composition.
Results
VITON achieves promising quantitative and qualitative results on a newly collected dataset and generates more realistic, appealing virtual try-on results than state-of-the-art generative models in a user study.
Takeaways & Limitations
The experiments indicate that 2D image-based synthesis can serve as an alternative to expensive 3D-based virtual try-on methods.
Takeaways & Limitations
Artifacts may occur near neck and collar regions because warping includes occluded clothing areas and the human parser treats neck and collar regions as background.
Abstract
from arXiv · showhide
We present an image-based VIirtual Try-On Network (VITON) without using 3D information in any form, which seamlessly transfers a desired clothing item onto the corresponding region of a person using a coarse-to-fine strategy. Conditioned upon a new clothing-agnostic yet descriptive person representation, our framework first generates a coarse synthesized image with the target clothing item overlaid on that same person in the same pose. We further enhance the initial blurry clothing area with a refinement network. The network is trained to learn how much detail to utilize from the target clothing item, and where to apply to the person in order to synthesize a photo-realistic image in which the target item deforms naturally with clear visual patterns. Experiments on our newly collected Zalando dataset demonstrate its promise in the image-based virtual try-on task over state-of-the-art generative models.
1. Introduction
VITON addresses the need for realistic online apparel previews without costly 3D measurements by transferring target clothing onto a person using only 2D RGB images. Its coarse-to-fine design combines a clothing-agnostic representation, coarse synthesis, masking, and refinement to preserve pose, natural deformation, and clothing details.
- Motivation: Online apparel shopping creates demand for virtual try-on, but existing systems often depend on costly 3D body measurements or annotated data.These measurements may come from depth cameras or be inferred from 2D images, while hardware and annotation costs inhibit large-scale deployment.
- Approach: VITON transfers a target clothing item from a product image onto the corresponding region of a clothed person using only 2D RGB images.The synthesized image is intended to retain the person’s body parts and pose while making the clothing deform naturally and preserving detailed visual patterns.
- Approach: VITON uses a clothing-agnostic person representation and a multi-task encoder-decoder to generate a coarse clothed-person image and a clothing-region mask.The mask guides subsequent warping of the target clothing item to account for deformations.
- Approach: A refinement network learns how to composite the warped clothing item with the coarse image, including natural deformations and detailed visual patterns.This coarse-to-fine structure is summarized as an encoder-decoder generator stage followed by a refinement stage.
- Related Work: Prior fashion-analysis and image-synthesis work addresses related tasks, but VITON focuses on precise clothing replacement in a reference image using 2D inputs.The paper distinguishes this task from methods that modify attributes, generate people regardless of pose, or use text to specify fashion items.
3. VITON
VITON uses a clothing-agnostic person representation and a coarse-to-fine pipeline to transfer target clothing while preserving the person’s pose, body shape, and identity. A refinement network warps clothing details and learns where to combine them with the coarse synthesis for more natural results.
- 3.1. Clothing-Agnostic Person Representation: Pose and body shape are both necessary for plausible synthesis: omitting pose obscures or misplaces hands, while body shape supports fitting and can help preserve pose.These observations include examples involving plus-size clothing and cases where body shape and pose benefit from each other.
- 3.1. Clothing-Agnostic Person Representation: VITON represents a person using pose, body shape, and face-and-hair features, concatenated into a 22-channel clothing-agnostic representation.The representation has spatial dimensions 256 × 192 and preserves information needed to model person-specific relations.
- 3.2. Multi-task Encoder-Decoder Generator: The encoder-decoder takes the target clothing image and person representation as input, producing a coarse synthesized image and a clothing-region mask.Its multi-task objective combines perceptual image reconstruction with L1 regression of the predicted clothing mask toward the parser-derived mask.
- 3.3. Refinement Network: Because coarse synthesis loses clothing graphics and textures, VITON warps the target clothing with a thin plate spline estimated by shape context matching.The warp uses the target clothing foreground mask and the predicted clothing mask of the person.
- 3.3. Refinement Network: The refinement network predicts a composition mask that controls how much information comes from the warped clothing versus the coarse image.A higher-layer VGG19 perceptual loss tolerates small warping displacements, while the negative L1 and total-variation terms encourage detail use and spatially smooth transitions.
4. Experiments
Experiments use a newly collected tops dataset and compare VITON with generative, refinement, encoder-decoder, and warping baselines. Qualitative studies, representation ablations, in-the-wild tests, and a user study show strong results while exposing failures on rare poses and major clothing-shape mismatches.
- Dataset: The dataset contains 16,253 person–clothing pairs, split into 14,221 training and 2,032 testing pairs, with shuffled target garments during testing.The study focuses on tops, while the authors state the method can also be trained for other clothing categories.
- Baselines: VITON is compared against PRGAN, CAGAN, CRN, an encoder-decoder variant, and non-parametric warped synthesis under matched image-generation settings.The comparisons isolate adversarial modeling, refinement, and learned composition from direct garment warping.
- Qualitative Results: VITON generates detailed and seamless virtual try-on results, whereas competing methods produce blur, artifacts, or unnatural pasted clothing in qualitative comparisons.Adversarial methods sharpen edges but introduce artifacts, while direct warping ignores inconsistencies between original and target garments.
- Representation Analysis: Body shape is critical for adjusting garment size, while body shape alone cannot resolve occlusion and pose ambiguity in complicated poses.Removing pose and body shape individually shows that the full clothing-agnostic representation is more comprehensive and effective than prior alternatives.
- Limitations and In-the-Wild Results: Failure cases arise from rarely seen poses, large mismatches between current and target clothing shapes, and artifacts near the neckline.In-the-wild COCO results suggest potential applications in generating people in clothing.
- User Study: Human preferences favor person-representation-guided methods, and VITON benefits from compositing coarse results with warped clothing rather than relying on either component alone.The study also finds that Inception Score is unsuitable for virtual try-on because it does not measure natural clothing transfer or pose preservation.
5. Conclusion
VITON transfers clothing from product images using only RGB inputs through a coarse-to-fine pipeline. Its coarse synthesis, refinement network, and dataset experiments achieve promising quantitative and qualitative results as an alternative to expensive 3D methods.
- VITON transfers a clothing item from a product image to a person using only RGB images.
- A multi-task encoder-decoder first generates a coarse result from a detailed clothing-agnostic person representation.
- A refinement network enhances the coarse result by learning the optimal composition.
- Experiments on a newly collected dataset achieve promising results both quantitatively and qualitatively.
- The results indicate that the 2D image-based synthesis pipeline can serve as an alternative to expensive 3D-based methods.
Detailed Network Structures
The supplementary figures detail VITON’s encoder-decoder generator and refinement network. They specify the convolutional operations, activation slope, and generator dropout used in these structures.
- Figure 1 shows the encoder-decoder generator, with blue encoding layers and green decoding layers.
- The generator uses 4 × 4 convolutions with stride 2 for convolution and stride 1/2 for deconvolution.
- The generator uses Leaky ReLU with negative slope 0.2 and dropout probability 0.5.
- Figure 2 shows the refinement network, whose convolutions are 3 × 3 with stride 1.
- The refinement network also uses Leaky ReLU with negative slope 0.2.
Person Representation Analysis
The person-representation analysis shows that pose and body shape materially affect virtual try-on quality. Pose is especially important for placing arms and hands, while body shape supports realistic fitting and can help preserve pose.
- Person Representation without Pose: Removing pose produces unsatisfactory coarse try-on results because the model cannot determine where arms and hands should occur.Without pose, hands may disappear and forearms become ambiguous because they can lie in front of, behind, or beside the body.
- Person Representation without Body Shape: Body shape is essential for generating good synthesized results, including virtual try-on of plus-size clothing.
- Person Representation without Body Shape: Body shape can help preserve a person’s pose, suggesting that body shape and pose benefit from each other.
- User Study: 67.6% of user-study trials preferred the full representation over the representation without body shape.
- User Study: 77.4% of user-study trials preferred the full representation over the representation without pose.
More Qualitative Results
Additional qualitative comparisons examine VITON against alternative methods and show multiple virtual try-on results. The examples include repeated trials in which one person wears different target clothing items.
- The supplementary qualitative comparisons support the same conclusions reported in the main paper.
- Figure 5 presents qualitative comparisons between VITON and other methods.
- Figure 6 presents virtual try-on results in which each row shows one person trying on different target clothing items.
Artifacts near Neck
Neck-region artifacts arise because the refinement stage warps clothing areas that should be occluded, especially neck tags and inner collars. Segmenting and removing these regions before warping eliminates the unwanted artifacts, while augmenting the parser addresses collar-style mismatches.
- Artifact cause: Neck artifacts stem from warping the whole clothing foreground, including neck tags and inner collars that should be occluded.These regions are therefore inappropriate inputs for the refined result.
- Artifact removal: A segmentation model removes neck regions from target clothing images before warping to eliminate the resulting artifacts.The FCN is trained using annotations for 2,000 training images.
- Artifact removal: The FCN-based correction removes unwanted neck areas and eliminates artifacts near the neck in the illustrated examples.
- Collar correction: Augmenting the human parser with neck annotations and retraining addresses discrepancies between target and predicted collar style and shape.The original parser treated neck and collar regions as background, causing the original collar style to persist.
Keep the original pants regions
VITON preserves original pants regions by segmenting the legs and regenerating them for seamless overlays, though artifacts can remain near the waist. The method also addresses collar regions through human parsing and handles whole-image generation without masks.
- Keep the original pants regions: Keeping pixels outside the clothing mask can create gaps when the target item is shorter, so VITON regenerates the legs for a seamless overlay.
- Keep the original pants regions: The human parser is augmented to correct neck segmentation and resolve inconsistencies between target and predicted collar styles.
- Keep the original pants regions: VITON segments the leg region and uses it as input while preserving the original pants, although artifacts remain near the waist.
- Keep the original pants regions: The method treats pants regions similarly to face and hair to retain the original pants regions.
More results
Additional refined VITON results show the updated method removing neck-region artifacts and inconsistencies while preserving original leg regions. The examples include both the visual try-on dataset and in-the-wild COCO images.
- More results: Updated refined VITON results remove artifacts and collar inconsistencies near the neck while keeping the original leg regions.
- More results: The displayed examples comprise two top rows from the visual try-on dataset and a final row of in-the-wild COCO results.