Source-linked AI summary
High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions
Sangyun Lee, Gyojung Gu, Sunghyun Park, Seunghwan Choi, Jaegul Choo
TL;DR
High-resolution virtual try-on suffers from misalignment and pixel-squeezing artifacts when clothing warping and segmentation generation operate separately. This paper unifies both processes and reports superior performance over existing methods at 1024×768 resolution.
Problem
Separate warping and segmentation modules cause misalignment and pixel-squeezing artifacts, degrading high-resolution virtual try-on results, especially when body parts occlude garments.
Method
The framework jointly deforms clothing and predicts the segmentation map, then uses discriminator rejection to filter incorrect segmentation predictions.
Results
At 1024×768 resolution, the method eliminates misalignment and pixel-squeezing artifacts while outperforming existing virtual try-on methods.
Takeaways & Limitations
The unified condition generator provides a high-resolution virtual try-on framework that handles garment occlusion and supports practical filtering of impaired segmentation results.
Takeaways & Limitations
Real-world use must account for out-of-distribution inputs, such as different camera views or images without humans, which can produce low-quality segmentation.
Abstract
from arXiv · showhide
Image-based virtual try-on aims to synthesize an image of a person wearing a given clothing item. To solve the task, the existing methods warp the clothing item to fit the person's body and generate the segmentation map of the person wearing the item before fusing the item with the person. However, when the warping and the segmentation generation stages operate individually without information exchange, the misalignment between the warped clothes and the segmentation map occurs, which leads to the artifacts in the final image. The information disconnection also causes excessive warping near the clothing regions occluded by the body parts, so-called pixel-squeezing artifacts. To settle the issues, we propose a novel try-on condition generator as a unified module of the two stages (i.e., warping and segmentation generation stages). A newly proposed feature fusion block in the condition generator implements the information exchange, and the condition generator does not create any misalignment or pixel-squeezing artifacts. We also introduce discriminator rejection that filters out the incorrect segmentation map predictions and assures the performance of virtual try-on frameworks. Experiments on a high-resolution dataset demonstrate that our model successfully handles the misalignment and occlusion, and significantly outperforms the baselines. Code is available at https://github.com/sangyun884/HR-VITON.
1 Soongsil University
At 1024×768 resolution, the proposed method outperforms VITON-HD [2] by handling both misalignment artifacts and occlusion-induced pixel-squeezing in try-on synthesis.
- At 1024×768, our method successfully handles misalignment and occlusion artifacts compared with VITON-HD [2].The comparison highlights both artifact types in try-on synthesis results.
- Misalignment between warped clothing and the segmentation map produces visible artifacts in VITON-HD [2].
- Body-part occlusion causes pixel-squeezing artifacts in VITON-HD [2].
1 Introduction
Image-based virtual try-on suffers from misalignment and pixel-squeezing artifacts because clothing warping and segmentation generation operate separately, especially under body-part occlusion. The proposed unified condition generator jointly produces aligned warped garments and segmentation maps, while discriminator rejection filters incorrect predictions and supports state-of-the-art performance on high-resolution data.
- Problem: Separate warping and segmentation modules create misalignment artifacts and excessive warping near garments occluded by body parts, limiting realistic poses and real-world applicability.Misalignment harms high-resolution perceptual quality, while information disconnection causes pixel-squeezing artifacts in occluded regions.
- Proposed Method: The unified try-on condition generator simultaneously predicts the warped garment and segmentation map, making them aligned and naturally handling body-part occlusions.The architecture removes the information disconnection between warping and segmentation generation.
- Results: The method achieves state-of-the-art performance on a high-resolution dataset while successfully handling misalignment and occlusion.The reported experiments evaluate the framework’s ability to address both identified artifact types.
- Proposed Method: Discriminator rejection filters incorrect segmentation-map predictions that would otherwise produce unnatural final results, assuring virtual try-on performance for real-world applications.The rejection mechanism is introduced as an additional safeguard for segmentation quality.
2 Related Work
Recent image-based virtual try-on methods typically separate segmentation generation, clothing warping, and image fusion, but misalignment between warped clothing and the person’s body still causes artifacts. Prior work includes normalization to alleviate misalignment and discriminator rejection sampling to improve generated-sample fidelity.
- Virtual Try-On Methods: Virtual try-on methods generally separate segmentation-map generation, clothing warping, and fusion into three modules.The fusion module synthesizes photo-realistic images from the generated conditions and warped clothing.
- Clothing Warping: Misalignment between warped clothes and the person’s body remains a source of artifacts, while VITON-HD normalization does not naturally fill misaligned regions with clothing texture.The paper therefore targets warped-clothing generation without misaligned regions.
- Rejection Sampling: Rejection-based methods filter low-quality generator outputs, including classifier-based rejection sampling, discriminator rejection sampling [1], and subsequent related approaches.Discriminator rejection sampling rejects generated samples at test time and can permit exact sampling from the data distribution under strict assumptions.
3 Proposed Method
HR-VITON unifies clothing warping and person-segmentation prediction in a try-on condition generator, then synthesizes the final image from their aligned outputs. It further uses body-part occlusion handling and discriminator rejection to suppress pixel-squeezing artifacts and incorrect segmentation maps.
- Proposed framework: The framework has a try-on condition generator that simultaneously deforms the clothing and predicts the segmentation map, followed by a try-on image generator.The final generator fuses the clothing-agnostic image, warped clothing, pose map, and predicted segmentation map.
- Try-on condition generator: The condition generator’s feature fusion blocks exchange information between clothing and segmentation pathways, enabling their mutual refinement and preventing misalignment.Two encoders feed a decoder whose fused features predict the segmentation map and appearance flow.
- Try-on condition generator: Body-part occlusion handling removes regions hidden by the body from the warped clothing and mask, eliminating pixel-squeezing artifacts.Condition aligning also removes non-overlapping clothing-mask regions to keep the warped clothing and segmentation map consistent.
- Try-on condition generator: The condition generator is trained end-to-end with cross-entropy, conditional GAN, L1, perceptual, and total-variation losses, while regularizing only the final appearance flow is vital for coarse-scale estimation.The losses supervise segmentation, clothing-mask alignment, appearance similarity, and flow smoothness.
- Discriminator rejection: At test time, discriminator rejection filters out incorrect segmentation maps by rejecting generated conditions whose acceptance probability falls below a threshold.The method constructs discriminator inputs from the segmentation map and the pose, agnostic-person, clothing, and clothing-mask conditions.
4 Experiments
Experiments on VITON-HD’s high-resolution dataset show that HR-VITON outperforms established baselines, while qualitative and ablation results support its condition-generation design. The evaluations also examine paired and unpaired settings, misalignment and occlusion handling, discriminator rejection, and comparisons with segmentation-free methods.
- Experimental Setup: On 13,679 frontal-view woman–top clothing pairs from VITON-HD, experiments use 11,647 training pairs and 2,032 test pairs at a native resolution of 1024×768.The study evaluates paired reconstruction and unpaired clothing-change settings using SSIM and LPIPS for paired evaluation, and FID and KID for unpaired evaluation.
- Comparison with Baselines: The proposed method produces more photo-realistic try-on images, preserves target-clothing details, generates natural necklines and body shapes, and provides higher-quality conditions than VITON-HD.Figures 4 and 5 support the qualitative comparison and show that condition quality is crucial for perceptually convincing synthesis.
- Additional Analysis: Discriminator rejection filters impaired segmentation maps caused mainly by preprocessing errors, while explicit segmentation helps distinguish regions to preserve or generate and enables better high-resolution clothing removal than PF-AFN.The rejected maps are considerably impaired, and PF-AFN fails to remove original clothing regions on the high-resolution dataset.
- Comparison with Baselines: HR-VITON outperforms CP-VTON, ACGPN, and VITON-HD on all evaluation metrics, especially at 1024×768, where CP-VTON and ACGPN cannot handle unpaired high-resolution try-on.Table 2 reports the quantitative baseline comparison, with KID values multiplied by 100; HR-VITON denotes the proposed model.
- Ablation Study: Feature fusion and condition aligning provide largely additive benefits, while removing both yields suboptimal results and demonstrates the necessity of information exchange between warping and segmentation generation.The ablation study evaluates the proposed components in the unpaired setting.
5 Discussion
The discussion identifies a limitation of image-based virtual try-on methods under out-of-distribution inputs. Discriminator rejection can filter low-quality segmentation predictions caused by camera-view shifts or inputs without humans.
- Limitation of Discriminator Rejection: Existing image-based virtual try-on approaches assume test images come from the same distribution as training images.
- Limitation of Discriminator Rejection: Real-world inputs may differ from training data because they use another camera view or contain no humans.
- Limitation of Discriminator Rejection: Discriminator rejection can filter low-quality segmentation predictions produced by such out-of-distribution inputs.
6 Conclusion
The proposed high-resolution virtual try-on architecture jointly performs clothing warping and segmentation generation while exchanging information between them. It eliminates misalignment and pixel-squeezing artifacts under body-part occlusion, while discriminator filtering removes impaired segmentation results.
- 6 Conclusion: The try-on condition generator jointly performs clothing warping and segmentation generation while exchanging information between the two processes.This unified architecture addresses the information disconnection between the stages.
- 6 Conclusion: The proposed generator eliminates misaligned regions and solves pixel-squeezing artifacts by handling occlusion from body parts.
- 6 Conclusion: The condition generator’s discriminator filters impaired segmentation results, providing a practical safeguard for virtual try-on.
APPENDIX · A. Implementation Details
The appendix specifies the architectures, training resolutions, objectives, and optimization settings for the try-on condition and image generators. It details feature fusion, multi-scale discrimination, loss design, and high-resolution inference preparation.
- A. Implementation Details: The appendix documents the detailed condition-generator architecture and its residual, scaling, and convolutional-channel conventions in Fig. 10.The figure defines ResBlock (n), Up/Down (f), and Conv (m) by output channels and scaling factors.
- A. Implementation Details: The try-on condition generator uses two five-block residual encoders and four feature fusion blocks to jointly produce flow and segmentation pathways.Encoder features generate the first flow map through concatenation and a 3×3 convolution, while segmentation features pass through two residual blocks.
- A. Implementation Details: The condition generator predicts ˆIc, ˆSc, and ˆS at 256×192 during training, then provides upscaled conditioning inputs for high-resolution synthesis.The segmentation map and appearance flow are upscaled to 1024×768 before entering the try-on image generator.
- A. Implementation Details: The condition-generator discriminator receives inputs downsampled by a factor of 2, uses dropout for training stability, and applies λCE=10, λV GG=10, and λT V=2.These settings increase the discriminator’s receptive field while stabilizing optimization and weighting the condition-generation losses.
- A. Implementation Details: The try-on image generator comprises residual blocks with upsampling layers, uses two multi-scale discriminators, and applies spectral normalization to every convolutional layer.Its detailed architecture is presented in Fig. 11.
- A. Implementation Details: The image-generator objective combines conditional adversarial, perceptual, and feature matching losses, following the losses used in SPADE and pix2pixHD.The formal objective weights these three loss terms through λTOIG, λVGG^TOIG, and λFM^TOIG.
- A. Implementation Details: The image generator uses learning rates of 0.0001 for the generator and 0.0004 for the discriminator, with Adam parameters β1=0.5 and β2=0.999.The appendix also specifies the relative loss weights and optimizer configuration for both modules.
B. Additional Experiments
Additional experiments show that the model outperforms VITON-HD* across all evaluation metrics at 1024×768, avoids misalignment and pixel-squeezing artifacts, and benefits from multi-scale losses for coarse-to-fine flow estimation. Qualitative studies further examine resolution changes, user preferences, and varied person–garment combinations.
- Additional Results: Additional qualitative experiments evaluate the model across different resolutions and combinations of different clothes and people.The paper presents resolution comparisons and 1024×768 synthesis examples across varied person–garment combinations.
- Comparison with the Variant of VITON-HD: The model consistently outperforms VITON-HD* on all evaluation metrics at 1024×768.The quantitative comparison is reported in Table 3, where KID is multiplied by 100.
- Comparison with the Variant of VITON-HD: Appearance-flow warping in VITON-HD* retains misalignment artifacts, while increased warping freedom worsens pixel-squeezing without occlusion handling.The qualitative comparison identifies misalignment and pixel-squeezing artifacts in VITON-HD* using green and red regions, respectively.
- Comparison with the Variant of VITON-HD: Our model successfully resolves both misalignment and pixel-squeezing artifacts in the qualitative comparison at 1024×768.This result appears in the third column of Fig. 14.
- Loss Analysis: Multi-scale L1/VGG losses enable meaningful coarse-scale intermediate flow estimation for coarse-to-fine appearance-flow generation.Without these losses, the model has difficulty learning flow estimation at a coarse scale.