Source-linked AI summary
Parser-Free Virtual Try-on via Distilling Appearance Flows
Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, Ping Luo
TL;DR
Human-parsing errors can create artifacts, and distilling parser-based try-on outputs limits a parser-free student to the teacher’s image quality. PF-AFN uses teacher-tutor-student distillation with real-image supervision and appearance-flow transfer, and reports highly photo-realistic results without human parsing. Its key limitation is that the earlier parser-based teacher can produce artifacts that bound student quality.
Problem
Inaccurate human parsing can create unrealistic try-on artifacts, while directly distilling parser-based outputs bounds a parser-free student by teacher image quality.
Method
PF-AFN treats parser-based fake images as tutor inputs, supervises the parser-free student with real person images, and distills appearance flows for dense garment-person correspondences.
Results
PF-AFN is reported to produce highly photo-realistic try-on images without human parsing and to outperform prior parser-free and state-of-the-art approaches.
Takeaways & Limitations
Real-image supervision and appearance-flow distillation support parser-free try-on while preserving accurate garment-person correspondences and image quality.
Takeaways & Limitations
The earlier parser-based teacher can produce large artifacts, which bound the student’s image quality when its outputs are used as direct supervision.
Abstract
from arXiv · showhide
Image virtual try-on aims to fit a garment image (target clothes) to a person image. Prior methods are heavily based on human parsing. However, slightly-wrong segmentation results would lead to unrealistic try-on images with large artifacts. Inaccurate parsing misleads parser-based methods to produce visually unrealistic results where artifacts usually occur. A recent pioneering work employed knowledge distillation to reduce the dependency of human parsing, where the try-on images produced by a parser-based method are used as supervisions to train a "student" network without relying on segmentation, making the student mimic the try-on ability of the parser-based model. However, the image quality of the student is bounded by the parser-based model. To address this problem, we propose a novel approach, "teacher-tutor-student" knowledge distillation, which is able to produce highly photo-realistic images without human parsing, possessing several appealing advantages compared to prior arts. (1) Unlike existing work, our approach treats the fake images produced by the parser-based method as "tutor knowledge", where the artifacts can be corrected by real "teacher knowledge", which is extracted from the real person images in a self-supervised way. (2) Other than using real images as supervisions, we formulate knowledge distillation in the try-on problem as distilling the appearance flows between the person image and the garment image, enabling us to find accurate dense correspondences between them to produce high-quality results. (3) Extensive evaluations show large superiority of our method (see Fig. 1).
1. Introduction
PF-AFN removes human-parsing inputs through teacher-tutor-student distillation, using real person images and appearance flows to improve parser-free try-on quality. The method is reported to outperform prior parser-free counterparts and state-of-the-art approaches.
- Inaccurate human segmentation can produce highly unrealistic try-on images with large artifacts, motivating parser-free methods.Prior virtual try-on systems commonly rely on masks for body parts such as the upper body, lower body, arms, face, and hair.
- WUTON trains a parser-free student to mimic fake try-on images from a parser-based teacher, so student quality remains bounded by teacher artifacts.The teacher-generated images can contain large artifacts that yield unsatisfactory student results.
- PF-AFN instead treats parser-based outputs as tutor knowledge and supervises the parser-free student with the original real person image.This teacher-tutor-student scheme uses the real image as self-supervision to correct artifacts in the tutor output.
- PF-AFN is reported to significantly outperform previous parser-free counterparts and to show large qualitative and quantitative superiority over state-of-the-art approaches.The contribution statement attributes this evaluation to experiments on popular datasets.
- Appearance-flow distillation helps PF-AFN find accurate dense correspondences between garments and persons for high-quality image generation.The formulation distills appearance flows between the person image and garment image rather than only transferring generated try-on images.
2. Related Work
Related work covers 2D and 3D virtual try-on, appearance flow, and knowledge distillation. PF-AFN is situated among 2D methods that avoid the measurements and computational demands of 3D approaches while addressing parsing dependence.
- Virtual Try-on: Because available 2D try-on datasets contain unpaired clothes and person images, earlier methods mainly mask clothing regions and reconstruct the person image.These approaches require accurate human parsing, and inaccurate parsing can produce visually terrible images with noticeable artifacts.
- Appearance Flow: Appearance flow denotes 2D coordinate vectors identifying source pixels used to synthesize each target pixel.The passage notes that flow estimation is limited on non-rigid clothing regions with large deformation.
- Appearance Flow: Prior 3D appearance-flow work synthesized person images in target poses by fitting a 3D model, but those flows are unavailable in 2D try-on.This distinguishes the supervision setting of prior 3D methods from 2D image-based try-on.
- Knowledge Distillation: Knowledge distillation transfers intrinsic information from a teacher network to train a student network and has also been extended to cross-modality transfer.In cross-modality transfer, a model using superior modalities supervises another model using weaker modalities.
3. Proposed Approach
PF-AFN combines a parser-based tutor, real-image teacher supervision, and appearance-flow distillation to generate parser-free try-on images. Its warping module progressively estimates dense correspondences while a smoothness constraint preserves garment details.
- Network Training: PF-AFN uses teacher-tutor-student distillation, treating parser-based outputs as tutor knowledge and real person images as teacher knowledge for student training.The student is trained with tutor-generated try-on images and corrected toward the original real image.
- Appearance Flow Warping Module (AFWM).: Both networks use AFWM to estimate dense 2D appearance flows that specify which garment pixels fill person-image locations.PFEN extracts pyramid features, while AFEN progressively refines flows across levels.
- Appearance Flow Warping Module (AFWM).: Progressive flow refinement captures long-range clothing-person correspondences, enabling the model to handle large misalignment and deformation.AFEN cascades flow estimation networks from coarse to fine pyramid levels.
- Appearance Flow Warping Module (AFWM).: The second-order smooth constraint encourages neighboring flows to remain co-linear, helping preserve tightly arranged clothing text and repeated patterns.The constraint is designed to reduce distortions in logos, stripes, and other garment characteristics.
- Appearance Flow Warping Module (AFWM).: Adding the smooth constraint retains target-clothing details such as stripes and characters without distortion.The comparison in Fig. 4(a) directly illustrates the visual effect of the constraint.
- Generative Module (GM): The generative modules synthesize try-on images from warped clothes and preserved person information, using pixel-wise and perceptual losses against the real image.PF-GM uses warped clothes and the tutor image, while training minimizes a weighted combination of L1, perceptual, and smoothness losses.
4. Experiments
Experiments evaluate PF-AFN across VITON, VITON-HD, and MPV using qualitative comparisons, FID, user studies, and ablations. PF-AFN produces realistic parser-free try-on results, outperforming parser-based and prior parser-free methods while its adjustable distillation and AFEN improve performance.
- Datasets and setup: Experiments use VITON, VITON-HD, and MPV, with parser-free testing inputs consisting of a reference person image and target clothes image.VITON and MPV use 256 × 192 images, while VITON-HD uses 512 × 384 images.
- Qualitative results: PF-AFN avoids artifacts seen in parser-based baselines, including broken sleeves, distorted embroideries, messy lower-body clothes, fake arms, and blurred hands.The baselines are affected by segmentation errors and limited warping flexibility.
- Qualitative results: PF-AFN handles complex postures and large clothes-person misalignment while preserving target-clothing characteristics, non-target clothes, and clear body parts.Its second-order smooth constraint supports long-range correspondences and helps avoid logo and embroidery distortion.
- Quantitative results: PF-AFN outperforms parser-based methods on VITON and WUTON on MPV according to FID, where lower scores indicate higher image quality.Table 1 reports FID comparisons across CP-VTON, ClothFlow, CP-VTON+, ACGPN, WUTON, and PF-AFN.
- Ablation study: Adjustable distillation improves FID from 11.40 without distillation and 10.86 with fixed distillation to 10.09.Only accurate feature representations and predicted flows from the parser-based network guide the parser-free student.
- Ablation study: The complete AFEN achieves the best FID among ablated variants, while removing refinement, correlation, or cascaded modules worsens results.A simple encoder-decoder produces unsatisfactory appearance flows for clothes warping.
5. Conclusion
The paper introduces teacher-tutor-student distillation for photo-realistic virtual try-on without human parsing. It uses real person images and appearance-flow distillation to improve dense correspondence and image generation, with evaluations showing superiority over state-of-the-art methods.
- Conclusion: PF-AFN treats parser-based outputs as tutor knowledge and uses original real person images as self-supervised teacher knowledge.This design aims to correct artifacts in fake tutor images while removing human parsing from the parser-free model.
- Conclusion: Appearance-flow distillation finds accurate dense correspondences between person and clothing images for high-quality generation.The method distills flows in addition to using real images as supervision.
- Conclusion: Extensive evaluations show the approach is superior to state-of-the-art methods for photo-realistic try-on without human parsing.The conclusion states the overall advantage without specifying a single benchmark value.