PaperlayerView exact revision on arXiv

Source-linked AI summary

Street TryOn: Learning In-the-Wild Virtual Try-On from Unpaired Person Images

Aiyu Cui, Jay Mahajan, Viraj Shah, Preeti Gomathinayagam, Chang Liu, Svetlana Lazebnik

arXiv:2311.16094v3cs.CVcs.GR

TL;DR

Studio-trained try-on methods struggle with the varied poses, backgrounds, and lighting of in-the-wild images. StreetTryOn introduces a benchmark for these settings, and the proposed method combines pose estimation with inpainting; it outperforms existing methods on StreetTryOn and is competitive on VITON-HD.

  • Problem

    Studio-trained try-on methods assume standard poses, white backgrounds, and uniform lighting, unlike the varied poses, camera angles, backgrounds, and lighting of in-the-wild images.

    Source: §1 · p. 2

  • Method

    The method introduces StreetTryOn and uses pretrained pose estimation and inpainting networks, with DensePose warping correction for transferring garments to and from in-the-wild images.

    Source: §4.2 · p. 4, §6 · p. 8

  • Results

    The method outperforms all existing methods on StreetTryOn, is competitive on VITON-HD, and achieves similar Street2Street results with in-domain or out-of-domain training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    StreetTryOn covers Shop2Street and Street2Street alongside Shop2Model and Model2Model, establishing a suite of tasks with garment and person inputs from varied sources.

    Source: §1 · p. 2

    On street images, complex backgrounds make FID less able to reflect improvement in texture refinement.

    Source: §5.2.2 · p. 8

Abstract

from arXiv · show

Most virtual try-on research is motivated to serve the fashion business by generating images to demonstrate garments on studio models at a lower cost. However, virtual try-on should be a broader application that also allows customers to visualize garments on themselves using their own casual photos, known as in-the-wild try-on. Unfortunately, the existing methods, which achieve plausible results for studio try-on settings, perform poorly in the in-the-wild context. This is because these methods often require paired images (garment images paired with images of people wearing the same garment) for training. While such paired data is easy to collect from shopping websites for studio settings, it is difficult to obtain for in-the-wild scenes. In this work, we fill the gap by (1) introducing a StreetTryOn benchmark to support in-the-wild virtual try-on applications and (2) proposing a novel method to learn virtual try-on from a set of in-the-wild person images directly without requiring paired data. We tackle the unique challenges, including warping garments to more diverse human poses and rendering more complex backgrounds faithfully, by a novel DensePose warping correction method combined with diffusion-based conditional inpainting. Our experiments show competitive performance for standard studio try-on tasks and SOTA performance for street try-on and cross-domain try-on tasks.

1. Introduction

In-the-wild try-on aims to let people visualize garments on casual photos, but diverse poses, backgrounds, and scarce paired data make studio methods difficult to apply. The paper introduces StreetTryOn and an unpaired method that combines DensePose warping correction with diffusion inpainting, reporting strong street-task performance and competitive studio results.

1. Introduction

  • In-the-wild try-on targets casual photos so people can visualize how garments would look on their own bodies.The paper describes this application as having significant commercial potential and as a key research direction as advanced image generation models become available.Source: §1 · p. 1
  • Street images add pose, camera-angle, background, and lighting diversity, while paired training images are difficult to collect for these settings.Studio methods trained on standard poses, white backgrounds, and uniform lighting struggle with limb reconstruction, warping, and background rendering in the street domain.Source: §1 · p. 2
  • StreetTryOn establishes a benchmark for street try-on and evaluates Shop2Street and Street2Street alongside traditional Shop2Model and Model2Model tasks.The suite is intended to assess robustness and cross-domain generalization between studio and street images.Source: §1 · p. 2
  • The proposed unpaired method uses DensePose to estimate garment warps, then corrects them and uses diffusion inpainting to remove the old garment, inpaint skin, and composite the target garment.Training-time augmentation mimics DensePose registration errors, and the skin-inpainting module addresses limb reconstruction.Source: §1 · p. 2
  • The method outperforms existing methods on StreetTryOn, competes on VITON-HD, and achieves similar Street2Street results with in-domain or out-of-domain training.The evaluation is presented as evidence of robustness in the hardest try-on setting, Street2Street.Source: §1 · p. 2

2. Related Work

Prior try-on benchmarks and methods largely rely on paired studio images, while existing unpaired or street-oriented approaches have important constraints. The paper positions its approach as using pretrained diffusion inpainting and lightweight components to learn from unpaired data.

Virtual Try-On Benchmarks

  • Existing Shop2Model benchmarks use paired studio person and garment images, and none of the existing datasets are representative of in-the-wild try-on.SHHQ-1.0 includes at least 25% studio model images aggregated from DeepFashion and the African fashion dataset.Source: §2 · p. 2

Virtual Try-On Approaches

  • Most top-performing Shop2Model methods train on paired datasets, while paired-data Model2Model methods cannot be trained where such data is unavailable.PASTAGAN and PASTAGAN++ are noted as prior unpaired methods trained on UPT; experiments show PASTAGAN++ cannot handle complex street backgrounds.Source: §2 · p. 2
  • StyleGAN-like garment-style-code approaches can struggle to preserve garment details, while PASTAGAN++ uses patch-based representations but has difficulty with varied street backgrounds.Dressing-in-the-wild requires videos for both training and inference to learn the person representation.Source: §2 · pp. 2–3
  • Prior warping methods generally learn from paired data; Pose-with-style also uses DensePose but requires paired data to refine warping and rendering.Other approaches described include pyramid flow prediction, garment-person correlation, StyleGAN flow prediction, local flows, and implicit flows via cross-attention.Source: §2 · p. 3

Diffusion Models

  • Pretrained diffusion inpainting can be used with lightweight additional components and relatively little unpaired data, unlike specialized methods requiring paired training.StableVITON learns garment warping through U-Net cross-attention from paired data; the proposed method trains DensePose-conditioned ControlNets for garment removal and compositing.Source: §2 · p. 3

3. StreetTryOn Benchmark

StreetTryOn provides filtered street-person images and test tuples for street and cross-domain garment-transfer tasks. Combining these with VITON-HD test sets creates a broader suite that also includes conventional studio tasks.

3. StreetTryOn Benchmark

  • StreetTryOn filters DeepFashion2 with annotations, person detection, and manual selection, yielding 12,364 training and 2,089 test images.The filtering removes images unsuitable for try-on, including partial-body views, large occlusions, non-frontal views, and dark lighting.Source: §3 · p. 3
  • Street2Street uses shuffled unpaired tuples from 909 top images and 1,190 dress images, while Shop2Street and Model2Street each use 909 cross-domain test tuples.Combining these test sets with VITON-HD Shop2Model and Model2Model tests gives a comprehensive evaluation suite.Source: §3 · pp. 3–4

4. Our Try-On Method

The method combines separately trained components to predict a try-on parse, warp the target garment, reconstruct exposed skin, and refine the composite. Its design addresses pose and alignment errors while preserving the input person’s face and background.

4. Our Try-On Method

  • The inference pipeline predicts a try-on parse, corrects a DensePose-based garment warp, inpaints the original garment area, composites the result, and refines the garment boundary.The method’s four learnable components are trained separately without paired data.Source: §4 · p. 4

4.1. TryOn Parse Estimator

  • The parse estimator uses a StyleGAN architecture with DensePose initialization and outfit-segment codes; it excludes skin regions to avoid sleeve-shape bias and trains with masking and segment transforms.At inference, the top segment comes from the target garment, while other outfit segments come from the person image.Source: §4.1 · p. 4

4.2. Warping Correction Module

  • DensePose warping can misregister loose garments and leave unwrapped areas, so a trained correction module refines its initial flow.The corrector is trained without paired data using synthetically perturbed person images and DensePose estimates to reconstruct the person image.Source: §4.2 · pp. 4–5

4.3. Inpainting

  • The pipeline separately inpaints the removed garment area with skin before compositing, then refines the garment boundary with diffusion inpainting.The boundary mask is eroded because misalignments tend to concentrate there; the original face is copied back to avoid distortion from changes outside the mask.Source: §4.3 · p. 5

5. Experiments

The experiments evaluate try-on across street, cross-domain, and studio settings, and report strongest results on street-facing tasks while approaching—but not matching—specialized methods on Shop2Model. Ablations examine whether background rendering and individual method components account for performance differences.

5. Experiments

  • The evaluation uses FID throughout and also reports SSIM and LPIPS when paired test tuples are available; target-person resolution is 512 × 320 for StreetTryOn and 512×384 for VITON-HD.The comparative evaluation includes paired VITON-HD training, unpaired VITON-HD-person training, and unpaired StreetTryOn-person training.Source: §5 · p. 5, §5.1 · p. 6

5.1.1 Garment Transfer from Person Images.

  • Competing methods perform worse than ours in FID on street images, where they often produce warping artifacts and struggle with complex backgrounds.The comparison includes methods retrained for Model2Model and PASTAGAN++ trained on street images.Source: Table 1 · p. 6, §5.1.1 · p. 7

5.1.2 Garment Transfer from Shop Images.

  • Our method achieves the best performance on Shop2Street, while on Shop2Model it comes close to—but does not match—highly tuned paired-data methods.The comparisons cover Shop2Street and Shop2Model, respectively.Source: §5.1.2 · p. 7, Table 2 · p. 7

5.2.1 Background rendering is not the only challenge.

  • Even after factoring out background rendering, the best prior method does not match ours, indicating that background rendering is not the only street-try-on challenge.The ablation compares foreground-only, foreground blended with an inpainted background, and end-to-end inference.Source: §5.2.1 · p. 7, Figure 6 · p. 8

5.2.2 Effectiveness of each component.

  • Without skin inpainting, outputs remain biased toward the original garment shape; without warping correction, high-frequency textures such as stripes are not properly reconstructed.The component ablation also finds that directly generating garments without warping fails to reconstruct garment details.Source: §5.2.2 · pp. 7–8

6. Conclusion

The paper introduces the StreetTryOn benchmark and a try-on method that transfers garments to and from in-the-wild images. The method remains limited by absent relighting, inherited hand-generation artifacts, and inconsistent garment-detail preservation.

6. Conclusion

  • The method still cannot relight images, inherits Stable Diffusion hand-generation artifacts, and does not always preserve garment details.The authors note that more fine-tuning and better correction modules may alleviate many of these issues.Source: §6 · p. 8

Appendices

The appendix describes the source and filtering requirements for the StreetTryOn benchmark. DeepFashion2 images need to show a sufficiently visible, frontal upper body in bright conditions.

A. Street TryOn Benchmark Details

  • DeepFashion2 contributes 191,961 training and 32,153 validation in-the-wild fashion images across 13 clothing categories, with extensive image annotations.Annotations include scale, occlusion, zoom, viewpoint, category, style, bounding boxes, dense landmarks, and pixel masks.Source: A. Street TryOn Benchmark Details · p. 11
  • Virtual try-on filtering requires a frontal person image with at least the full upper body visible, without large occlusion, and in relatively bright lighting.The source images cannot be used directly because they do not consistently meet these requirements.Source: A. Street TryOn Benchmark Details · p. 11

A.1. Data Filtering Processes

StreetTryOn is built by filtering DeepFashion2 images for virtual try-on suitability, then cropping and resizing the detected person. The process yields 12,364 training and 2,089 validation images.

A.1. Data Filtering Processes

  • The first filtering stage keeps images labeled frontal viewpoint, no zoom-in, and slight occlusion, while excluding customer-sourced images.After this stage, 15,556 training and 2,401 test images remain; Figure 9 shows examples excluded as infeasible.Source: §A.1 · p. 11, Figure 9 · p. 11
  • The second stage uses DensePose detections to discard images without detected people, horizontal poses, or bounding boxes with aspect ratios above 5:8.The retained person crops are padded to a 5:16 aspect ratio and resized to 512 × 320.Source: §A.1 · p. 11
  • 12,364 training and 2,089 validation images remain after filtering and processing the DeepFashion2 images.The selected examples are shown in Figure 10.Source: §A.1 · p. 11, Figure 10 · p. 11

A.2. Data Statistics

The appendix describes StreetTryOn’s garment and test-attribute distributions, and explains why DeepFashion2 garment IDs cannot reliably produce paired images. It also presents cosine perturbation to mimic DensePose misalignment during training without paired data.

A.2. Data Statistics

  • The benchmark includes diverse garment categories, and its test images have six manually labeled attributes for future difficulty-level divisions.The labels cover full-body and frontal views, arm placement, torso occlusion, watermarks, and padding; Figures 11 and 12 show distributions.Source: §A.2 · p. 11, Figure 11 · p. 12, Figure 12 · p. 12
  • DeepFashion2 garment IDs are too noisy for directly constructing try-on pairs because color variations are not distinguished, and 14% of filtered images have no same-ID image.Building paired images would require additional manual annotation.Source: §A.2 · p. 12, Figure 13 · p. 12

B. DensePose Perturbation with Cosine Noise

  • Cosine noise is added to DensePose pixel coordinates during training to mimic imperfect alignment at inference without requiring paired data.The perturbation uses randomly sampled coefficients for each UV map and simulates the misalignment that distorts garment warping.Source: B. DensePose Perturbation with Cosine… · p. 12

C. Additional Implementation Details

The appendix specifies diffusion-inpainting settings and garment DensePose preprocessing, then presents additional visual examples for studio and street try-on.

C.1. Pre-trained Diffusion Models

  • Both diffusion inpainters use 20 inference steps and guidance scale 7.5, with negative prompts tailored to skin generation and refinement.The skin prompt excludes clothing terms; the refinement prompt lists image artifacts and anatomy errors.Source: §C.1 · pp. 12–13
  • The inpainter starts denoising from concatenated noise, a latent masked image, and a mask; its noise uses predicted-parse class mean features with timestep 999.The prose describes this as a replacement for directly using Gaussian noise.Source: §C.1 · p. 13

C.2. Garment DensePose Detection

  • Garment DensePose is detected using Cui et al.’s [4] algorithm at 256 × 192 resolution, then resized to the desired resolutions.Figure 15 shows detected garment DensePose examples.Source: Figure 15 · p. 13, §C.2 · p. 14

D.1. Shop2Model Test

  • Additional Shop2Model examples on VITON-HD compare the method with PFAFN [11], FS-VTON [17], SDAFN, and GP-VTON [33].Figures 16 and 17 show the examples; the passage does not state a comparative outcome.Source: §D.1 · p. 14

D.2. Street2Street Test

  • Additional Street2Street examples show top and dress try-on, including intermediate outputs.Figures 18 and 19 contain the examples.Source: §D.2 · p. 14

D.3. Ablation Study

The appendix adds visual examples for the component ablation and for studio and street try-on; it does not report specific ablation outcomes in these passages.

D.3. Ablation Study

  • Figure 20 provides more ablation examples intended to validate the effectiveness of each component in the proposed design.The passages do not identify individual components or report specific outcomes.Source: §D.3 · p. 14, Figure 20 · p. 19
  • Figures 16 and 17 provide additional Shop2Model test examples on the VITON-HD benchmark.The captions identify the benchmark and test setting but do not state comparative results.Source: Figure 16 · p. 15, Figure 17 · p. 16
  • Figures 18 and 19 provide additional Street2Street examples for top and dress try-on, respectively.The captions identify the try-on categories but do not state comparative outcomes.Source: Figure 18 · p. 17, Figure 19 · p. 18

Related papers

Loading 2311.16094v3…