PaperlayerView exact revision on arXiv

Source-linked AI summary

WildVidFit: Video Virtual Try-On in the Wild via Image-Based Controlled Diffusion Models

Zijian He, Peixin Chen, Guangrun Wang, Guanbin Li, Philip H. S. Torr, Liang Lin

arXiv:2407.10625v1cs.CV

TL;DR

Video virtual try-on must preserve garment appearance while adapting to people’s poses, but existing approaches face data, computation, and occlusion challenges. WildVidFit uses image-based controlled diffusion and pretrained-model guidance for temporal coherence. Its TikTok experiments demonstrate effectiveness in dynamic, real-world scenarios.

  • Problem

    Video try-on must preserve a specific garment’s appearance while adapting to the source person’s pose and body shape, but existing methods face costly video data and computation and misalignment from occlusions and garment deformation.

    Source: §1 · p. 2

  • Method

    WildVidFit uses a one-stage image-based controlled diffusion model conditioned on garment and person, with diffusion guidance from pretrained models for temporal coherence.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    WildVidFit experiments on TikTok demonstrate effectiveness in dynamic, real-world scenarios, highlighting practicality and versatility.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The method enables video try-on within a predominantly image-based framework while achieving inter-frame consistency.

    Source: §5 · p. 14

    The one-stage network fails to reproduce tiny, complex garment patterns, and inaccurate parsing can produce unreasonable garment masks and imperfect restoration.

    Source: B Failure Cases and Limitations · p. 21

Abstract

from arXiv · show

Video virtual try-on aims to generate realistic sequences that maintain garment identity and adapt to a person's pose and body shape in source videos. Traditional image-based methods, relying on warping and blending, struggle with complex human movements and occlusions, limiting their effectiveness in video try-on applications. Moreover, video-based models require extensive, high-quality data and substantial computational resources. To tackle these issues, we reconceptualize video try-on as a process of generating videos conditioned on garment descriptions and human motion. Our solution, WildVidFit, employs image-based controlled diffusion models for a streamlined, one-stage approach. This model, conditioned on specific garments and individuals, is trained on still images rather than videos. It leverages diffusion guidance from pre-trained models including a video masked autoencoder for segment smoothness improvement and a self-supervised model for feature alignment of adjacent frame in the latent space. This integration markedly boosts the model's ability to maintain temporal coherence, enabling more effective video try-on within an image-based framework. Our experiments on the VITON-HD and DressCode datasets, along with tests on the VVT and TikTok datasets, demonstrate WildVidFit's capability to generate fluid and coherent videos. The project page website is at wildvidfit-project.github.io.

1 Introduction

WildVidFit targets in-the-wild video try-on challenges: costly video data and computation, limited generalization, and misalignment from occlusions and garment deformation. It uses a one-stage image-based diffusion model and pretrained-model guidance for temporal coherence; TikTok experiments demonstrate effectiveness in dynamic, real-world scenarios.

1 Introduction

  • Existing video try-on methods face costly, high-quality video-data and computational requirements, limited generalization, and misalignment from limb occlusions and garment deformation.Source: §1 · p. 2
  • The approach separates fine-grained image try-on for complex movement and occlusion from extending results to video with frame coherence.Source: §1 · p. 2
  • WildVidFit is a video-training-free virtual try-on framework using image-based controlled diffusion models for realistic video try-on.Source: §1 · p. 2
  • Diffusion guidance improves temporal consistency using pretrained video models and image self-supervised models to establish latent-space correspondence between frame features.Source: §1 · p. 3
  • WildVidFit experiments on TikTok demonstrate effectiveness in dynamic, real-world scenarios, highlighting practicality and versatility.Source: §1 · p. 3

2 Related Work

Prior image try-on methods commonly use warping and blending, while diffusion models offer an alternative for generation and garment-detail preservation. Video methods add temporal modules, but wild movements expose challenges associated with labeled video data and explicit warping.

2 Related Work

  • Image try-on methods commonly decompose generation into warping and blending; VITON uses TPS warping, while ClothFlow estimates flow with a neural network.Source: §2 · p. 3
  • LaDI-VTON [29] and DCI-VTON [12] incorporate diffusion into two-stage try-on, whereas TryOnDiffusion preserves garment details and handles pose and body changes in one network.TryOnDiffusion's two parallel UNets and super-resolution module bring substantial computational cost when extending it to video synthesis.Source: §2 · pp. 3–4
  • Video methods FW-GAN, MVTON, and ClothFormer use temporal or memory modules, but complex wild-video movements remain challenging amid costly labeled data and explicit-warping limitations.Source: §2 · p. 4

3 Method

WildVidFit treats video try-on as conditional image generation using person and garment representations, with diffusion guidance to improve temporal coherence. Its one-stage network handles garment fitting, while additional modules support longer clips and preserve details outside the garment region.

3 Method

  • WildVidFit generates a try-on sequence from a person video and target garment using a one-stage network conditioned on human motion and garment texture, plus diffusion guidance for temporal coherence.Preprocessing forms person representations from cloth-agnostic images and pose maps, and garment representations from the garment image and edge map.Source: §3 · pp. 4–5

3.1 One-stage Image Try-On Network

  • The one-stage network conditions on cloth-agnostic images and pose maps, and uses garment-image cross-attention for implicit warping rather than explicit optical-flow estimation.The garment edge map emphasizes details to preserve, while hierarchical feature aggregation occurs in the UNet decoder.Source: §3.1 · p. 6

3.2 Temporal Coherent Editing using Diffusion Guidance

  • VideoMAE reconstruction loss and DINO-V2 feature similarity guide iterative denoising to improve clip smoothness and limit feature distances between adjacent frames.The losses guide sampling without fine-tuning the diffusion network; the implementation computes them only on the garment area because of memory limits.Source: §3.2 · pp. 6–8
  • Longer videos are generated from overlapping short clips, averaging the latent representations of clips that include each frame during temporal co-denoising.Clip strides are typically L//2 or L//4; this strategy is used to ensure temporal smoothness.Source: §3.2 · p. 8

3.3 Other module for Enhanced Performance

  • The autoencoder uses mask-aware skip connections to add encoder features in unmasked regions, and fully cross-frame attention replaces self-attention during sequential inference.The skip connection uses the resized garment mask; cross-frame attention operates over all latent frames in a clip.Source: §3.3 · p. 8

4 Experiments

Experiments evaluate image try-on on VITON-HD and DressCode, video try-on on VVT and TikTok, and the contributions of edge maps, guidance scale, and temporal modules. WildVidFit leads the reported image-quality comparisons and outpaces image-based video methods while matching ClothFormer on VVT.

4 Experiments

  • The experiments cover image try-on on VITON-HD and DressCode, video try-on on VVT and TikTok, and ablations of edge maps, guidance scale, and temporal modules.The setup uses VITON-HD, DressCode-Upper, VVT, and TikTok data; it trains the UNet decoder and condition branch while keeping the encoder frozen.Source: §4 · p. 9, §4.1 · p. 9

4.2 Image Try-on Results

  • WildVidFit achieves state-of-the-art image visual quality under paired and unpaired evaluation on VITON-HD and DressCode.The qualitative comparison also highlights clothing-characteristic and texture retention, including garments with intricate folds.Source: §4.2 · pp. 10–11, Figure 4 · p. 10, Table 1 · p. 11

4.3 Video Try-On Results on VVT Dataset

  • WildVidFit outpaces image-based approaches and matches video-based ClothFormer on the VVT video-try-on comparison.The evaluation uses VFID on 36-frame clips and a user survey.Source: §4.3 · p. 11, Table 3 · p. 12

4.4 In-the-Wild Video Virtual Try-On

  • On TikTok dance videos, WildVidFit transfers DressCode and VITON-HD garments to TikTok reference videos after joint training on all three datasets.The figure presents DressCode garments in the first row and VITON-HD garments in the second.Source: Figure 5 · p. 12, §4.4 · p. 12
  • In the TikTok comparison, HR-VTON fails and LaDI-VTON performs poorly under limb occlusion, while WildVidFit reproduces garment details through motion.The authors attribute LaDI-VTON’s difficulty primarily to warping challenges.Source: §4.4 · pp. 12–13, Figure 6 · p. 13

4.5 Ablation Study

  • With edge maps introduced through cross-attention and garment guidance scale set to 2, the method reports KID 8.67 and FID 0.10.The ablation reports improvement from adding edge maps.Source: §4.5 · p. 13, Table 4 · p. 14
  • Sequentially adding cross-frame attention, LMAE guidance, LSIM guidance, and co-denoising improves the direct image-sequence baseline; diffusion guidance also smooths videos and rectifies image inconsistencies.The figure illustrates garment flaws without diffusion guidance.Source: Figure 7 · p. 13, Table 5 · p. 14, §4.5 · p. 14

5 Conclusions

WildVidFit is an image-based virtual try-on framework designed for frequent movement and significant occlusions in wild videos. Experiments report state-of-the-art image try-on performance and progress on video try-on in the wild.

5 Conclusions

  • WildVidFit uses a one-stage image diffusion model conditioned on garment and person, with diffusion guidance for inter-frame consistency in video try-on.The conclusion attributes robust performance to training on many image pairs.Source: §5 · p. 14

APPENDIX

The appendix supplements the main text with additional visual results and a discussion of failure cases and limitations.

APPENDIX

  • The appendix includes further image and video try-on visualizations, TikTok results, and a discussion of failure cases and method limitations.The listed visualizations include DressCode image try-on and VVT video try-on comparisons.Source: APPENDIX · p. 19

A.1 Image Try-on Results on DressCode

The qualitative results show strong garment-detail reproduction on DressCode and robust try-on across varied poses in video comparisons.

A.1 Image Try-on Results on DressCode

  • WildVidFit handles complex garment designs and significantly surpasses other methods in the reported qualitative comparison.The passage highlights multi-layered wrinkled sleeves and lace garments.Source: §A.1 · p. 19
  • WildVidFit preserves striped-garment textures, while other methods produce blurred outcomes.Source: §A.1 · p. 19
  • On VVT, WildVidFit is robust across poses and forms coherent video sequences compared with ClothFormer [22], HR-VOTN [25], and LaDI-VTON [29].Source: Figure 8 · p. 19

A.2 Qualitative Comparison on VVT Dataset

On VVT, WildVidFit is robust to varied poses and performs on par with ClothFormer in temporal consistency, but has difficulty with very fine clothing details.

A.2 Qualitative Comparison on VVT Dataset

  • WildVidFit performs on par with ClothFormer in temporal consistency, while HR-VTON and LaDI-VTON lack a temporal module.Source: §A.2 · p. 19
  • WildVidFit is more robust than HR-VTON, LaDI-VTON, and ClothFormer when generating try-on results across varied poses.Source: §A.2 · p. 19
  • WildVidFit has difficulty reproducing very fine details such as small text, while HR-VTON and ClothFormer produce blurred textures in side views.Source: §A.2 · p. 19

A.3 More Video Try-on Results on TikTok Dataset

TikTok examples show continuous, smooth videos with good garment appearance; the supplied failure-case figures identify image-try-on and segmentation errors.

A.3 More Video Try-on Results on TikTok Dataset

  • On TikTok videos, WildVidFit maintains garment appearance and produces continuous, smooth results.Source: §A.3 · p. 20
  • The image try-on failure-case figure marks differences in examples from DressCode and VITON-HD.Source: Figure 9 · p. 20
  • The segmentation failure cases show inaccurate masks and imperfect lower-garment restoration.The figure caption identifies inappropriate masked regions and lower-garment restoration errors.Source: Figure 10 · p. 20

B Failure Cases and Limitations

WildVidFit's limitations include losing tiny, complex garment patterns and errors from inaccurate garment parsing that affect both garment regions.

B Failure Cases and Limitations

  • The one-stage try-on network fails to reproduce tiny, complex patterns such as small garment writing.The authors suggest latent-space implicit warping may lose overly detailed textures during compression.Source: B Failure Cases and Limitations · p. 21
  • Parsing errors can mask lower-garment regions inappropriately, leading to imperfect lower-garment restoration and adversely affecting upper-garment try-on.Source: B Failure Cases and Limitations · p. 21
  • The described parsing failure involves segmentation that does not distinguish between upper and lower garments.Source: B Failure Cases and Limitations · p. 21

Related papers

Loading 2407.10625v1…