PaperlayerView exact revision on arXiv

Source-linked AI summary

VITON-DiT: Learning In-the-Wild Video Try-On from Human Dance Videos via Diffusion Transformers

Jun Zheng, Fuwei Zhao, Youjiang Xu, Xin Dong, Xiaodan Liang

arXiv:2405.18326v2cs.CV

TL;DR

Existing video try-on methods rely on labor-intensive paired data and have limited performance on casually captured videos. VITON-DiT applies a DiT-based framework to unpaired human dance videos, and the paper reports improved spatio-temporal consistency in real-world scenarios.

  • Problem

    Existing video try-on methods rely on paired garment-person datasets whose labor-intensive construction limits scalability and generalization.

    Source: §1 · p. 2

  • Method

    VITON-DiT combines a garment extractor, spatio-temporal denoising DiT, and identity-preservation ControlNet with attention fusion for unpaired video try-on.

    Source: §1 · pp. 2–3

  • Results

    VITON-DiT is reported to outperform previous methods in spatio-temporal consistency under real-world scenarios with diverse clothing and complex body movements.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    The paper concludes that VITON-DiT offers a scalable approach for using readily available unlabeled video data in video try-on.

    Source: §1 · p. 3, §5 · p. 11

Abstract

from arXiv · show

Video try-on stands as a promising area for its tremendous real-world potential. Prior works are limited to transferring product clothing images onto person videos with simple poses and backgrounds, while underperforming on casually captured videos. Recently, Sora revealed the scalability of Diffusion Transformer (DiT) in generating lifelike videos featuring real-world scenarios. Inspired by this, we explore and propose the first DiT-based video try-on framework for practical in-the-wild applications, named VITON-DiT. Specifically, VITON-DiT consists of a garment extractor, a Spatial-Temporal denoising DiT, and an identity preservation ControlNet. To faithfully recover the clothing details, the extracted garment features are fused with the self-attention outputs of the denoising DiT and the ControlNet. We also introduce novel random selection strategies during training and an Interpolated Auto-Regressive (IAR) technique at inference to facilitate long video generation. Unlike existing attempts that require the laborious and restrictive construction of a paired training dataset, severely limiting their scalability, VITON-DiT alleviates this by relying solely on unpaired human dance videos and a carefully designed multi-stage training strategy. Furthermore, we curate a challenging benchmark dataset to evaluate the performance of casual video try-on. Extensive experiments demonstrate the superiority of VITON-DiT in generating spatio-temporal consistent try-on results for in-the-wild videos with complicated human poses.

1 Introduction

VITON-DiT targets video try-on beyond product garments and simple scenes with a DiT-based framework for casually captured videos. It combines garment-detail recovery with identity and pose preservation, and uses training and inference strategies for long videos.

1 Introduction

  • Existing video try-on methods rely on paired garment-person data, whose labor-intensive construction limits scalability and generalization.VITON-DiT instead uses unpaired human dance videos with a multi-stage training procedure.Source: §1 · pp. 2–3
  • The paper introduces a benchmark designed to evaluate video try-on under real-world complexities.It also reports that performance scales with increased data volume.Source: §1 · p. 3
  • Experiments report realistic try-on results and superiority in unconstrained scenes with complex motion and backgrounds.Figure 1 also illustrates diverse clothing, including flawed non-product garments, and dancing against real-world backgrounds.Source: Figure 1 · p. 2, §1 · p. 3
  • VITON-DiT is presented as the first DiT-based video try-on network, with spatio-temporal generation for casually captured human videos.Its garment extractor, denoising DiT, and identity-preservation ControlNet are connected through attention fusion.Source: §1 · pp. 2–3
  • Attention fusion combines extracted garment features with person-denoising features to recover clothing details in generated videos.The framework also uses an identity-preservation ControlNet to preserve pose and identity.Source: §1 · pp. 2–3
  • Random selection during training and Interpolated Auto-Regressive sampling at inference are proposed to facilitate long-video generation.The paper describes generation of sequences spanning tens of seconds.Source: §1 · pp. 2–3

2 Related Work

Prior video try-on work includes GAN-based warping-and-fusion methods and a UNet-based diffusion method. Video-generation research also explores temporal extensions to image models, while DiT-based approaches support longer video generation.

Video Virtual Try-on

  • GAN-based video try-on methods warp garments and fuse them with the person, but inaccurate optical flow can cause garment-person misalignment.FW-GAN uses preceding-frame optical flow for temporal coherence, while ClothFormer integrates garment and person features with a dual-stream transformer.Source: §2 · p. 3
  • Tunnel Try-on uses a UNet-based diffusion model that handles camera movement and preserves clothing textures, but its demos cover product images and only a few seconds.The paper contrasts this scope with VITON-DiT's stated support for diverse garments and sequences spanning tens of seconds.Source: §2 · p. 3

Diffusion Models for Video Generation

  • Video LDM and AnimateDiff add temporal components to pretrained text-to-image models, whereas DiT-based text-to-video methods can generate videos spanning tens of seconds.The paper says VITON-DiT is built on Open-Sora and is the first DiT-based video virtual try-on model.Source: §2 · p. 3

3 Method

VITON-DiT combines a spatio-temporal denoising DiT, a garment extractor, and an identity-preserving ControlNet. It uses attention fusion and long-video training and inference strategies to retain garment detail and generate videos up to 20 seconds.

3 Method

  • VITON-DiT connects a spatio-temporal denoising DiT, garment extractor, and ID ControlNet through attention fusion to generate try-on videos while retaining clothing details and the target person’s pose and identity.Attention fusion sums attentive outputs from garment and person features.Source: §3 · pp. 3–4

3.1 Preliminary

  • LDMs diffuse compact VAE-encoded latents rather than high-dimensional pixels, and VITON-DiT uses a spatio-temporal DiT backbone suited to human video with limited computational resources.The DiT replaces the U-Net denoiser used in the basic LDM formulation.Source: §3.1 · pp. 4–5

3.2 Spatio-Temporal Diffusion Transformer (ST-DiT)

  • Each ST-DiT block applies spatial self-attention, temporal self-attention, and prompt cross-attention to video tokens, with skip connections around each layer.The temporal layer integrates dynamic information across frames; the default prompt is “a dancing person.”Source: §3.2 · p. 5

3.3 Garment Extractor

  • The garment extractor processes a single clothing image and injects its features into the denoising DiT and ControlNet through spatial cross-attention followed by summed feature outputs.The added cross-attention layer is intended to improve garment-feature perception and reduce input ambiguity.Source: §3.3 · pp. 5–6

3.4 Identity Preservation ControlNet (ID ControlNet)

  • The ID ControlNet processes agnostic images, DensePose, and masks, then adds feature residuals to the denoising DiT to preserve identity, pose, and background.It is a trainable replica of the denoising DiT’s front half; the authors report rational output despite visibly artifacted DensePose guidance.Source: §3.4 · p. 6

3.5 Strategies for Long Video Generation

  • For long-video generation, the method combines random condition and frame-sampling strategies during training with interpolated autoregressive inference.Random agnostic-condition swap is a prerequisite for the IAR inference strategy.Source: §3.5 · p. 6, Figure 3 · p. 6, §3.5.1 · p. 7

3.5.1 Random Selection Training Strategies

  • Training randomly replaces selected agnostic images with ground-truth frames and sets their masks to zero for the ControlNet, while varying frame-sampling stride to expose diverse body views.The denoising DiT input remains unchanged during condition swapping; stride ranges depend on each dataset’s average clip length.Source: §3.5.1 · p. 7

3.5.2 Interpolated Auto-Regressive (IAR) Inference.

  • IAR divides a video into sub-videos, generates starting keyframes from the conditions, then fills missing frames autoregressively while replacing agnostic conditions with interpolated frames when available.The authors state this helps avoid quality degradation from occlusions while retaining smooth autoregressive footage.Source: Figure 3 · p. 6, §3.5.2 · p. 7

4.1 Implementation Details

VITON-DiT is trained in stages using unpaired dance-video data, with image pretraining followed by video fine-tuning. The implementation uses two resolutions and different frame counts for comparison and demo models.

Dataset Construction

  • The collected dance-video dataset contains over 15,000 high-quality clips, supplemented by FashionVideo and TikTok data, with 50 diverse clips reserved as a benchmark.Clips were filtered to exclude multiple people or clips showing only a small portion of a person.Source: §4.1 · p. 7

Multi-Stage Self-Supervised Training

  • Self-supervised training proceeds from garment-extractor image pretraining to ID ControlNet image pretraining and then video fine-tuning, while keeping the denoising ST-DiT SSA frozen.The first stage reconstructs a person image from parsed clothing; the next two stages progressively add trainable modules.Source: §4.1 · p. 7

Hyper-parameters Setting

  • The low-resolution model uses 192x256 resolution and 36 frames for comparisons and ablations, while the 384x512 model uses 16 frames for demos.Both use 28 ST-DiT blocks, patch size 2, hidden dimension 1152, and AdamW with a fixed 1e-5 learning rate.Source: §4.1 · p. 8

Qualitative comparison

  • The qualitative comparison reports that VITON-DiT preserves garment shape and color and maintains clothing-person alignment across camera distances.The caption identifies this as a comparison with baselines.Source: Figure 4 · p. 8

4.2 Qualitative Results

On the VVT dataset, the qualitative comparison describes distinct failure patterns among GAN-based and diffusion-based baselines. VITON-DiT is presented as preserving garment details and temporal consistency in varied scenes.

GAN-based baselines

  • PBAFN and ClothFormer show clothing-person misalignment, while ClothFormer also produces blurry images and distorted cloth texture.The passage attributes the GAN-based methods’ misalignment to inaccurate garment warping.Source: §4.2 · p. 8

Diffusion-based baselines

  • StableVITON produces relatively accurate single frames but noticeable inter-frame jitters, while Tunnel Try-on shows clothing color discrepancy against ground truth.Tunnel Try-on is described as supervised and restricted to product clothing images.Source: §4.2 · p. 8

VITON-DiT results

  • VITON-DiT preserves garment shape, color, and clothing-person alignment as subjects move across camera distances.The chest lettering follows the input shape and color as the subject approaches the camera.Source: §4.2 · p. 8

In-the-wild results

  • Additional VITON-DiT results include complex dance motions against real-world backgrounds, with high-detail preservation and temporal consistency.The passage connects attention fusion with adapting to different human movements and clothing.Source: §4.2 · p. 8

4.3 Quantitative Results

The quantitative evaluation uses single-frame image metrics and video metrics intended to capture visual quality and temporal consistency. The reported comparison describes lower image scores but better video metrics for VITON-DiT.

Evaluation metrics

  • Single-frame evaluation uses SSIM and LPIPS, while video performance is assessed with VFID using I3D and 3D-ResNeXt101 feature extractors.VFID uses 3D convolution to evaluate visual quality and temporal consistency.Source: §4.3 · p. 8

Evaluation setup

  • Table 1 reports quantitative comparisons on the VVT dataset, and the paper provides an implementation of VFID metrics for fair comparison.The stated reason is the absence of open-source VFID metric code.Source: §4.3 · p. 8, Table 1 · p. 9

Quantitative comparison

  • VITON-DiT has inferior SSIM and LPIPS scores on product clothing images but outperforms other methods on video metrics.The passage attributes the image-score drop to unpaired training images parsed from person images, which may contain defects such as missing holes.Source: §4.3 · p. 9

4.4 Ablation Study

The ablations examine data scale, video fine-tuning strategy, attention fusion, and IAR. They report benefits from high-quality data, full tuning, attention fusion with SCA, and IAR, while describing the conditions used for each comparison.

4.4 Ablation Study

  • The ablations use 192x256 resolution and include experiments on the collected dataset and VITON-HD.The study is described as verifying the effectiveness of model components.Source: §4.4 · p. 9

4.4.1 Data Scaling

  • Greater quantities of high-quality data improve visual performance, while low-quality clips with extremely small or invalid people harm the model.Even less high-quality Data-S significantly outperforms Data-F; Data-M and Data-L also produce reasonable results with inaccurate pose guidance.Source: Figure 5 · p. 9, §4.4.1 · p. 10

4.4.2 Training Strategy for Video Fine-tuning

  • Full-tuning is preferred to freeze-tuning, yielding more faithful qualitative and quantitative results.The ablation compares training configurations for the third stage.Source: §4.4.2 · p. 10, Figure 6 · p. 10

4.4.3 Effectiveness of Attention Fusion

  • With SCA in the ID ControlNet, the full model recovers garment texture better than the tested variants.This spatial-dimension ablation follows the second training stage without TSA loaded and is evaluated on VITON-HD.Source: Table 3 · p. 9, §4.4.3 · p. 10, Figure 6 · p. 10

4.4.4 Effectiveness of IAR

  • IAR handles occlusions more robustly and recovers texture details better than vanilla autoregressive inference, with benchmark metrics also favoring IAR.The comparison is shown in Figure 6(c) and reported in Table 4.Source: §4.4.4 · p. 10, Table 4 · p. 10, Figure 6 · p. 10

5 Conclusions

VITON-DiT combines DiT-based video try-on with unpaired training to support scalable generation. Its attention fusion and inference strategies recover clothing details in videos spanning tens of seconds, while experiments report improved spatio-temporal consistency in real-world scenarios.

5 Conclusions

  • VITON-DiT is the first DiT-based video try-on network proposed for scalable unpaired video virtual try-on.The authors present it as a step toward using readily available unlabeled video data.Source: §5 · p. 11
  • Attention fusion, novel random selection strategies, and IAR enable faithful clothing-detail recovery in generated videos spanning tens of seconds.The authors describe random selection as a training strategy and IAR as an inference strategy.Source: §5 · p. 11
  • Experiments show VITON-DiT handles diverse clothing and complex body movements, outperforming previous methods in spatio-temporal consistency under real-world scenarios.Source: §5 · p. 11

Related papers

Loading 2405.18326v2…