PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamVVT: Mastering Realistic Video Virtual Try-On in the Wild via a Stage-Wise Diffusion Transformer Framework

Tongchun Zuo, Zaiyu Huang, Shuliang Ning, Ente Lin, Chao Liang, Zerong Zheng, Jianwen Jiang, Yuan Zhang, Mingyuan Gao, Xin Dong

arXiv:2508.02807v1cs.CV

TL;DR

Video virtual try-on methods struggle with fine-grained garment preservation and temporal consistency because they rely on scarce paired data and underuse pretrained priors and test-time inputs. DreamVVT addresses this with a two-stage DiT framework that creates multimodal keyframe try-on guidance before generating the final video. Experiments report better garment-detail preservation and temporal stability than existing methods across real-world scenarios.

  • Problem

    Existing methods struggle to preserve fine-grained garment details and temporal consistency in unconstrained scenarios while relying on scarce paired data and underusing pretrained priors and test-time inputs.

  • Method

    DreamVVT generates multimodal keyframe try-on images, then feeds them with skeleton maps and fine-grained descriptions into a pretrained video model enhanced with LoRA adapters.

  • Results

    DreamVVT surpasses existing methods in preserving detailed garment content and maintaining temporal stability across diverse real-world scenarios.

  • Takeaways & Limitations

    Stage-wise integration of keyframe appearance, motion guidance, unpaired data, and pretrained video-model priors supports high-fidelity virtual try-on in unrestricted scenarios.

  • Takeaways & Limitations

    Precomputed agnostic masks can compromise foreground objects and complex scenes, and complex garment-interaction motions remain difficult for the method.

Abstract

from arXiv · show

Video virtual try-on (VVT) technology has garnered considerable academic interest owing to its promising applications in e-commerce advertising and entertainment. However, most existing end-to-end methods rely heavily on scarce paired garment-centric datasets and fail to effectively leverage priors of advanced visual models and test-time inputs, making it challenging to accurately preserve fine-grained garment details and maintain temporal consistency in unconstrained scenarios. To address these challenges, we propose DreamVVT, a carefully designed two-stage framework built upon Diffusion Transformers (DiTs), which is inherently capable of leveraging diverse unpaired human-centric data to enhance adaptability in real-world scenarios. To further leverage prior knowledge from pretrained models and test-time inputs, in the first stage, we sample representative frames from the input video and utilize a multi-frame try-on model integrated with a vision-language model (VLM), to synthesize high-fidelity and semantically consistent keyframe try-on images. These images serve as complementary appearance guidance for subsequent video generation. \textbf{In the second stage}, skeleton maps together with fine-grained motion and appearance descriptions are extracted from the input content, and these along with the keyframe try-on images are then fed into a pretrained video generation model enhanced with LoRA adapters. This ensures long-term temporal coherence for unseen regions and enables highly plausible dynamic motions. Extensive quantitative and qualitative experiments demonstrate that DreamVVT surpasses existing methods in preserving detailed garment content and temporal stability in real-world scenarios. Our project page https://virtu-lab.github.io/

1. Introduction

Existing video virtual try-on methods struggle with garment-detail preservation and temporal consistency in unconstrained scenarios because they depend on scarce paired data, disrupt pretrained priors, and lack sufficient test-time appearance information. DreamVVT addresses these issues with a two-stage DiT framework that combines keyframe try-on images, multimodal motion guidance, unpaired data, and pretrained video-generation priors.

  • Existing methods struggle to preserve fine-grained garment details and temporal consistency under complex motion, dynamic scenes, and diverse character styles.They rely heavily on insufficient paired clothing-video data collected mainly in homogeneous indoor environments.
  • End-to-end training limits effective use of unpaired data, pretrained visual-model priors, and additional information available at inference.Full fine-tuning with limited data can disrupt pretrained priors and degrade generated-video quality and temporal stability.
  • Front-view garment images alone can produce implausible unseen-region results when people turn or camera viewpoints change substantially.
  • DreamVVT first generates high-fidelity, semantically consistent keyframe try-on images using motion-selected frames, VLM descriptions, garment images, and a LoRA-equipped multi-frame try-on model.These keyframe images provide complementary appearance guidance for later video generation.
  • The second stage combines temporally smoothed skeletal features, fine-grained video-LLM descriptions, and aligned keyframe images in a pretrained video model with LoRA adapters.The framework also uses multi-task learning to preserve controllability across modalities.
  • DreamVVT outperforms existing methods in preserving high-fidelity garment details and maintaining temporal stability across diverse real-world scenarios.

2. Related work

Prior image-based methods improve garment realism but largely remain single-view, while video-based methods struggle to use diverse unpaired data without disrupting pretrained priors. DreamVVT builds on these limitations by combining multi-frame try-on with pose- and multimodally guided video generation for unconstrained scenarios.

  • 2.1. Image-based Virtual Try-on: Image-based virtual try-on methods improve realism and garment preservation through reference architectures and LoRA, but mostly remain limited to conventional single-view settings.DreamVVT extends this direction toward instruction-guided multi-frame try-on for downstream video generation.
  • 2.2. Video-based Virtual Try-on: Video-based try-on approaches use temporal attention, clothing-consistency strategies, or DiT backbones, yet struggle to preserve both visual fidelity and motion consistency in unconstrained inputs.Their limitations include inability to leverage broad unpaired data and disruption of pretrained priors.
  • 2.3. Pose guided Human Video generation: Human-video generation methods commonly combine reference encoders with pose guiders, but whole-image references can degrade garment details during large subject or camera rotations.DreamVVT therefore uses multiple try-on keyframes, tracking-based cropping, and multi-task learning.

3. Methodology

DreamVVT uses two sequential stages: motion-aware keyframe try-on produces consistent garment appearances, then multimodal video generation combines those appearances with pose and textual motion guidance. Its inputs and training design target detail preservation and temporal coherence in complex scenes.

  • 3. Methodology: DreamVVT samples motion-varied keyframes, generates consistent try-on images, and synthesizes the final video from keyframe appearance, pose, and textual conditions.
  • 3.1. Input conditions: Character crops, pose conditions, agnostic masks, agnostic images, and tightly cropped garment images form the model’s principal input conditions.Tracking crops preserve character detail, while masks reduce original-garment information leakage.
  • 3.2.1. Keyframe Sampling: Keyframes are selected by scoring skeletal-motion similarity to a frontal A-pose anchor, weighting by subject area, and enforcing a minimum score interval to reduce redundancy.
  • 3.2.2. Multi-frame Try-on Model: The multi-frame try-on model uses LoRA-modified MMDiT blocks, a parameter-sharing reference branch, multimodal tokenization, cross-keyframe attention, and VLM-rewritten garment descriptions.These components target coherent results with consistent garment details across keyframes.
  • 3.3. Stage2: Multi-modal Guided Virtual Try-on Video Generation: Stage two encodes cropped skeletons into temporally smoothed pose latents and concatenates agnostic, mask, noise, and pose information into video tokens.
  • 3.3. Stage2: Multi-modal Guided Virtual Try-on Video Generation: Qwen2.5-VL supplies attribute-disentangled motion and visual descriptions, while keyframe images provide appearance tokens for joint spatial-temporal alignment.The appearance descriptions are replaced with target-garment descriptions during inference.
  • 3.3. Stage2: Multi-modal Guided Virtual Try-on Video Generation: The video model freezes text streams, inserts LoRA adapters into duplicated video and image streams, and uses joint self-attention before DiT denoising.A multi-task training schedule randomly combines text, pose, and keyframe conditions.
  • 3.3. Stage2: Multi-modal Guided Virtual Try-on Video Generation: The experiments compare one- and two-keyframe inputs and full-parameter fine-tuning against LoRA adapters on garment, physical-realism, and temporal-consistency outcomes.The supplied figure and table captions identify these ablation and quantitative comparison scopes without reporting their cell values.

4.1. Datasets

The study combines a large human-centric video collection and multi-view image pairs with established try-on datasets, then evaluates on indoor and in-the-wild benchmarks. The in-the-wild benchmark specifically targets varied motion, scenes, garments, and character styles.

  • The collected human-centric video dataset contains 69,643 samples with unrestricted subject and camera movement and dynamic scenes.
  • More than one million same-person multi-view image pairs were gathered from public websites and mixed with VITON-HD, DressCode, and ViViD for training.
  • Evaluation uses the 180-sample ViViD-S indoor dataset and an 81-sample Wild-TryOnBench covering motion, scene, garment-input, and character-style variation.

4.2. Implementation Details

The implementation uses a DiT-based training and inference setup with specified optimization, sampling, and evaluation protocols for paired and unpaired try-on scenarios.

  • 4.2. Implementation Details: DreamVVT trains with AdamW at a constant 2e-5 learning rate, 0.01 weight decay, and 1.0 gradient clipping on eight NVIDIA H20 GPUs for about 10 days.
  • 4.2. Implementation Details: Inference uses Euler sampling for 50 steps with classifier-free guidance 2.5; keyframe results are sampled three times, while video results use one run.
  • 4.2.1. Evaluation: Evaluation combines VFID for unpaired video quality, SSIM and LPIPS for paired similarity, and human scores for garment detail, physical realism, and temporal consistency.

4.3. Qualitative Comparison

DreamVVT produces more realistic and spatiotemporally smooth try-on videos than the compared methods across ViViD-S and WildTryOn, including challenging in-the-wild conditions.

  • 4.3. Qualitative Comparison: DreamVVT demonstrates strong superiority over CatV2TON, MagicTryOn, and GPT4o+VACE in realistic, spatiotemporally smooth results across ViViD-S and WildTryOn.
  • 4.3. Qualitative Comparison: CatV2TON and MagicTryOn degrade on out-of-domain WildTryOn cases and can become blurry during 360-degree rotations with severe self-occlusion.
  • 4.3. Qualitative Comparison: GPT4o+VACE struggles to preserve person identity and reproduce cloth details, whereas DreamVVT supports arbitrary resolutions, frame rates, and indoor or outdoor scenarios.

4.4. Quantitative Comparison

DreamVVT achieves state-of-the-art quantitative performance across the ViViD-S and WildTryOn benchmarks, with especially strong results in unpaired and WildTryOn evaluations.

  • 4.4. Quantitative Comparison: DreamVVT achieves state-of-the-art performance among compared baselines on both ViViD-S and WildTryOn.On ViViD-S, it has the lowest VFID scores in unpaired try-on; in paired try-on, it is best on VIDI and LPIPS and second-best on VIDR and SSIM.
  • 4.4. Quantitative Comparison: On WildTryOn, DreamVVT outperforms existing baselines and achieves the best performance on all three reported metrics.

4.5. Ablation Study

The ablations show that multiple keyframes improve garment and motion understanding, while LoRA adapters preserve pretrained text control and improve physical realism.

  • 4.5. Ablation Study: The ablation study evaluates LoRA use and different keyframe counts through human evaluation on the WildTryOn benchmark.
  • 4.5.1. Key frame number: A single keyframe can cause blurry outputs or artifacts because it provides insufficient garment detail information in complicated scenarios such as turning around.
  • 4.5.1. Key frame number: Two keyframes produce clearer, more plausible results than one keyframe, improving detail preservation and slightly increasing physical realism and temporal consistency.
  • 4.5.2. LoRA adapters for video generation model: LoRA training better preserves pretrained text-control capability than full-parameter fine-tuning and generates more physically realistic clothing-interaction videos.
  • 4.5.2. LoRA adapters for video generation model: The LoRA variant significantly improves physical realism without compromising garment detail preservation or temporal consistency.

5. Discussions

DreamVVT is a stagewise DiT framework that combines keyframe try-on with multimodal guided video generation for unconstrained virtual try-on. Experiments report improved garment-detail preservation and temporal consistency, while limitations remain for large masks and complex garment interactions.

  • DreamVVT integrates keyframe try-on with multimodal guided video generation to leverage unpaired data, pretrained priors, and test-time inputs.
  • Extensive experiments show improved garment-detail preservation and temporal consistency under unrestricted scenarios across diverse garments.
  • Large precomputed agnostic masks may compromise foreground-object and complex-scene integrity when accommodating arbitrary garment styles.
  • Complex garment-interaction motions remain difficult because of pretrained-model generation limits and insufficient fine-grained action captioning.

7. Supplementary Material

The supplementary material describes curated video and image datasets, keyframe sampling, long-term video continuation, and additional qualitative and ablation results. These components support diverse training scenarios, motion-aware frame selection, longer generation, and LoRA-based garment interaction.

  • 7.1.1. Video Dataset Construction: Approximately 102K videos were segmented into 187K clips, filtered for quality, motion, and text-based occlusion, and annotated with pose plus environment, appearance, and motion descriptions.
  • 7.1.2. Image Dataset Construction: A clean set of 1.01 million images was obtained from 1.3 million collected images after quality, appearance-consistency, and person-size filtering, then captioned with viewpoint, clothing, and scene information.
  • 7.2.1. Keyframe Sampling Algorithm: Keyframe sampling scores frames using skeletal motion similarity and subject area ratio, then selects frames separated by a minimum score interval.
  • 7.2.2. Long-term Video Generation: Long-term generation reuses the previous segment’s final latent frame directly, avoiding repeated decoder–encoder passes that accumulate errors as sequence length increases.
  • 7.3. More Results: Additional comparisons cover ViViD and WildTryOn, while ablations show LoRA-adapter training enables garment interaction more effectively than full-parameter finetuning.
Loading 2508.02807v1…