ViViD: Video Virtual Try-on using Diffusion Models

Motivation

• The task aims to synthesize realistic videos that preserve target-clothing identity while keeping the rest of the source video unchanged.Source: §1 · p. 1

Problem

• Frame-wise image try-on causes flickers and artifacts because it neglects temporal consistency.Source: §1 · pp. 1–2

Limitations of prior work

• Existing video methods remain limited by low-quality data, warping dependence, spatial-temporal inconsistency, and unstable GAN-based generation.The commonly used VVT dataset has 791 videos at 256 × 192 and only upper-body garments.Source: §1 · pp. 2–3

Dataset

• ViViD provides 9,700 paired garments and high-resolution videos at 832 × 624, totaling 1,213,694 frames across three garment classes.The dataset is reported as more than 5 × larger than VVT in the introduction.Source: §1 · p. 3

Framework

• ViViD introduces a diffusion-based video try-on framework combining garment features, pose signals, attention fusion, and temporal modules.The Garment Encoder preserves clothing semantics, while the Pose Encoder and temporal modules support spatial-temporal consistency.Source: §1 · p. 3

Evidence

• Quantitative and qualitative experiments validate ViViD’s ability to synthesize high-quality, temporally consistent try-on videos.Source: §1 · p. 3

2.1 Image-based Virtual Try-on

• Image-based try-on methods preserve garment details effectively but do not accurately capture dynamic interactions between clothing and the human body.Diffusion-based methods such as LaDI-VITON [29], Multimodal Garment Designer [3], and StableVITON extend this line of work.Source: §2.1 · p. 3

2.2 Video-based Virtual Try-on

• Video-based try-on methods target temporal consistency through flow, warping, memory refinement, or occlusion-aware modules, yet still produce rudimentary apparel with flicker and limited garment coverage.These methods focus only on upper-body try-on and are constrained by the lack of high-quality public datasets.Source: §2.2 · p. 4

2.3 Video Editing and Image Animation

• Text-driven video editing preserves motion and structure, but text alone cannot accurately describe complex clothing characteristics.FateZero [30] uses attention-map fusion, while Tune-A-Video [37] uses temporal modeling and DDIM inversion.Source: §2.3 · p. 4
• Image animation can synthesize pose-matched videos but may alter content outside the clothing region and miss facial motion beyond pose signals.MagicAnimate [40] and Animate Anyone [19] use an additional UNet and body-pose conditions.Source: §2.3 · p. 4

Dataset scope

• ViViD contains 9,700 clothing-video pairs at 832 × 624 across upper-body, lower-body, and dress garments, totaling 1,213,694 frames.It is described as the largest, most diverse, and highest-resolution video virtual try-on dataset to date.Source: §3 · p. 4

Motivation

• The dataset addresses VVT’s limitations of 791 videos, 256 × 192 resolution, upper-body-only garments, and repetitive catwalk scenes.Source: §3 · p. 4

Data collection

• Data are collected from e-commerce clothing images and indoor videos showing people performing movements such as walking, swaying, and turning.Source: §3 · p. 4

Annotation

• Each sample is augmented with clothing-agnostic maps, masks, human keypoints, semantic parsing, DensePose, clothing masks, and garment categories.OpenPose, SCHP, DensePose, SAM, and BLIP-2 provide these annotations or categories.Source: §3 · p. 5

4.1 Proposed Method

• ViViD treats video virtual try-on as video inpainting, conditioning the UNet on noisy video, clothing-agnostic video, and mask inputs.The expanded UNet input uses nine channels: four noisy-video, four agnostic-video, and one mask channel.Source: §4.1 · p. 5
• The Garment Encoder extracts fine-grained clothing semantics and injects them into the UNet through attention feature fusion.It processes the latent garment image and resized garment mask, while CLIP image embeddings provide global clothing semantics.Source: §4.1 · pp. 5–7
• Pose and temporal modules improve motion-aware synthesis by encoding human pose and modeling information across video frames.The Pose Encoder uses DensePose signals, while Temporal Modules apply self-attention along the frame dimension.Source: §4.1 · pp. 6–7

4.2 Training Strategy

• The framework uses a single-stage objective that jointly trains on image and video data, with garment and mask representations supplied to the training loss.The loss conditions on concatenated noisy-video, agnostic-video, and mask latents together with concatenated garment and mask latents.Source: §4.2 · p. 7
• Image-video joint training freezes temporal modules for image samples and trains them for video clips, combining garment-detail learning with temporal modeling.The strategy samples image data with probability controlled by λ and video sequences otherwise.Source: §4.2 · p. 7

5.1 Datasets

• The ViViD dataset contains 9,700 high-resolution video-image pairs, split into 7,759 training and 1,941 test videos, totaling 1,213,694 frames.Source: §5.1 · p. 8

5.2 Implementation Details

• Image-video joint training uses image samples with frozen temporal modules and continuous 24-frame video sequences with active temporal modules.Source: §5.2 · p. 8

5.3 Qualitative Results

• ViViD generates high-quality try-on videos with preserved clothing details and temporal consistency, including during large movements such as turning.Source: §5.3 · p. 8

5.4 Quantitative Results

• ViViD surpasses previous image-based and video-based methods on the VVT dataset in image quality and temporal consistency.Source: §5.4 · pp. 8–9, Table 1 · p. 9

5.5 Ablation Study

• Removing or replacing the Garment Encoder fails to preserve fine clothing details, supporting its role with attention feature fusion.Source: §5.5 · p. 9, Figure 7 · p. 9
• Two-stage image-then-video training loses clothing details because data-format and context shifts impair retention of learned texture and style features.Source: §5.5 · p. 9

• ViViD uses diffusion models for video virtual try-on and produces high visual quality with temporal consistency.Source: §6 · p. 9
• The paper introduces a large multi-category dataset with high-resolution image-video pairs for video virtual try-on.Source: §6 · p. 9

• Training uses AdamW at 1 × 10−5, 512 × 384 inputs, 24-frame clips sampled at rate 4, four 80GB A100 GPUs, 60,000 steps, and fp16 precision.Source: A More implementation details · p. 13
• Because temporal modules train on 24-frame clips, sliding-window denoising with overlap and noise interpolation generates longer spatial-temporally consistent videos.Source: A More implementation details · p. 13
• Spatial augmentation applies flips and affine shifts or scaling, while illumination changes are excluded because they may cause garment color deviation.Source: A More implementation details · p. 13

• ViViD applies to a wide variety of clothing scenarios in additional generated try-on results.Source: B More experimental results · p. 13
• Figures 8 and 9 provide more generated try-on results, with the first and fifth rows in Figure 9 showing source videos.Source: Figure 8 · p. 13, Figure 9 · p. 14
• Figure 10 provides additional generated try-on results, with the first and fourth rows showing source videos.Source: Figure 10 · p. 15

Limitations

• Indoor-scene training limits ViViD, which exhibits noticeable artifacts in outdoor scenarios.The authors note that similar issues occur in OOTDiffusion [38] and call for in-the-wild video try-on data.Source: C Discussion · p. 16, Figure 11 · p. 16

Potential societal impact

• Accurate virtual try-on may require body measurements and shape data, creating potential privacy-breach risks.Source: C Discussion · p. 16
• Video virtual try-on could be misused for deceptive content, raising ethical and legal concerns that require regulatory guidance.Source: C Discussion · p. 16