PaperlayerView exact revision on arXiv

Source-linked AI summary

The devil is in the details: Enhancing Video Virtual Try-On via Keyframe-Driven Details Injection

Qingdong He, Xueqin Chen, Yanjie Pan, Peng Tang, Pengcheng Xu, Zhenye Gan, Chengjie Wang, Xiaobin Hu, Jiangning Zhang, Yabiao Wang

arXiv:2512.20340v3cs.CV

TL;DR

DiT-based video virtual try-on still struggles with fine-grained garment dynamics and background integrity, while added interaction modules increase computational cost and public datasets remain limited. KeyTailor injects details guided by sampled keyframes, alongside the ViT-HD dataset. Experiments report stronger garment fidelity and background integrity across dynamic and static scenarios.

  • Problem

    DiT-based VVT methods struggle with fine-grained garment dynamics and background integrity, incur added computational costs, and rely on limited public datasets.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    KeyTailor samples informative keyframes and uses garment-detail and background-optimization modules to inject enriched details into standard DiTs.

    Source: §1 · pp. 2–3, §3.1 · p. 4

  • Results

    KeyTailor outperforms state-of-the-art baselines in garment fidelity and background consistency across dynamic and static scenarios.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    KeyTailor and ViT-HD provide reported baselines and resources for future research in video and image virtual try-on.

    Source: §5 · p. 9

    KeyTailor still struggles to reliably handle complex garment–body interaction motions, given limits in pretrained-model generative capacity and fine-grained action captioning.

    Source: J. Limitations · p. 15

Abstract

from arXiv · show

Although diffusion transformer (DiT)-based video virtual try-on (VVT) has made significant progress in synthesizing realistic videos, existing methods still struggle to capture fine-grained garment dynamics and preserve background integrity across video frames. They also incur high computational costs due to additional interaction modules introduced into DiTs, while the limited scale and quality of existing public datasets also restrict model generalization and effective training. To address these challenges, we propose a novel framework, KeyTailor, along with a large-scale, high-definition dataset, ViT-HD. The core idea of KeyTailor is a keyframe-driven details injection strategy, motivated by the fact that keyframes inherently contain both foreground dynamics and background consistency. Specifically, KeyTailor adopts an instruction-guided keyframe sampling strategy to filter informative frames from the input video. Subsequently,two tailored keyframe-driven modules, the garment details enhancement module and the collaborative background optimization module, are employed to distill garment dynamics into garment-related latents and to optimize the integrity of background latents, both guided by keyframes.These enriched details are then injected into standard DiT blocks together with pose, mask, and noise latents, enabling efficient and realistic try-on video synthesis. This design ensures consistency without explicitly modifying the DiT architecture, while simultaneously avoiding additional complexity. In addition, our dataset ViT-HD comprises 15, 070 high-quality video samples at a resolution of 810*1080, covering diverse garments. Extensive experiments demonstrate that KeyTailor outperforms state-of-the-art baselines in terms of garment fidelity and background integrity across both dynamic and static scenarios.

1. Introduction

Video virtual try-on aims to replace a person’s clothing while preserving motion and visual consistency, but current methods struggle with garment details, background integrity, computational cost, and limited training data. KeyTailor uses sampled keyframes to enrich garment and background details, and experiments report stronger fidelity and consistency across dynamic and static scenarios.

1. Introduction

  • Existing methods lose fine-grained garment dynamics, background detail and temporal consistency, while added DiT interaction modules raise computational costs and public datasets remain limited.Reported garment issues include missing wrinkles and incorrect garment details; background issues include blurred textures, temporal artifacts, and incoherence.Source: §1 · p. 2
  • KeyTailor samples informative keyframes and uses two lightweight modules to enhance garment dynamics and optimize background integrity before injecting details into DiTs.The modules are garment dynamic details enhancement and collaborative background details optimization; pose and mask latents are among the other conditions.Source: §1 · p. 3
  • KeyTailor uses standard DiTs without additional interaction layers and fine-tunes them with a LoRA adapter to reduce computational demand.The paper points to Figure 2(b) and 2(c) as evidence of the reduced computational demand.Source: §1 · p. 3
  • ViT-HD is a large-scale, high-definition dataset collected from multiple e-commerce platforms, comprising 15,070 video samples at 810×1080 resolution across various garment styles.Source: §1 · p. 3
  • KeyTailor outperforms state-of-the-art baselines in garment fidelity and background consistency across dynamic and static scenarios.Experiments cover ViT-HD, two VVT datasets, and two image-based virtual try-on datasets.Source: §1 · p. 3

2. ViT-HD Dataset

ViT-HD is introduced to address the limited scale and quality of existing public VVT datasets. It contains 15,070 high-definition samples with diverse garment styles, collected and filtered to preserve subject integrity and consistent color.

2. ViT-HD Dataset

  • Existing VVT datasets are limited in scale and quality, with low resolution or simple, repetitive runway scenes.The passage notes that these datasets fall short of the need for large, high-quality video data.Source: §2 · p. 3
  • ViT-HD contains 15,070 video samples at 810×1080 resolution and spans diverse garment styles.The dataset is presented as expanding the scale and quality of available VVT resources.Source: §2 · p. 3
  • The collection pairs high-resolution garment images with corresponding model showcase videos, then derives pose and masked videos using established processing pipelines.Samples are categorized as upper-body, lower-body, or full-body outfits.Source: §2 · pp. 3–4
  • Processing discards videos with many frames of incomplete clothing occlusion and removes overexposed opening frames to preserve subject integrity and consistent color tones.Source: §2 · p. 4

3. Method

KeyTailor uses instruction-guided keyframes to enrich garment and background latents, then injects those details alongside pose, mask, and noise information into DiT generation. Its tailored modules address garment variation and background detail while retaining the original DiT architecture.

3.1. Keyframe-driven Details Injection

  • IKS parses view–action instructions into target views and actions, creates multi-anchor pose frames, then ranks frames by motion difference and garment-area ratio with redundancy- and time-aware dual selection.Frames are sorted by score before the dual-selection strategy constrains score differences and temporal intervals.Source: §3.1 · p. 4
  • GDDE enriches an edited first-frame try-on representation with keyframe-derived garment variations, including backside textures and wrinkles caused by raised arms.It encodes segmented garment regions from keyframes and distills their features into the initial garment latent.Source: §3.1 · p. 4
  • CBDO combines coarse global background encoding with fine-grained keyframe detail enrichment, using the frame with the highest background completeness as its supplementary input.The background latent blend uses a balance weight α, set to 0.3 by default; the module addresses subtle details lost when garment regions are inpainted in the agnostic video.Source: §3.1 · pp. 4–5

3.2. Virtual Try-on Video Generation

  • Generation patchifies pose and mask latents, fuses them with enhanced garment latents, adds noise, and injects optimized background latents before DiT denoising.Garment latents replace text tokens in cross-attention; LoRA finetunes DiT attention modules, and the generated tokens are decoded into video sequences.Source: §3.2 · pp. 5–6

4. Experiments

KeyTailor is evaluated across video and image try-on benchmarks, with comparisons showing gains over state-of-the-art methods and ablations assessing its components. The experiments also examine computational overhead, qualitative results, and user preferences.

4.1. Setups: Datasets and Details

  • Training combines ViT-HD and ViViD, while evaluation includes ViViD-S, VVT, VITON-HD, and DressCode.The base model is Wan2.1-I2V-14B-720P; training uses 81-frame samples for 14,500 iterations, and video inference uses 25 steps.Source: §4.1 · p. 6

4.2. Performance comparison with SOTA methods

  • KeyTailor outperforms state-of-the-art methods across nearly all metrics in paired and unpaired video settings on ViT-HD, VVT, and ViViD.The quantitative comparisons are reported in Tables 2 and 3.Source: §4.2 · p. 6, Table 2 · p. 7, Table 3 · p. 7
  • KeyTailor also delivers better performance than baselines on image-based virtual try-on tasks in static scenarios.The image comparisons use VITON-HD and DressCode.Source: §4.2 · p. 6, Table 4 · p. 7
  • Qualitative results show finer garment detail retention and more consistent backgrounds, while most study participants preferred KeyTailor over state-of-the-art methods.The comparisons include ViViD, ViT-HD, in-the-wild examples, and person-to-video garment transfer.Source: §4.2 · p. 6, Figure 5 · p. 7, Figure 6 · p. 8
  • KeyTailor adds 2.10% in parameters over its backbone, versus 15.11% for MagicTryOn and 157.10% for ViViD.It has approximately 194K FLOPs and 12.58 s/it, and trains on 0.2057B parameters versus 16.446B for MagicTryOn.Source: §4.2 · p. 6, Table 5 · p. 8

4.3. Ablation Study

  • The full KeyTailor model outperforms all ablation variants, and removing any component degrades performance, most notably garment dynamics distillation.Removing components also produces color shifts, distorted backgrounds, hallucinated objects, and skin-tone changes.Source: §4.3 · pp. 7–8

5. Conclusion

KeyTailor pairs a keyframe-driven DiT video try-on model with the ViT-HD dataset, and the conclusion reports stronger fidelity, background integrity, and temporal consistency across video and image try-on tasks. The section also reviews DiT and LoRA preliminaries and defines scores for motion, garment area, and background integrity.

5. Conclusion

  • KeyTailor and ViT-HD are presented as a model-and-dataset contribution, with experiments reporting superior garment fidelity, background integrity, and temporal consistency on video and image try-on tasks.The conclusion describes ViT-HD as a large-scale, high-definition dataset and says the work establishes baselines and resources for future research.Source: §5 · pp. 8–9

A. Related Work

  • DiT-based try-on methods use clothing branches and added interaction modules, while KeyTailor injects keyframe-driven details without explicitly modifying the DiT architecture.The related-work discussion identifies limited garment dynamics, background detail, computational overhead, and publicly available dataset scale as unresolved concerns.Source: A. Related Work · p. 12

B. Preliminary

  • Wan is described as a video DiT with patchifying, Transformer-block, and unpatchifying stages, using alternating cross-attention and self-attention for long-context instruction following.The preliminary section also introduces LoRA as low-rank adaptation of pretrained weights that reduces trainable parameters.Source: B. Preliminary · pp. 12–13

C. Score Definition

  • The motion difference score measures how a frame’s motion differs from anchor-frame motion, while the garment-area ratio measures the fraction of the frame occupied by clothing.The motion score uses skeleton-direction differences; garment regions are extracted with HumanParsing.Source: C. Score Definition · p. 13
  • The background integrity score measures preserved-background clarity and proportion, with clarity based on background edge density and normalized average gradient strength.The edge map is computed with Sobel and a fixed threshold of 50.Source: C. Score Definition · p. 13

D. Metrics

The metrics section specifies measures for video-generation quality and image virtual try-on.

D. Metrics

  • Video generation is evaluated with SSIM, LPIPS, and VFID, while image virtual try-on additionally uses FID and KID alongside SSIM and LPIPS.SSIM measures structural similarity, LPIPS perceptual differences, and VFID temporal consistency and overall video quality.Source: D. Metrics · p. 13

E. Training Details

Training details describe parameter-efficient LoRA finetuning and a flow-matching objective that trains the model to predict the true velocity.

E. Training Details

  • LoRA parameters in the DiT blocks, single-image try-on model, mask guider, and pose guider are finetuned from pretrained weights while remaining modules stay frozen.The described finetuning applies LoRA to attention and feed-forward transformations for lightweight adaptation.Source: E. Training Details · p. 13
  • Flow matching trains the model to learn the true velocity while maintaining equivalence with the maximum-likelihood objective.The model-predicted velocity is denoted by u(xt, L, ctxt, t; θ).Source: E. Training Details · p. 14

F. Settings of Ablation Study

The ablation settings compare component removals and alternative keyframe sampling choices, while the sampling procedure scores frames and selects keyframes under temporal and score-difference constraints.

F. Settings of Ablation Study

  • Ablations remove or replace instruction-guided sampling, garment-detail enhancement, keyframe-aware LoRA, background optimization, or weighted fusion to evaluate their respective contributions.Variants also test random selection, motion-only guidance, using the first frame alone, and restricting the keyframe set to one frame.Source: F. Settings of Ablation Study · p. 14

G. Instruction-Guided Keyframe Sampling

  • Instruction-guided sampling parses view and action targets, then scores candidate frames using instruction relevance, motion difference, garment area, and occlusion-related criteria.The settings specify a maximum of three keyframes for short videos and six for long videos, with an interval threshold of video duration divided by five.Source: F. Settings of Ablation Study · p. 14
  • Candidates are sorted by score and selected to reduce redundancy and preserve temporal spacing, using score-difference and temporal-interval thresholds.The procedure uses a minimum temporal distance from already selected frames and appends candidates that meet the selection conditions.Source: F. Settings of Ablation Study · p. 14

H. Hyperparameter Studies

The study finds three keyframes best balance view and motion coverage with generation quality, and selects λ = 0.5 and α = 0.3 for the other tested hyperparameters.

H. Hyperparameter Studies

  • K = 3 yields the best performance (SSIM=0.9066, LPIPS=0.0397), outperforming K = 1, 2, 4, and 5 on both metrics.Three keyframes cover the front view, back view, and hand-raising action; additional keyframes introduce unnecessary or overlapping information.Source: Table A1 · p. 15, H. Hyperparameter Studies · p. 15
  • λ = 0.5 gives the best performance by balancing garment-area visibility with motion cues.Values away from 0.5 reduce SSIM and increase LPIPS.Source: H. Hyperparameter Studies · p. 15
  • α = 0.3 is optimal; larger values increase perceptual distortion despite minor structural-similarity gains.α controls reliance on the general background latent code versus the frame representation with the highest background completeness.Source: H. Hyperparameter Studies · p. 15

I. More Qualitative Results

Additional comparisons on ViViD and ViT-HD show KeyTailor producing more natural videos with finer garment dynamics, coherent backgrounds, and temporal consistency.

I. More Qualitative Results

  • Across ViViD and ViT-HD comparisons, KeyTailor produces more realistic, natural videos with finer garment dynamics, coherent background details, and temporal consistency.Source: §I · p. 15

J. Limitations

KeyTailor still struggles to reliably handle complex garment–body interaction motions.

J. Limitations

  • Complex garment–body interaction motions remain difficult to handle reliably, given pretrained-model capacity and fine-grained action-captioning challenges.Source: J. Limitations · p. 15

Related papers

Loading 2512.20340v3…