SwiftTry: Fast and Consistent Video Virtual Try-On with Diffusion Models

• Over 60% faster runtime than ViViD (Fang et al. 2024) is reported for SwiftTry, which preserves garment texture detail and consistency.The introduction also states that SwiftTry outperforms existing video virtual try-on methods in accuracy and efficiency.Source: Figure 1 · p. 1, Introduction · p. 2
• Video try-on methods can flicker between frames, while rendering high-quality results over long sequences remains computationally expensive.These limitations undermine realism and practical use.Source: Introduction · p. 1
• Existing datasets lack the diverse in-the-wild scenes and movements needed for evaluation, and inaccurate human-parsing masks can degrade results.VVT has basic garments, uniform backgrounds, static camera angles, and repetitive motions; ViViD still struggles with complex movements and diverse backgrounds.Source: Introduction · p. 1
• TikTokDress provides 817 videos with realistic scenes, diverse garments, complex movements, varied backgrounds, and high resolution for training and evaluating video try-on models.Source: Introduction · p. 2
• SwiftTry adds temporal attention to a diffusion-based image virtual try-on UNet and uses ShiftCaching to smooth transitions between video chunks while reducing redundant computation.ShiftCaching is applied at inference, while temporal attention is incorporated for training on video try-on data.Source: Introduction · p. 2

Image Virtual Try-On

• Traditional image try-on commonly uses a two-stage GAN pipeline that warps clothing before fusing it with the person image, but can generalize poorly to complex backgrounds.Warping approaches include thin-plate splines, spatial transformer networks, and flow estimation.Source: Related Work · p. 2
• Diffusion-based image try-on methods improve realism, but extending them to video can cause inter-frame inconsistency and flickering.TryOnDiffusion uses two U-Nets and requires many same-person, different-pose image pairs; StableVITON uses zero cross-attention for garment conditioning, while IDM-VTON combines low-level GarmentNet features with IP-Adapter semantics.Source: Related Work · p. 2

Video Virtual Try-On

• Video try-on methods use optical flow, memory refinement, transformers, or diffusion, while prior work has limits in video length or dataset complexity.FW-GAN warps preceding frames, MV-TON refines prior-frame features, and ClothFormer combines TPS- and appearance-based warping; Tunnel Try-On handles camera movement but its demos span only a few seconds, while ViViD has simple backgrounds and movements. SwiftTry proposes temporal smoothness across chunks and caching to reduce redundant computation during long-video inference.Source: Related Work · p. 2

Methods

• SwiftTry aims to generate a garment-conditioned video that preserves the person’s motion, source background, and garment color and texture.Because paired source and target videos with identical motion are difficult to collect, training uses a single video with garment regions masked and inpainted using the garment image as guidance.Source: Methods · p. 2

Overall Architecture

• SwiftTry trains an image try-on diffusion model, then adapts its Main U-Net for video with temporal attention layers.The image model uses a 13-channel input combining latent noise, masked-image latents, a binary mask, and a pose map; a Garment U-Net supplies garment features through Reference Attention.Source: Overall Architecture · pp. 2–3, Figure 2 · p. 3
• Temporal attention operates across frames at matching latent locations, factorized after spatial attention to avoid full 3D attention; training updates only the temporal layers.The model adds sinusoidal frame-position encodings and trains the temporal layers on video data while keeping other layers unchanged.Source: Overall Architecture · pp. 3–4

ShiftCaching Technique

• ShiftCaching shifts non-overlapping video chunks between DDIM steps and uses cached features for partial Main U-Net computation to avoid recomputing overlapping regions.Previous overlap-based methods trade speed for consistency: small overlaps can cause flickering and texture artifacts, while large overlaps slow inference. Masked Temporal Attention blocks information flow from less accurate cached features to more accurate ones while allowing the reverse, supporting smoothness and quality.Source: ShiftCaching Technique · p. 4

TikTokDress Dataset

• TikTokDress pairs high-resolution videos with garments, poses, and precise binary cloth masks, targeting complex backgrounds, diverse movements, and balanced gender representation.The dataset is intended to address limitations of single-image datasets and VVT, including restricted poses, uniform movement, white backgrounds, and low resolution.Source: TikTokDress Dataset · pp. 4–5
• Manually corrected garment masks improve video try-on quality, while standard segmenters used by VITON-HD, DressCode, and VVT struggle with complex videos.Mask quality is described as crucial for try-on results; the cited comparison is qualitative.Source: TikTokDress Dataset · p. 5
• TikTokDress covers dancing and other dynamic poses, varied camera distances, indoor and outdoor backgrounds with complex lighting, and garments from T-shirts to chainmail tops.The range is intended to address real-world challenges in video virtual try-on.Source: TikTokDress Dataset · p. 5

Video collection and annotation.

• The dataset contains 817 video-garment pairs and over 270,000 RGB frames sampled at 30 frames per second, with 2D keypoints and dense poses computed using DWPose and DensePose.Videos are 10–30-second TikTok clips; the collection was expanded for background, skin-tone, and clothing diversity, and low-quality videos or garments were excluded.Source: Video collection and annotation · p. 5
• SAM 2 masks for clothing and arms required an algorithm for selecting frames and prompts, plus manual refinement for complex garments (Ravi et al. 2024).The procedure addresses SAM 2’s sensitivity to prompt points and particular frames.Source: Video collection and annotation · p. 5

Experiments

• The evaluation uses VVT and TikTokDress with paired SSIM and LPIPS, unpaired VFID, and FPS; training first adapts garment and inpainting modules, then adds temporal attention.VVT has 791 paired videos at 256 × 192 resolution; TikTokDress has 693 training and 124 testing videos at 540 × 720. Temporal attention is initialized from AnimateDiff weights.Source: Experiments · pp. 5–6

Comparisons with Prior Approaches

• On VVT, SwiftTry excels in VFID, performs competitively in SSIM and LPIPS, and reaches 2.27 FPS with ShiftCaching, over 1.5 times faster.Table 3 reports the VVT comparison; the prose gives no exact SSIM, LPIPS, or VFID values.Source: Table 3 · p. 6, Comparisons with Prior Approaches · p. 6
• On TikTokDress, compared methods can produce accurate individual frames but often flicker and lack temporal coherence and motion handling across frames.The dataset comparison uses 540 × 720 videos with varied backgrounds, dynamic movements, and diverse body poses.Source: Experiments · p. 5, Table 4 · p. 6, Comparisons with Prior Approaches · p. 6
• Prior methods show garment-texture variation across frames and significant jitters between adjacent frames.These qualitative issues are shown for TikTokDress and VVT comparisons.Source: Comparisons with Prior Approaches · p. 6, Figure 7 · p. 7, Figure 8 · p. 7

Ablation Study

• Random shifts provide the best consistency; partially computing 50% of frames accelerates inference 1.5 times while maintaining comparable quantitative metrics.This is the ShiftCaching ablation on VVT.Source: Ablation Study · p. 6, Table 5 · p. 6
• Half Attention performs best, allowing bad features from timestep t + 2 to access good features from timestep t + 1 while good features interact with each other.The study examines masking choices for temporal attention on VVT.Source: Table 6 · p. 6, Ablation Study · p. 7
• Matching training and inference video chunk lengths at N = 16 yields the best results.The study compares inference chunks of 8, 16, and 24 frames with training set to 16 frames.Source: Table 7 · p. 6, Ablation Study · p. 7

• ShiftCaching maintains temporal smoothness across video chunks while reducing redundant computations during video inference.The paper says this improves the efficiency and quality of video virtual try-on and makes it more practical for real-world applications.Source: Conclusion · p. 7
• TikTokDress is a video virtual try-on dataset featuring diverse backgrounds, complex movements, and high-resolution videos.It is designed to address limitations of existing datasets and support future research.Source: Conclusion · p. 7

• The supplementary material provides a comprehensive overview of the proposed method’s training process.Source: Supplementary Material · p. 10
• It elaborates on the masking strategies used in the attention module.Source: Supplementary Material · p. 10
• It gives detailed information about how the TikTokDress dataset was created.Source: Supplementary Material · p. 10
• It presents additional qualitative results on both the VVT and TikTokDress datasets.Source: Supplementary Material · p. 10

Optimization

• Both stages use Adam (Kingma and Ba 2014) with β1 = 0.9, β2 = 0.999, and a fixed learning rate of 1e −5, plus Min-SNR Weighting (Hang et al. 2023) with γ = 5.0.Training uses mixed-precision FP16.Source: Training Detail · p. 10

Hardware

• Training runs in PyTorch with Diffusers (von Platen et al. 2022) on one 40GB A100; stage 1 takes approximately 3 days and stage 2 around 30 hours.FPS benchmarks are also measured on the A100 GPU.Source: Training Detail · p. 10

Training

• Stage 1 optimizes the Main UNet decoder, Garment UNet, and Pose Encoder on 1024×768 VITON-HD; stage 2 optimizes only motion-related modules using 16-frame clips at 512 × 384.All other modules are fixed in stage 1.Source: Training Detail · p. 10

• Half Attention yields the best performance, transferring accurate t + 1 features to query frames.Each query frame attends to accurate features, enabling their transfer to less accurate ones.Source: Different Masking Attention Mechanisms · p. 10
• Full Attention lets every frame attend to all others, risking less accurate features influencing accurate ones.In the illustrated masks, white cells represent 0 and gray cells represent −∞.Source: Different Masking Attention Mechanisms · p. 10, Figure 9 · p. 10
• Quarter Attention lets less accurate frames attend to all others while preventing accurate features from attending to less accurate ones.This is intended to preserve the quality of accurate features.Source: Different Masking Attention Mechanisms · p. 10, Figure 9 · p. 10
• Causal Attention makes each frame’s computation depend only on succeeding frames, which contain less noise.Source: Different Masking Attention Mechanisms · p. 10, Figure 9 · p. 10

TikTokDress Dataset

• The dataset spans indoor and outdoor backgrounds, varied camera angles and person distances, Black, Asian, and White skin tones, and multiple garment types.Examples include T-shirts, sweaters, crop tops, long-sleeve T-shirts, and dresses.Source: TikTokDress Dataset · p. 10
• Incorrect or incomplete garment masks can produce hallucinated clothing shapes or stripes, while precise masks yield more accurate try-on results.The examples show both a tank-top-like output from an incorrect mask and extra red stripes when the garment is not fully masked.Source: TikTokDress Dataset · p. 10, Figure 10 · p. 11
• Agnostic masks combine SAM2 garment and arm segmentations from a clear frame with the most visible body parts, then are dilated three to seven times.Garment prompts use body-center and shoulder positives; arm prompts use elbow and wrist positives, with specified negative points for both. Frame selection ranks visible parts first, then pose score.Source: TikTokDress Dataset · pp. 10–11
• Garment images are sourced through Google Lens at the highest available resolution, with manual keyword searches as fallback; videos are excluded if no suitable image is found.Manual searches use garment type, color, and brand keywords.Source: TikTokDress Dataset · p. 11

Qualitative results on the VVT dataset

• On VVT, SwiftTry preserves garment textures and reduces flickering relative to ViViD, including flicker in the garment’s “GAP” lettering caused by ViViD’s temporal averaging.The authors attribute smoother, more stable results to ShiftCaching.Source: Qualitative results on the VVT dataset · p. 11, Figure 13 · p. 12

Qualitative results on the TikTokDress dataset.

• On TikTokDress, SwiftTry produces consistent garment textures and smooth, continuous long videos, with results reported as superior to those trained on VVT.The method uses SAM 2 (Ravi et al. 2024), manually refined dataset masks, and SAM 2 frame selection; examples span varied backgrounds and show texture consistency over time.Source: Qualitative results on the TikTokDress… · p. 12, Figure 12 · p. 12, Figure 14 · p. 13