Source-linked AI summary
FlowVVTON: Flow-Guided Mask-Free Video Virtual Try-On
Shengyao Chen, Xianbing Sun, Liqing Zhang, Jianfu Zhang
TL;DR
Video virtual try-on struggles with mask and keypoint failures under motion and lacks explicit supervision for cross-frame correspondence. FlowVVTON removes those preprocessing dependencies through a two-stage, flow-guided training framework, achieving a 5.7× VFID-R improvement over SwiftTry while using optical flow only during training.
Problem
Video try-on must maintain garment consistency during motion, but unreliable masks and keypoints cause artifacts while attention-only methods lack explicit motion supervision.
Method
FlowVVTON trains mask-free spatial alignment first, then applies a per-layer flow-warped latent loss across generation-model layers for multi-scale temporal supervision.
Results
5.7× VFID-R improvement over SwiftTry is reported on TikTokDress, with multi-scale flow supervision contributing to temporal consistency.
Takeaways & Limitations
The framework performs mask-free video try-on using raw video frames and a garment image, without flow computation during inference.
Takeaways & Limitations
The method depends on optical-flow quality during training, uses a UNet-specific architecture, and faces current resolution constraints.
Abstract
from arXiv · showhide
Video virtual try-on aims to transfer a target garment onto a moving person across video frames. Current methods rely on human parsing masks or pose keypoints that frequently fail under large motions and occlusions, causing boundary artifacts and temporal inconsistency. A further limitation is that most approaches rely solely on attention mechanisms for temporal modeling, providing no explicit motion supervision. We propose FlowVVTON, a mask-free framework that eliminates parsing mask dependency entirely. Optical flow is used solely as a training-time supervision signal: a flow-warped latent loss, applied across all layers of the generation model, enforces multi-scale temporal consistency by aligning adjacent-frame features under explicit physical motion constraints. A two-stage training strategy establishes mask-free spatial alignment before introducing flow-guided temporal supervision. Experiments on TikTokDress show that FlowVVTON outperforms baselines by substantial margins, particularly in temporal consistency (5.7$\times$ VFID-R improvement over SwiftTry), while requiring no segmentation masks, pose keypoints, or region annotations at any stage.
FlowVVTON: Flow-Guided Mask-Free Video Virtual Try-On
The paper is authored by Shengyao Chen, Xianbing Sun, Liqing Zhang, and Jianfu Zhang.
- The authors are Shengyao Chen, Xianbing Sun, Liqing Zhang, and Jianfu Zhang.
Introduction
FlowVVTON addresses temporal inconsistency and boundary artifacts in video try-on by removing preprocessing masks and adding explicit flow-guided supervision. Its two-stage training aligns garments spatially before enforcing multi-scale temporal consistency, improving results on TikTokDress.
- Motivation: Video try-on must preserve garment stability across frames despite motion, rotation, and limb occlusion.Failure appears as flickering, texture drift, and boundary artifacts.
- Motivation: Parsing masks and pose keypoints become unreliable under fast motion, self-occlusion, and challenging lighting, propagating errors into generated boundaries and textures.
- Motivation: Attention-only temporal modeling lacks explicit motion supervision and can break down during rapid movements or large body rotations.
- Method: Optical flow provides dense pixel correspondence and temporal motion information for mask-free video try-on when used as training-time supervision.
- Method: FlowVVTON uses a per-layer flow-warped latent loss to align adjacent-frame features, while optical flow is omitted during inference.The loss supplies multi-scale temporal supervision across the generation model.
- Results: 5.7× VFID-R improvement over SwiftTry demonstrates substantially better temporal consistency on TikTokDress.
- Contribution: FlowVVTON requires no parsing masks, pose keypoints, or region annotations and targets robust performance under large body motions.
- Training Strategy: The two-stage pipeline first establishes mask-free spatial alignment, then introduces flow-guided temporal supervision for cross-frame consistency.
Related Work
Related work progresses from image-based and frame-independent try-on toward temporally modeled video generation, mask-free conditioning, and optical-flow guidance. FlowVVTON differs by applying a per-layer flow-warped latent loss to adjacent-frame UNet features.
- Image Virtual Try-On: Image virtual try-on evolved from warp-and-blend GAN pipelines to diffusion models with garment feature injection.
- Video Virtual Try-On: Video try-on requires temporal consistency, while early frame-independent methods used post-hoc smoothing and lacked joint spatiotemporal modeling.
- Mask-Free Try-On: Mask-free methods avoid upstream parser error propagation, but video approaches have used sparse keypoints, coarse boxes, or pose-driven animation instead of dense masks.
- Optical Flow for Video Generation: Optical flow has served as a loss, conditioning signal, or model-internal representation for improving video generation.
- Optical Flow for Video Generation: FlowVVTON applies a per-layer flow-warped latent loss to align adjacent-frame UNet features under motion constraints.This provides explicit geometric supervision for temporal consistency.
Method
FlowVVTON uses a mask-free two-UNet latent-diffusion architecture for garment transfer, combining mutual self-attention with temporal attention. Its two-stage training adds confidence-weighted, multi-scale flow supervision across main-UNet layers while preserving spatial alignment.
- Architecture: A reference UNet extracts multi-scale garment features and injects them into the main video-generating UNet through mutual self-attention.The main UNet uses frame features as queries and garment features as keys and values, enabling spatial garment transfer without explicit warping.
- Temporal modeling: Only temporal attention layers are trained after inflating a 2D UNet, while spatial weights and the reference UNet remain frozen to preserve image-pretrained spatial quality.Temporal attention is inserted after each spatial block to model motion across video frames.
- Mask-free input: The main UNet receives concatenated noisy latents and VAE-encoded source-video latents, avoiding parsing masks, pose keypoints, and region labels.The source latent retains background, body contours, lighting, and motion information.
- Flow supervision: Optical flow is used only during training to align adjacent-frame features through a flow-warped latent loss applied at every main-UNet layer.The loss operates in feature space, providing semantic multi-scale supervision without requiring inference-time flow.
- Flow supervision: Confidence-weighted, multi-stride flow supervision downweights unreliable occlusion regions and captures adjacent-, medium-, and long-range temporal consistency.Forward-backward consistency produces confidence weights, while strides 1, 2, and 4 use directly computed flows with decreasing weights.
- Training strategy: Training proceeds in two stages: image-level fine-tuning first establishes mask-free per-frame garment transfer, then video-level flow-guided training activates temporal learning.Stage 2 freezes Stage 1 spatial weights and introduces the flow loss across all main-UNet layers alongside temporal attention.
Experiments
FlowVVTON is evaluated on TikTokDress using paired and unpaired settings, with quantitative, qualitative, ablation, and flow-quality analyses. It achieves strong temporal consistency and visual quality, while multi-scale flow supervision and 32 RAFT iterations provide the best reported trade-offs.
- Evaluation Setup: Evaluation uses paired and unpaired TikTokDress settings, reporting SSIM, LPIPS, and VFID metrics with I3D and ResNeXt backbones.VFIDI3D measures spatiotemporal dynamics, while VFIDRN captures spatial texture quality.
- Qualitative Comparison: Qualitatively, FlowVVTON produces sharper garment textures, cleaner boundaries, and more stable texture adherence than SwiftTry under large body motions.The comparison highlights rapid subject rotation and crossed arms as challenging cases for the mask-based pipeline.
- Quantitative Comparison: FlowVVTON achieves the best paired results across all four reported metrics, with a 6× VFID-R improvement over SwiftTry (0.71 vs. 4.30).The paired evaluation is reported on the TikTokDress test set.
- Quantitative Comparison: 5.7× VFID-R gain over SwiftTry (0.60 vs. 3.42) is reported for FlowVVTON in the unpaired setting.CatV2TON and MagicTryOn perform worse than SwiftTry in this comparison.
- Ablation Study: Removing flow supervision raises VFID-I from 30.99 to 32.48 and VFID-R from 0.60 to 1.24, while removing confidence weighting degrades VFID-R to 0.67.The qualitative ablation associates confidence-weighting artifacts with occlusion boundaries.
- Ablation Study: Strides {1, 2, 4} achieve VFID-R of 0.60, a 17% improvement over stride-1 only, whereas stride-8 slightly degrades VFID-R to 0.67.The results support balancing temporal coverage against flow reliability.
- Flow Quality Analysis: 32 RAFT iterations are selected because 16 degrades VFID-I from 30.99 to 31.46, while 48 yields only marginal VFID-I improvement to 30.92 with identical VFID-R.The 48-iteration flow is visually indistinguishable from the 32-iteration result, despite higher per-iteration training cost.
Conclusion
FlowVVTON removes preprocessing mask dependencies by using optical flow only as a training-time, per-layer flow-warped latent loss. The framework reports robust temporal consistency and identifies flow-warped loss and multi-scale strides as essential, while retaining limitations in flow quality, architecture, and resolution.
- FlowVVTON is a mask-free video virtual try-on framework that eliminates preprocessing mask dependencies.
- Optical flow is used solely during training through a per-layer flow-warped latent loss to improve temporal consistency and reduce boundary artifacts.The loss is applied across the generation model's layers and is especially relevant under large motions.
- The approach depends on optical flow quality during training, uses a UNet-specific architecture, and faces current resolution constraints.