Source-linked AI summary
LiveVVT: High-Fidelity Video Virtual Try-On in Real Time
Yushe Cao, Shikun Feng, Ruxiang Duan, Liyong Wang, Dianxi Shi, Chun Yu, Junliang Xing
TL;DR
Continuous video virtual try-on needs high fidelity without the latency and computation of full-clip bidirectional diffusion, but simple causal conversion degrades quality. LiveVVT uses rolling bounded-bidirectional diffusion, complementary memories, and progressive distillation, achieving strong fidelity with 26× lower latency for real-time streaming.
Problem
Continuous source-conditioned streaming VVT remains largely unexplored because full-clip bidirectional processing is slow, while naive causal conversion disrupts pretrained priors and long-term consistency.
Method
LiveVVT combines staggered-noise local bidirectional denoising, causal recurrence, complementary temporal and appearance memories, and progressive distillation for few-step streaming generation.
Results
LiveVVT demonstrates strong fidelity on paired and unpaired long-sequence benchmarks while achieving 26× lower latency.
Takeaways & Limitations
LiveVVT supports high-fidelity real-time video virtual try-on with bounded look-ahead and recurrent generation.
Takeaways & Limitations
Performance may become less reliable when observed appearance differs markedly from the frontal reference because of severe illumination changes, self-occlusion, extreme articulation, or large out-of-plane rotation.
Abstract
from arXiv · showhide
Diffusion-based Video Virtual Try-On (VVT) achieves high visual fidelity through bidirectional spatio-temporal modeling, but complete-clip dependence incurs prohibitive latency and computational overhead in practical continuous deployment. Naively enforcing causality disrupts pretrained bidirectional priors and substantially degrades synthesis quality. We introduce LiveVVT, a rolling streaming diffusion framework that preserves bounded bidirectional modeling within causal recurrent generation. Within a fixed-size window, LiveVVT jointly denoises multiple video chunks under bounded look-ahead, preserving local bidirectional interactions while emitting one clean chunk per iteration. Beyond the window, two complementary memories sustain long-term consistency: a bounded temporal memory propagates recent dynamics and occlusion context, whereas a persistent global appearance memory, constructed once from the target garment and a frontal try-on keyframe, anchors garment details and dressed appearance throughout the stream. We further introduce a progressive distillation framework integrating bidirectional VVT learning, teacher-trajectory regression for causal few-step adaptation, and Collaborative Matching Distillation, which couples teacher-distribution matching with rolling flow matching on real videos to align optimization with recurrent inference. Experiments on paired and unpaired long-sequence benchmarks demonstrate superior generation quality over similarly sized models, with $26\times$ lower latency and $11\times$ higher throughput, enabling high-fidelity real-time streaming VVT.
1 Introduction
LiveVVT addresses the latency, computation, and consistency challenges of streaming video virtual try-on by combining bounded local bidirectional denoising with causal recurrence and complementary memories. Progressive distillation transfers offline bidirectional priors to recurrent few-step generation for high-fidelity real-time VVT.
- Problem: Streaming VVT remains challenging because clip-based bidirectional methods require future frames and independently process windows, causing high latency and redundant computation.Continuous source-conditioned streaming that jointly delivers high fidelity and low latency remains largely unexplored.
- Motivation: Simply imposing causal attention on an offline model disrupts pretrained bidirectional spatio-temporal priors, while unbounded history is impractical and aggressive truncation causes appearance drift.Recurrent generation also differs from offline denoising, motivating dedicated adaptation rather than direct causal conversion.
- LiveVVT framework: LiveVVT combines staggered-noise joint denoising within a fixed-size window with causal recurrence across windows, enabling high-fidelity streaming VVT under bounded look-ahead.It emits one clean chunk per iteration while retaining local bidirectional interactions.
- Long-term consistency: Two complementary memories sustain long-term consistency: bounded temporal memory propagates recent dynamics and occlusion context, while persistent global appearance memory preserves garment details and dressed appearance.The global memory avoids repeated reference encoding throughout the stream.
- Progressive distillation: Progressive distillation transfers high-fidelity offline priors to recurrent few-step generation through bidirectional VVT learning, teacher-trajectory regression, and Collaborative Matching Distillation.These components adapt the model to the recurrent streaming setting.
2 Related Work
Virtual try-on has progressed from explicit garment warping and composition to conditional diffusion for improved realism and detail preservation, while autoregressive video generation has advanced from discrete visual tokens to continuous latent modeling. LiveVVT extends these directions with source-conditioned streaming try-on using bounded rolling denoising and complementary temporal and appearance memories for low-latency, high-throughput inference.
- Image Virtual Try-On: LiveVVT enables source-conditioned streaming try-on through bounded rolling denoising, recurrent temporal memory, and persistent global appearance memory.These mechanisms support low-latency, high-throughput real-time streaming inference.
- Image Virtual Try-On: Image virtual try-on evolved from explicit garment warping and composition toward conditional diffusion that improves realism and fine-grained detail preservation.The passage cites Han et al. 2018, 2019; Yang et al. 2020; Ge et al. 2021 for earlier approaches and Zhu et al. 2023; Choi et al. 2024; Xu et al. for diffusion-based methods.
- Autoregressive Video Generation: Autoregressive video generation has progressed from discrete visual-token prediction to continuous latent-space modeling.Recent work adapts bidirectional diffusion models into few-step causal generators through trajectory- or distribution-level distillation, self-generated rollouts, and adversarial post-training.
3 Method
LiveVVT enables causal real-time video virtual try-on by jointly denoising staggered chunks within a fixed rolling window, while temporal and global memories preserve continuity and appearance. Progressive distillation transfers bidirectional VVT capabilities to causal few-step recurrent generation and aligns the rollout with real-video states.
- Dual-scale memory: LiveVVT constructs its appearance anchor from a frontal A-Pose source keyframe and its condition, selecting the closest-pose training frame when no reference is available.The pose-alignment distance determines the training-frame selection.
- Rolling streaming generation: LiveVVT jointly advances staggered-noise chunks in a fixed-size rolling window, using bounded look-ahead to emit one clean chunk per update without accessing future frames.Only the leading chunk reaches the clean state; active chunks interact bidirectionally, while fixed-window processing keeps per-update computation and activation memory independent of stream length.
- Dual-scale memory: A bounded temporal memory propagates recent motion and occlusion context, while persistent global appearance memory anchors garment details and dressed appearance across the stream.Temporal memory caches key–value features from completed clean latents; global memory is computed once from garment features and a frontal try-on reference, then reused by every window.
- Progressive distillation: Progressive distillation transfers bidirectional VVT priors to causal few-step inference through bidirectional learning, teacher-trajectory regression, and Collaborative Matching Distillation.Teacher-trajectory regression adapts the student’s attention pattern and denoising trajectory, while CoMD couples teacher-distribution matching with Rolling Flow Matching on real videos to supervise recurrent deployment states.
4 Experiments
Experiments show that LiveVVT delivers the strongest long-sequence video quality while enabling real-time streaming with low first-chunk latency, high sustained throughput, and stable per-update cost. Ablations further show that complementary memories and progressive distillation are essential for preserving appearance and adapting bidirectional VVT to rolling causal inference.
- Long-sequence try-on quality: LiveVVT delivers the strongest video quality on long-sequence try-on benchmarks despite causal recurrence, bounded look-ahead, and few-step sampling.Two-stage image-to-video pipelines consistently underperform video-native methods because animating one try-on frame misses evolving deformation, occlusion, and person–garment interactions.
- Real-time efficiency: 1.56 seconds first-chunk latency and 22.39 FPS sustained throughput establish LiveVVT as a high-fidelity real-time streaming system.First-chunk latency controls responsiveness, while sustained throughput controls continuity during subsequent updates.
- Long-horizon consistency: Figure 3 shows that LiveVVT maintains long-horizon garment consistency better than MagicTryOn under window boundaries and large viewpoint changes.MagicTryOn exhibits appearance drift across clip windows, including pronounced drift in the shoulder-bag strap.
- Long-stream scalability: LiveVVT’s fixed-size rolling window stabilizes per-update latency at approximately 0.5 seconds as sequence length grows, unlike offline bidirectional methods.Its total inference time increases smoothly with stream duration, whereas offline methods incur rapidly growing per-update and end-to-end latency.
- Complementary memory mechanisms: Global frontal try-on memory reduces appearance drift, while temporal memory provides broader video-level gains across long sequences.Ablations retain the garment memory component and isolate the frontal try-on component Af and temporal memory H on ViViD-SL.
- Progressive distillation: Progressive distillation is necessary for rolling inference: teacher-trajectory regression adapts causal few-step generation, and CoMD yields further gains across all metrics.Directly applying bounded-window recurrent inference to Stage I substantially degrades quality, while Stage II provides a stable initialization for the student.
5 Conclusion
LiveVVT is a rolling diffusion framework for high-fidelity streaming video virtual try-on. It preserves local bidirectional modeling with bounded look-ahead while using temporal and global appearance memories to maintain consistency across long streams.
- LiveVVT uses a fixed staggered-noise window to preserve local bidirectional spatio-temporal modeling under bounded look-ahead while emitting one clean chunk per update.
- Complementary temporal and persistent global appearance memories propagate recent motion and occlusion context while anchoring garment details and dressed appearance across long streams.
- The framework further uses progressive distillation that integrates bidirectional VVT learning and teacher-trajectory regression.
A Additional Quantitative Comparisons
On the longer TikTokDress-L benchmark, LiveVVT achieves the strongest paired and unpaired performance, combining video-level realism with frame-level garment fidelity across extended, unconstrained sequences.
- A Additional Quantitative Comparisons: TikTokDress-L comprises the 60 longest TikTokDress test videos, spanning 352–693 frames with diverse motions, viewpoint changes, and garment categories.The benchmark extends evaluation to longer and more unconstrained real-world sequences.
- A Additional Quantitative Comparisons: LiveVVT achieves the best paired performance on TikTokDress-L across all four metrics: VFIDI 19.668, VFIDR 0.412, SSIM 0.842, and LPIPS 0.103.These results indicate high video-level realism without sacrificing frame-level garment structure or perceptual fidelity.
- A Additional Quantitative Comparisons: LiveVVT ranks first on both unpaired metrics, obtaining VFIDI 30.447 and VFIDR 0.762 for arbitrary target garments.The unpaired setting evaluates generalization to unmatched person–garment combinations without frame-wise ground truth, using VFID for video-level realism and distributional consistency.
B Fast Image Try-On
LiveVVT supports fast single-image try-on through the same multimodal model used for streaming video, without a task-specific image branch. On VITON-HD, it achieves strong paired fidelity and similarity metrics while substantially reducing single-image inference time.
- Training protocol: LiveVVT uses one multimodal model for streaming video and single-image try-on, training images as one-frame videos without requiring a frontal keyframe.Training combines dedicated video data with VITON-HD and DressCode image try-on data.
- Image quality: On the 2,032-image VITON-HD test set at 512 × 384, LiveVVT achieves the best paired FID (5.910) and KID (0.419), with the lowest paired LPIPS (0.054).Its paired SSIM is 0.873, close to the strongest result of 0.883.
- Inference speed: 0.31 seconds per image makes LiveVVT 16.94×, 7.74×, 3.65×, and 23.29× faster than IDM-VTON, OOTDiffusion, CatVTON, and UniFit, respectively.The comparison uses 512 × 384 single-image inference; the baselines require 5.25, 2.40, 1.13, and 7.22 seconds, respectively.
C Additional Ablation Studies · C.1 Analysis of the Hyperparameter λ · C.2 Cached vs. Repeated Garment Conditioning
These ablations show that CoMD benefits from complementary real-video trajectory supervision, while cached garment conditioning preserves comparable quality more efficiently than repeated encoding. The λ study favors balancing teacher-prior transfer with real-trajectory supervision rather than letting RFM dominate, and the conditioning study finds no consistent quality advantage from recomputation.
- C.1 Analysis of the Hyperparameter λ: CoMD combines teacher-distribution matching with real-video rolling-flow supervision through LCoMD = LDMD + λLRFM.RFM reconstructs deployment-aligned temporal caches, global appearance memory, and staggered-noise schedules from real-video windows.
- C.1 Analysis of the Hyperparameter λ: Introducing RFM improves both VFID metrics and SSIM for λ ∈ {0.1, 0.2, 0.4} over DMD-only training on ViViD-SL paired evaluation.The accompanying LPIPS increase indicates that RFM should remain complementary to DMD rather than dominate the objective.
- C.1 Analysis of the Hyperparameter λ: λ = 0 reduces CoMD to DMD-only training, whereas nonzero λ values add direct supervision on real rolling trajectories.This supervision improves video-level fidelity and structural consistency beyond teacher-distribution matching alone, while the LPIPS increase argues against allowing RFM to dominate.
- C.2 Cached vs. Repeated Garment Conditioning: LiveVVT encodes the unchanged target garment once and stores its attention key/value features in persistent global appearance memory.This Cached design is compared with Repeated Encoding, which recomputes garment conditioning at every rolling update instead of retaining garment KV features.
- C.2 Cached vs. Repeated Garment Conditioning: Cached conditioning performs better on both VFID metrics, while Repeated Encoding yields small gains in SSIM and lower LPIPS.The metric-dependent results show no consistent quality advantage for recomputing garment conditioning.
- C.2 Cached vs. Repeated Garment Conditioning: Throughput decreases from 22.39 to 19.73 FPS when garment conditioning is recomputed with Repeated Encoding.The efficiency cost is therefore more pronounced for Repeated Encoding despite its comparable generation quality.
D Additional Visual Comparisons · E Long-Sequence Try-On Examples
Additional paired and unpaired long-sequence comparisons show that LiveVVT preserves garment detail and sequence-level appearance consistency across garment types, viewpoint changes, rotations, scale changes, occlusion, and challenging in-the-wild motion. Compared with competing methods, it reduces texture, color, boundary, and window-transition failures while retaining robust capability despite occasional local artifacts.
- D Additional Visual Comparisons: ViViD and CatV2TON recover overall garment layout but often lose fine texture, deviate in color, or create boundary artifacts, especially under large nonrigid deformation.The failures affect collars, hems, and occluded regions, where consistent modeling across successive frames is required.
- D Additional Visual Comparisons: MagicTryOn’s independently processed windows can produce locally plausible outputs that change texture or color at window boundaries because they lack an explicit sequence-level appearance state.This limitation is illustrated by a visible color shift in later frames of the generated shorts.
- D Additional Visual Comparisons: LiveVVT maintains comparable local detail while preserving stable garment color and texture across complete sequences, unlike MagicTryOn’s visible later-frame appearance shift.Figure 10 specifically shows MagicTryOn’s color shift in black shorts, whereas LiveVVT remains consistent.
- E Long-Sequence Try-On Examples: Unpaired long-sequence examples test independently selected garments across tops, summer outfits, and dresses, including full-body rotations and substantial subject-scale changes from camera motion.These conditions require the generated garment to remain identifiable as visible regions change.
- E Long-Sequence Try-On Examples: On in-the-wild videos with complex backgrounds, rapid articulated motion, and severe self-occlusion, LiveVVT generally preserves garment category, appearance, and temporal continuity.The results extend beyond controlled benchmark scenes and support practical interactive streaming use.
- E Long-Sequence Try-On Examples: Occasional local artifacts remain around fine garment boundaries and under extreme motion, identifying limitations for future improvement despite robust long-sequence try-on capability.The remaining issues occur in challenging in-the-wild conditions rather than undermining overall sequence continuity.
F Limitations and Discussion
LiveVVT’s streaming design is most reliable within a defined operating regime: reference-dependent appearance memory and bounded future context can limit performance under large appearance changes or abrupt motion. Remaining errors are mainly localized, while current frame-level and distributional metrics incompletely capture temporal consistency.
- Operating Regime: Reference-dependent appearance memory can become less reliable when observed appearance departs markedly from the garment and frontal source-person reference.The reference anchor preserves person-specific dressed appearance over extended streams, but its effectiveness is conditioned on similarity to the reference.
- Residual Errors: Residual failures are predominantly localized to fine garment boundaries, heavily occluded regions, and rapidly moving body parts, appearing as small distortions or texture inconsistencies.SSIM and LPIPS assess frame-level correspondence, whereas VFID assesses distributional video quality, so evaluation should also include explicit temporal-consistency measures.
- Efficiency Trade-off: Fixed-size rolling windows, cached garment features, and few-step sampling create a context–latency trade-off that may disadvantage abrupt motion or rapid appearance changes.These choices make per-update cost largely insensitive to sequence length while restricting future context available at each emission; adaptive windowing or content-aware memory updates are suggested remedies.