PaperlayerView exact revision on arXiv

Source-linked AI summary

RealVVT: Towards Photorealistic Video Virtual Try-on via Spatio-Temporal Consistency

Siqi Li, Zhengkai Jiang, Jiawei Zhou, Zhihong Liu, Xiaowei Chi, Haoqian Wang

arXiv:2501.08682v2cs.CVcs.GR

TL;DR

Video try-on must maintain garment appearance as people move and change viewpoint, a challenge that becomes especially difficult in long sequences. RealVVT builds on pretrained video models with attention mechanisms and pose-guided generation. The authors report state-of-the-art performance and high-quality, coherent try-on videos, while noting that inconsistent dataset masks and pose annotations remain a challenge.

  • Problem

    Video try-on remains challenged by preserving consistent garment appearance through changing poses and viewpoints, with framewise errors accumulating in long sequences.

    Source: §1 · p. 2

  • Method

    RealVVT builds on pretrained video models and combines clothing-temporal attention, agnostic-mask-guided attention loss, and pose-guided keyframe generation for long videos.

    Source: §1 · p. 2, §3.5 · p. 6

  • Results

    RealVVT achieves state-of-the-art performance and produces high-quality virtual try-on videos with temporal coherence and garment consistency.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The framework produces high-resolution, photorealistic images and is presented as suitable for practical virtual try-on applications.

    Source: §5 · p. 8

    Existing video datasets contain inaccurate, temporally inconsistent agnostic masks and DensePose extractions, and substantial effort remains to address their instability.

    Source: B. Limitations & Discussion · p. 11

Abstract

from arXiv · show

Virtual try-on has emerged as a pivotal task at the intersection of computer vision and fashion, aimed at digitally simulating how clothing items fit on the human body. Despite notable progress in single-image virtual try-on (VTO), current methodologies often struggle to preserve a consistent and authentic appearance of clothing across extended video sequences. This challenge arises from the complexities of capturing dynamic human pose and maintaining target clothing characteristics. We leverage pre-existing video foundation models to introduce RealVVT, a photoRealistic Video Virtual Try-on framework tailored to bolster stability and realism within dynamic video contexts. Our methodology encompasses a Clothing & Temporal Consistency strategy, an Agnostic-guided Attention Focus Loss mechanism to ensure spatial consistency, and a Pose-guided Long Video VTO technique adept at handling extended video sequences.Extensive experiments across various datasets confirms that our approach outperforms existing state-of-the-art models in both single-image and video VTO tasks, offering a viable solution for practical applications within the realms of fashion e-commerce and virtual fitting environments.

1. Introduction

Video virtual try-on must preserve garment appearance as people move, change pose, and shift viewpoint; RealVVT addresses spatial, temporal, and long-video consistency. The authors report superior performance across image and video datasets for short- and long-video try-on.

1. Introduction

  • Video try-on must preserve garment shape, style, and texture across changing poses and viewpoints while maintaining spatial and temporal consistency.Frame-by-frame generation can accumulate errors over time, especially with complex movements and occlusions.Source: §1 · p. 2
  • RealVVT is built on diffusion models and uses a Reference Net and CLIP Encoder for garment features alongside a Denoising UNet for the input video.The overview figure omits the VAE encoder and decoder for clarity.Source: §1 · p. 2, Figure 2 · p. 3
  • Agnostic Mask-Guided Attention Loss prioritizes wearable regions and reduces attention to non-wearable areas to preserve garment shape, style, texture, and spatial alignment.It also appropriately fills mask regions across the sequence.Source: §1 · p. 2
  • Clothing & Temporal Consistency Attention integrates reference and temporal information across frames to reduce flickering and misalignment during pose changes or camera shifts.The mechanism uses interactive information between two U-Nets.Source: §1 · p. 2
  • Pose-guided Long VVT uses pose inputs and iterative keyframe replacement to preserve motion realism and clothing coherence in long video sequences.The strategy estimates motion and viewpoint changes.Source: §1 · p. 2
  • Experiments on multiple high-quality image and video datasets demonstrate superior performance in both short- and long-video virtual try-on tasks.The paper presents RealVVT as a framework for high-quality, temporally coherent try-on results.Source: §1 · p. 2

2. Related Work

Prior video try-on work includes optical-flow-based GAN methods and diffusion approaches, with reported challenges in alignment, detail preservation, and temporal coherence. Video diffusion builds on pretrained image models or temporal modeling, while longer, higher-resolution generation can raise consistency and computational challenges.

2.1. Video Virtual Try-On

  • RealVVT is presented as a one-stage video try-on framework intended to provide high-quality synthesis with spatiotemporal consistency and efficiency.The related-work discussion contrasts it with WildVidFit's two-stage training and stated limitations in temporal coherence and fine-detail preservation.Source: §2.1 · p. 3
  • GAN-based try-on methods warp garments with optical flow, but inaccurate flow can cause misalignment; diffusion methods use pretrained weights and can produce higher generation quality.The section also notes that Tunnel Try-on handles camera movement and preserves clothing textures.Source: §2.1 · p. 3

2.2. Video Generation via Diffusion Models

  • Video diffusion methods add temporal layers to pretrained image diffusion models or train video models from scratch, yet texture consistency and temporal coherence remain challenging.The section describes SVD as combining pretrained image diffusion with temporal components.Source: §2.2 · p. 3
  • DiT structures capture spatiotemporal dependencies, but their computational demands increase with resolution and sequence length, constraining high-fidelity video try-on applications.RealVVT builds on pretrained SVD to improve spatial and temporal consistency over prior models.Source: §2.2 · p. 3

3. Proposed Approach

RealVVT frames video try-on as exemplar-based video inpainting and builds on a dual U-Net architecture. It adds mask-guided spatial attention, clothing-aware temporal attention, and pose-guided keyframe iteration for long videos.

3. Proposed Approach

  • The approach introduces spatial, temporal, and long-video strategies to address try-on generation, with the method organized around those three aims.The section presents an Agnostic-guided Attention Focus Loss, a temporal-consistency method, and a pose-guided strategy for long video generation.Source: §3 · p. 4

3.1. Preliminary

  • The proposed approach uses Stable Video Diffusion’s EDM denoiser, whose output combines a skip-scaled noisy input with a preconditioned U-Net prediction conditioned on the input.The conditioning input can include the first frame in SVD or cloth information in this approach; EDM preconditioning parameters depend on noise level.Source: §3.1 · p. 4

3.2. Overview

  • RealVVT treats try-on as exemplar-based video inpainting, with a Denoising U-Net processing noisy and agnostic video latents plus garment masks, conditioned on reference-cloth features and pose information.A ReferenceNet encodes fine-grained garment features, while the Denoising U-Net is initialized from Stable Video Diffusion.Source: §3.2 · p. 4

3.3. Agnostic Mask-Guided Attention for Clothing Consistency

  • The agnostic-mask attention loss emphasizes target-garment attention within the removed-clothing region and suppresses it outside, while accommodating target garments that differ in shape, style, or coverage.The model must also infer contextual details such as skin tone or limb shape in masked areas not covered by the replacement garment.Source: §3.3 · pp. 4–5

3.4. Clothing&Temporal Consistency Attention

  • Clothing & Temporal Consistency Attention incorporates temporal information into clothing-specific attention, combining current-frame and selected-frame features to align garments across distant frames and viewpoint changes.The selection strategy uses adjacent-frame differences rather than always choosing the first frame as the temporal reference.Source: §3.4 · p. 5

3.5. Pose-guided Long VVT

  • Pose-guided long VVT selects keyframes by DensePose motion and iteratively replaces agnostic keyframe latents, using overlapping segments when keyframes exceed the fixed generation length.The strategy estimates pose and viewpoint changes and uses the preceding segment’s denoised results for overlapping frames.Source: §3.5 · p. 6

4. Experiments

Experiments evaluate RealVVT on video and image virtual try-on datasets, compare it with prior methods, and ablate its spatial and temporal consistency components. Results show advantages across the evaluated datasets, while the ablation examines how the components affect garment fitting and temporal stability.

4. Experiments

  • The experiments jointly train on image and high-resolution video datasets, using 512 × 384 training resolution and evaluating the image datasets at 1024 × 768.The evaluation includes VVT at 512 × 384, VITON-HD, and the DressCode categories.Source: §4 · p. 6

4.1. Comparison with State-of-the-Art Methods

  • On VVT, RealVVT generally outperforms prior approaches, with significant improvements in SSIM, LPIPS, and VFIDResNeXt.The comparison is on the VVT dataset.Source: Table 1 · p. 7, §4.1 · p. 7
  • On ViViD, RealVVT maintains clothing color, shape, and fine details, while StableViton flickers and ViViD struggles with garment dynamics and design details.The example uses a long-sleeved, textured target garment.Source: §4.1 · p. 7, Figure 3 · p. 7
  • For a small garment over a large agnostic mask, ViViD produces unsuitable gap fills or sleeve artifacts, whereas RealVVT generates the expected garment shape.The comparison tests models trained at 512×384 and evaluated at 832×624 resolution.Source: §4.1 · p. 7, Figure 4 · p. 7
  • RealVVT demonstrates a clear performance advantage across VITON-HD and the three DressCode test subsets.The reported image comparisons cover VITON-HD and DressCode UpperBody, LowerBody, and Dresses.Source: §4.1 · pp. 7–8, Table 2 · p. 7, Table 3 · p. 8

4.2. Ablation Study

  • With C&T and λagn = 0.5, VFIDI3D and VFIDResNeXt decrease by 2.072 and 0.0597, respectively, with visual results showing improved frame-to-frame continuity.The ablation attributes the combined setting’s results to enhanced stability and garment consistency across the sequence.Source: Figure 5 · p. 8, §4.2 · p. 8
  • Increasing λagn improves quantitative metrics and helps recover the expected short-sleeved dress under occlusion, but sleeves still extend in later frames.At λagn = 0.05, the agnostic loss has minimal impact; the reported sleeve extension shows temporal consistency remains suboptimal.Source: §4.2 · p. 8

5. Conclusion

RealVVT combines spatial and temporal consistency mechanisms with pose-guided long-video generation to produce high-quality virtual try-on videos. The authors report state-of-the-art performance and potential for practical virtual try-on applications.

5. Conclusion

  • RealVVT is designed to generate accurate virtual try-on videos with robust spatial and temporal consistency.It is built on a dual U-Net structure.Source: §5 · p. 8
  • Its innovations include Clothing & Temporal Consistency Attention, Agnostic Mask-Guided Loss, and Pose-guided Long VVT.Source: §5 · p. 8
  • Experiments demonstrate state-of-the-art performance, while the framework generates high-resolution, photorealistic images suited to practical virtual try-on applications.The authors identify potential for enhancing realism and user engagement in e-commerce platforms.Source: §5 · p. 8

A. More Image Dataset Visual Results

RealVVT’s DressCode visual comparisons cover upper-body, lower-body, and dress try-on, extending evaluation to male try-on tasks. The results highlight garment-detail preservation, including for lower-body garments.

A. More Image Dataset Visual Results

  • DressCode covers upper-body, lower-body, and dress changes for female and male clothing, allowing demonstration of male try-on applicability.The main-text visualizations focus on ViViD, a high-resolution video dataset primarily featuring female clothing changes.Source: A. More Image Dataset Visual Results · p. 11
  • Most concurrent works discussed emphasize upper-body clothing, likely reflecting the greater availability of upper-body training pairs in existing datasets.VITON-HD contains only upper-body examples, and upper-body pairs dominate DressCode.Source: A. More Image Dataset Visual Results · p. 11
  • RealVVT better preserves lower-body garment color, shape, and details such as trouser-leg features than StableVITON in Fig. 7.The comparison uses DressCode’s original unpaired image pairs without manual matching.Source: A. More Image Dataset Visual Results · p. 11

B. Limitations & Discussion

The discussion identifies segmentation and pose-extraction errors in existing try-on datasets, including temporal jitter in video inputs. It also presents visual comparisons of RealVVT with StableVITON on upper- and lower-body garments.

B. Limitations & Discussion

  • Existing image- and video-based try-on datasets have inaccurate masks and DensePose extractions, including oversized or incomplete clothing masks and missed visible limbs.Automated segmentation methods SAPIENS and SAM2 yielded limited improvement and did not provide satisfactory segmentation performance.Source: B. Limitations & Discussion · p. 11
  • Image-based tools used to generate video masks and DensePose can introduce temporal inconsistency and frame-to-frame jitter.Counteracting the input discontinuities and instability still requires considerable effort, motivating better video try-on datasets.Source: B. Limitations & Discussion · p. 11
  • The DressCode-Upper comparison reports better preservation of target-garment shape, color, and collar details by RealVVT than StableVITON.Source: Figure 6 · p. 12
  • On DressCode-Lower, RealVVT is reported to better resist upper-body clothing and environmental influences while retaining pants characteristics.Source: Figure 7 · p. 13

Related papers

Loading 2501.08682v2…