PaperlayerView exact revision on arXiv

Source-linked AI summary

BooM-VVT: Boosting Mask-Free Video Virtual Try-On with Image-Level Pseudo Data

Wei Zhang, Xin Li, Peishu Shi, Jialin Gao, Xuekang Peng, Zhichao Lian, Yeying Jin

arXiv:2609.04120v1cs.CV

TL;DR

Video virtual try-on is limited by mask dependence, costly video-level pseudo-data construction, coarse keyframe selection, and scarce multi-view data. BooM-VVT addresses these issues with mask-free localization, garment-sensitive keyframes, shared 3D positional encodings, and OmniView, achieving stronger temporal consistency and garment fidelity than existing methods while retaining dependencies on visual priors.

  • Problem

    VVT methods remain vulnerable to masks, expensive video-level pseudo data, garment-insensitive keyframe sampling, and scarce multi-view try-on data.

  • Method

    BooM-VVT combines multi-stage image-level pseudo-data training, Garment-Sensitive Keyframe Sampling, Frame-Shared 3D-RoPE, and the OmniView multi-view dataset.

  • Results

    BooM-VVT achieves superior temporal consistency and garment fidelity over existing methods in challenging try-on videos.

  • Takeaways & Limitations

    The framework supports mask-free try-on video generation across diverse viewpoints and try-on tasks while reducing reliance on costly video-level pseudo data.

  • Takeaways & Limitations

    GSKS relies on DWPose and SAM, while extreme occlusions, inaccurate detections, and illumination changes can affect performance.

Abstract

from arXiv · show

Video virtual try-on (VVT) aims to generate realistic videos of a person wearing a target garment. Recent methods leverage a keyframe-driven video generation paradigm to improve in-the-wild performance, yet they still rely on masks to localize try-on regions, making them vulnerable to large motions and severe occlusions. Although mask-free image-based try-on methods have shown promising results by leveraging large-scale pseudo data, extending this paradigm to videos remains difficult, as constructing video-level pseudo data is prohibitively expensive. Furthermore, coarse keyframe sampling and the scarcity of multi-view try-on data limit existing keyframe-driven methods in maintaining garment consistency and handling diverse try-on tasks. To address these challenges, we propose BooM-VVT, a mask-free VVT framework built upon the keyframe-driven paradigm. To achieve mask-free VVT, we introduce a multi-stage training strategy that leverages image-level pseudo data for mask-free localization learning, substantially reducing the need for costly video-level pseudo data. To improve garment consistency, we propose Garment-Sensitive Keyframe Sampling, which selects keyframes based on garment-relevant body regions to better capture garment appearance. We further introduce Frame-Shared 3D-RoPE to establish spatiotemporal correspondences between keyframes and target video frames for accurate garment-detail transfer. Finally, we construct OmniView, a large-scale multi-view try-on dataset to support reliable try-on video generation under complex camera viewpoints and diverse try-on tasks. Extensive experiments demonstrate that BooM-VVT achieves superior temporal consistency and garment fidelity over existing methods. Project page: https://boomvvt.github.io/boomvvt.

1 Introduction

BooM-VVT addresses mask dependence, inefficient video-level pseudo-data construction, garment-insensitive keyframe selection, and limited multi-view data in video virtual try-on. It combines mask-free localization, garment-sensitive keyframes, spatiotemporal alignment, and OmniView to improve garment consistency and diverse-view generation.

  • Existing VVT methods struggle with garment fidelity and temporal consistency because paired garment–video data are scarce and scene diversity is limited.
  • Mask-based localization is vulnerable to large motions and occlusions, while extending image-level pseudo-data methods to video is costly because video-level pseudo data are expensive to construct.
  • BooM-VVT uses multi-stage training to learn mask-free try-on-region localization from image-level pseudo data, reducing reliance on video-level pseudo data.
  • Garment-Sensitive Keyframe Sampling selects frames using garment-relevant body regions, and Frame-Shared 3D-RoPE establishes correspondences for accurate garment appearance transfer.
  • OmniView provides large-scale multi-view try-on data to support reliable generation across diverse viewpoints and try-on tasks.

2 OmniView Dataset

OmniView is constructed to overcome limitations in existing multi-view try-on datasets. It expands garment quality, category coverage, viewpoint completeness, and pseudo-data support for diverse try-on tasks.

  • Existing multi-view try-on datasets are limited in scale, garment diversity, and completeness.
  • OmniView contains 6,110 samples, each with person images from at least two viewpoints, and 88% include back-view images.
  • Each OmniView sample provides front- and back-view flat garment images for more complete garment references.
  • OmniView covers upper-body, lower-body, full-body, and an additional outerwear garment category.
  • The dataset includes synthesized pseudo-labeled data supporting cross-category and layered try-on.

3 Method

BooM-VVT uses garment-sensitive keyframe selection and a two-stage keyframe-driven generation pipeline for mask-free video try-on. Its design combines multi-view image try-on, video synthesis, and shared positional encoding to transfer garment details consistently.

  • Garment-Sensitive Keyframe Sampling: GSKS evaluates garment-relevant body regions and selects a high-capacity primary keyframe before adding frames with complementary viewpoint coverage.
  • Overview: The framework selects informative keyframes, generates their try-on images, and synthesizes the final try-on video under keyframe guidance.
  • Garment-Sensitive Keyframe Sampling: For upper-body garments, GSKS considers shoulders and arms as relevant limbs and the upper torso as the relevant body region.
  • Keyframe Try-on: Selected keyframes are horizontally concatenated and processed in one image-model forward pass, enabling cross-frame information interaction before being split into individual try-on frames.
  • Keyframe-Guided Video Generation: The video generator encodes video, pose, and keyframe tokens, concatenates keyframe and video tokens, and decodes the denoised representation into the final try-on video.
  • Frame-Shared 3D-RoPE: Frame-Shared 3D-RoPE assigns aligned keyframe and video-frame tokens identical positional encodings to reduce their relative positional distance and strengthen attention.
  • Multi-Stage Training: Multi-stage training first learns video generation with mask-provided regions and then learns mask-free localization using image-level pseudo pairs.

4 Experiments

BooM-VVT is evaluated on indoor and in-the-wild benchmarks using qualitative comparisons, quantitative metrics, and component ablations. The experiments show stronger garment fidelity, temporal consistency, and mask-free localization with efficient use of pseudo data.

  • Evaluation Setup: The evaluation uses ViViD-S with 180 samples and WildVVT with 100 samples covering diverse motions, scenes, and garment categories.Generated videos are assessed with FVD, SSIM, LPIPS, and judge-based quality dimensions.
  • Qualitative Comparison: BooM-VVT produces more realistic and temporally coherent results than ViViD, CatV2TON, and MagicTryOn on ViViD-S and WildVVT.Competing mask-based methods show artifacts under garment-shape changes, turning motions, and severe occlusions.
  • Quantitative Comparison: Keyframe-driven BooM-VVT variants consistently outperform non-keyframe-driven baselines on both ViViD-S and WildVVT.Because public implementations of existing keyframe-driven methods are unavailable, BooM-VVT Stage 1 serves as the comparison baseline.
  • Quantitative Comparison: BooM-VVT achieves the best performance across all quality metrics and substantially outperforms MagicTryOn at comparable inference cost.The comparison includes both quality metrics and WildVVT inference time and GPU memory usage.
  • Ablation Study: Removing GSKS or Frame-Shared 3D-RoPE degrades performance, particularly garment consistency, while separate positional encodings weaken garment-detail transfer.The ablation reports incorrect shoulder straps and missing white chest stripes without Frame-Shared 3D-RoPE.
  • Ablation Study: With only 500 video-level pseudo samples, the full model reaches performance comparable to the variant without Stage 2 trained with 2,500 samples.Stage 2 improves mask-free localization and fine-tuning efficiency across all tested pseudo-video scales.

5 Conclusion

BooM-VVT is a mask-free, keyframe-driven video virtual try-on framework that combines image-level pseudo-data training, garment-sensitive keyframe selection, Frame-Shared 3D-RoPE, and the OmniView dataset. Experiments show improved temporal consistency and garment fidelity in unconstrained real-world scenarios.

  • Conclusion: BooM-VVT learns mask-free try-on localization from image-level pseudo data, reducing reliance on costly video-level pseudo data.Its multi-stage training strategy supports mask-free video virtual try-on with minimal video-level pseudo data.
  • Conclusion: GSKS selects garment-relevant keyframes, while Frame-Shared 3D-RoPE establishes spatiotemporal correspondences for garment appearance transfer.These components target garment consistency and fine-grained detail preservation.
  • Conclusion: OmniView provides multi-view garment references and broader category coverage for challenging viewpoints and diverse try-on tasks.The dataset supports layered, cross-category, and multi-view try-on settings.
  • Conclusion: Extensive experiments demonstrate superior temporal consistency and garment fidelity under unconstrained real-world scenarios.The reported conclusion covers the framework's overall experimental performance.

6 Limitations

The paper’s context identifies limitations in virtual try-on paradigms and diffusion-transformer foundations, while the framework addresses mask dependence through keyframe-driven generation.

  • Image try-on methods use warping-and-fusion or diffusion-based reference injection, but dense preprocessing remains common.
  • Video try-on must preserve temporal coherence, motivating temporal modules, garment encoders, memory refinement, and video DiT architectures.
  • Mask-based try-on offers spatial controllability but depends on mask quality, whereas mask-free methods implicitly localize try-on regions from data.
  • Keyframe-driven methods decompose VVT into keyframe try-on and keyframe-guided video generation, enabling use of unpaired videos and pretrained backbones.
  • Existing keyframe-driven methods still rely on explicit masks, limiting robustness in challenging real-world scenarios.
  • GSKS selects garment-relevant limbs and body regions using DWPose and SAM, with K=2, λ=0.2, and Δmin = 4 frames in the default setting.

C.2 Evaluation Metrics

The evaluation uses VLM-based consistency and quality metrics, supplements them with human validation, and discusses a limitation in commonly used VFID implementations.

  • The VLM protocol evaluates Appearance & Background Consistency, Garment Consistency, and Overall Visual Quality with Gemini 3.1 Pro.
  • Each VLM dimension is scored independently from 0 to 5 using the source video, target garment image, and generated try-on video.
  • A human study found strong agreement between Gemini-based and human judgments, supporting the reliability of the VLM evaluation.
  • VFID is included for comparability with prior work but is not used as the primary metric in the main paper.
  • The open-source VFID toolkit evaluates only the first 10 frames, so it cannot fully assess videos with later motion or content changes.

C.3 Pseudo-Label Data Construction

The pseudo-data pipeline combines image-level editing and verification with model-generated video pseudo data for BooM-VVT’s multi-stage training strategy.

  • The strategy uses image-level pseudo data for Stage 2 and video-level pseudo data for Stage 3.
  • Qwen3-VL-32B analyzes each person image and generates a garment-editing instruction for pseudo-image construction.
  • GSKS determines garment-related limb sets and body regions before scoring and selecting informative, complementary keyframes.
  • Qwen-Image-Edit-2511 modifies the garment region, while Qwen3-VL-32B verifies instruction consistency and rejects failed samples.
  • The pipeline constructs 30K pseudo images spanning diverse garment types, body poses, and real-world backgrounds.
  • Video-level pseudo try-on videos are synthesized with the Stage 1 BooM-VVT model because existing open-source video editors lack sufficient quality.

D.1 Additional Quantitative Results

Additional quantitative evaluation with corrected VFID shows BooM-VVT ranks among the strongest methods across ViViD and WildVVT settings, especially on WildVVT.

  • BooM-VVT achieves the best corrected-VFID results in three of four ViViD and WildVVT settings and ranks second in the remaining setting.
  • On WildVVT, BooM-VVT achieves the lowest VFIDI by a clear margin, indicating stronger generalization in unconstrained real-world scenarios.
  • Human evaluation on WildVVT uses 10 evaluators, the same ABC, GC, and OVQ dimensions, and a 0–5 rating scale.

D.1.1 Human Evaluation Validation.

Human evaluation on WildVVT assesses generated videos along three quality dimensions, and BooM-VVT achieves the highest scores across all of them. The human judgments strongly agree with Gemini-based evaluation.

  • BooM-VVT achieves the highest human evaluation scores across ABC, GC, and OVQ.ABC measures Appearance & Background Consistency, GC measures Garment Consistency, and OVQ measures Overall Visual Quality on a 0–5 scale.
  • Pearson correlation of approximately r=0.9 indicates strong agreement between human scores and Gemini-based evaluation.
  • Human evaluation uses Appearance & Background Consistency, Garment Consistency, and Overall Visual Quality as its three dimensions.Evaluators score each dimension from 0 to 5, with higher scores indicating better quality.

D.2 Additional Qualitative Results

Additional qualitative results evaluate BooM-VVT under large motion, severe occlusion, object interaction, and diverse try-on tasks. Across these settings, the method preserves garment appearance, temporal consistency, interaction relationships, and viewpoint-dependent details.

  • Large motion: Under large motion, BooM-VVT produces more stable garment appearance and better temporal consistency than existing methods.Its full mask-free model also outperforms the mask-dependent BooM-VVT (Stage 1) in garment consistency, indicating that inaccurate fast-motion masks can degrade localization.
  • Severe occlusion: Under severe occlusion, BooM-VVT maintains plausible garment appearance despite large portions of the body being occluded.Existing methods often produce obvious artifacts or incorrect garment structures in this setting.
  • Object interaction: During object interaction, BooM-VVT better preserves person–object relationships than existing methods.Other methods may introduce artifacts in interaction regions or distort and erase interacting objects.
  • Diverse try-on tasks: OmniView enables layered, cross-category, and multi-view try-on, demonstrating generalization across diverse try-on scenarios.The settings use sequential layer-by-layer dressing, text-guided cross-category information, and garment images from different viewpoints.

D.4 Additional Ablation Results

Ablation studies examine garment-sensitive keyframe sampling, the OmniView dataset, and the GSKS weighting factor. The results support more informative garment-focused keyframes, improved cross-view and multi-view-input consistency, and stable performance across a broad weighting range.

  • Garment-Sensitive Keyframe Sampling: GSKS selects keyframes using garment-relevant regions and complementary viewpoint coverage, improving fine-grained garment-detail preservation.Compared with global pose-difference sampling, it supports more faithful logos, patterns, and other garment details in keyframe images and final videos.
  • OmniView ablation: OmniView improves cross-view consistency in keyframe try-on results.Without OmniView, garment details such as belts can appear in one viewpoint and disappear in another.
  • OmniView ablation: OmniView improves handling of multi-view garment inputs by preserving sharper textures and more accurate garment structures.The model trained without OmniView can generate incorrect back-view textures even when front- and back-view references are provided.
  • Weighting-factor sensitivity: When λ is between 0.2 and 0.6, SSIM stays around 0.871 and LPIPS around 0.062 on WildVVT.Performance remains stable across a wide range of λ values, while overly large values cause slight degradation.
  • Evaluation protocol: The VLM-based evaluation scores Appearance & Background Consistency, Garment Consistency, and Overall Visual Quality on integer scales from 0 to 5.
  • Qualitative evaluations: Qualitative evaluations cover large motion, severe occlusion, object interaction, and multiple try-on task configurations.The figures include layered, cross-category, and multi-view try-on, plus single-view and front-and-back multi-view garment inputs.
Loading 2609.04120v1…