PaperlayerView exact revision on arXiv

Source-linked AI summary

iTryOn: Mastering Interactive Video Virtual Try-On with Spatial-Semantic Guidance

Jun Zheng, Zhengze Xu, Mengting Chen, Jing Wang, Jinsong Lan, Xiaoyong Zhu, Kaifu Zhang, Bo Zheng, Xiaodan Liang

arXiv:2605.21431v2cs.CV

TL;DR

Existing video virtual try-on methods largely omit active human-garment interaction, so iTryOn introduces spatial-semantic guidance for interactive try-on and achieves state-of-the-art performance on interactive and traditional benchmarks.

  • Problem

    Existing video virtual try-on methods and benchmarks focus on passive subjects, leaving active human-garment interaction and its ambiguity and sparse deformation events insufficiently addressed.

  • Method

    iTryOn uses multi-level interaction injection with 3D hand priors, synchronized captions, and action-aware constraint loss within a video diffusion Transformer.

  • Results

    iTryOn achieves state-of-the-art performance on interactive and traditional benchmarks, with interaction success rates over 61% versus less than 49% for existing methods.

  • Takeaways & Limitations

    The work establishes Interactive VVT as a benchmarked task and provides a framework that generates physically plausible interactions while retaining state-of-the-art visual quality on traditional VVT.

  • Takeaways & Limitations

    iTryOn lacks explicit garment-semantic reasoning and does not yet quantify fine-grained physical accuracy, sometimes producing pantomimed infeasible actions.

Abstract

from arXiv · show

Video Virtual Try-On (VVT) aims to seamlessly replace a garment on a person in a video with a new one. While existing methods have made significant strides in maintaining temporal consistency, they are predominantly confined to non-interactive scenarios where models merely showcase garments. This limitation overlooks a crucial aspect of real-world apparel presentation: active human-garment interaction. To bridge this gap, we introduce and formalize a new challenging task: Interactive Video Virtual Try-On (Interactive VVT), where subjects in the video actively engage with their clothing. This task introduces unique challenges beyond simple texture preservation, including: (1) resolving the semantic ambiguity of interactions from standard pose information, and (2) learning complex garment deformations from video where interactive moments are sparse and brief. To address these challenges, we propose iTryOn, a novel framework built upon a large-scale video diffusion Transformer. iTryOn pioneers a multi-level interaction injection mechanism to guide the generation of complex dynamics. At the spatial level, we introduce a garment-agnostic 3D hand prior to provide fine-grained guidance for precise hand-garment contact, effectively resolving spatial ambiguity. At the semantic level, iTryOn leverages global captions for overall context and time-stamped action captions for localized interactions, synchronized via our novel Action-aware Rotational Position Embedding (A-RoPE). Extensive experiments demonstrate that iTryOn not only achieves state-of-the-art performance on traditional VVT benchmarks but also establishes a commanding lead in the new interactive setting, marking a significant step towards more dynamic and controllable virtual try-on experiences.

1. Introduction

Interactive Video Virtual Try-On extends VVT beyond temporally coherent garment replacement to videos containing active human-garment interactions. iTryOn addresses interaction ambiguity and sparse physical events with a video diffusion Transformer, multi-level interaction guidance, and an action-aware constraint loss, achieving state-of-the-art performance on interactive and traditional benchmarks.

  • Motivation: Video Virtual Try-On generates temporally coherent videos of people wearing new garments while preserving garment texture and structure across frames.VVT must also capture garment drape, flow, and response to movement.
  • Problem: Existing VVT methods and benchmarks operate exclusively in non-interactive scenarios, limiting their ability to model active human-garment engagement.This limitation motivates formalizing Interactive VVT as a distinct task.
  • Challenges: Interactive VVT must resolve semantic ambiguity because standard 2D keypoints lack the 3D orientation and shape needed to distinguish gestures such as tucking in a shirt.It must also learn physically plausible garment deformation from interactive moments that are complex, brief, and sparse.
  • Method: iTryOn uses a large-scale video diffusion Transformer with multi-level interaction injection and an action-aware constraint loss.Its spatial guidance uses a garment-agnostic 3D hand prior for precise hand-garment contact, while synchronized captions provide semantic guidance.
  • Results: iTryOn achieves state-of-the-art performance on both interactive and traditional VVT benchmarks.The reported result covers both the newly formalized interactive setting and established VVT evaluation.

2. Related Work

Recent open-source video generation models have accelerated progress in Video Virtual Try-On, evolving from image-model adaptations toward powerful spatiotemporal Diffusion Transformer architectures.

  • Video Virtual Try-On: Open-source video generation models have catalyzed significant advancements in Video Virtual Try-On.The cited works include Xu et al. (2024), Karras et al. (2024), Fang et al. (2024), Wang et al. (2024), Li et al. (2025a), Zheng et al. (2025), Chong et al. (2025b), Li et al. (2025b), and Zuo et al. (2025).
  • Video Virtual Try-On: Early diffusion-based VVT methods adapted image generation models for video tasks.ViViD introduced a large-scale VVT dataset while repurposing an image-generation model for video virtual try-on.
  • Video Generation Models: Diffusion Transformers emerged as state-of-the-art video-generation architectures after Sora, replacing earlier image-model adaptations with full spatiotemporal modeling.AnimateDiff adapted image models with temporal modules, whereas Hunyuan-DiT and Wan2.1 embraced full spatiotemporal architectures.

3. Methodology

iTryOn formalizes Interactive Video Virtual Try-On as generating a video that preserves a person’s identity and motion while rendering a target garment responding realistically to interaction. Its methodology combines an annotated dataset, a conditional DiT framework, garment-agnostic 3D-hand spatial guidance, action-aware semantic timing, and interaction-focused evaluation and training.

  • Task Formulation: Interactive VVT generates a target-garment video that preserves the source subject’s identity and motion while modeling garment responses during interaction.The task maps a source video and target garment image to a synthesized output video.
  • Challenges: The methodology addresses interaction ambiguity caused by 2D pose projections losing depth cues, hand shape, and hand orientation.These missing signals hinder recognition of approaching, pulling, pressing, and other contact-related motions.
  • Dataset: VVT-Interact contains 5,292 high-quality video-garment pairs spanning six interaction categories, each with global and time-stamped action captions.The annotations supervise training and provide ground truth for Interaction Success Rate evaluation.
  • Framework: iTryOn uses a conditional Diffusion Transformer with parallel trainable modules that inject context, 3D-hand, semantic, and interaction-focused guidance.Inputs are encoded into latent space by a frozen Wan encoder, denoised iteratively in the DiT, and decoded into the output video.
  • Spatial Guidance: A garment-agnostic 3D hand prior supplies fine-grained spatial guidance for distinguishing hand shape and orientation during physical garment contact.The prior is extracted with HaMeR and processed by an Interaction Guider, alongside pose and agnostic-video Context Blocks.
  • Semantic Guidance: Action captions specify interaction semantics, while A-RoPE aligns them temporally to video segments and excludes non-interactive segments through null-caption keys.The separation hyperparameter k is set to 4, and A-RoPE is applied to all video queries but only meaningful action-caption keys.
  • Training Objective: The action-aware constraint loss applies an additional penalty exclusively to interaction-frame latents, with λ set to 0.5 in the experiments.A binary interaction mask selects the frames receiving the extra penalty, while the standard diffusion loss covers all frames.

4. Experiments

Experiments evaluate iTryOn on both traditional and interactive video virtual try-on, using visual and interaction fidelity metrics. iTryOn surpasses baselines in visual quality, motion coherence, and semantically successful interactions, with ablations attributing gains to combined spatial and semantic guidance.

  • Datasets: Evaluation covers the ViViD non-interactive benchmark and the newly introduced VVT-Interact interactive benchmark.ViViD contains 7,759 training and 180 testing paired videos at 624×832 resolution.
  • Evaluation Metrics: The experiments measure visual fidelity with SSIM, LPIPS, and VFID, and interaction fidelity with temporal coherence and the proposed Interaction Success Rate.ISR uses Qwen-VL to verify whether intended actions are semantically recognizable and coherent with hand motion.
  • Visual Fidelity: iTryOn outperforms all baselines on SSIM, LPIPS, and VFID, indicating stronger garment texture preservation and fewer flickering artifacts.These results are reported for visual fidelity on VVT-Interact.
  • Interaction Fidelity: iTryOn achieves the lowest FVD and an ISR success rate above 61%, compared with below 49% for existing methods.The results indicate smoother motion and more semantically correct physical interactions than baseline models.
  • Qualitative Comparison: Qualitative comparisons show baselines producing implausible garment deformations or misinterpreting zippering and hem-pulling actions as hand motion without garment engagement.The failures occur on the VVT-Interact dataset.
  • Ablation Study: Ablations show that dataset training alone is insufficient, spatial guidance enables hand-garment contact, and semantic guidance plus AC loss resolves interaction ambiguity.The results identify the synergistic combination of spatial and semantic guidance as essential for high-fidelity interactions.

5. Limitations

iTryOn remains limited by missing explicit garment-semantic reasoning and the lack of a metric for fine-grained physical accuracy. These gaps can cause pantomimed infeasible actions and leave physical correctness difficult to quantify.

  • Semantic reasoning: The model lacks explicit garment-semantic reasoning, sometimes generating pantomimed actions for infeasible interactions such as unzipping a seamless T-shirt.This limitation concerns understanding whether a requested garment interaction is physically meaningful.
  • Evaluation: Although ISR evaluates semantic success, fine-grained physical accuracy remains difficult to quantify and is left as an open challenge.The authors discuss both limitations in Appendix A.1.

6. Conclusion

The work formalizes Interactive Video Virtual Try-On, establishes infrastructure for its evaluation, and introduces iTryOn to address interaction ambiguity and sparsity.

  • 6. Conclusion: The paper introduces Interactive VVT, constructs the first large-scale VVT-Interact dataset, proposes the ISR metric, and presents iTryOn with multi-level interaction injection and action-aware constraint loss.These contributions target the core challenges of ambiguity and sparsity in interactive video virtual try-on.

Impact Statement

The work addresses ethical considerations in constructing the VVT-Interact dataset and applying its generative model. It uses publicly available videos from trusted sources and privacy-preserving data processing and model design.

  • The authors explicitly consider ethical implications surrounding the VVT-Interact dataset and generative model.
  • The dataset uses publicly available videos from trusted sources whose creators implicitly or explicitly consented to public sharing.
  • The data processing pipeline and model design are inherently privacy-preserving to further protect personal identity.

A. Appendix · A.1. Limitations and Future Work

The appendix identifies two key areas for future exploration: handling physically implausible interactions and developing metrics specifically for interaction fidelity. When requested actions are infeasible, iTryOn degrades to a non-interactive result while preserving the input hand.

  • A.1. Limitations and Future Work: The authors identify two key areas for future exploration in iTryOn.These areas are handling implausible interactions and evaluating interaction fidelity.
  • A.1. Limitations and Future Work: iTryOn assumes that the input action caption describes a physically feasible interaction with the target garment.This assumption limits behavior in edge cases where the requested interaction is physically impossible.
  • A.1. Limitations and Future Work: An “unzipping” action on a T-shirt without a zipper is an example of an implausible interaction the model cannot physically execute.The limitation arises because the target garment lacks the feature required by the specified action.
  • A.1. Limitations and Future Work: For implausible interactions, the framework gracefully degrades to a non-interactive virtual try-on result.It cannot execute the intended physical effect, but still produces a non-interactive outcome.
  • A.1. Limitations and Future Work: In these edge cases, the framework faithfully preserves the input hand while foregoing the intended physical garment effect.The supplied passage describes this preservation as part of the non-interactive fallback behavior.
  • A.1. Limitations and Future Work: Interactive VVT lacks specialized evaluation metrics for measuring whether generated physical interactions are correct.This is described as a primary challenge in the nascent field of Interactive VVT.
  • A.1. Limitations and Future Work: Standard SSIM, LPIPS, FVD, and VFID metrics assess overall visual quality and temporal consistency rather than interaction correctness.The passage distinguishes pixel-level metrics, SSIM and LPIPS, from video-level metrics, FVD and VFID.
  • A.1. Limitations and Future Work: These metrics cannot distinguish physically plausible fabric stretch from visually coherent but incorrect interaction behavior.This motivates future metrics designed specifically for interaction fidelity.

A.2. Data Annotation Pipeline

The VVT-Interact dataset is created through an annotation pipeline with two primary components: VLM-based semantic annotation and 3D hand prior generation.

  • Annotation Pipeline: The annotation pipeline comprises VLM-based annotation for semantic guidance and 3D hand prior generation.It provides a detailed description of the process used to create the VVT-Interact dataset.

A.2.1. VLM-BASED ANNOTATION FOR SEMANTIC GUIDANCE

This section describes a Qwen-VL-32B-based pipeline for generating semantic captions, classifying clothing interactions, and identifying coherent interaction timestamps. It combines single-pass structured annotation with per-frame classification and morphological smoothing to reduce temporal-label noise.

  • Annotation Model: Qwen-VL-32B performs all semantic annotations, covering caption generation and timestamp annotation.The model was selected based on superior preliminary-evaluation performance.
  • Caption and Interaction Type Annotation: A single-pass prompt produces a JSON object containing a high-level motion description and specific interaction type.The predefined categories are Adjusting the collar, Adjusting the hem, Rolling/Unrolling sleeves, Putting on/Taking off clothes, Pulling at clothes, and Other interactions.
  • Timestamp Annotation and Smoothing: Per-frame binary classification identifies precise temporal boundaries for clothing interactions.The raw labels can contain isolated misclassifications and other noise.
  • Timestamp Annotation and Smoothing: Morphological opening followed by closing treats the label sequence as a 1D signal, removing spurious predictions and forming coherent contours.This smoothing procedure is applied to noisy raw binary labels from the VLM.
  • Interaction Definition: The annotation prompt defines manipulative interaction as purposeful action that adjusts, fastens, or changes the garment, excluding passive contact and no contact.Examples include pulling, tugging, or stretching the fabric.

A.2.2. VLM MODEL SELECTION … A.6.2. ANALYSIS OF PERFORMANCE DRIVERS

The appendix validates iTryOn’s annotation, spatial guidance, action-aware semantic guidance, and training choices, while showing state-of-the-art non-interactive performance. Ablations identify effective components and hyperparameters, and analysis attributes ViViD gains to the backbone and general-purpose strategies.

  • A.2.2. VLM MODEL SELECTION: Qwen-VL-32B achieves the highest F1-score and precision for binary per-frame interaction annotation and is selected for large-scale automated labeling.The comparison used 1,000 manually annotated frames, despite ambiguity in identifying interaction start and end frames.
  • A.3. 3D Hand Prior Annotation: HaMeR generates per-frame 3D hand meshes and poses, rendered into 2D spatial guidance that exceeds 95% accuracy on a manually inspected random subset.The framework remains robust to minor inaccuracies in the 3D hand prior.
  • A.4.2. QUANTITATIVE ABLATION STUDY: Categorical time-stamped action captions resolve interaction ambiguity, while A-RoPE synchronizes semantic guidance with video frames and prevents leakage into non-interactive frames.The ablation study reports that action captions improve most metrics, and A-RoPE produces an additional gain, especially on temporal-consistency metrics.
  • A.4.1. MOTIVATION: AMBIGUITY IN GLOBAL CAPTIONS: Global VLM captions can describe distinct garment interactions with generic verbs, producing confusing supervision that favors non-interactive try-on generation.Examples distinguish rolling sleeves from adjusting the hem, although both may be labeled “adjusts.”
  • A.4.2. QUANTITATIVE ABLATION STUDY: k = 4 yields the best overall performance in the A-RoPE separation-scale ablation.The separation scale controls how distinctly different action segments are encoded in positional space.
  • A.5. Ablation Study of Action Constraint Loss Weight: λ = 0.5 yields the best overall performance for AC loss, which increases supervision on interactive frames to counteract non-interactive-frame dominance.The coefficient controls the strength of reconstruction supervision applied through the action mask.
  • A.6.1. PERFORMANCE ON THE VIVID BENCHMARK: On the non-interactive ViViD benchmark, iTryOn achieves a decisive lead across almost all metrics, surpassing MagicTryOn in VFID and SSIM with greater parameter efficiency.Qualitatively, it preserves intricate garment details and structural integrity during motion, while baselines show blurring or temporal flickering.
  • A.6.2. ANALYSIS OF PERFORMANCE DRIVERS: ViViD performance is attributed to Wan2.1-VACE’s reference-guided editing pretraining, flow-matching loss weighting, and early-step Interval Guidance using CFG during the first 10%-40%.These general-purpose strategies provide consistent or substantial gains and form the foundation for interaction-specific innovations.
Loading 2605.21431v2…