Source-linked AI summary
Vanast: Virtual Try-On with Human Image Animation via Synthetic Triplet Supervision
Hyunsoo Cha, Wonjung Woo, Byungjun Kim, Hanbyul Joo
TL;DR
Existing two-stage systems separate virtual try-on from pose-driven animation, creating consistency and efficiency limitations. Vanast unifies both tasks through synthetic triplet supervision and a Dual Module video-diffusion architecture. It reports stronger garment transfer, pose adherence, and identity preservation than the compared pipelines, while supporting zero-shot garment interpolation.
Problem
Two-stage virtual try-on and animation pipelines can produce identity drift, garment distortion, compounding artifacts, computational inefficiency, and front–back inconsistency.
Method
Vanast combines scalable synthetic triplet construction with a Dual Module video-diffusion architecture that separately supports human animation and garment transfer.
Results
Vanast achieves the best performance across all reported metrics against compared two-stage pipelines and ablation baselines, with qualitative results showing accurate pose and garment transfer while preserving identity.
Takeaways & Limitations
The unified framework supports identity-consistent garment animation across diverse garment settings, including zero-shot garment interpolation and multi-garment transfer without finetuning.
Abstract
from arXiv · showhide
We present Vanast, a unified framework that generates garment-transferred human animation videos directly from a single human image, garment images, and a pose guidance video. Conventional two-stage pipelines treat image-based virtual try-on and pose-driven animation as separate processes, which often results in identity drift, garment distortion, and front-back inconsistency. Our model addresses these issues by performing the entire process in a single unified step to achieve coherent synthesis. To enable this setting, we construct large-scale triplet supervision. Our data generation pipeline includes generating identity-preserving human images in alternative outfits that differ from garment catalog images, capturing full upper and lower garment triplets to overcome the single-garment-posed video pair limitation, and assembling diverse in-the-wild triplets without requiring garment catalog images. We further introduce a Dual Module architecture for video diffusion transformers to stabilize training, preserve pretrained generative quality, and improve garment accuracy, pose adherence, and identity preservation while supporting zero-shot garment interpolation. Together, these contributions allow Vanast to produce high-fidelity, identity-consistent animation across a wide range of garment types.
1. Introduction
Vanast unifies virtual try-on and pose-driven human animation in a single framework, using scalable triplet supervision and Dual Modules to preserve identity, garments, and pose.
- Motivation: Two-stage virtual try-on and animation pipelines can cause identity drift, garment distortion, compounding artifacts, computational inefficiency, and front–back inconsistency.These limitations motivate a unified, single-stage generation framework.
- Vanast Framework: Vanast directly synthesizes garment-transferred human animation videos from a human image, garment images, and a pose guidance video.
- Synthetic Triplet Supervision: The training data pipeline generates identity-preserving alternative-outfit images, captures full upper and lower garment triplets, and assembles diverse in-the-wild triplets.These strategies address the absence of public triplet datasets and single-garment limitations in online sources.
- Dual Module Architecture: The Dual Module architecture preserves the pretrained video backbone while adding dedicated pathways for garment transfer and pose guidance.It is designed to converge quickly, maintain generative fidelity, and improve garment accuracy, pose adherence, and identity preservation.
- Capabilities: The modular conditioning structure supports zero-shot garment interpolation without additional finetuning while preserving identity.
2. Related Work
Prior virtual try-on and human-animation methods rely on separate stages or image-based mechanisms that limit temporal consistency and robustness. Vanast instead uses a video-native diffusion framework with pose-conditioned generation for unified animation.
- Virtual Try-On: Early virtual try-on methods use parsing, geometric alignment, 2D warping, and appearance blending, but robustness decreases with pose variation, occlusion, and garment deformation.
- Virtual Try-On: Video applications of virtual try-on can exhibit temporal flickering and identity drift because they lack mechanisms for cross-frame consistency.
- Vanast: Vanast generates videos directly from a subject using pose-conditioned diffusion, unifying virtual try-on and temporal animation.
- Diffusion-Based Human Animation: Image synthesis models emphasize identity preservation and compositional control but typically require a separate animation stage for temporal coherence.
3. Method
Vanast generates temporally coherent garment-transferred human animation from a human image, garment images, and motion guidance in one framework. Its method combines synthetic triplet construction with Dual Modules that separately integrate animation and garment-transfer conditions.
- 3. Method: Vanast takes a human image wearing an arbitrary garment, target garment images, motion guidance, and a text prompt to generate an F-frame animation sequence.The generated sequence depicts the same person wearing the target garment items and supports multiple garments or accessories.
- 3.1. Synthetic Triplet Dataset Generation: Training uses triplets containing a differently dressed human image, target garment images, and a ground-truth video of the person moving in the target garment.The motion sequence is extracted from the ground-truth video using an off-the-shelf 2D keypoint estimator.
- 3.1. Synthetic Triplet Dataset Generation: The dataset pipeline combines online garment-video pairs, diffusion-generated alternative-outfit human images, in-the-wild triplets, and a studio dataset for multiple garments.The in-the-wild strategy extracts garments and human images directly from videos, while the studio data supports multiple garment items.
- 3.1. Synthetic Triplet Dataset Generation: Alternative human images are created by selecting suitable video frames, constructing garment-region inpainting masks, and prompting diffusion inpainting with diverse garment types and colors.Frame selection considers face visibility, open eyes, near-frontal pose, and a quality score of at least 95/100; the pipeline also uses an auxiliary synthesized image for mask extraction.
- 3.2. Model Architecture: The architecture uses a Human Animation Module for human and pose inputs and a Garment Transfer Module for garment inputs within a distributed, cascaded video diffusion transformer.The backbone is divided into specialized modules that progressively integrate contextual information across multiple representation levels.
- 3.2. Model Architecture: Vanast supports smooth garment interpolation by varying γ from 0 to 1 when combining garment-module features.The interpolation ratio controls the transition between garment conditions and is intended to produce semantically coherent interpolations.
4. Experiments
Experiments compare Vanast with two-stage subject-to-image and virtual try-on pipelines, then test architectural components and transfer settings. Vanast achieves stronger pose following, garment transfer, identity preservation, and broad zero-shot transfer capabilities across the reported evaluations.
- Experimental Setup: The evaluation uses 9,135 videos and compares Vanast against two-stage baselines combining subject-to-image or image virtual try-on models with animation models.The comparisons include 16 model combinations for the two-stage baselines.
- Comparisons: Vanast achieves the best performance across all metrics against subject-to-image-based baselines, with more accurate pose following and garment transfer and stronger identity preservation.These findings are reported quantitatively in Table 1 and qualitatively in Figure 4.
- Comparisons: Vanast outperforms virtual try-on-based combinations on every metric except SSIM, where its score remains comparable to the best-performing method.Qualitative comparisons show Vanast outputs most closely resemble the ground truth among these baselines.
- Ablation Study: The full model achieves the highest performance across all ablation metrics, while Single Module, Backbone-LoRA, and w/o SynthHuman show pose-control or garment-transfer weaknesses.The full model most closely resembles ground truth; Single Module is vulnerable to pose conditions, while the other variants struggle with accurate garment transfer.
- Applications: Vanast supports single, simultaneous upper-and-lower, in-the-wild, and interpolated garment transfer without additional training.Reported results preserve garment details and identity consistency, maintain temporal consistency for in-the-wild inputs, and produce smooth interpolation of both garment regions.
5. Conclusion
Vanast unifies garment transfer and human animation from a single human image, garment images, and pose guidance. It uses scalable triplet supervision and a Dual Module architecture to improve fidelity, pose adherence, identity preservation, and temporal coherence.
- Vanast synthesizes garment-transferred human animation videos directly from a single human image, garment images, and a pose guidance video.
- Scalable triplet supervision combines in-the-wild videos with complementary upper–lower garment captures to support diverse, identity-preserving garment variations.
- The Dual Module architecture improves garment fidelity, pose adherence, and identity preservation while supporting zero-shot garment interpolation and multi-garment transfer without finetuning.
- Vanast consistently surpasses two-stage pipelines and generalizes effectively to in-the-wild garment images with strong temporal coherence.