Source-linked AI summary
Tunnel Try-on: Excavating Spatial-temporal Tunnels for High-quality Virtual Try-on in Videos
Zhengze Xu, Mengting Chen, Zhao Wang, Linyu Xing, Zhonghua Zhai, Nong Sang, Jinsong Lan, Shuai Xiao, Changxin Gao
TL;DR
Video try-on must preserve clothing detail while modeling coherent motion, a gap left by prior methods. Tunnel Try-on uses diffusion generation with clothing-focused tunnels, smoothing, attention cues, and environment context, and reports superior qualitative and quantitative comparisons across complex scenarios.
Problem
Video try-on requires preserving clothing details and coherent motion, while prior methods are limited in complex scenes, diverse clothing, and free-form movement.
Method
Tunnel Try-on uses a diffusion framework with a zoomed focus tunnel, Kalman-smoothed crops, tunnel position embeddings, and an environment encoder.
Results
Tunnel Try-on outperforms existing alternatives in both qualitative and quantitative comparisons and produces high-fidelity results across complex backgrounds and diverse clothing types.
Takeaways & Limitations
Tunnel Try-on is presented as a practical video try-on tool for fashion-industry applications and future virtual try-on research.
Takeaways & Limitations
Prior video try-on methods remain limited to simple motions, backgrounds, textures, and tight-fitting tops, motivating Tunnel Try-on’s broader target setting.
Abstract
from arXiv · showhide
Video try-on is a challenging task and has not been well tackled in previous works. The main obstacle lies in preserving the details of the clothing and modeling the coherent motions simultaneously. Faced with those difficulties, we address video try-on by proposing a diffusion-based framework named "Tunnel Try-on." The core idea is excavating a "focus tunnel" in the input video that gives close-up shots around the clothing regions. We zoom in on the region in the tunnel to better preserve the fine details of the clothing. To generate coherent motions, we first leverage the Kalman filter to construct smooth crops in the focus tunnel and inject the position embedding of the tunnel into attention layers to improve the continuity of the generated videos. In addition, we develop an environment encoder to extract the context information outside the tunnels as supplementary cues. Equipped with these techniques, Tunnel Try-on keeps the fine details of the clothing and synthesizes stable and smooth videos. Demonstrating significant advancements, Tunnel Try-on could be regarded as the first attempt toward the commercial-level application of virtual try-on in videos.
1. Introduction
Video try-on must preserve clothing appearance while maintaining coherent human motion, but prior approaches struggle with inter-frame consistency and complex scenes. Tunnel Try-on addresses this with a diffusion framework that zooms into clothing-focused tunnels and smooths their motion.
- Video try-on must preserve clothing appearance and human motion, while offering a more comprehensive presentation than image try-on.Its interactive use lets consumers explore clothing without physical try-on.
- Frame-by-frame application of image try-on methods causes significant inter-frame inconsistency and unacceptable video outcomes.Prior video methods use optical-flow-based warping and temporal smoothing to improve consistency.
- Tunnel Try-on extends an image-based diffusion baseline with Main and Reference U-Nets plus Temporal-Attention for video virtual try-on.The Reference U-Net extracts and injects fine clothing details, while the Main U-Net performs inpainting.
- The focus tunnel crops and zooms clothing regions so people remain centered, maximizing preservation of fine reference-clothing details.Kalman filtering is used to construct smoother crop sequences despite camera and human movement.
- Extensive experiments show Tunnel Try-on significantly outperforms other video virtual try-on methods and achieves state-of-the-art performance in complex scenarios.The framework combines tunnel construction, Kalman smoothing, tunnel position embedding, and environment context in attention layers.
2. Related Work
Related work spans GAN- and diffusion-based image try-on, optical-flow video methods, and pose-driven image animation. These approaches remain limited in complex real-world video settings, motivating a diffusion model without explicit warp modules and with focus-tunnel guidance.
- Image virtual try-on methods mainly use GAN-based or diffusion-based generators, with GAN systems typically separating clothing warping from image blending.Warping modules align clothing semantically to the human body before generation.
- Video try-on methods use optical flow or temporal smoothing to improve coherence, but prior systems generally target minor motion, simple backgrounds, and simple clothing textures.They have also focused only on tight-fitting tops, limiting real-world applicability.
- Tunnel Try-on removes explicit warp modules and uses diffusion generation with focus tunnels to handle varied person-background relationships in real-world videos.The strategy targets diverse clothing, complex backgrounds, free-form movement, and changing person size, proportion, and position.
- Image-animation pipelines combine image try-on with pose-driven video generation, but lack of human-video guidance can produce static backgrounds and poor environmental blending.Large person movements can also yield strange results when actions are driven solely by pose sequences.
3. Method
Tunnel Try-on extends diffusion-based image try-on to videos with a focus tunnel that emphasizes clothing details, while temporal and environmental conditioning support coherent generation.
- Overall Architecture: Tunnel Try-on combines a Main U-Net, Ref U-Net, CLIP image encoder, pose conditioning, temporal attention, tunnel embedding, and environment encoding for video try-on.The Main U-Net processes masked video features and noise, while reference clothing features, pose maps, tunnel information, and environmental context provide complementary guidance.
- Focus Tunnel Extraction: The focus tunnel crops, pads, and resizes clothing regions to the model resolution before blending generated outputs back into the original video.Pose maps identify upper- or lower-body bounding boxes, whose coordinates are expanded to cover the clothing.
- Tunnel Smoothing: Frame-by-frame tunnel extraction can produce jitter and jumps, making temporal attention harder to optimize and reducing temporal consistency.The problem arises because independent frame processing ignores inter-frame relationships and camera or person motion.
- Tunnel Smoothing: Kalman filtering smooths tunnel coordinates, and a subsequent low-pass filter removes high-frequency jitter from exceptional cases.The algorithm takes raw tunnel coordinates and tunnel length as input and outputs smoothed tunnel coordinates.
- Tunnel Embedding: Tunnel Embedding encodes image size, tunnel center, and tunnel size before injecting positional information into temporal attention.It is designed to help temporal attention handle the larger camera movements introduced by zoomed focus-tunnel inputs.
4. Experiments
Tunnel Try-on is evaluated against existing methods on the VVT dataset and on a collected real-world dataset, using qualitative, quantitative, user-study, and ablation analyses. It achieves stronger visual and temporal results while its focus tunnel, tunnel enhancement, and environment encoder address detail preservation, flicker, continuity, and background context.
- Comparisons with Existing Alternatives: Qualitatively, Tunnel Try-on preserves correct clothing letters as the person moves closer, whereas per-frame diffusion baselines show changing letters and adjacent-frame jitters.GAN-based warping methods also struggle when the person becomes smaller, while ClothFormer produces blurry results with color deviations.
- Comparisons with Existing Alternatives: Tunnel Try-on outperforms existing methods on VVT in SSIM, LPIPS, and VFID, indicating stronger visual quality and temporal continuity.LPIPS shows a substantial advantage and is described as more aligned with human visual perception than SSIM.
- Comparisons with Existing Alternatives: On the collected dataset, Tunnel Try-on handles varied person-to-camera scales, movements, backgrounds, and clothing categories including pants, skirts, and cross-category try-on.The collected dataset is intended to address the limited diversity and simplicity of VVT.
- Ablation Study: The focus tunnel improves SSIM and LPIPS but lowers VFID, while tunnel enhancement improves VFID and further increases SSIM.This ablation indicates that zooming improves frame quality but can introduce flicker that reduces temporal consistency before enhancement is added.
- Qualitative Analysis: The focus tunnel preserves garment details by zooming into clothing regions, correcting logo distortions that appear without the close-up strategy.The qualitative ablation specifically highlights improved preservation of garment detail after zooming.
- Ablation Study: Tunnel enhancement stabilizes clothing textures over time, while the environment encoder improves background generation around garments by correcting nearby wall and zebra-crossing textures.The environment encoder has little quantitative impact but contributes contextually consistent background details.
5. Conclusion
Tunnel Try-on is presented as the first diffusion-based video visual try-on model, outperforming existing alternatives in qualitative and quantitative comparisons. Its focus tunnel, tunnel enhancement, and environment encoding support diverse camera movements, human motions, complex backgrounds, and clothing types.
- Tunnel Try-on outperforms all existing alternatives in both qualitative and quantitative comparisons.
- The model adapts to diverse camera movements and human motions in videos using focus tunnel, tunnel enhancement, and environment encoding.
- Trained on real datasets, Tunnel Try-on handles complex backgrounds and diverse clothing types while producing high-fidelity try-on results.