Source-linked AI summary
Single Stage Virtual Try-on via Deformable Attention Flows
Shuai Bai, Huiling Zhou, Zhikang Li, Chang Zhou, Hongxia Yang
TL;DR
Virtual try-on must produce realistic fittings from an in-shop garment and a reference person, despite prior methods’ multi-stage pipelines and reliance on potentially inaccurate intermediate labels. This paper proposes a single-stage, pose-guided deformable-attention-flow method that estimates self- and cross-flows. It reports superior try-on performance and applicability to multi-view synthesis and image animation.
Problem
Virtual try-on remains challenging under complex poses and large deformations, with existing methods facing misalignment or artifacts and often using multi-stage pipelines or potentially inaccurate intermediate predictions.
Source: §1 · pp. 1–2Method
SDAFN uses pose keypoints as guidance and estimates self- and cross-deformable attention flows to warp garments and synthesize the person in one end-to-end stage.
Source: §1 · p. 2Results
The method reports superior performance on virtual try-on and extension to multi-view synthesis and image animation.
Source: §1 · p. 3, §6 · p. 14Takeaways & Limitations
The approach produces photo-realistic fitting results end to end using only pose-map guidance and is also evaluated on multi-view synthesis and image animation.
Source: §1 · p. 3, §6 · p. 14
Abstract
from arXiv · showhide
Virtual try-on aims to generate a photo-realistic fitting result given an in-shop garment and a reference person image. Existing methods usually build up multi-stage frameworks to deal with clothes warping and body blending respectively, or rely heavily on intermediate parser-based labels which may be noisy or even inaccurate. To solve the above challenges, we propose a single-stage try-on framework by developing a novel Deformable Attention Flow (DAFlow), which applies the deformable attention scheme to multi-flow estimation. With pose keypoints as the guidance only, the self- and cross-deformable attention flows are estimated for the reference person and the garment images, respectively. By sampling multiple flow fields, the feature-level and pixel-level information from different semantic areas are simultaneously extracted and merged through the attention mechanism. It enables clothes warping and body synthesizing at the same time which leads to photo-realistic results in an end-to-end manner. Extensive experiments on two try-on datasets demonstrate that our proposed method achieves state-of-the-art performance both qualitatively and quantitatively. Furthermore, additional experiments on the other two image editing tasks illustrate the versatility of our method for multi-view synthesis and image animation.
1 Introduction
Virtual try-on remains difficult under complex poses and large deformations, while existing pipelines can depend on noisy intermediate predictions. The paper proposes a single-stage, pose-guided approach that estimates deformable attention flows and reports strong try-on performance and extensions to other image-editing tasks.
1 Introduction
- Existing try-on methods struggle with misalignment or visible artifacts under complex poses and large deformations.Multi-stage approaches combine garment warping, structure estimation, and image synthesis, while some methods use intermediate labels.Source: §1 · pp. 1–2
- Flow-based warping can predict inaccurate structure, external-model predictions can introduce artifacts, and GANs can alter garment color or style.Additional flow constraints may fail on complex poses.Source: §1 · p. 2
- DAFlow estimates multiple flows for different semantic areas and merges feature- and pixel-level information through attention.The resulting flows are used to warp clothes and synthesize the human torso and shadows at the same time.Source: §1 · pp. 2–3
- SDAFN uses self- and cross-deformable attention flows to generate fitting results in a single end-to-end stage.Self- and cross-DAFlows address the reference person and garment images, respectively.Source: §1 · p. 2
- The method reports superior performance on virtual try-on and extension to multi-view synthesis and image animation.The conclusion describes the fitting results as photo-realistic and produced end to end using only a pose map as guidance.Source: §1 · p. 3
2 Related Work
Prior work explores 2D try-on pipelines, spatial-transform methods for garment warping, and attention mechanisms for image tasks. The paper situates its approach against their reliance on intermediate information, flow limitations, and the computational cost of dense attention.
2.1 Virtual Try-on.
- 2D virtual try-on methods range from coarse-to-fine and segmentation-guided pipelines to approaches using densepose or UV descriptors, while some reduce inference-time reliance on extra information.The passage notes that increasingly complex pipelines can rely on external information and incur artifacts from inaccurate intermediate labels.Source: §2.1 · p. 3
2.2 Spatial Transform Module.
- Spatial-transform methods warp garments to match a person's pose, using TPS, learned TPS parameters, or cascaded dense flows.ClothFlow and PFAPN add smoothness constraints, but large discrepancies between target and original clothing remain challenging.Source: §2.2 · pp. 3–4
2.3 Efficient Attention Mechanism.
- Dense attention can be computationally costly at high feature resolution, motivating sparse attention approaches.The paper positions its approach as extending learned sparse attention to pixel-level image transformation by combining attention with flow operations.Source: §2.3 · p. 4
3 Our Approach
SDAFN performs virtual try-on in a single stage, estimating flows and merging image information through a cascade. Its approach combines deformable attention flows, feature warping, and multiscale training losses.
3.1 Architecture
- Pose keypoints guide a three-part architecture: pyramid feature extraction, cascade flow estimation, and shallow encoder-decoder generation.Separate reference and garment branches extract features; cascade estimation addresses large deformations from coarse to fine.Source: §3.1 · p. 5
- Shared shallow encoders and decoders transform and merge person and garment features with DAWarp before producing the try-on image.Each encoder and decoder has two convolution layers without downsampling.Source: §3.1 · p. 5
3.2 Deformable Attention Flows
- DAFlow combines multiple flow fields and attention weights to preserve texture while estimating garment structure beyond what a single flow can capture.DAWarp integrates features from multiple locations; image-level warping can also generate realistic neck and arm parts absent from the original image.Source: §3.2 · pp. 6–7
3.3 Deformable Attention Flow Network.
- At each scale, DAFN uses Self-MFE, Cross-MFE, and Refine-MFE to estimate and refine self- and cross-flow fields and attention maps.The cascade upsamples prior-scale flows and attention maps for finer estimation.Source: §3.3 · p. 7
3.4 Train Losses
- Training compares generated images with targets across scales using L1, perceptual, and style losses, without flow constraints or regularization.The perceptual loss measures L1 distances between VGG-19 feature maps; output-nearer scales receive larger weights.Source: §3.4 · p. 8
4 Experiments
Experiments on VITON and MPV assess SDAFN’s qualitative outputs, paired and unpaired performance, and component contributions. The results cover pose and viewpoint variation, higher-resolution inference, and ablations of sampling and model modules.
4.1 Datasets.
- The experiments use VITON and MPV; MPV includes multiple views, and its back-tagged images are removed because target garments are front-facing.VITON has 14,221 training and 2,032 testing image pairs; MPV has 4,175 test pairs.Source: §4.1 · pp. 8–9
4.2 Implementation Details.
- The five-layer FPN encoders and per-layer multi-flow estimators use K = 6; training uses AdamW for 200 epochs with shared settings across both datasets.Experiments were run in PyTorch on Tesla V100 GPUs with batch size 8.Source: §4.2 · p. 9
4.3 Qualitative Results.
- For VITON, SDAFN preserves garment patterns and body characteristics in challenging poses and misalignments, while comparison methods show artifacts, blur, or sensitivity to segmentation errors.The reported examples include stripes around the forearm and waist, side views, and garment-sleeve discrepancies.Source: §4.3 · pp. 9–10, Figure 4 · p. 10
- On MPV, SDAFN captures texture and pose across clothing and viewpoint variations and generates realistic body parts unseen in the reference images.Figure 5 places reference poses in the first row and target garments in the first column.Source: §4.3 · p. 10, Figure 5 · p. 11
- At 512 × 384 inference resolution, the model trained at 256 × 192 uses linearly interpolated flows to retain finer texture than image-level interpolation, without retraining.The comparison includes clarity of clothing characters and patterns.Source: §4.3 · p. 10, Figure 6 · p. 11
4.4 Quantitative Results.
- In paired VITON tests, SDAFN outperforms the baselines on all reported metrics, with the best FID and IS scores and higher SSIM and PSNR scores; unpaired tests report FID improvements over parser-based and parser-free methods.The paired comparison uses Table 1; the unpaired comparison covers VITON and MPV.Source: Table 1 · p. 10, §4.4 · pp. 11–12, Table 2 · p. 12
4.5 Ablation Study.
- Sampling performance improves as K increases, then drops slightly beyond 6; the study sets K to 6 as a balance between performance and computational cost.Increasing K consumes more GPU memory but adds limited time consumption.Source: §4.5 · p. 13, Table 3 · p. 13
- Adding cascaded flow estimation, a shallow encoder and decoder, and DAFN increases SSIM and PSNR, with DAFN contributing the most and the full combination performing best.The effects of the modules are reported in Table 4.Source: §4.5 · p. 13, Table 4 · p. 13
5 Applications on Other Tasks
The framework is adapted for multi-view synthesis and image animation by removing self-MFE for single-image transformations. The reported examples show chair views with reconstructed occluded parts and pose reconstruction during complex motion.
Multi-view synthesis
- For single-image transformations, the authors remove self-MFE from DAFN; multi-view synthesis learns structural correlations across views of the same object.The multi-view experiment uses ShapeNet chairs.Source: §5 · p. 13
- The model reconstructs occluded chair parts and predicts accurate textures across novel views from a one-shot input.The experiment uses the ShapeNet chairs dataset.Source: §5 · pp. 13–14
Image animation
- The FashionVideo examples reconstruct the input pose during complex motion, including a view of the body's back.The experiments use 500 training videos and 100 testing videos.Source: §5 · p. 14, Figure 9 · p. 14
6 Conclusions
The paper concludes that its single-stage try-on framework uses pose-map guidance to generate photo-realistic fitting results end to end. It reports superior try-on performance and extension to multi-view synthesis and image animation.
6 Conclusions
- The proposed method achieves superior virtual try-on performance and extends to multi-view synthesis and image animation.The conclusion also describes photo-realistic torso and fitting-clothes synthesis with 3D shadows.Source: §6 · p. 14
- The single-stage framework generates photo-realistic fitting results end to end using only a pose map as guidance.Source: §6 · p. 14
- DAFlow estimates accurate structure while retaining vivid texture for photo-realistic human-torso and fitting-clothes synthesis with 3D shadows.Source: §6 · p. 14