ShoeFit: A New Dataset and Dual-image-stream DiT Framework for Virtual Footwear Try-On

1 Introduction

• VFTON faces data scarcity, mismatched shoe and foot viewpoints, and background-induced color contamination that complicate faithful footwear try-on.The paper notes that no open-source footwear try-on dataset was available.Source: §1 · p. 2
• MVShoes provides 7,305 annotated high-resolution triplets with two product views and a corresponding try-on result.The dataset covers diverse shoe categories and scenarios, with matching supported by a data-cleaning pipeline.Source: §1 · p. 2
• ShoeFit uses a dual-stream DiT with Multi-View Conditioning and LayeredRefAttention to address viewpoint mismatch and background-induced distortion.Its decoupled denoising and conditioning branches generate try-on results; LayeredRefAttention uses background features to modulate shoe features.Source: §1 · pp. 2–3
• The authors report that quantitative and qualitative experiments validate the approach's superiority.The conclusion reports improved detail fidelity and texture consistency across diverse scenarios.Source: §1 · pp. 2–3

2 Related Works

• GAN-based garment try-on commonly uses a two-stage process: warp clothing to the desired shape, then fuse it with the person image.More recent approaches incorporate pretrained diffusion models as priors for virtual try-on.Source: §2 · p. 3
• Recent garment try-on methods integrate garment features through parallel U-Nets and self-attention or concatenate garment and person images spatially.The passage identifies OOTDiffusion, IDM, and MMTryon as parallel-U-Net examples, and CatVTON as a concatenation example.Source: §2 · p. 3
• Existing garment-focused try-on methods lack support for faithful 3D multi-view shoes, while controllable diffusion methods rely on conditioning described as too coarse for VFTON fidelity demands.The passage highlights footwear's varied materials, geometric variability, and stringent fidelity requirements.Source: §2 · p. 3

3 MVShoes Dataset

• MVShoes contains 7,305 high-resolution image triplets, each with two product views and a corresponding try-on result, plus foot landmarks and shoe-agnostic masks.These annotations are intended to facilitate precise alignment.Source: §3 · p. 4

3.1 MVShoes Collection

• The collection draws on varied model scenarios and shoe imagery, then uses automated matching, manual filtering, blur removal, and pose and mask extraction to form try-on triplets.The initial collection comprised approximately 25,000 raw shoe product image sets, each with two to five images.Source: §3.1 · p. 4

3.2 Dataset Statistic

• The dataset includes four model scenarios: top-down and horizontal foot close-ups, half-body models, and full-body models.Horizontal foot close-ups are the largest proportion; preserving side details is challenging when viewpoints are misaligned.Source: §3.2 · p. 4
• MVShoes uses a two-level product-category label set designed to align its category distribution with everyday use.The passage notes background-color contamination from reflections, particularly for boots, leather shoes, and high heels.Source: §3.2 · p. 4, Figure 3 · p. 4

4.1 Dual-image-stream DiT Framework

• ShoeFit removes FLUX’s text stream and initializes a conditioning image stream from part of the denoising-stream architecture and weights, decoupling shoe-detail preservation from generation.The paper reports approximately 40% parameter savings, including T5 and CLIP models.Source: §4.1 · p. 5

4.2 Multi-view Conditioning

• Reference Concatenation spatially concatenates product views for conditioning-stream processing, letting human-image queries attend across shoe features to align target regions with product details.The denoising stream also receives noisy human latents, a shoe-agnostic masked image, an agnostic mask, and foot-pose landmarks.Source: §4.2 · p. 6
• 3D RoPE extends FLUX’s 2D positional encoding to distinguish view identity from within-image height and width, addressing semantic jumps at concatenation boundaries.The positional encoding includes a view channel and separate height and width positions.Source: §4.2 · p. 6

4.3 LayeredRefAttention Module

• LayeredRefAttention uses background features to modulate foreground shoe features, while restricting denoising attention to foreground content to preserve intrinsic color and texture.This targets background-color contamination while retaining illumination cues that can help infer shoe appearance.Source: §4.3 · pp. 6–7

5.1 Experimental Setup

• Evaluation uses MVShoes at 768 × 768 resolution, with 6,305 training pairs and 1,000 test pairs; paired and unpaired evaluations use different metrics.LPIPS, SSIM, and DISTS measure paired reconstruction, while FID and KID assess unpaired authenticity; baselines include FLUX variants, OOTDiffusion, CatVTON, and a two-view CatVTON variant.Source: §5.1 · pp. 7–9

5.2 Qualitative Results

• In hard scenes, ShoeFit reproduces shoe structure, decouples environmental lighting, avoids background confusion, and handles uncommon poses.The qualitative comparison reports extreme color variations for OOTDiffusion and condition mismatches for FLUX.1-Fill-Redux and FLUX.1-Fill.Source: Figure 5 · p. 8, §5.2 · p. 9

5.3 Quantitative Results

• ShoeFit significantly surpasses all baselines under both single-view and multi-view conditioning in the reported quantitative evaluations.The passage attributes this result to specialized parameters in the Dual-image-stream DiT framework and background decoupling through LayeredRefAttention.Source: §5.3 · p. 9

5.4 Ablation Study

• The dual-image-stream framework addresses artifacts and incorrect structures, while LayeredRefAttention avoids color contamination that appears with environmental lighting, shadows, or background patches.Removing shoe-image backgrounds instead worsens color artifacts, particularly for highly saturated hues; the authors hypothesize that this removes useful illumination cues.Source: Figure 6 · p. 8, §5.4 · p. 9
• Multi-View Conditioning reduces artifacts in unseen areas by adding another product view, and the proposed 3D RoPE improves performance without additional computational overhead.The single-view ablation often generates unrealistic speculations and artifacts where conditional information is insufficient.Source: Figure 6 · p. 8, §5.4 · p. 10

5.5 Inference Time and Memory Usage

• On an NVIDIA H20, ShoeFit with the Dual-image-stream architecture outperforms CAT-VTON in inference speed and memory efficiency.The authors attribute this to removing the text stream and its T5 and CLIP encoders; Table 3 compares inference time and GPU memory usage.Source: §5.5 · p. 10, Table 3 · p. 10

6 Conclusion

• MVShoes contains 7,305 pairs of high-quality multi-view garment-model samples, and ShoeFit combines a dual-stream architecture with Multi-View Conditioning and LayeredRefAttention.The paper reports improved detail fidelity and texture consistency across diverse scenarios and describes the method as a strong baseline for future VFTON research.Source: §6 · p. 10
• The authors warn that ShoeFit could be misused to produce media breaching intellectual-property rights or privacy norms and advocate conscientious use.Source: §6 · p. 10
• ShoeFit occasionally struggles to render small logos and intricate text, a limitation the authors identify as particularly relevant to e-commerce sellers.They propose explicit 3D geometric and material priors as future work for more robust multi-view representation and refined visual fidelity.Source: §6 · p. 10

A Limitations and Future Work

• ShoeFit occasionally renders small logos and intricate text inaccurately because they occupy little image area and contain high-frequency variation.The authors note that these patterns matter to e-commerce sellers because they convey brand information.Source: A Limitations and Future Work · p. 16
• The authors plan supplementary conditioning to pre-detect and enhance the injection of small patterns.Source: A Limitations and Future Work · p. 16
• Future work will introduce explicit 3D geometric and material priors for more robust multi-view representation and refined visual fidelity.Source: A Limitations and Future Work · p. 16

• The model can struggle to render small logos and intricate text because they occupy little image area and contain high-frequency variation.Source: Figure 7 · p. 16
• The data pipeline classifies shoe and model images, removes duplicates and mismatched pairs, segments shoes, then manually filters triplets and extracts foot poses and shoe masks.The described tools include Qwen2.5-VL, CLIP, SAM, DINO, DWPose, and GroundingDINO.Source: B MVShoes Dataset Supplement · pp. 16–17
• MVShoes statistics are presented for shoe subcategories and human scenes, while the supplement adds quantitative tables and further visual samples.The main body provides only a rough visual presentation of these statistics because of space constraints.Source: B MVShoes Dataset Supplement · pp. 16–17, Table 4 · p. 18, Table 5 · p. 18
• Fine-tuning DWPose on diverse lower-body and close-up leg poses addresses the difficulty of estimating partial-leg and full-lower-body poses for footwear try-on.Source: B MVShoes Dataset Supplement · p. 17
• The leg-pose-aware mask combines pose information with an expanded shoe-detection box to support cross-category try-on while maintaining anatomical alignment and occlusion.Source: B MVShoes Dataset Supplement · p. 17
• Severe rear-shoe occlusion can defeat keypoint and shoe detection, classify the rear shoe as background, and leave only the front foot in the inpainting mask.The resulting incomplete mask can cause virtual try-on to fail; the authors describe this as a challenge for explicit masking methods.Source: B MVShoes Dataset Supplement · p. 17
• ShoeFit learns stochastic shoe-volume expansion in real-world pairs, such as outward bulging from stiff tongues, and reproduces the corresponding deformation.Source: B MVShoes Dataset Supplement · p. 17

C.1 Preliminary

• FLUX.1 uses a variational autoencoder with encoder E and decoder D for latent-space image representations, and RoPE [58] rotates attention queries and keys to capture relative positional relationships in that space.Source: §C.1 · p. 18
• Flow matching optimizes a velocity field that maps noise toward data, with the loss taking the expected squared difference between the conditional model velocity and target vector field.The expectation spans time, latent variables, conditions, and noise.Source: §C.1 · p. 19

C.2 LayeredRefAttention in Single-stream DiT Block

• In the single-stream DiT block, an SE block and spatial average pooling compute channel-specific weights, then linear layers extract background modulation parameters to modulate foreground shoe features.The product-image shoe mask is represented by M_fg, and subsequent operations follow regular single-stream DiT blocks.Source: §C.2 · p. 19

D.1 Baseline Training Details

• All models are trained on MVShoes at 768 × 768 resolution, using 6305 pairs for training and 1000 for testing.Source: §D.1 · p. 19
• Flux.1 Fill uses rank-64 LoRA on attention modules and pose-injecting ControlNet; Redux adds trainable SigLip projections, while OOTDiffusion follows its official pipeline with ControlNet.Flux.1 Fill trains for six days on eight A100 GPUs; Redux retains those settings, while OOTDiffusion trains for six days on four A100 GPUs.Source: §D.1 · pp. 19–20
• CatVTON-Flux-Lite-2V concatenates footwear and human images, keeps the FLUX-lite text stream, and trains for seven days on twelve A100 GPUs.Source: §D.1 · p. 20

D.2 Data Augmentation

• Augmentations independently apply horizontal flips, up to 10% padding-based resizing, hue shifts from −5 to +5, and contrast factors from 0.8 to 1.2, each with 50% probability.Each operation is applied simultaneously to footwear and model images.Source: §D.2 · p. 20

D.3 LayeredRefAttention Hyperparameters

• LayeredRefAttention adds a foreground linear layer and an SE block with reduction rate 4, using hidden-size features.Figure 10 shows the module within a single-stream DiT block.Source: Figure 10 · p. 19, §D.3 · p. 20

E More visual results

• Visual ablations highlight improvements from all three components of the method.Source: E More visual results · p. 20, Figure 9 · p. 22
• ShoeFit substantially improves rendering fidelity and robustness on challenging real-world product shoes in the MVShoes comparisons.All methods are shown with single-view conditioning for a fair comparison.Source: E More visual results · p. 20, Figure 11 · p. 23
• ShoeFit synthesizes high-fidelity, detail-faithful try-on results in additional MVShoes examples.Source: E More visual results · p. 20, Figure 12 · p. 24
• Early training can produce structural distortions, while inaccurate masks for tall footwear can cause artifacts or fitting errors.Source: E More visual results · p. 20, Figure 13 · p. 25
• High-quality try-on results on samples outside MVShoes demonstrate the model’s reported stability and generalization.Source: E More visual results · p. 20, Figure 14 · p. 25