PaperlayerView exact revision on arXiv

Source-linked AI summary

FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On

Johanna Karras, Yuanhao Wang, Yingwei Li, Ira Kemelmacher-Shlizerman

arXiv:2604.08526v1cs.CVcs.GR

TL;DR

Fit-aware virtual try-on lacks large-scale data with precise size annotations and paired images, especially for ill-fitting garments. FIT addresses this gap with physics-based synthetic data and fit-aware modeling, demonstrating accurate garment-fit visualization.

  • Problem

    Fit-aware virtual try-on lacks large-scale data with precise person and garment measurements, ill-fit examples, and paired images for supervised training.

  • Method

    FIT uses physics simulation and generative re-texturing to create photorealistic, size-annotated try-on triplets with paired person images.

  • Results

    The resulting baseline fit-aware VTO model accurately showcases garment fit according to person and garment measurements.

  • Takeaways & Limitations

    The results support physics-informed synthetic supervision as a promising approach to overcoming size-annotated data scarcity in virtual try-on.

  • Takeaways & Limitations

    The dataset is limited to upper-body garments, standardized front-facing views, casual poses, and simple structural designs supported by GarmentCode.

Abstract

from arXiv · show

Given a person and a garment image, virtual try-on (VTO) aims to synthesize a realistic image of the person wearing the garment, while preserving their original pose and identity. Although recent VTO methods excel at visualizing garment appearance, they largely overlook a crucial aspect of the try-on experience: the accuracy of garment fit -- for example, depicting how an extra-large shirt looks on an extra-small person. A key obstacle is the absence of datasets that provide precise garment and body size information, particularly for "ill-fit" cases, where garments are significantly too large or too small. Consequently, current VTO methods default to generating well-fitted results regardless of the garment or person size. In this paper, we take the first steps towards solving this open problem. We introduce FIT (Fit-Inclusive Try-on), a large-scale VTO dataset comprising over 1.13M try-on image triplets accompanied by precise body and garment measurements. We overcome the challenges of data collection via a scalable synthetic strategy: (1) We programmatically generate 3D garments using GarmentCode and drape them via physics simulation to capture realistic garment fit. (2) We employ a novel re-texturing framework to transform synthetic renderings into photorealistic images while strictly preserving geometry. (3) We introduce person identity preservation into our re-texturing model to generate paired person images (same person, different garments) for supervised training. Finally, we leverage our FIT dataset to train a baseline fit-aware virtual try-on model. Our data and results set the new state-of-the-art for fit-aware virtual try-on, as well as offer a robust benchmark for future research. We will make all data and code publicly available on our project page: https://johannakarras.github.io/FIT.

1 Introduction

Existing virtual try-on systems produce photorealistic garment transfers but largely ignore person–garment size compatibility, limiting realistic fit simulation and experimentation. FIT addresses this gap with a large-scale, measurement-annotated benchmark, synthetic physics-based data pipeline, and fit-aware model.

  • Motivation: Recent VTO methods often transfer garment appearance without accounting for person or garment sizes, failing to answer how a garment will fit the user.This limits the accuracy and reliability of simulated try-on experiences and prevents experimentation with fitted or oversized looks.
  • Motivation: Fit-aware try-on is difficult because existing real-world datasets lack precise measurements and examples where garments are substantially too large or too small.Retail catalog imagery generally lacks these ill-fit cases, creating a major data gap.
  • FIT Dataset and Pipeline: FIT introduces the first large-scale, size-aware VTO benchmark for diverse upper-garment fit scenarios, using procedurally generated 3D garments with exact measurements and simulated draping on parametric bodies.The synthetic strategy addresses limitations of real-world data collection through GarmentCode and physics simulation.
  • FIT Dataset and Pipeline: The dataset pipeline combines physics simulation and generative re-texturing to create photorealistic, geometrically grounded try-on triplets while preserving paired identities.Paired data addresses the scarcity of identical-subject, identical-pose examples with different garments and avoids limitations of pseudo triplets.
  • FIT Dataset and Pipeline: FIT contains 1.13M training and 1K test samples spanning men’s and women’s upper-garments, with try-on, layflat garment, paired person images, and person and garment measurements.The dataset includes extreme ill-fits, such as a size 3XL garment draped onto a size XS person.
  • Fit-Aware Model: A baseline Fit-VTO model incorporates person and garment measurements to visualize garment appearance together with accurate garment fit.The model is demonstrated by fine-tuning Flux.1-dev with a custom dataset and measurement encoder.

2 Related Works

Prior virtual try-on research largely lacks explicit size annotations and ill-fitting examples, while existing methods often prioritize appearance and alignment over physically accurate fit. Recent approaches address fit through geometric heuristics or coarse labels, whereas FIT uses synthetic 3D draping and exact measurements to provide precise control.

  • Fit-Aware Datasets: Standard 2D VTO benchmarks predominantly contain well-fitted garments and lack the diverse fit conditions needed for size-aware training.Examples include ViTON, ViTON-HD, DressCode, Street-TryOn, and LAION-Garment.
  • Fit-Aware Datasets: GarmentCode enables synthetic garment generation across varied garment and body sizes through sewing patterns with explicit size parameters.Existing 3D clothed-human datasets often make accurate garment-measurement extraction from raw meshes infeasible.
  • Image-Based Virtual Try-On: Image-based VTO methods follow mask-based or mask-free paradigms, with recent diffusion architectures achieving photorealism but largely neglecting physical garment fit.Mask-based methods use segmentation-guided inpainting, while mask-free methods synthesize without segmentation priors.
  • Image-Based Virtual Try-On: Mask-free methods commonly train on hallucinated pseudo-triplets, inheriting teacher artifacts and geometric inconsistencies.FIT instead simulates actual draping dynamics on 3D bodies to produce ground-truth pairs with precise geometry and segmentation.
  • Fit and Size Control: Prior fit-control methods use geometric heuristics or coarse labels, limiting generalization to complex poses and precise control.FIT conditions its fit-aware model on exact metric measurements rather than noisy intermediate values.

3 Fit-Inclusive Try-on (FIT) Dataset

FIT is a large-scale dataset of over 1.13 million try-on samples with precise body and garment measurements spanning diverse bodies, poses, designs, and garment fits. Its construction combines procedurally generated and physically draped garments, geometry-preserving photorealistic re-texturing, and controllable paired-image generation.

  • 3.2 Data Generation: GarmentCode procedurally generates measured 3D garments and physics-simulates their draping across human bodies with varying sizes to produce diverse garment fits.The pipeline uses cross-draping: patterns generated for multiple body sizes are draped onto a single target human, with separate bottom- and top-garment simulation for realistic layering.
  • 3.1 Dataset Statistics: 1,137,282 training and 1000 test samples cover 168 body shapes, 528 poses, and 158,483 unique top and garment designs across loose-to-tight fits.The body shapes include 82 men’s and 86 women’s bodies in sizes XS-3XL, and the test set is balanced across gender and body-size distributions.
  • 3.2 Data Generation: The procedural framework extracts garment measurements in centimeters from sewing patterns and body measurements from GarmentCode’s parametric body model.Garment metrics include length, bust circumference, sleeve length, waist, and out-seam length; body metrics include height, bust, waist, and hips.
  • 3.3 Re-texturing: A diffusion-based re-texturing pipeline converts synthetic renderings into photorealistic try-on images while strictly preserving garment and subject geometry through surface-normal conditioning.The model is based on Flux.1-dev and uses normal maps as a geometry-preserving bridge between synthetic and real domains.
  • 3.4 Paired Data Generation: Fixing a subject’s shape and pose while changing garments enables ground-truth paired synthetic renderings and paired reference-image synthesis for supervised training.The primary try-on image is generated with the re-texturing pipeline, while a separate conditional process synthesizes the paired reference image.

4 Fit-Aware Virtual Try-On

Fit-VTO predicts a person wearing a target garment according to both person and garment measurements. It uses a flow-matching diffusion architecture finetuned from FLUX.1-dev, with measurement conditioning and additional online fashion data for robustness.

  • Model formulation: Fit-VTO synthesizes a try-on image conditioned on the person image, garment image, garment measurements, and person measurements.The model predicts person p wearing garment g according to measurements m_p and m_g.
  • Training data: 330,559 online fashion images and corresponding layflat garment images augment FIT training, using null measurements when ground-truth measurements are unavailable.This augmentation is intended to improve robustness to diverse real-world garments and poses.
  • Training data: FIT person and garment measurements are normalized between 0 and 1.The normalization applies to all measurements in FIT data samples.
  • Architecture: The architecture is a flow-matching diffusion model that predicts velocity from noisy ground-truth images across diffusion timesteps.Training enforces consistency between predicted velocity and the ground-truth velocity v_t = x_0 − z_0.
  • Architecture: The network is finetuned from the 12-billion-parameter FLUX.1-dev model using lightweight LoRA parameters while conditioning measurements through cross-attention.Positional encodings represent each measurement, and measurement conditioning replaces T5 text conditioning in the single-stream and double-stream blocks.

5 Experiments

Experiments evaluate the synthetic paired-image data and Fit-VTO across image quality, identity and garment-fit fidelity. Results show that Fit-VTO preserves appearance and identity while modeling size-dependent fit, generalizing to real images and benefiting from combined FIT, real-world, and measurement supervision.

  • Paired Image Generation Evaluation: Paired-image comparisons show VLM baselines degrade pose, identity, and body-shape preservation, while VTO and inpainting baselines introduce visual artifacts.The proposed approach produces highly consistent paired data.
  • Fit-Aware VTO Evaluation: Fit-VTO generates high-quality try-on images that preserve person identity and garment appearance while representing tight, perfect, and loose fits.These qualitative results are shown on the synthetic FIT test dataset.
  • Fit-Aware VTO Evaluation: Fit-VTO adjusts garment fit realistically when garment size varies with the person fixed, while maintaining consistent person and garment appearance.The paper also reports independent garment-measurement controllability, extending to real-world human images.
  • Fit-Aware VTO Evaluation: Compared with related methods, Fit-VTO more accurately reflects fit relative to person and garment sizes, whereas competing methods can produce overly loose or tight results despite strong texture warping.Fit-VTO also generalizes to real-world VITON-HD images without measurements.
  • Ablations: Ablations show that FIT supervision improves size-aware fitting, real-world data supports VITON-HD generalization, and the full model best balances both benchmarks.FIT-only training degrades on VITON-HD, while removing FIT data weakens person–garment size modeling; combining FIT supervision, real-world data, and the measurement encoder yields robust, size-sensitive behavior.

6 Scope and Limitations

FIT is a proof-of-concept with intentionally constrained scope, showing that physics-informed synthetic supervision can teach models to respect metric sizing while exposing important limitations. The dataset and model poorly represent varying tightness and independently changing correlated measurements.

  • Scope: FIT focuses exclusively on upper-body garments in standardized front-facing settings, constraining its current scope.The authors position the work as an initial exploration and foundation for scaling toward complex, in-the-wild scenarios.
  • Scope: Synthetic, physics-informed supervision validates the hypothesis that generative models can respect precise metric sizing.The work demonstrates a promising strategy for overcoming scarce size-annotated virtual try-on data.
  • Technical limitations: The dataset and VTO model do not represent varying degrees of tightness well because tight and very tight garments appear nearly identical when fitted to the skin.The authors note that the simulated appearance fails to capture the distinct wearing experience associated with different tightness levels.
  • Technical limitations: Fit-VTO is sensitive to measurement correlations, limiting its ability to alter individual measurements independently.Increasing width frequently also causes slight increases in length and sleeve length.

7 Conclusions and Future Work · Appendix

The paper introduces FIT as a large-scale fit-aware virtual try-on dataset and benchmark, alongside Fit-VTO, a model that uses person–garment size annotations to improve fit accuracy. Extensive comparisons show Fit-VTO’s advantage in modeling garment fit according to person and garment measurements.

  • 7 Conclusions and Future Work: FIT is introduced as the first large-scale dataset and benchmark for fit-aware virtual try-on.It contains over 1.13M samples.
  • 7 Conclusions and Future Work: FIT comprises over 1.13M samples for fit-aware virtual try-on.
  • 7 Conclusions and Future Work: The paper presents Fit-VTO, a novel model designed to leverage FIT’s person–garment size annotations.
  • 7 Conclusions and Future Work: Fit-VTO is evaluated through extensive comparisons with related and ablated methods.
  • 7 Conclusions and Future Work: Fit-VTO demonstrates a clear advantage in modeling accurate garment fit.
  • 7 Conclusions and Future Work: Garment-fit accuracy is assessed according to person and garment measurements.

A Additional FIT Dataset Details … A.3 Measurement Statistics

The FIT dataset appendix explains its coarse size-labeling scheme, broad coverage of plausible person–garment fit pairings, exclusion of extreme implausible pairings, and measurement-statistics reporting for bodies and garments.

  • A.1 Size Categorization: Coarse labels such as XS, L, and XL abbreviate full person–garment measurements in figures.These labels are based on average measurement ranges and are used only for visualization and grouping, not Fit-VTO training or evaluation.
  • A.1 Size Categorization: The coarse size labels do not replace the full measurements used for Fit-VTO training or evaluation.Their role is limited to visualization and grouping purposes.
  • A.2 Garment Fit Distribution: The dataset represents reasonable fit scenarios ranging from very tight to very loose person–garment pairings.Examples include an XL person wearing an M garment and an XS person wearing a 2XL garment.
  • A.2 Garment Fit Distribution: Garments more than 3 sizes smaller than the person are excluded as implausible fit pairings.A 3XL person wearing an XS garment illustrates an excluded combination.
  • A.3 Measurement Statistics: The appendix reports minimum, mean, maximum, and standard deviation statistics for body measurements in the FIT dataset.These statistics are presented for the body meshes in Table 3, with measurements reported in centimeters.
  • A.3 Measurement Statistics: The appendix reports minimum, mean, maximum, and standard deviation statistics for garment measurements in the FIT dataset.These statistics are presented for the garment meshes in Table 4, with measurements reported in centimeters.
  • A.3 Measurement Statistics: The body and garment measurement statistics indicate that FIT covers a wide range of body shapes and garment sizes.The reported statistics summarize the distributions of the body and garment meshes.

A.4 Additional Dataset Examples … B.2 Boxmesh Realignment

The appendix provides additional FIT try-on triplets and details a cross-draping pipeline for generating ill-fitting garments. Because cross-draping can misalign garment boxmesh panels with target bodies, boxmesh realignment aligns patterns to reference garments and target meshes for successful simulation.

  • A.4 Additional Dataset Examples: The dataset includes additional examples of try-on image triplets in Figure 9.
  • B Additional Details on Data Generation Pipeline: The standard GarmentCode pipeline generates sewing patterns from design templates and target body parameters, producing garments well-fitted to the wearer.
  • B.1 Cross Draping vs. Linear Size Change: Simple linear scaling of garment parameters cannot model ill-fitting garments because real-world garment grading rules are nonlinear and differ from geometric scaling.
  • B.1 Cross Draping vs. Linear Size Change: Cross-draping applies a sewing pattern to a target body mesh of a different size to create oversized or undersized garment examples.
  • B.2 Boxmesh Realignment: Cross-draping can misalign boxmesh panels with target body parts, causing draping errors and failures.
  • B.2 Boxmesh Realignment: Boxmesh realignment aligns an ill-fitting target pattern’s panels to a well-fitted reference pattern generated for the target body size.The reference pattern is generated on the target body size and supplies the spatial locations used for alignment.
  • B.2 Boxmesh Realignment: Aligning the target pattern to the reference pattern ensures alignment with the target body mesh and supports successful simulation.
  • B.2 Boxmesh Realignment: Significant differences between the human model’s arm angle and initialized sleeve-panel angle can cause arm–sleeve penetration.

B.3 Retexturing Model Training Data · B.4 Reposing

The retexturing model is trained on real-world person images enriched with estimated geometry cues and structured garment prompts for paired generation. To diversify poses beyond static A-pose simulations, the dataset reposes samples into randomly selected casual stances from 528 target poses.

  • B.3 Retexturing Model Training Data: 50k real-world person images form the retexturing model’s training dataset.The images come from VITON-HD and scraped online modeling images with studio backgrounds.
  • B.3 Retexturing Model Training Data: Scraped images depict people modeling in front of the camera against studio backgrounds.
  • B.3 Retexturing Model Training Data: Sapiens estimates normal and segmentation maps to provide geometric guidance for retexturing.
  • B.3 Retexturing Model Training Data: Gemini generates garment-texture prompts in a structured two-sentence format, with one sentence per garment piece.For paired generation, swapping only the top garment updates the first sentence corresponding to the top.
  • B.3 Retexturing Model Training Data: Paired generation is enabled by updating only the prompt sentence corresponding to the swapped top garment.
  • B.4 Reposing: 528 distinct target poses expand dataset diversity beyond GarmentCode’s static A-pose simulations.Each sample is reposed into a randomly chosen pose from the pool, prioritizing casual stances common in real-world try-on scenarios.

C Additional Resizing Results … E.1 VITON-HD Preprocessing

Fit-VTO supports resizing individual garment measurements while preserving unaffected dimensions and appearance, with additional tests on real-world images. Its limitations include restricted tightness control, correlated measurements that can cause unintended changes, and pseudo-pair preprocessing for VITON-HD.

  • C Additional Resizing Results: Fit-VTO adjusts specific garment dimensions in response to measurement changes while preserving non-adjusted dimensions and person and garment appearance.This controllability is evaluated in Figure 10.
  • C Additional Resizing Results: Additional resizing examples demonstrate Fit-VTO on real-world person images using measurements captured from human subjects and garments randomly selected from FIT test data.The real-world examples combine captured person measurements with randomly chosen FIT test garments and measurements.
  • D Failure Cases: The method has limited ability to represent varying degrees of garment tightness because of limitations in GarmentCode.The paper leaves improving this capability to future work.
  • D Failure Cases: Correlations among FIT garment measurements can cause changing one measurement to unintentionally change another in Fit-VTO.The paper gives larger width correlating positively with larger length as an example.
  • E.1 VITON-HD Preprocessing: For VITON-HD preprocessing, pseudo paired-person images are generated for every image because paired data are unavailable.The images are generated using Nano Banana Pro, with prompts described in Section F.4.
  • E.1 VITON-HD Preprocessing: When applying Fit-VTO to VITON-HD, person and garment measurements are set to the null value -1.The null value matches the dropout value used during training.

E.2 Implementation Details · F LLM and VLM Prompts

The paper specifies evaluation settings for baseline VTO systems and documents the exact LLM/VLM prompts used for image editing, garment description, try-off generation, paired-person generation, and quality assurance.

  • E.2 Implementation Details: Any2AnyTryon uses the official implementation and the “dev_lora_any2any_multi” checkpoint for all evaluations.
  • E.2 Implementation Details: COTTON uses its official code and checkpoint, with default try-on mode on VITON-HD and r=length/bust scaling on FIT.
  • E.2 Implementation Details: Nano Banana Pro receives paired-person and layflat garment images, person-garment measurements, and a prompt specifying the target garment and measurements.
  • E.2 Implementation Details: IDM-VTON uses official code, original VITON-HD agnostic masks, FIT masks from preprocessing, and officially recommended hyperparameters.
  • F LLM and VLM Prompts: The paper provides exact prompts for all calls to Gemini LLM and Nano Banana Pro VLM models.
  • F.1 Head & Shoes Generation: Head-and-shoe editing adds photorealistic hair behind the shoulders and visible shoes while preserving pose, garment properties, and image layout.
  • F.2 Prompt Generation: Garment prompts describe top and bottom garments in two sentences, invent new texture, logo, and design details, and remain under 50 words.
  • F.3 Garment Try-Off · F.4 Paired Person Image Generation: Try-off generation creates a top-only product image on white, while paired-person generation changes only the upper garment and preserves all other image content.
Loading 2604.08526v1…