PaperlayerView exact revision on arXiv

Source-linked AI summary

RefTon: Reference person shot assist virtual Try-on

Liuzhuozheng Li, Yue Gong, Shanyuan Liu, Dengyang Jiang, Zanyi Wang, Bo Cheng, Yuhang Ma, Leibucha Wu, Dawei Leng, Yuhui Yin

arXiv:2511.00956v6cs.CV

TL;DR

Virtual try-on methods often rely on auxiliary inputs, and garment-only images may not show worn appearance or fine details. RefTon adapts Flux-Kontext for person-to-person try-on with unpaired clothed-person references; the authors report state-of-the-art performance across settings. Its single-model design supports mask-based and mask-free try-on with varying reference conditions.

  • Problem

    Prior virtual try-on approaches rely on external models and auxiliary inputs, while garment images alone may not reveal details visible when the garment is worn.

    Source: §1 · p. 2

  • Method

    RefTon adapts Flux-Kontext with a two-stage training strategy and unpaired clothed-person references as additional visual guidance.

    Source: §1 · p. 2

  • Results

    RefTon consistently outperforms existing methods quantitatively and qualitatively, achieving state-of-the-art performance in all settings.

    Source: §5 · p. 9

  • Takeaways & Limitations

    One model supports mask-based and mask-free try-on while integrating varying numbers and types of references.

    Source: §1 · p. 3

Abstract

from arXiv · show

We introduce RefTon, a flux-based person-to-person virtual try-on framework that enhances garment realism through unpaired visual references. Unlike conventional approaches that rely on complex auxiliary inputs such as body parsing and warped mask or require finely designed extract branches to process various input conditions, RefTon streamlines the process by directly generating try-on results from a source image and a target garment, without the need for structural guidance or auxiliary components to handle diverse inputs. Moreover, inspired by human clothing selection behavior, RefTon leverages additional reference images (the target garment worn on different individuals) to provide powerful guidance for refining texture alignment and maintaining the garment details. To enable this capability, we built a dataset containing unpaired reference images for training. Extensive experiments on public benchmarks demonstrate that RefTon achieves competitive or superior performance compared to state-of-the-art methods, while maintaining a simple and efficient person-to-person design.

1. Introduction

Existing virtual try-on methods often depend on auxiliary models and inputs, while garment-only images may not reveal worn appearance or fine details. RefTon combines a streamlined person-to-person design with clothed-person references and a reference-data pipeline.

1. Introduction

  • Prior virtual try-on methods rely on external models and auxiliary conditions, increasing framework complexity and making results sensitive to input quality.Examples include pose estimators, parsing and segmentation models, and cloth masks.Source: §1 · p. 2
  • Garment-only images may not reveal details such as translucent materials or lace collars, and public datasets lack corresponding clothed-person references.The paper notes that worn references can reveal details that are difficult to discern in isolated garment images.Source: §1 · p. 2
  • RefTon adapts Flux-Kontext for try-on with a two-stage training strategy, enabling inference from a source image and target garment without auxiliary structural inputs.It also uses images of other people wearing the target garment as visual guidance.Source: §1 · p. 2
  • The authors build the VFR dataset on existing benchmarks and report state-of-the-art garment-detail preservation and strong generalization to in-the-wild person–clothing scenarios.The framework supports varying numbers and types of references and both mask-based and person-to-person try-on in one model.Source: §1 · pp. 2–3

2. Related Works

Flow matching offers a simulation-free training objective for flow-based generation, while prior diffusion-based try-on methods use varied forms of guidance and garment alignment. RefTon is positioned as reference-guided generation supporting both mask-based and mask-free settings.

2.1. Generative Model via Flow Matching

  • Flow Matching learns a time-dependent vector field with a simulation-free objective that avoids numerical integration during training.The paper describes Flux-Kontext inputs as encoded latent sequences concatenated with Gaussian noise.Source: §2.1 · p. 3

2.2. Diffusion-based Virtual Try-on

  • Diffusion-based garment-transfer methods use text guidance or inpainting, semantic control, and two-stage garment alignment and fusion.The cited examples include DiffusionCLIP, DCI-VTON, and IDM-VTON.Source: §2.2 · p. 3
  • Prior person-to-person methods may require additional conditions or lack unified mask-based and mask-free settings; RefTon uses clothed references without a try-off stage.The paper contrasts this with try-off–then–try-on pipelines, which it says introduce error accumulation and garment-detail loss.Source: §2.2 · p. 3

3. Method

RefTon adapts flow-based image generation to person-to-person try-on, progressively transforming noise into a garment-conditioned target image. Its method combines two-stage training, multi-condition position indexing, and synthesized references showing the target garment on different people.

3.1. Preliminary

  • RefTon adapts Flux.1 to generate a target image by transforming Gaussian noise under conditions that include the person, target garment, and optional visual references.The method frames try-on as conditional generation of an image of a person wearing the target cloth.Source: §3.1 · pp. 3–4

3.2. Person To Person Virtual Try-on Model with Two Stage Training

  • Stage 1 synthesizes unpaired person images with a mask-based try-on model; Stage 2 trains on those images or agnostic images to fit a target garment directly.Stage 1 training pairs must use the same garment category to avoid unrealistic mask-region mismatches; the two input types are sampled equally in Stage 2.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Multi-input Training and Adaptation

  • RefTon replaces Flux-kontext’s binary position flag with condition labels and independently generated, resolution-rescaled spatial indices for multi-input RoPE.The indices distinguish person, garment, and reference inputs while preserving spatial alignment across different resolutions.Source: §3.3 · p. 5, Figure 4 · p. 5

3.4. Virtual-Tryon Generation with Extra Visual Reference

  • The reference pipeline synthesizes images of different people wearing the target garment, preserving its appearance while varying the wearer and non-target clothing.Reference construction uses target-image editing and prompt descriptions; the resulting references augment existing benchmarks in the VFR dataset for training and evaluation.Source: §3.4 · pp. 5–6, Figure 5 · p. 6

4. Experiments

RefTon is evaluated on public benchmarks in paired, unpaired, mask-based, and mask-free settings, with additional tests on mixed-dataset training and position-index ablations. Across these evaluations, reference inputs improve results, while the model also performs strongly without masks.

4. Experiments

  • Experiments use VITON-HD and DressCode, with additional StreetTryOn tests; single-dataset models are trained independently, while an additional mixed-VFR model is used to assess generalization.For quantitative evaluation, generated images are resized to 512 × 384; cross-dataset results are reported at 1024 × 768.Source: §4 · p. 7

4.1. Quantative result

  • On DressCode, RefTon outperforms all baselines; adding references further improves results and establishes a new state-of-the-art, while mask-free accuracy is comparable to or better than prior baselines.The comparison covers paired and unpaired settings; +R denotes reference-image input and MF denotes mask-free input.Source: §4.1 · p. 7, Table 1 · p. 8
  • On StreetTryOn, RefTon achieves state-of-the-art performance despite not being trained on that domain, and can extract garment information directly from a reference image.The evaluation omits cloth-only input and uses the person image as a reference in the person-to-person setting.Source: §4.1 · pp. 7–8, §4 · p. 7, Table 2 · p. 8

4.2. Qualitative comparison

  • Qualitatively, RefTon preserves lace transparency, perforations, patterns, and logos, and renders reflective fabrics more realistically than prior methods.Reference images further improve generation quality, while mask-free inputs retain garment length, structure, and design.Source: Figure 7 · p. 7, §4.2 · p. 8

4.3. Training on VRF Dataset and Evaluation

  • Training on 103,936 mixed-dataset image pairs yields performance comparable to or better than OOTDiffusion across VITON-HD and DressCode, including when RefTon was not trained on either dataset individually.The comparison covers paired and unpaired settings, with gains on most metrics, including FID and LPIPS.Source: §4.3 · pp. 8–9, Table 3 · p. 8

4.4. Ablation Study

  • Across mask-based and mask-free modes, reference images improve results, while switching to mask-free inputs causes only mild metric fluctuations.Qualitative examples show reference cues resolving ambiguous garment structures and materials; mask-free outputs reduce dependence on mask quality.Source: §4.4 · p. 9, Figure 8 · p. 9
  • At 0.25× and 0.5× condition scales, the rescaled position index outperforms the original Flux-Kontext index on FID and KID in both masked and mask-free settings.The evaluation uses VITON-HD with cloth, dense pose, warp mask, and reference-image conditions.Source: §4.4 · p. 9, Table 4 · p. 9

5. Conclusion

RefTon supports mask-based and mask-free inference and uses reference images to guide virtual try-on. The paper attributes its garment-detail fidelity and benchmark performance to this framework, its rescaled position index, and its reference-data generation pipeline.

5. Conclusion

  • RefTon supports mask-based and mask-free inference, uses reference images, and achieves state-of-the-art performance across settings while preserving translucent fabrics, intricate designs, and fine details.The framework adapts Flux-Kontext for multi-condition inputs at varying resolutions with a rescaled position index and trains using a reference-data pipeline integrating Qwen2.5-VL and Flux-Kontext.Source: §5 · p. 9

A. Generation of Reference Image

The reference-image pipeline edits a target-person image so another person wears the target garment, while preserving garment details and varying the person, other clothing, and action. It automates reference creation using Flux-Kontext and generated appearance descriptions.

A. Generation of Reference Image

  • Reference images depict another person wearing the target garment, generated by editing the target-person image while preserving the garment’s details.The editing prompt instructs Flux-Kontext to keep the target cloth unchanged.Source: A. Generation of Reference Image · p. 13
  • The reference person is made significantly different from the target person to reduce the shortcut of copying the reference and ignoring the other inputs.The pipeline uses opposite descriptions of the target person’s appearance as positive and negative prompts.Source: A. Generation of Reference Image · p. 13
  • A vision-language model automatically generates descriptions of the target person and their opposite, which are used to edit the target image into a reference.The paper names Qwen2.5-VL as an example model for automating this description step.Source: A. Generation of Reference Image · p. 13
  • Outfit and action descriptions vary non-target garments and actions across dress, upper-body, and lower-body scenarios, with accessories added for diversity.The editable garment description is chosen to differ from the target garment.Source: A. Generation of Reference Image · p. 13

B. Further Quantitative Evaluation on High Resolution

High-resolution evaluation reports strong performance across paired and unpaired settings on VITON-HD and DressCode. The appendix also reports DressCode subset results, with reference inputs improving results and mask-free variants preserving garment style and pose alignment.

B. Further Quantitative Evaluation on High Resolution

  • On DressCode’s upper-body, lower-body, and dress subsets, RefTon outperforms all baselines; reference inputs further improve results, and mask-free results retain garment styles and pose alignment.The reported mask-free accuracy is comparable to or better than prior baselines.Source: B. Further Quantitative Evaluation on… · p. 13, Table S5 · p. 15
  • At 1024 resolution, RefTon and RefTon+R perform strongly on nearly all metrics, while mask-free variants deliver particularly strong results across the reported settings.The comparison covers VITON-HD and DressCode under paired and unpaired protocols.Source: B. Further Quantitative Evaluation on… · pp. 13, 15, Table S6 · p. 15

C. Additional Qualitative Results

Qualitative comparisons present RefTon results across benchmark datasets, clothing types, input settings, and in-the-wild conditions. The reported strengths include detailed garment preservation, mask-free capability, and reference-guided refinement.

C. Additional Qualitative Results

  • RefTon is reported to deliver the best visual quality with the fewest required inputs, more faithful StreetTryOn details, and the strongest mask-free capability.Adding a reference further improves intricate details such as lace, transparency, and texture.Source: C. Additional Qualitative Results · p. 15, Figure S11 · p. 16
  • On VITON-HD, RefTon preserves dense textures, irregular motifs, and high-frequency clothing patterns with minimal distortion.The generated patterns remain clear, sharp, and recognizable.Source: C. Additional Qualitative Results · p. 15, Figure S12 · p. 17
  • On VITON-HD, RefTon maintains garment geometry, coherent contours, and plausible spatial arrangements for challenging structures and silhouettes.The examples include multi-layered regions, unique silhouettes, and uncommon shapes.Source: C. Additional Qualitative Results · pp. 15–16, Figure S13 · p. 18
  • Across DressCode upper-body, lower-body, and dress subsets, results retain fabric characteristics and realistic garment alignment in unpaired, mask-free settings.The examples span tops, pants, skirts, and full-body dresses, with stable structure and smooth boundaries.Source: C. Additional Qualitative Results · p. 16, Figure S14 · p. 19
  • In-the-wild results across diverse poses, lighting, and cluttered backgrounds preserve body pose, identity cues, and background consistency; references further improve garment geometry and texture.The mask-free pipeline transfers garments without human parsing masks or pose estimators.Source: C. Additional Qualitative Results · pp. 15–16

RefTON/MF+R

The figures show qualitative results for DressCode lower-body and dress subsets in the unpaired setting, with reference and mask-free input variants.

RefTON/MF+R

  • The lower-body DressCode examples show the unpaired setting, with reference images and mask-free inputs marked as variants.“MF” denotes use of the original person image instead of a masked agnostic image.Source: Figure S15 · p. 20
  • In both subsets, compare the results with a reference image against mask-free inputs using the original person image.The captions identify the input variants but do not state a comparative outcome.Source: Figure S15 · p. 20, Figure S16 · p. 21
  • The DressCode dress-subset examples likewise show the unpaired setting, with reference and mask-free input variants.“MF” denotes use of the original person image instead of a masked agnostic image.Source: Figure S16 · p. 21

Related papers

Loading 2511.00956v6…