PaperlayerView exact revision on arXiv

Source-linked AI summary

CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation

Seungyong Lee, Hyun Jun Jang, Sangoh Kim, Sungjoon Park

arXiv:2607.09362v1cs.CVcs.AI

TL;DR

Current virtual try-on systems offer limited control over garment size, style, and placement. CtrlVTON addresses this gap with instance segmentation and mask-controlled image editing, achieving state-of-the-art task performance and substantially better spatial adherence than proprietary editors while remaining competitive on fidelity.

  • Problem

    Current virtual try-on methods provide limited control over garment size, style, and spatial placement.

  • Method

    The paper introduces VIP-SAM for visual-instance-prompt segmentation and CtrlVTON, an image-editing framework using segmentation masks for pixel-level garment control.

  • Results

    VIP-SAM achieves state-of-the-art performance across its evaluations, while CtrlVTON provides substantially better spatial control than proprietary baselines and remains competitive on fidelity metrics.

  • Takeaways & Limitations

    The framework provides a unified approach to garment swapping, layering, selective switching, and multi-garment composition with pixel-precise spatial control.

Abstract

from arXiv · show

Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. Yet most systems give the user little control over how a garment should be worn -- its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions. First, we define and solve Visual-Instance-Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level segmentation. Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds segmentation masks as pixel-level control over garment layout, including style, size, and spatial placement on the body. VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity.

1 Introduction

CtrlVTON addresses limited user control in virtual try-on by recasting it as image editing and using segmentation masks to control garment style, size, and placement. The paper also introduces VIP-Seg and VIP-SAM to segment a specific reference garment instance on a person.

  • Motivation: Current VTO methods provide limited control over garment size and style despite improved photorealism and garment fidelity.Examples include loose versus fitted sizing and tucked versus untucked or zipped versus unzipped styles.
  • CtrlVTON: CtrlVTON recasts VTO as image editing and uses segmentation masks for pixel-level control over garment style, size, and placement.The framework handles diverse garment categories and display formats while unifying garment swap, layering, selective switching, and multi-garment try-on.
  • VIP-Seg and VIP-SAM: VIP-Seg formalizes locating the same garment instance from a support image in a query image of a person wearing it.This visual-instance-prompt task is distinct from category-level visual-reference-prompt segmentation.
  • VIP-Seg and VIP-SAM: VIP-SAM achieves state-of-the-art results on a fashion-domain dataset and standard benchmarks repurposed for the application.The segmentation pipeline provides the automatic, scalable garment masks required by the models.

2 Related Work

Prior virtual try-on methods progressed from geometric warping and GAN rendering to diffusion and editing-based approaches, but editing-based systems sacrificed precise spatial control. CtrlVTON addresses this gap with segmentation-mask control, while VIP-SAM targets instance-level garment discrimination beyond category-level segmentation.

  • Virtual Try-On: Diffusion-based virtual try-on improved realism and detail fidelity over early geometric-warping and GAN-based methods, which struggled with complex poses, fine textures, and occlusion.Early methods geometrically warped garment features before GAN rendering; later diffusion methods used latent diffusion and parallel UNets.
  • Virtual Try-On: Inpainting-based VTO lets users specify edited regions, but its masks create difficulties with complex poses, occlusions, and identity drift.Editing-based methods subsequently removed reliance on inpainting masks.
  • Controllable Editing: Editing-based methods remove inpainting limitations but lose spatial control, leaving users unable to specify how a garment is worn.M&M VTO and PromptDresser offer coarse text-guided layouts for multi-garment outfits but lack precise placement and layered-garment support.
  • Controllable Editing: CtrlVTON closes this gap by incorporating segmentation masks into an editing framework to recover precise garment-layout control.The related-work passage presents segmentation masks as the mechanism for restoring spatial control within editing-based VTO.
  • Visual-Instance-Prompt Segmentation: SAM and SAM2 accept spatial prompts but cannot specify a target instance through a separate reference image, motivating visual reference prompting.SAM produces class-agnostic masks from points, boxes, or coarse masks, while SAM2 propagates identities across video frames through memory.
  • Visual-Instance-Prompt Segmentation: Existing visual-reference methods generally evaluate category-level correspondence, whereas VIP-SAM addresses instance-level discrimination among same-category garments.The paper’s setting requires singling out a specific garment, such as one of several similarly colored layered shirts.

3 Visual-Instance-Prompt Segmentation (VIP-Seg)

VIP-Seg is defined as an instance-level segmentation task that uses a support image to identify the exact same object instance in a query image. It differs from category-level segmentation by resisting same-class distractors.

  • Task definition: VIP-Seg segments a specific object instance in a query image using a support image of that object.The support may be a studio flatlay of a particular garment, while the query may show a person wearing it.
  • Task definition: The task targets the exact same instance, rather than any object belonging to the same class.This makes VIP-Seg distinct from VRP-Seg as described in the passage.
  • Task definition: VIP-Seg must distinguish the target instance even when same-class distractors appear in the query image.An example is a shirt worn over another shirt of similar color.

4 Controllable Virtual Try-On

CtrlVTON reframes virtual try-on as full-image editing, preserving non-target content while using semantic tokens and pixel-level masks to control garment replacement and placement. It is trained on synthesized person–garment triplets and extended to multiple garments without architectural changes.

  • Editing formulation: CtrlVTON conditions on the entire input image, enabling pose, identity, other garments, accessories, and background to flow through while modifying only the targeted region.This avoids the mask-tightness trade-off of inpainting, where undersized masks leak garment pixels and oversized masks erase identity cues.
  • Training data: Training uses synthesized triplets (p, pref, gref) and corresponding garment masks, with source pairs assembled from public, commercial, and in-house datasets spanning diverse garments and image formats.VITON-HD-edit contains 2,032 images and supports image-editing VTO, VIP-Seg, and spatially-controllable VTO.
  • Model design: CtrlVTON-base uses garment-class and task tokens for semantic control, while CtrlVTON adds pixel-level spatial control through Mp, Mpref, and Mgref masks with a LoRA adapter.The task token supports full_swap, partial_swap, and add operations; user-provided masks can adjust garment style, fit, or placement at inference.
  • Mask injection: Mask latents are channel-wise concatenated with image latents, doubling channel dimension without changing the H × W token grid, attention cost, or conditioning-token count.The base flow-matching DiT is frozen while LoRA parameters are trained on its linear projection blocks.
  • Multiple garments: The model extends to multiple garments by ingesting multiple references and color-coded RGB masks, preserving per-garment controllability without architectural changes.Training combines single- and multi-garment data in one stage, and all Sec. 5 experiments use the multi-garment model.

5 Experiments

Experiments show that VIP-SAM achieves state-of-the-art visual-instance segmentation, while CtrlVTON performs strongly across single- and multi-garment try-on and enables precise mask- and token-based control. CtrlVTON also provides substantially better spatial control than proprietary editing models on mask-controllable try-on.

  • VIP-SAM evaluation: VIP-SAM isolates the queried garment instance more reliably than VRP-SAM-style methods, including for layered and similarly textured garments.Existing methods may include surrounding garments or the entire body, whereas VIP-SAM consistently isolates only the queried instance.
  • VIP-SAM evaluation: VIP-SAM achieves state-of-the-art performance across the fashion, COCO-20i, and PASCAL-5i evaluations.The category-level benchmarks are modified to train and evaluate on the same category, targeting instance-level segmentation.
  • CtrlVTON-base evaluation: CtrlVTON-base outperforms inpainting-based baselines and is competitive with or surpasses recent editing-based models on most single-garment VTO metrics.Evaluation uses VITON-HD and a clothes-only subset of OmniTry Bench containing 2,250 samples.
  • CtrlVTON-base evaluation: CtrlVTON-base surpasses FastFit, OmniTry, and BootComp in multi-garment try-on on DressCode-MR and Garments2Look.These baselines represent multi-garment architectures, sequential single-garment inference, and person regeneration from references, respectively.
  • Task-token control: Changing only the task token unifies full_swap, partial_swap, and add, producing full replacement, selective replacement, and layering behaviors.The task-token interface preserves existing garments for selective swap and layering as specified by each operation.
  • Mask-controllable try-on: CtrlVTON achieves substantially better spatial control than four proprietary editing models on VITON-HD-edit using user-provided garment masks.The evaluation compares mask adherence using IoU, dHu, and dH, with baselines receiving the person, reference garment, and control mask as reference images.

6 Conclusion

The paper presents CtrlVTON as a controllable virtual try-on framework that makes controllability a first-class objective through garment re-identification and pixel-precise mask conditioning. By varying only the input mask, it enables fine-grained control over garment styling, layering, sizing, and outfit composition.

  • Contributions: CtrlVTON treats controllability as a first-class objective in virtual try-on by recasting try-on as an editing problem over (pref, gref, p).This formulation sidesteps structural limitations of inpainting.
  • Contributions: VIP-SAM re-identifies a specific garment across flatlay, on-person, and in-the-wild images, producing masks for mask-conditioned virtual try-on.The masks support garment-specific control rather than only category-level segmentation.
  • Contributions: A lightweight mask-conditioning LoRA adds pixel-precise spatial control to CtrlVTON.Mask conditioning provides the mechanism for controlling garment layout.
  • Fine-grained control: Varying only the input mask generates diverse outputs for single-garment styling, layered outer-over-inner styling, and full-outfit composition.Demonstrated controls include tucking, buttoning, sleeve length, zipping, cuffing, and sizing.

Supplementary Material … C Data Pipeline Details

The supplementary material compares VIP-SAM and VRP-SAM resource requirements and illustrates how mask design and surrounding context constrain inpainting-based virtual try-on outputs.

  • A VIP-SAM: Resource Comparison: VIP-SAM (ViT-B/ResNet-50) requires substantially more training resources than VRP-SAM (ViT-B/ResNet-50).The difference arises because VIP-SAM computes gradients through layers inside the ViT backbone.
  • A VIP-SAM: Resource Comparison: Table 5 compares peak training memory, peak inference memory, FLOPs, total parameters, and trainable parameters.The comparison covers visual-reference segmentation methods.
  • A VIP-SAM: Resource Comparison: PerSAM and Matcher are training-free, so some resource metrics are not applicable to them.Table 5 denotes non-applicable metrics with “–”.
  • B Inpainting Failure Modes: Visual Examples: The supplementary material uses Fig. 11 to justify the claim that inpainting-based VTO is fundamentally limited by the mask itself.The figure presents two examples with four annotated failure regions.
  • B Inpainting Failure Modes: Visual Examples: The first three failure regions show that generated outputs are highly sensitive to mask size and shape.These examples directly connect mask geometry to output variation.
  • B Inpainting Failure Modes: Visual Examples: The fourth failure region shows that generated outputs are also entangled with context outside the mask.Thus, the output depends on both the mask and surrounding image context.

C.1 Data Sources and Proportions · C.2 Masking Strategies for Synthetic pref · C.3 Quality-Control Protocol

CtrlVTON’s training corpus combines public, commercial, and in-house person–garment pairs, while synthetic references are produced with complementary masking strategies and selected through VLM, automatic, and human quality control. The protocol balances garment-shape variation with context preservation and rejects candidates whose edited silhouettes leak the original garment.

  • C.1 Data Sources and Proportions: The training corpus draws person–garment pairs from public VTO datasets, licensed commercial fashion-retailer datasets, and in-house data.Table 6 reports source-level proportions and garment-category coverage.
  • C.1 Data Sources and Proportions: The corpus is analyzed by garment cardinality, garment category, and task-token assignment to support capabilities evaluated in the main paper.Table 7 reports these distributions, including mixed task-token labels for some multi-garment samples.
  • C.2 Masking Strategies for Synthetic pref: Synthetic references are generated with inpainting and editing models using training-free inpainting methods, collectively termed an inpainting system.These systems are used to obtain synthetic training images.
  • C.2 Masking Strategies for Synthetic pref: Three masking strategies trade off context preservation, garment-shape variation, and localization, ordered from the frequently used default to the surgical exception.The fixed strategy is insufficient because different images require different trade-offs.
  • C.2 Masking Strategies for Synthetic pref: The default box mask excludes face, hair, hands, and held items while enabling the highest garment-shape diversity through a silhouette-independent bounding box.The box begins from a garment-agnostic body region produced by human segmentation models.
  • C.2 Masking Strategies for Synthetic pref: Loose masks preserve segmentation contours for unsafe free-form edits, while tight masks are used for more surgical synthesis; at least one candidate is generated per strategy.Loose masks help preserve complex backgrounds and held accessories, and the best candidate is selected using VLM and human screening.
  • C.3 Quality-Control Protocol: For each source pair, four to five candidates pass through VLM screening, automatic silhouette-leakage filtering, and final human review.The VLM compares each candidate with the source person using four yes/no questions, including identity and pose preservation.
  • C.3 Quality-Control Protocol: Candidates with contour match fraction above threshold are discarded because CMF detects silhouette leakage more effectively than IoU in partial-overlap cases, after which three annotators select the most plausible survivor.Human reviewers check identity drift, shadow inconsistency, and implausible garment proportions; the top candidate is retained.

D Training Details … F.2 VLM-as-Judge: GTC / PBC / PR

The paper details training protocols, mask-injection design choices, and evaluation procedures for garment fidelity and VLM-judged try-on quality. VIP-SAM training uses task-specific dataset settings, while evaluation measures garment similarity, person/background preservation, and physical realism.

  • D Training Details: VIP-SAM training splits the fashion dataset into 85:7.5:7.5 train, validation, and test sets, using batch size 16 for 100 epochs.The configuration largely follows VRP-SAM, with the batch size limited by an L40 GPU for the ViT-B / ResNet-50 setup.
  • D Training Details: COCO-20i and PASCAL-5i use VRP-SAM’s class splits, average results across splits, and evaluate on training classes to target same-instance finding.VIP-SAM, VRP-SAM, and ProSAM are trained for 50 epochs in this setting.
  • D Training Details: ProSAM outperforms VRP-SAM on PASCAL-5i but not on COCO-20i, where masks contain more connected components on average: 1.91 versus 1.77.The connected-component count is presented as a proxy for multiple same-category instances of varying size and shape.
  • E Mask Injection Strategies: Channel-Wise vs. Token-Wise: The mask-injection ablation compares channel-wise and token-wise concatenation from the same base checkpoint after the same number of training steps.Token-wise concatenation treats mask tokens as additional reference tokens, whereas channel-wise concatenation preserves spatial alignment with corresponding image tokens.
  • F.1 Garment-fidelity Metrics (M-DINO, M-CLIP-I): Garment fidelity is measured by cropping the generated garment with its VIP-SAM mask, normalizing it on white, and comparing it with the reference in DINO and CLIP-image embedding spaces.Both metrics report cosine similarity in [−1, 1], with higher values better; M-DINO emphasizes fine-grained structure, while M-CLIP-I reflects category-level coherence.
  • F.1 Garment-fidelity Metrics (M-DINO, M-CLIP-I): M-DINO generally scores lower than M-CLIP-I because it penalizes geometric variation, so the metrics should be interpreted together for structural and category fidelity.A method strong on both metrics is faithful at both levels, whereas a method strong only on M-CLIP-I may preserve category but not precise local structure.
  • F.2 VLM-as-Judge: GTC / PBC / PR: Gemini 3.0 Flash judges try-on outputs on GTC, PBC, and PR, each scored as a strict float from 0 to 5.GTC assesses garment details and global properties; PBC assesses identity, body, belongings, and background; PR assesses drape, contact, occlusion, layering, lighting, and shadows.

Reproducibility and the Choice of a Proprietary VLM Judge

Using Gemini 3.0 Flash as a VLM judge limits reproducibility because proprietary models may change or disappear without notice and cannot be independently audited. An open-weight Qwen3-VL-8B-Instruct comparison shows heavily overlapping confidence intervals despite quality differences revealed by proprietary-judge scores.

  • Proprietary VLM limitations: Gemini 3.0 Flash creates a reproducibility limitation because proprietary VLMs may change or be deprecated without notice and cannot be independently audited.This limitation arises from dependence on the provider’s underlying model and inaccessible internals.
  • Proprietary VLM limitations: Proprietary VLM judges are also used by concurrent VTO evaluation work: OpenVTON-Bench uses Gemini 2.0 Flash, while CORAL uses GPT-5.The passage identifies reliance on proprietary judges as a limitation shared beyond this work.
  • Open-weight judge comparison: Qwen3-VL-8B-Instruct scores on VITON-HD-edit used the same GTC/PBC/PR prompt and mean ± 95% confidence intervals over 5 independent runs.The table compares these open-weight results with Gemini 3.0 Flash scores for the same methods in Table 4.
  • Open-weight judge comparison: The open-weight and proprietary-VLM confidence intervals overlap heavily despite real quality differences revealed by the proprietary-judge scores.This comparison cautions that overlapping intervals do not necessarily reflect the quality differences reported in Table 4.

F.3 Task-Following Consistency (TFC) for Token Control · F.4 Mask-Adherence Metrics (IoU, dHu, dH)

The paper evaluates task-token control with TFC, a strict [0, 5] criterion applied separately to full_swap, partial_swap, and add operations. It evaluates mask adherence by comparing re-extracted garment masks with control masks using IoU, Hu-moment distance, and symmetric Hausdorff distance to assess region, global shape, and boundary fidelity.

  • F.3 Task-Following Consistency (TFC) for Token Control: TFC measures how faithfully each task token’s specified operation is realized, using a strict score from 0 to 5.The grader receives the input person, reference garment, generated output, and intended task token.
  • F.3 Task-Following Consistency (TFC) for Token Control: full_swap replaces every matching-class garment with the reference and leaves no original garment of that class.The operation is evaluated separately for each task token.
  • F.3 Task-Following Consistency (TFC) for Token Control: partial_swap replaces exactly one matching-class garment while preserving all other garments, including same-class items.This token tests targeted replacement rather than complete class-level replacement.
  • F.3 Task-Following Consistency (TFC) for Token Control: add preserves the original outfit and layers the reference garment on top as an additional garment.CtrlVTON-base is reported in isolation because no open-weight VTO system exposes comparable operation-level discrete-token control.
  • F.4 Mask-Adherence Metrics (IoU, dHu, dH): For each generated try-on image, VIP-SAM re-extracts the garment mask, which is compared against the input control mask under three complementary criteria.These criteria assess region overlap, global shape, and boundary deviation.
  • F.4 Mask-Adherence Metrics (IoU, dHu, dH): IoU measures control-mask and generated-region overlap, with higher values indicating better region-level agreement.IoU can miss localized boundary failures and shape distortions that preserve overall area.
  • F.4 Mask-Adherence Metrics (IoU, dHu, dH): Hu-moment distance dHu measures global shape agreement using the Euclidean distance between seven translation-, scale-, and rotation-invariant Hu vectors.The invariants are computed from area-normalized central moments.
  • F.4 Mask-Adherence Metrics (IoU, dHu, dH): Symmetric Hausdorff distance dH captures worst-case contour deviation in pixels, complementing IoU and dHu for localized boundary fidelity.Lower is better, and the combined metrics provide a balanced view of region, global shape, and boundary fidelity.

F.5 Supplementary evaluation of mask-adherence metrics using SAM3

The supplementary evaluation tests whether VIP-SAM-based metrics are self-referential by recomputing mask extraction and scores with independent, text-prompted SAM3. Results show only slight shifts across all five methods, while qualitative examples examine CtrlVTON under disagreements between the segmentation models.

  • Motivation: VIP-SAM-based evaluation could be self-referential because VIP-SAM extracts garment masks during both training-data construction and evaluation.This raises the possibility that strong scores reflect overfitting to VIP-SAM’s garment-mask definition.
  • Evaluation setup: The evaluation re-runs inference-time mask extraction and metric computation on VITON-HD-edit with independent, text-prompted SAM3.Neither the segmenter nor the prompting modality overlaps with those used during training.
  • Results: All five methods show only slight score shifts relative to the VIP-SAM-based results in Tab. 4.Table 11 reports the SAM3 re-evaluation for all five methods on VITON-HD-edit.
  • Qualitative analysis: Qualitative examples test CtrlVTON when VIP-SAM and SAM3 disagree on garment masks while the garment and person references remain fixed.The examples include disagreements over garment hemline length.

G Additional Qualitative Results

Additional qualitative results show CtrlVTON across diverse single- and multi-garment settings, improved try-on quality from curated data, and responsiveness to user-provided masks. They also visualize mask-following behavior and identify failures arising from semantic-only guidance or poorly drawn masks.

  • Single-garment results: Additional single-garment comparisons span diverse garment types, person images, and reference representations.These results complement Fig. 7.
  • Multi-garment results: CtrlVTON-base composes multiple reference garments in one forward pass, unlike FitDit and OmniTry, which apply garments sequentially.FastFit and BootComp natively accept multiple garments.
  • Data curation: Curated-data training reduces silhouette leakage and produces more physically plausible garment fit and body contact than the baseline.The baseline’s garment outline can follow the reference person’s worn garment rather than the reference garment.
  • Mask control: Varying only the input control mask generates multiple outputs for the same person–garment pair, demonstrating mask-controllable try-on.The results complement Fig. 10 of the main paper.
  • Mask-following metrics: Mask overlays show that large discrepancies produce low IoU and high dH, while subtle global-shape mismatches appear in dHu.The visualizations confirm that CtrlVTON follows requested layouts more closely than proprietary baselines.
  • Failure cases: CtrlVTON-base can produce stylistically or physically implausible garments, while CtrlVTON’s responsiveness to control masks makes poorly drawn masks a main weakness.Examples include an untucked one-piece bodysuit and fabric appearing torn at the torso.
Loading 2607.09362v1…