CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation

1 Introduction

• CtrlVTON recasts try-on as image editing and uses segmentation masks for fine-grained control over garment style, size, and placement while matching the strongest VTO systems on image quality.The image-editing formulation is intended to avoid failure modes of inpainting-based VTO.Source: §1 · p. 2
• CtrlVTON handles tops, bottoms, dresses, shoes, and bags across flatlay, on-person, and in-the-wild formats, including layering, selective garment switching, and multi-garment try-on.The framework also unifies standard garment swap.Source: §1 · p. 2
• VIP-Seg asks a model to locate the same garment instance from a support image in a query image, unlike category-level VRP-Seg studied in VRP-SAM and related work.The task involves same-category distractors, heavy occlusion, and non-rigid deformation between flatlay and on-body imagery.Source: §1 · p. 2
• VIP-SAM achieves state-of-the-art performance on a purpose-built fashion benchmark and standard category-level benchmarks repurposed for instance-level segmentation.The model is designed for the VIP-Seg task.Source: §1 · p. 2
• A data pipeline synthesizes person, garment, and changed-garment triplets with masks, and yields VITON-HD-edit, a benchmark for editing VTO, mask-controllable VTO, and instance-level visual-prompt segmentation.The pipeline provides an automatic, scalable way to segment each reference garment within a person image.Source: §1 · p. 2

2.1 Virtual Try-On

• Diffusion improved VTO realism, but inpainting masks remain difficult for complex poses, occlusions, and identity preservation; editing-based methods remove mask reliance but lose spatial control.Earlier approaches warped garment features and rendered with GANs, while later work adopted diffusion and DiT backbones [13] [28]. CtrlVTON uses segmentation masks to restore pixel-level control within an editing framework.Source: §2.1 · pp. 2–3

2.2 Controllable Image Editing

• Existing editing interfaces offer structural, coarse-layout, mask-inpainting, or reference-guided control, but none jointly provides reference fidelity, user-specified layout, and input-person identity preservation.This combination is required for VTO.Source: §2.2 · p. 3

2.3 Visual-Reference-Prompt Segmentation

• Visual-reference-prompt segmentation methods target objects from support images, but—with the exception of SAM3—rely on feature correspondence and category-level evaluation rather than distinguishing a specific garment instance.SAM and SAM2 use spatial prompts without a separate reference image; VIP-SAM addresses instance-level discrimination, such as selecting one garment among similar layered items.Source: §2.3 · pp. 3–4

3 Visual-Instance-Prompt Segmentation (VIP-Seg)

• VIP-Seg segments the exact instance shown in a support image, not merely any object of the same class, despite same-category distractors, occlusion, and non-rigid deformation.The task targets matching garments across studio flatlay and on-person e-commerce images.Source: §3 · p. 4
• Prior VRP-SAM-style methods struggle with layered garments and similar colors because frozen query encoding and support-query matching must both disambiguate corresponding objects.The paper reports these failures empirically and points to Fig. 5.Source: §3 · p. 4
• VIP-SAM injects support-image features into SAM or SAM2 at intermediate encoder stages through cross-attention adapters, conditioning query features from earlier than the prompt level.It supports SAM’s ViT or SAM2’s Hiera backbone and interchangeable support encoders including ResNet-50, DINOv2, and DINOv3 [53].Source: §3 · p. 4
• The VIP-Seg fashion dataset pairs garments with person images and annotated on-person garment regions, comprising roughly 10K human-labeled masks.Each example is a (garment, person, mask) triplet spanning diverse garment categories.Source: §3 · p. 4
• VIP-SAM supplies the accurate garment-instance masks required to train CtrlVTON’s spatial control, addressing existing methods’ inability to reliably isolate the referenced garment on a person.These masks are the prerequisite that enables the controllable VTO framework in Sec. 4.Source: §3 · p. 4

4.1 From Inpainting to Editing

• CtrlVTON avoids the mask-tightness trade-off of inpainting by editing from a same-person, same-pose, same-background reference image and learning to preserve those attributes while changing the garment.Inpainting can leak garment pixels with undersized masks or erase identity cues with oversized masks, and can bias outputs toward mask shape and outside-mask context.Source: §4.1 · p. 5

4.2 Data Curation

• Training triplets pair target images and garments with synthetic reference-person images, while VIP-SAM and dichotomous segmentation supply garment masks for controllable editing.Synthetic references are generated by inpainting a plausible unpaired garment into the target person image while preserving pose, identity, and background.Source: §4.2 · p. 5
• Applying the preparation pipeline to all 2,032 VITON-HD test images produces VITON-HD-edit, a public benchmark for image-editing VTO, VIP-Seg, and spatially controllable VTO.Source: §4.2 · p. 6

4.3 Model

• CtrlVTON-base uses garment-class and task tokens to specify garment type and whether to replace all matching garments, replace one, or add the reference garment over existing clothing.The garment classes are upper, lower, full, shoes, and bag; tasks are full_swap, partial_swap, and add.Source: §4.3 · p. 6
• CtrlVTON adds pixel-level control by training a LoRA adapter on the frozen base model with masks for the target person, reference person, and reference garment.At inference, users can edit a VIP-SAM mask for garment style, fit, or placement before running CtrlVTON; reference-person masks can also be all-white, all-black, SAM-derived, or handcrafted.Source: §4.3 · pp. 6–7
• The model handles multiple reference garments without architectural changes, using distinct shared RGB mask colors to preserve per-garment control; experiments use the combined single- and multi-garment model.Source: §4.3 · p. 7

5.1 VIP-SAM

• VIP-SAM achieves state-of-the-art performance across its fashion segmentation benchmark and category-level evaluations run with same-category training and evaluation.Qualitative examples show VIP-SAM isolating queried instances in layered or similarly textured garments where late feature-matching methods fail.Source: §5.1 · pp. 7–8, Table 1 · p. 7, Figure 5 · p. 8

5.2 CtrlVTON Experimental Setup

• Evaluations cover single-garment, multi-garment, and mask-control settings, with comparisons against open-weight VTO models and proprietary editors for mask control.The benchmarks include VITON-HD, OmniTry Bench, DressCode-MR, Garments2Look, and VITON-HD-edit; multi-garment mask control is qualitative because no public benchmark is available.Source: §5.2 · p. 8
• The evaluation combines garment-crop embedding similarity, VLM judgments of garment transfer, person-background consistency and realism, and mask-adherence measures.Mask adherence compares VIP-SAM-extracted output masks with input masks using IoU, Hu moment distance dHu, and symmetric Hausdorff distance dH; VTO evaluation follows an unpaired protocol.Source: §5.2 · pp. 8–9, Figure 6 · p. 9

5.3 Single- and Multi-Garment Virtual Try-On

• On single-garment VTO, CtrlVTON-base outperforms inpainting-based baselines and is competitive with or surpasses editing-based models on most metrics.The evaluation uses VITON-HD and the clothes-only OmniTry Bench subset.Source: §5.3 · p. 9, Table 2 · p. 9
• On multi-garment VTO, CtrlVTON-base surpasses FastFit, OmniTry, and BootComp across the reported evaluation.These baselines use multi-garment architectures, sequential single-garment inference, and person regeneration from references, respectively.Source: §5.3 · p. 10, Table 3 · p. 10
• Changing only the task token lets CtrlVTON-base perform full swap, selective swap, or layering for the same person-garment pair.The standard VTO benchmarks cover only full_swap; the paper also reports additional token-following evaluations in the supplement.Source: §5.3 · pp. 10–11, Figure 8 · p. 11

5.4 Mask-Controllable Try-On

• CtrlVTON achieves substantially better mask adherence than four proprietary editors while remaining competitive on garment-fidelity metrics.On VITON-HD-edit, spatial control is evaluated with IoU, dHu, and dH; the proprietary models often fail to align garments with the provided mask.Source: §5.4 · p. 11, Table 4 · p. 11
• Mask controls support adjustments to fit and placement, layering, selective swaps, and per-garment spatial control in multi-garment settings.Color-coded masks let users specify styling details such as tuck depth, zipper endpoint, sleeve roll, and oversized fit.Source: §5.4 · p. 11

• VIP-SAM re-identifies a specific garment across flatlay, on-person, and in-the-wild images, producing masks for mask-conditioned virtual try-on.This addresses instance-level garment identification across different image contexts.Source: §6 · p. 12
• CtrlVTON treats try-on as editing over (pref, gref, p) triplets and adds pixel-precise spatial control through lightweight mask-conditioning LoRA.This formulation sidesteps structural limitations of inpainting-based try-on.Source: §6 · p. 12
• The resulting unified framework supports garment swapping, layering, and multi-garment composition, while VITON-HD-edit provides a public testbed for editing-based and spatially controllable try-on.Source: §6 · p. 12

A VIP-SAM: Resource Comparison

• VIP-SAM, especially its Hiera variants, achieves higher accuracy with a smaller resource footprint than SAM-ViT-H-based baselines.The comparison covers memory, FLOPs, and parameters; the largest Hiera-L + DINOv3-H variant has inference memory comparable to PerSAM/Matcher while reaching state-of-the-art accuracy.Source: A VIP-SAM: Resource Comparison · p. 17, Table 5 · p. 17
• VIP-SAM (ViT-B/ResNet-50) requires substantially more training resources than VRP-SAM (ViT-B/ResNet-50), but their inference resources are comparable.The passage attributes the training difference to gradient computation inside the ViT backbone.Source: A VIP-SAM: Resource Comparison · p. 17, Table 5 · p. 17
• VIP-SAM’s smaller variants use less inference memory and fewer FLOPs than VRP-SAM/ProSAM despite training more parameters.The passage attributes this to the lighter Hiera backbones.Source: A VIP-SAM: Resource Comparison · p. 17, Table 5 · p. 17
• The resource comparison reports peak training and inference memory, FLOPs, total parameters, and trainable parameters; PerSAM and Matcher are training-free.The table marks metrics that do not apply with a dash.Source: Table 5 · p. 17, A VIP-SAM: Resource Comparison · p. 17

B Inpainting Failure Modes: Visual Examples

• Inpainting-based VTO outputs are highly sensitive to the size and shape of the mask.The paper illustrates this limitation with two examples and four annotated failure regions.Source: B Inpainting Failure Modes: Visual… · p. 17
• Inpainting-based VTO outputs are also entangled with context outside the mask.This is identified as the fourth annotated failure region in the examples.Source: B Inpainting Failure Modes: Visual… · p. 17

C.1 Data Sources and Proportions

• CtrlVTON’s training pairs come from public VTO datasets, retailer-licensed commercial datasets, and in-house datasets; Table 6 reports coarse source proportions and garment-category coverage.Source: §C.1 · p. 17, Table 6 · p. 18
• The corpus is broken down by single- versus multi-garment samples, garment category, and task-token assignment, with mixed marking samples whose garments have different tokens.The per-sample mixed label has no per-garment analogue, since each garment has exactly one task token. Table 7 reports these statistics.Source: §C.1 · p. 17, Table 7 · p. 19

C.2 Masking Strategies for Synthetic pref

• Synthetic images are generated with inpainting or editing models using training-free inpainting methods, and the mask controls how much of the original image can change.A single fixed mask is insufficient because images require different trade-offs among context preservation, garment-shape variation, and localization.Source: §C.2 · p. 18
• The default box mask excludes the face, hair, hands, and held items while freeing the inpainting system from the original garment silhouette, enabling the greatest garment-shape diversity.It is derived from an axis-aligned bounding box around the garment-agnostic body region.Source: §C.2 · p. 18
• Loose masks preserve the garment-agnostic segmentation contour for complex backgrounds or accessories, while tight, dilated VIP-SAM instance masks localize edits such as partial_swap.The loose mask confines edits to the body silhouette; the tight mask leaves the rest of the body untouched.Source: §C.2 · p. 19

C.3 Quality-Control Protocol

• For each source pair, 4–5 synthetic candidates pass through a VLM screen requiring preserved identity, pose, and background, and a garment-confined edit before further review.At least one candidate is generated per masking strategy, and the VLM and human screening select the best pref for each image.Source: §C.3 · p. 19, §C.2 · p. 19
• The silhouette-leakage filter discards candidates whose VIP-SAM masks have a contour match fraction above a threshold, using the maximum of directional contour-match fractions.CMF detects bulk contour overlap within a pixel tolerance; unlike IoU, it can catch leakage when contours substantially overlap despite only partial silhouette overlap, and it is robust to one mask being a subset of the other.Source: §C.3 · pp. 19–20, Figure 12 · p. 20
• Three annotators review candidates that survive both filters, catch remaining failures, and select the most physically plausible candidate for each source pair.They may identify subtle identity drift, shadow inconsistency, or implausible garment proportions; the selected pref is retained and the others are discarded.Source: §C.3 · p. 20

D Training Details

• VIP-SAM uses an 85:7.5:7.5 train/validation/test split, batch size 16, and 100 epochs; few-shot models train for 50 epochs and evaluate on training classes.CtrlVTON-base fine-tunes FLUX.2 Klein [3] at approximately 1MP resolution with global batch size 128 and approximately 20 H200-GPU-days of compute.Source: D Training Details · pp. 20–21
• ProSAM outperforms VRP-SAM on PASCAL-5i but not COCO-20i; COCO masks average 1.91 components versus 1.77, potentially making regularized embeddings less suited to same-instance identification.The passage notes that connected components imperfectly proxy object counts and that ProSAM’s embedding regularization can average over visually distinct same-category instances.Source: D Training Details · pp. 20–21

E Mask Injection Strategies: Channel-Wise vs. Token-Wise

• Channel-wise mask concatenation preserves each mask’s pixel alignment with its image, while token-wise concatenation requires attention to recover correspondence and increases quadratic self-attention cost.The two strategies are compared under the same training budget on VITON-HD-edit using parameter count, throughput, peak inference VRAM, mask adherence, and perceptual metrics; supplied passages give no metric values.Source: E Mask Injection Strategies:… · p. 21, Table 8 · p. 21

F Evaluation Metrics: Details

• The evaluation appendix adds garment-fidelity, VLM-as-judge, and mask-adherence details; it compares identically processed results and reference garments in two embedding spaces, reporting cosine similarity from −1 to 1, with higher values better.Source: F Evaluation Metrics: Details · p. 21, §F.1 · p. 22

F.1 Garment-fidelity Metrics (M-DINO, M-CLIP-I)

• M-DINO captures fine-grained garment structure, whereas M-CLIP-I primarily reflects category-level coherence; the metrics are most informative when read together.The passage explains that DINO penalizes geometric variation more than CLIP, so a high M-CLIP-I with low M-DINO can indicate category preservation without precise local structure.Source: §F.1 · p. 22

F.2 VLM-as-Judge: GTC / PBC / PR

• Gemini 3.0 Flash scores garment transfer consistency, person-background consistency, and physical realism on a strict 0–5 scale, with 5 meaning no perceivable issues.The criteria cover garment details and global properties, preservation of the person and surroundings, and plausible drape, contact, occlusion, and lighting.Source: §F.2 · p. 22
• The judge prompt also checks multi-garment completeness and accuracy, preservation of non-target items and background, and plausible layering without fabric floating, clipping, or unrealistic merging.It additionally assesses lighting and shadow consistency on the generated garments.Source: §F.2 · p. 23

Reproducibility and the Choice of a Proprietary VLM Judge

• Open-weight Qwen3-VL-8B-Instruct [1] could not reliably rank five VITON-HD-edit methods: its GTC and PR confidence intervals overlapped substantially, while PBC gaps were only a few hundredths.Table 9 reports means and 95% confidence intervals over five independent runs, using the same GTC/PBC/PR system prompt as Appendix F.2.Source: Reproducibility and the Choice of a… · p. 23, Table 9 · p. 23
• Gemini 3.0 Flash poses a reproducibility limitation because its provider may change or deprecate it without notice, and its internals cannot be independently audited.The passage notes that this reliance is shared by concurrent VTO evaluation work: OpenVTON-Bench [39] uses Gemini 2.0 Flash and CORAL [33] uses GPT-5.Source: Reproducibility and the Choice of a… · p. 23
• The section concludes that proprietary VLM judges are required to assess VTO outputs at retail-grade image quality because open-weight models lack the needed discriminative power.The reported open-weight evaluation used the same five methods evaluated on VITON-HD-edit in Tab. 4 of the main paper.Source: Reproducibility and the Choice of a… · p. 23

F.3 Task-Following Consistency (TFC) for Token Control

• TFC scores task-token adherence separately for each operation on a [0, 5] scale, with strict grading that deducts for any deviation from the intended operation.GTC, PBC, and PR are task-agnostic; TFC is the additional task-specific criterion used for Table 10.Source: §F.3 · pp. 23–25
• full_swap replaces every matching-class garment, partial_swap replaces exactly one while preserving the others, and add preserves the outfit while layering the reference garment on top.The grader sees the input person, reference garment, generated output, and intended task token.Source: §F.3 · p. 24
• CtrlVTON-base achieves consistently high TFC across full_swap, partial_swap, and add on OmniTry-Bench, while GTC remains stable across tokens; no open-weight VTO comparison is available.Table 10 evaluates the clothes-only subset with other inputs held fixed while varying the task token; the reported takeaway is that token-level control does not trade off against garment fidelity.Source: Table 10 · p. 24, §F.3 · p. 24

F.4 Mask-Adherence Metrics (IoU, dHu, dH)

• For each generated try-on image, the garment mask is re-extracted with VIP-SAM and compared with the control mask using region overlap, global shape, and boundary deviation.The three criteria are illustrated in Fig. 19.Source: §F.4 · p. 25
• IoU measures overlap between the control and generated masks, with higher values better, but can miss localized boundary failures and shape distortions that preserve area.It is computed as the intersection area divided by the union area of the two masks.Source: §F.4 · p. 25
• The Hu-moment distance dHu compares the masks’ seven translation-, scale-, and rotation-invariant Hu invariants to capture global shape agreement beyond overlap.It is the Euclidean distance between the invariant vectors of the control mask and re-extracted mask.Source: §F.4 · p. 25
• The symmetric Hausdorff distance dH measures worst-case contour deviation in pixels, capturing localized failures such as a sleeve extending beyond the requested mask; lower is better.Together with IoU and dHu, it provides a view of region, global-shape, and worst-case boundary fidelity.Source: §F.4 · p. 26

F.5 Supplementary evaluation of mask-adherence metrics using SAM3

• Because VIP-SAM builds training masks and evaluates outputs, the authors test for self-referential evaluation using independent, text-prompted SAM3 [5].SAM3 replaces VIP-SAM for inference-time mask extraction and metric computation on VITON-HD-edit, avoiding overlap in segmenter and prompting modality with training.Source: §F.5 · p. 26
• Under SAM3, all five methods shift only slightly, but CtrlVTON remains far ahead of every proprietary baseline on all three spatial-control metrics.The ranking is unchanged from the VIP-SAM-based evaluation in Tab. 4 of the main paper.Source: Table 11 · p. 26, §F.5 · p. 26
• When VIP-SAM and SAM3 disagree on hemline length or garment identity, CtrlVTON follows whichever mask it receives rather than a fixed shape prior.The examples include SAM3 capturing a cardigan where VIP-SAM captures the garment reference.Source: Figure 14 · p. 27

Single-garment try-on

• Additional single-garment results compare diverse garment types, person images, and reference representations.They complement Fig. 7.Source: Figure 15 · p. 28

Multi-garment try-on

• CtrlVTON-base composes multiple garments in one forward pass, compared with multi-garment baselines that either accept multiple garments natively or apply them sequentially.FastFit and BootComp natively accept multiple garments; FitDit and OmniTry are run sequentially, one garment at a time.Source: Figure 16 · p. 29

Data curation

• CtrlVTON-base trained on curated triplets resolves silhouette leakage and implausible fit, showing that quality-controlled supervision can turn the same backbone into a high-quality VTO model.The triplets pass a three-stage quality-control funnel.Source: Figure 17 · p. 30

Mask-controllable try-on

• Varying only the control mask for a fixed person–garment pair produces multiple mask-controllable try-on outputs.These results complement Fig. 10 of the main paper.Source: Figure 18 · p. 31

Mask-following metrics

• Mask overlays show that CtrlVTON respects requested layouts while proprietary baselines do not, with visible discrepancies reflected in low IoU and high dH.Subtle global-shape mismatches are reflected in dHu.Source: Figure 19 · p. 32

Failure cases

• CtrlVTON-base can render garments in stylistically or physically implausible ways, while CtrlVTON’s output quality depends directly on the quality of the input mask.Poorly drawn masks can lead CtrlVTON to ignore the person reference, synthesize garments in arbitrary regions, or extend garments beyond the masked area.Source: Figure 20 · p. 33