PaperlayerView exact revision on arXiv

Source-linked AI summary

JewelTry: Mask-Free Scale Aware Jewelry Virtual Try-On

Xinlei Niu, Peixia Li, Jun Wang, Chenchen Xu, Jiayu Yang, Jing Zhang, Pulak Purkait, Hongdong Li

arXiv:2609.16626v1cs.CV

TL;DR

Jewelry VTON lacks reliable mask-free scale guidance and must preserve fine structural details while placing small, rigid objects realistically. The paper introduces JVTO-Bench and JewelTry, combining product-scale conditioning with attention-based jewelry preservation. Experiments report a balance across visual fidelity, background preservation, object consistency, and scale accuracy, while absolute scale control remains limited.

  • Problem

    Jewelry VTON lacks suitable benchmarks and mask-free methods with explicit product-scale guidance, despite requiring accurate placement, scale, and fine-grained structural preservation.

  • Method

    The paper introduces JVTO-Bench and JewelTry, which uses a scale adapter, single-directional condition attention, and attention refinement loss for mask-free jewelry VTON.

  • Results

    JewelTry achieves a balance among visual fidelity, background preservation, object consistency, and scale accuracy, establishing a strong baseline for mask-free, scale-aware jewelry VTON.

  • Takeaways & Limitations

    JVTO-Bench and JewelTry provide a benchmark and baseline for developing and evaluating scale-aware jewelry VTON without masks.

  • Takeaways & Limitations

    JewelTry mainly provides relative rather than absolute scale control, and repeated inference can produce slight output variations.

Abstract

from arXiv · show

Virtual try-on (VTON) enables customers to visualize how fashion products appear when worn and has become an important technology for online shopping. While recent advances have substantially improved garment VTON, jewelry remains a challenging and underexplored category due to its small size, rigid structure, and sensitivity to fine-grained visual details. Realistic jewelry VTON requires not only faithful appearance transfer but also accurate scale and placement relative to the wearer. Existing jewelry VTON methods typically rely on mask guidance, whereas mask-free approaches lack explicit guidance for modeling the product scale. To bridge this gap, we introduce JVTO-Bench, a benchmark dataset for scale-faithful jewelry VTON, providing reference source target triplets with real-world product-scale annotations across four major jewelry categories. Building upon this benchmark, we propose JewelTry, a mask-free diffusion framework for scale-aware jewelry VTON. JewelTry incorporates a scale adapter that encodes product dimensions into a scale token, enabling the model to learn scale relationships between jewelry items and surrounding human anatomy in-context. To further improve jewelry consistency, we introduce a single-directional condition attention mechanism and an attention refinement loss that preserve both coarse geometry and fine-grained structural details of the reference jewelry. Extensive experiments show that JewelTry achieves a balance among visual fidelity, background preservation, object consistency and scale accuracy, establishing a strong baseline for mask-free, scale-aware jewelry virtual try-on.

Introduction

Jewelry VTON remains challenging because realistic results must preserve fine appearance while placing rigid, small objects at plausible scales, yet suitable mask-free benchmarks are scarce. The paper introduces JVTO-Bench and JewelTry to support and improve scale-aware, mask-free jewelry try-on.

  • Introduction: Mask-free jewelry VTON is practically important because standalone photos and textual metadata can create misleading expectations about jewelry appearance and size.Realistic try-on must preserve detailed appearance while rendering jewelry at physically plausible sizes and locations on diverse human models.
  • Introduction: JVTO-Bench provides reference-source-target triplets with real-world scale annotations across four jewelry categories for mask-free training and evaluation.The benchmark includes training and test splits and is designed specifically for scale-faithful jewelry VTON.
  • Introduction: JewelTry is a mask-free, scale-aware framework that encodes product dimensions into a scale token for learning jewelry-to-anatomy scale relationships.The framework targets physically plausible scale while avoiding manually specified masks or explicit geometric supervision.
  • Introduction: Single-directional condition attention and attention refinement loss preserve coarse geometry and fine-grained jewelry structure during diffusion.The attention mechanism limits contamination of reference condition tokens by noisy latent features, while the refinement loss supervises attention toward the jewelry region.
  • Introduction: Extensive experiments establish JewelTry as a strong baseline for scale-aware and mask-free jewelry VTON.The reported evaluation covers visual fidelity, background preservation, object consistency, and scale faithfulness.

JVTO-Bench Dataset

JVTO-Bench provides a scale-annotated jewelry VTON benchmark, while JewelTry combines scale conditioning, directional attention, and attention-based supervision to preserve jewelry structure and placement.

  • JVTO-Bench Dataset: Over 23K images across rings, earrings, necklaces, and bracelets form JVTO-Bench, with each sample organized as a reference-source-target triplet and product scale in inches.The dataset is intended for scale-faithful, mask-free jewelry VTON.
  • JVTO-Bench Dataset: The dataset pipeline collects diverse jewelry imagery, filters product-person matches, and converts retained pairs into cropped, captioned, try-on triplets.Object removal and difference-map refinement address residual jewelry and shadow artifacts in try-off images.
  • Scale Adapter: JewelTry uses a scale adapter to convert product measurements in inches into scale tokens, helping model relative jewelry scale in context.The framework overview identifies the scale adapter as one of three key components.
  • Single Directional Condition Attention: Single-direction condition attention blocks noisy latent tokens from interfering with jewelry condition tokens while retaining jewelry-to-latent structural guidance.This design targets condition collapse caused by bidirectional feature exchange and preserves rigid topology and fine-grained details.
  • Training Loss: The attention refinement loss aligns an attention-derived jewelry soft mask with the ground-truth mask, improving spatial correspondence, placement, scale fidelity, and structural preservation.The soft mask is extracted from reference-query and noisy-latent-key attention and aggregated across heads with normalization.
  • Training Loss: The overall objective combines global velocity prediction with object-region and attention losses, maintaining person, clothing, background, jewelry structure, and fine details.The velocity loss supplies image-level supervision, while the other losses focus on the target jewelry region.

Experiment and Results

JewelTry is evaluated on JVTO-Bench and OmniTry-Bench using image fidelity, background preservation, and object consistency metrics.

  • The evaluation measures image fidelity, background preservation, and object consistency for jewelry virtual try-on.FID measures distributional similarity between generated and ground-truth images, while non-jewelry preservation and object consistency are assessed with additional metrics.
  • The experiments compare JewelTry with five baselines spanning zero-shot and adapted methods.
  • JVTO-Bench evaluation uses 375 test samples across rings, earrings, necklaces, and bracelets.
  • The reported comparison includes Ground Truth, JewelTry, Qwen-image-edit, OmniTry, Qwen-JVTON, Any2Any TryOn, and InsertAnything.

Ablation study

The ablation study shows that JewelTry’s components target complementary aspects of reference consistency, scale awareness, target alignment, and object consistency, while scale control remains implicit and imperfect.

  • The full JewelTry model achieves the best object consistency while maintaining background preservation and improving jewelry placement.Adding the attention refinement loss also increases IoU, indicating improved placement correctness.
  • Single-directional condition attention produces the largest single gain in DINOref by preserving reference jewelry structure.
  • The scale adapter further enhances scale awareness and increases IoU, with small metric changes attributed to inference variance.
  • Repeated generations converge to similar relative jewelry scales, indicating consistent implicit size control.
  • JewelTry may fail at extreme scales because implicit in-context learning and real-world jewelry scale bias limit control.

Conclusion

The paper introduces JVTO-Bench and JewelTry for mask-free, scale-aware jewelry virtual try-on, combining scale modeling with mechanisms that preserve jewelry structure and appearance.

  • JVTO-Bench provides triplet samples and product-scale annotations across four major jewelry categories.
  • JewelTry uses a scale adapter, single-directional condition attention, and an attention refinement loss to support scale-aware jewelry fidelity without masks.
  • Extensive experiments report a balance among visual fidelity, background preservation, object consistency, and scale accuracy.

[Supplementary Material] JewelTry: Mask-Free Scale Aware Jewelry Virtual

JVTO-Bench is a large real-world jewelry VTON dataset with category-specific scale annotations, triplet images, and supporting try-off refinement procedures.

  • JVTO-Bench contains approximately 23k samples spanning rings, earrings, necklaces, and bracelets.
  • Each sample includes ground-truth try-on, try-off person, and reference jewelry images, plus scale, product type, captions, and target annotations.
  • The final test set contains 59 ring, 104 earring, 106 necklace, and 106 bracelet products after product-level separation and filtering.
  • The dataset uses category-specific 2D physical dimensions such as length, width, diameter, or pendant size to represent product scale.
  • Qwen-Object-Remover outputs are refined with SSIM-based difference maps to reduce residual jewelry and shadow artifacts in try-off images.
  • The SSIM-based mask is applied manually only when object-removal results show obvious quality degradation because it is not robust across all images.

2 Implementation details

The implementation combines scale-aware conditioning, attention-based jewelry-region processing, and evaluation procedures designed to compare mask-free try-on fairly.

  • 2 Implementation details: JewelTry is fine-tuned from Qwen-Image-Edit-2511 with LoRA adapters on JVTO-Bench for 200k iterations using eight H200 GPUs.The adapters use rank 16 and alpha 16 in self-attention blocks, with batch size 8 and learning rate 5×10^-5.
  • 2 Implementation details: The scale adapter projects product dimensions into an embedding, combines it with jewelry-category information, and maps the result into text-token space.Scale tokens are randomly dropped in 25% of training cases so inference can operate without scale annotations.
  • 2 Implementation details: Attention refinement extracts cross-attention maps from transformer blocks 51–60, averages and normalizes them, and uses them to guide jewelry-region modeling.Grounded object masks are obtained from attention responses, while Grounded DINO and SAM detect jewelry regions for masking or metric calculation.
  • 2 Implementation details: Comparisons use shared scale-aware prompts for relevant baselines, rectangular rather than explicit insertion masks for InsertAnything, and random inference seeds.On OmniTry-Bench, scale information is unavailable, so JewelTry drops the scale token and scale text.
  • 2 Implementation details: ScaleErr measures the absolute log-ratio between predicted and ground-truth jewelry dominant dimensions, with zero indicating a perfect size match.The symmetric metric penalizes objects rendered too large or too small by the same factor.

3 Quantitative comparison across jewelry categories

Across jewelry categories, JewelTry performs strongly on both localized and larger accessories, achieving the best results in three categories and remaining competitive on earrings.

  • 3 Quantitative comparison across jewelry categories: JewelTry achieves the best performance on rings, necklaces, and bracelets, while ranking second on earrings with only a −0.0309 DINOp gap to the best method.Figure 5 compares DINOp for person/background preservation and CLIPTar for object consistency across the four categories.
  • 3 Quantitative comparison across jewelry categories: The category-wise results indicate generalization across jewelry types with different spatial scales, attachment regions, and structural characteristics.Rings and bracelets represent small localized regions, whereas necklaces require placement around complex neck and upper-body regions.

4 Result variance and significance test

Significance testing reports balanced improvements over zero-shot and trained-on-benchmark baselines, including significant gains in object consistency, background preservation, and scale faithfulness.

  • 4 Result variance and significance test: JewelTry significantly outperforms the zero-shot baselines on object consistency, background preservation, and scale faithfulness across multiple metrics.Reported metrics include DINOTar, IoU, and LPIPSp, with p-values as low as 3.87 × 10^-47.
  • 4 Result variance and significance test: Compared with trained-on-benchmark baselines, Qwen-JVTON and JewelTry also show significant gains on object consistency, background preservation, and scale faithfulness.The reported comparisons use DINORef, LPIPSp, and IoU p-values of 8.64 × 10^-3, 4.78 × 10^-7, and 2.19 × 10^-2.
  • 4 Result variance and significance test: The authors conclude that significance tests support a more balanced trade-off between object consistency and background preservation together with stronger scale-aware ability.The attention-refinement discussion motivates block-wise attention supervision based on observed attention behavior across MMDiT transformer blocks.

5.2 Timestep binary gate

The timestep gate restricts attention-refinement supervision to low-noise samples, while classifier-free guidance combines scale, text, person, and jewelry conditions during inference.

  • 5.2 Timestep binary gate: Attention refinement uses BCE and Dice supervision on cross-attention maps only when the noise level is below τ = 0.5.Samples with σ_b ≥ τ contribute zero attention loss because spatial structure is considered unreliable at high noise levels.
  • 5.2 Timestep binary gate: During inference, scale, text, person-image, and reference-jewelry conditions are combined into a unified classifier-free guidance signal.The person condition remains in the negative sample while scale, text, and jewelry conditions are dropped, preserving person structure while emphasizing jewelry-related conditions.
  • 5.2 Timestep binary gate: CFG = 1 provides insufficient scale guidance, whereas CFG values of 10 or 15 can overamplify conditions and cause unrealistic results or jewelry structural collapse.The qualitative comparison identifies an intermediate guidance setting as necessary for balancing conditioning strength.

7 More study on single direction condition attention mechanism

This study examines how block-wise masking choices affect single-direction condition attention in JewelTry, balancing object consistency against background preservation. Blocking reverse paths to both person and jewelry tokens improves consistency but can degrade background quality.

  • Single-direction condition attention blocks information flow from noisy latent tokens to jewelry condition tokens during training and improves coarse-level jewelry consistency during inference.
  • The M P&J mask extends blocking to both person and jewelry tokens to test effects on background preservation and object consistency.
  • Blocking reverse attention paths from noisy latent tokens to person and reference jewelry tokens further improves object consistency.
  • M P&J slightly degrades background preservation, producing higher LPIPSp despite its object-consistency improvement.

8 More visualization

The visual studies show JewelTry preserving jewelry structure and fine-grained details across categories and comparison settings. They also illustrate its quantitative category coverage, attention behavior, guidance-scale sensitivity, and trade-offs among masking strategies.

  • JewelTry preserves fine-grained textual details in generated jewelry, including details important for earrings and necklaces.
  • Across additional JVTO-Bench and OmniTry-Bench comparisons, JewelTry shows better structural fidelity, texture preservation, and fine-grained detail than selected competitive baselines.
  • JewelTry achieves superior object consistency and balanced background preservation across the four JVTO-Bench jewelry categories.
  • The attention visualizations examine Qnoise·KT maps across transformer blocks for earrings and necklaces, including person and reference attention maps averaged over inference timesteps.
  • The inference visualization compares JewelTry outputs under different classifier-free guidance scale values, while the masking comparison exposes a consistency–background-preservation trade-off.
  • Additional examples demonstrate JewelTry’s ability to preserve fine-grained textual details in jewelry.
Loading 2609.16626v1…