PaperlayerView exact revision on arXiv

Source-linked AI summary

ZooClaw-FashionSigLIP2: Distilled Fine-tuning for Robust Fashion Retrieval

Siqiao Xue, Chunxue Xu

arXiv:2606.27708v1cs.CV

TL;DR

Fashion retrieval must balance in-domain gains with out-of-distribution generalization. ZooClaw-FashionSigLIP2 uses distilled full fine-tuning plus WISE-FT interpolation, and leads or ties on every metric across the benchmark suite.

  • Problem

    Fashion retrieval lacks evidence on methods that simultaneously support short user queries, richer descriptions, and out-of-distribution catalogs.

  • Method

    The model applies full fine-tuning with knowledge distillation on curated fashion data, followed by WISE-FT interpolation with the base checkpoint.

  • Results

    ZooClaw-FashionSigLIP2 leads or ties on every metric of every benchmark in the evaluation suite.

  • Takeaways & Limitations

    The released model, benchmark, and pooled evaluation artifacts support future research in fashion retrieval and benchmark quality assessment.

  • Takeaways & Limitations

    Fashion200k’s public ground truth links queries to caption-source images, systematically favoring models trained on the same caption distribution.

Abstract

from arXiv · show

Adapting a foundation vision-language encoder to a specialized retrieval task creates a fundamental tradeoff: gains on the target distribution come at the cost of the foundation model's broad generalization, and fashion retrieval is a stringent instance of this problem. We present ZooClaw-FashionSigLIP2, a fashion-specialized SigLIP2-base model that resolves this tradeoff with a simple recipe -- full fine-tuning with knowledge distillation on curated in-domain data, followed by \wiseft~\citep{wortsman2022wiseft} weight interpolation with the base model -- and outperforms LoRA, larger backbones (up to 1B parameters), and external training data. Under fair evaluation, ZooClaw-FashionSigLIP2 outperforms all baselines on every benchmark in our suite. In addition, we release ZooClaw-Fashion, a new high-quality fashion retrieval benchmark, and a systematic quality analysis of widely-used benchmarks that exposes and mitigates structural biases in their public ground truth. We open-source the model weights and all evaluation artifacts to facilitate future research.

1 Introduction

Fashion retrieval must handle both short production keyword queries and diverse out-of-distribution catalogs, creating a tension between domain gains and broad generalization. ZooClaw-FashionSigLIP2 addresses this through distilled fine-tuning and WISE-FT interpolation, leading or tying on every evaluated metric while introducing a benchmark and revised evaluation artifacts.

  • Motivation: Production fashion search is dominated by short keyword queries, while most vision-language benchmarks and training recipes use detailed natural-language descriptions.Practical models therefore need to support both query types.
  • Problem: Domain-specific fine-tuning improves in-domain performance but creates a tension with out-of-distribution generalization across unseen catalogs and query styles.The passage identifies this as a fundamental production challenge.
  • Method: ZooClaw-FashionSigLIP2 uses full fine-tuning with knowledge distillation on curated in-domain data, followed by WISE-FT interpolation between fine-tuned and base checkpoints.The recipe was identified through systematic experimentation on the SigLIP2 family.
  • Results and resources: Under fair evaluation, ZooClaw-FashionSigLIP2 leads or ties on every metric of every benchmark, outperforming Marqo-fashionCLIP, Marqo-fashionSigLIP, and the zero-shot SigLIP2 family.The evaluation also re-examines Fashion200k using 102,494 held-out judgments after identifying bias in its public ground truth.
  • Results and resources: The authors open-source the model weights, the ZooClaw-Fashion benchmark, and pooled evaluation artifacts.These releases are intended to facilitate future research.

2 Related Work

Related work spans vision-language encoders for retrieval, fashion-specific models and benchmarks, and weight-space methods for improving accuracy or out-of-distribution robustness. SigLIP2 uses a sigmoid loss that avoids the need for large batch sizes, while WISE-FT interpolates pretrained and fine-tuned weights.

  • VLEs for retrieval: CLIP established contrastive image-text retrieval, while later work advanced architecture, training efficiency, and domain specialization.SigLIP and SigLIP2 improved architecture, OpenCLIP improved training efficiency, and FashionCLIP specialized models for fashion.
  • VLEs for retrieval: SigLIP2 replaces softmax-based InfoNCE with a sigmoid loss that eliminates the need for large batch sizes.
  • Fashion retrieval: FashionCLIP, Marqo-fashionCLIP, and Marqo-fashionSigLIP improve in-domain performance but often sacrifice generalization.Related benchmarks include DeepFashion, Fashion200k, and LookBench.
  • Model soups and weight-space ensembling: Model soups average multiple fine-tuned models, whereas WISE-FT interpolates zero-shot and single fine-tuned checkpoints to improve OOD robustness.These approaches rely on fine-tuned and pretrained models often occupying the same loss basin, enabling linear combinations.

3 Method

ZooClaw-FashionSigLIP2 fine-tunes SigLIP2-base for fashion retrieval using multi-task graded-relevance contrastive learning and image-encoder distillation, then interpolates the result with the base model to balance specialization and OOD generalization.

  • Method overview: The method uses a SigLIP2-base vision-language encoder with image and text encoders mapped into a shared embedding space, followed by three stages.These stages are multi-task contrastive fine-tuning with knowledge distillation, WISE-FT interpolation, and interpolation-coefficient selection.
  • Contrastive training: Generalized Contrastive Loss incorporates graded relevance scores as soft labels and downweights highly relevant in-batch negatives that may be false negatives.The image-to-text loss mirrors text-to-image loss, and each task optimizes both retrieval directions.
  • Contrastive training: Short-query and long-query retrieval are combined as Lcon = λsLshort + λlLlong with λl=1.0 and λs=0.5.The formulation addresses both keyword and attribute-rich queries without defining separate image-to-text tasks.
  • Knowledge distillation: A frozen base-model teacher provides image-encoder Learning without Forgetting distillation, with total loss L = Lcon + λLwF · LLwF and λLwF = 1.0.The regularizer is intended to mitigate in-domain fine-tuning’s degradation of OOD generalization.
  • WISE-FT interpolation: WISE-FT constructs θα = (1 −α) · θ0 + α · θft, sweeping α over ten values from 0.0 to 1.0 and selecting the point maximizing the minimum cross-benchmark margin.The interpolation trades off in-domain specialization against OOD retention, with α selected on the evaluation benchmarks.

4 Data Construction

The paper constructs training data and the ZooClaw-Fashion benchmark from a structured proprietary fashion catalog, using Gemma-4-31B to generate complementary short and long queries. Evaluation spans in-domain and out-of-distribution benchmarks, with pooled re-evaluation correcting limitations in Fashion200k’s original ground truth.

  • Data source: The catalog contains cleaned product images and structured attributes, organized into nine main fashion categories.Attributes include title, brand, color, category, sub-category, material, style, occasion, demographic, and pattern.
  • Training data: Training uses three catalog-derived scales: ZC-TRAIN-S (200K), ZC-TRAIN-M (400K), and ZC-TRAIN-L (800K+) image–text pairs.Some experiments additionally incorporate MARQO-FASHION1 (∼733K pairs) as external training data.
  • Query generation: Gemma-4-31B generates short and long queries from each product, conditioning on sampled attributes for concise searches and the full attribute set for verbose descriptions.Short queries are ≤8 words and preserve uniquely identifying attributes despite 50% per-attribute dropping; long queries are 30–60 words and average ∼40 words.
  • Evaluation benchmarks: ZooClaw-Fashion evaluates 2K queries against 12K catalog images using paired short and long queries with under-specified but unambiguous short-query construction.The benchmark uses the evaluation partition and the same attribute-sampling and LLM-rewrite pipeline as training.
  • Evaluation benchmarks: Evaluation also includes H&M as a secondary OOD benchmark and Fashion200k as the primary OOD benchmark, using 2K queries and 201,624 product images for Fashion200k.H&M contains 105K catalog images, while Fashion200k provides long natural-language descriptions with mean length ∼30 words.
  • Ground-truth correction: Fashion200k is re-evaluated with TREC-style pooled relevance because its original qrels mark only caption-source images and systematically favor models trained on the same caption distribution.Top-k results from all systems are pooled, graded on a 1–5 scale by Gemma-4-31B, and treated as relevant at threshold ≥3.

5 Experiments

Experiments show that ZooClaw-FashionSigLIP2 leads every benchmark metric, with full fine-tuning and WISE-FT interpolation preserving strong in-domain gains while improving out-of-distribution retrieval. Analyses indicate that larger backbones, LoRA, external data, and greedy soups do not reliably improve this tradeoff.

  • Main results: ZooClaw-FashionSigLIP2 leads on every metric across ZooClaw-Fashion, H&M, and Fashion200k under the reported ground-truth protocols.Fashion200k uses TREC-style pooled qrels to mitigate bias in the public ground truth.
  • Analysis I: LoRA vs. full fine-tuning: Full fine-tuning consistently outperforms LoRA on in-domain retrieval and provides a stronger foundation for model-soup interpolation.Despite extensive searches over rank, regularization, adapter scope, and data size, no LoRA configuration matches the base model on Fashion200k; the low-rank constraint concentrates updates in dominant directions.
  • External-data analysis: Adding MARQO-FASHION external data consistently hurts both in-domain and OOD performance across 8+ experiments.The paper attributes this degradation to distributional interference caused by data curated for a different base model and training recipe.
  • Analysis V: Model soup interpolation sweep: The deployed model uses WISE-FT interpolation at α=0.4, within the robust α ∈[0.3, 0.6] window where it beats every baseline.In-domain R@10 rises sharply through α=0.4, while pooled Fashion200k R@10 peaks around α ∈ [0.3, 0.5].
  • Analysis VI: WISE-FT vs. greedy model soup: The greedy model soup achieves higher in-domain recall but drops below the deployed model on Fashion200k R@10, favoring WISE-FT for the tradeoff.WISE-FT interpolates directly between the base and fine-tuned models, whereas greedy soup averages checkpoints along the training trajectory.

6 Benchmark Quality and Pooled Re-evaluation

This section shows that public Fashion200k ground truth is structurally biased, while pooled re-evaluation provides a fairer assessment in which ZooClaw-FashionSigLIP2 leads on graded relevance. The pooled qrels and evaluation artifacts are released to support reproducible future evaluation.

  • Public Ground-Truth Bias: Only 37.5% of public Fashion200k query–ground-truth pairs are clearly relevant, while 22% are clearly wrong, with an average grade of 3.35/5.The released qrels map each query to the single image whose caption generated it, rather than using independent relevance annotation.
  • Pooled Re-evaluation: 102,494 unique query–image pairs from twelve systems were pooled and graded under a TREC-style protocol, with ZooClaw-FashionSigLIP2 leading on graded relevance.Gemma-4-31B assigned grades using a 1–5 rubric.
  • Pooled Re-evaluation: 35,570 judgments on ZooClaw-Fashion and 43,940 on H&M yielded identical pooled and original rankings for the three listed systems.On both clean benchmarks, pooled nDCG@10 also agrees with R@10 on every reported pairwise comparison.
  • Evaluation Release: The 102,494 graded Fashion200k qrels are released as srpone/fashion200k-pooled-eval alongside, but not replacing, the original benchmark.The release supports reproducible fashion-retrieval evaluation by future systems.

7 Conclusion

ZooClaw-FashionSigLIP2 is a domain-adapted fashion retrieval model that leads or ties on every metric across the benchmark suite under fair evaluation. The paper attributes this performance to distilled full fine-tuning with WISE-FT, validates it with pooled re-evaluation, and releases the model and evaluation artifacts.

  • Performance: ZooClaw-FashionSigLIP2 leads or ties on every metric of every benchmark under fair evaluation.The conclusion describes it as a domain-adapted fashion retrieval model.
  • Method: Full fine-tuning with knowledge distillation followed by WISE-FT outperforms LoRA, greedy model soups, larger backbones, and LLM-based text encoders.The conclusion identifies this recipe as more effective than each listed alternative.
  • Resources: The authors open-source the model weights, ZooClaw-Fashion evaluation benchmark, and srpone/fashion200k-pooled-eval qrels to support future research.They also report analyzing quality issues in the public Fashion200k ground truth.
  • Evaluation: Pooled re-evaluation preserves rankings on the two clean benchmarks but flips Fashion200k in ZooClaw-FashionSigLIP2’s favour.The table attributes this change to caption-source bias, while the conclusion states that held-out pooled evaluation gives ZooClaw-FashionSigLIP2 the lead over Marqo-fashionSigLIP on every graded relevance metric.

Appendices … A.2 ZooClaw-Fashion and H&M validation

The appendices detail the pooled-judging methodology, graded-relevance metrics, and validation showing that pooled rankings preserve the original rankings on ZooClaw-Fashion and H&M. ZOOCLAW-FASHIONSIGLIP2 leads every H&M metric under pooled evaluation.

  • Appendices: The appendix provides the full pooled-judging protocol, per-benchmark validation tables, and threshold sensitivity analysis for graded relevance.These materials support the re-evaluation described in Section 6.
  • A.1 Pooled judging protocol: For each benchmark, the candidate pool contains every contributing model’s top-10 retrievals per query, with duplicate (query, image) pairs judged once.The protocol uses a held-out judge model for each unique pooled pair.
  • A.1 Pooled judging protocol: Each pooled image-query pair receives a 1–5 relevance rating based on how accurately the natural-language description matches the rendered image.The rubric ranges from 1 = completely wrong to 5 = excellent match, and requires only a single numeric response.
  • A.1 Pooled judging protocol: Gemma-4-31B serves as judge because it matches the family used for Fashion200k captions and showed consistent instruction-following in spot-checks.Table 6 summarizes pool sizes and contributing models across benchmarks.
  • A.1 Pooled judging protocol: The evaluation reports MRR@10 with binary relevance g ≥τ and nDCG@10 with gains 2g −1 for thresholds τ ∈{3, 4, 5}.nDCG@10 uses the full graded distribution and is treated as the headline rank-aware metric.
  • A.2 ZooClaw-Fashion and H&M validation: On both clean benchmarks, pooled and original rankings of SigLIP2-base, Marqo-fashionSigLIP, and ZOOCLAW-FASHIONSIGLIP2 are identical.Pooled nDCG@10 agrees with R@10 on every pairwise comparison.
  • A.2 ZooClaw-Fashion and H&M validation: 35,570 judgments evaluate ZooClaw-Fashion and 43,940 evaluate H&M; ZOOCLAW-FASHIONSIGLIP2 leads every H&M metric.Both tables use short queries and threshold ≥3, and both report agreement between original R@10 and pooled nDCG@10 rankings.

A.3 Threshold sensitivity on Fashion200k · A.4 Caveats

Across three relevance thresholds, ZOOCLAW-FASHIONSIGLIP2 leads or ties the headline systems on every reported metric, while the evaluation remains conditional on a single judge, shallow pooling, and pool-positive recall. These caveats limit interpretation of absolute scores, though relative model comparisons use the same judging conditions.

  • A.3 Threshold sensitivity on Fashion200k: A.3 Threshold sensitivity on Fashion200k: At threshold 3, ZOOCLAW-FASHIONSIGLIP2 leads Marqo-fashionSigLIP on both metrics.The supplied passage reports this comparison under the generously judged relevance threshold.
  • A.3 Threshold sensitivity on Fashion200k: A.3 Threshold sensitivity on Fashion200k: At thresholds 4 and 5, ZOOCLAW-FASHIONSIGLIP2 continues to lead on nDCG@10 and leads on MRR@10 at threshold 4.The supplied passage is truncated before specifying the remaining threshold-5 MRR@10 comparison.
  • A.3 Threshold sensitivity on Fashion200k: A.3 Threshold sensitivity on Fashion200k: ZOOCLAW-FASHIONSIGLIP2 leads or ties on every metric at every threshold in the Fashion200k pooled re-evaluation.Table 9 pools 102,494 judgments and includes 1,972, 1,647, and 1,267 queries with at least one grade-≥3, grade-≥4, or grade-≥5 image, respectively.
  • A.4 Caveats: A.4 Caveats: All judgments come from Gemma-4-31B, and inter-judge agreement with another VLM has not yet been measured.Absolute pooled numbers are treated as judge-conditional, while relative comparisons use the same judge across models.
  • A.4 Caveats: A.4 Caveats: The pool covers only contributing models’ top-10 results, so unseen documents in a future model’s true top-10 receive an implicit grade of 0.The released pipeline supports incrementally adding a new model’s top-10 and judging only genuinely new pairs.
  • A.4 Caveats: A.4 Caveats: R@10 under pooled qrels is recall-of-the-pool rather than corpus-wide recall because the corpus-wide number of relevant images is unknown.The denominator is the number of grade-≥3 images for that query inside the pool, applied symmetrically to every model.

B Evaluation Benchmark Construction

The evaluation suite uses text queries, product-image corpora, and query-to-item ground truth, centered on the newly constructed ZooClaw-Fashion benchmark. It also includes H&M and Fashion200k, with Fashion200k evaluated using released pooled qrels to mitigate caption-source bias.

  • ZooClaw-Fashion: ZooClaw-Fashion is derived from an internal catalog containing 12K products across 1,355 fine-grained categories.Its corpus contains 12K product images with structured metadata spanning product, demographic, and style attributes.
  • ZooClaw-Fashion: 2K sampled ZooClaw-Fashion products receive short partial queries generated by attribute sampling and natural-language rewriting.Queries are capped at 8 words, average approximately 5 words, and retain enough attributes to uniquely identify the target while preserving partial-information retrieval.
  • ZooClaw-Fashion: ZooClaw-Fashion also provides long queries averaging approximately 42 words, generated from all 8 structured attributes as detailed third-person descriptions.Generation uses greedy decoding with temperature = 0 and a 100-token limit.
  • H&M: H&M uses 105K product images from the Personalized Fashion Recommendations dataset and applies the same two-stage short-query pipeline to 2K randomly sampled products.Its metadata covers 131 product types, and the sampled queries average approximately 6 words.
  • Fashion200k: Fashion200k uses 201,624 product images and 2K query-document mappings from the Marqo FashionCLIP evaluation suite rather than newly generated queries.Because the original qrels are caption-source biased, the main-paper metrics use the released pooled qrels.
Loading 2606.27708v1…