PaperlayerView exact revision on arXiv

Source-linked AI summary

FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning

Haokun Wen, Xuemeng Song, Xinghao Xie, Xiaolin Chen, Xiangyu Zhao, Weili Guan

arXiv:2605.22552v1cs.CVcs.MM

TL;DR

Fashion image retrieval must handle diverse query formats and search intentions, but existing approaches target narrow tasks. FashionLens addresses this with U-FIRE and a unified MLLM-based framework using task-adaptive query calibration and sampling. It outperforms existing methods on standard tasks and generalizes to complex, unseen scenarios.

  • Problem

    Existing fashion retrieval methods focus on narrow tasks despite diverse query formats and search intentions, motivating a unified evaluation and retrieval framework.

  • Method

    FashionLens combines the U-FIRE benchmark with an MLLM-based framework using task-aligned query calibration and gradient-guided adaptive sampling.

  • Results

    FashionLens outperforms existing methods on standard retrieval tasks and generalizes to complex, unseen scenarios.

  • Takeaways & Limitations

    U-FIRE provides a standardized testbed for evaluating versatile fashion retrieval across fragmented datasets and unseen tasks.

Abstract

from arXiv · show

Fashion image retrieval is a cornerstone of modern e-commerce systems. A unified framework that supports diverse query formats and search intentions is highly desired in practice. However, existing approaches focus on narrow retrieval tasks and do not fully capture such diversity. Therefore, in this work, we aim to develop a unified framework capable of handling diverse realistic fashion retrieval scenarios, achieving truly versatile fashion image retrieval. To establish a data foundation, we first introduce U-FIRE, a comprehensive benchmark that consolidates fragmented fashion datasets into a unified collection, supplemented by two manually curated datasets for testing generalization. Building upon this, we propose FashionLens, a unified framework based on Multimodal Large Language Models. To handle divergent matching objectives, we design a Proposal-Guided Spherical Query Calibrator that dynamically shifts query representations into task-aligned metric spaces via adaptive spherical linear interpolation. Additionally, to mitigate the optimization imbalance caused by varying task complexities and data scales, we develop a Gradient-Guided Adaptive Sampling strategy that automatically re-weights tasks based on realtime learning difficulty and the data scale prior. Experiments on U-FIRE show that FashionLens achieves state-of-the-art performance across diverse retrieval scenarios and generalizes robustly to unseen tasks. The data and code are publicly released at https://github.com/haokunwen/FashionLens.

I. INTRODUCTION

Fashion image retrieval must accommodate diverse query formats and search intentions, but task-specific models and heterogeneous data create fragmented solutions and training imbalance. FashionLens addresses these challenges with the U-FIRE benchmark, a unified MLLM-based framework, and task-adaptive mechanisms for representation calibration and optimization.

  • Fashion retrieval spans images, text, sketches, videos, and multimodal queries, with intentions including similar, compatible, and attribute-specific item search.
  • Task-specific retrieval models produce fragmented solutions with limited data efficiency, motivating a unified architecture for diverse query formats and search intentions.
  • U-FIRE unifies 15 datasets spanning 9 existing fashion retrieval tasks and adds 2 new tasks for out-of-distribution evaluation.
  • Divergent matching objectives cause feature interference because fashion matching ranges from fine-grained visual identity to abstract stylistic harmony.
  • Uneven data scales and heterogeneous task complexities create severe training imbalance, exacerbated by the difficulty of acquiring fashion data.
  • FashionLens uses an MLLM backbone with a Proposal-Guided Spherical Query Calibrator and gradient-guided adaptive sampling to align representations and balance heterogeneous-task learning.

II. RELATED WORK … A. Instruction Annotation

The paper situates FashionLens within fashion vision-language pre-training and universal multimodal retrieval, then introduces U-FIRE as a unified, instruction-augmented benchmark for diverse fashion retrieval tasks and OOD evaluation. Its instruction annotation represents each sample as a raw query, search instruction, and target image, with varied templates to improve linguistic robustness.

  • A. Fashion Vision-Language Pre-training: Fashion vision-language pre-training methods address heterogeneous tasks such as retrieval, category recognition, and fashion captioning within single models.FAME-ViL uses three task modes with data-size-proportional multi-task sampling, while FashionSAP adds attribute information for fine-grained representations.
  • III. U-FIRE BENCHMARK: U-FIRE comprises 11 tasks, with nine supporting training, validation, and testing and two unseen tasks reserved exclusively for evaluation.This task organization explicitly supports evaluation on unseen retrieval scenarios.
  • A. Instruction Annotation: The benchmark uses placeholders for category, attribute, and text description information in its instruction templates.The notation [C], [ATTR], and [TEXT] denotes these respective placeholders.
  • B. General-domain Universal Multimodal Retrieval: General-domain universal multimodal retrieval aggregates diverse datasets and increasingly uses multimodal large language models as unified encoding backbones.UniIR established the M-BEIR benchmark by aggregating 10 datasets, followed by MLLM-based methods including VLM2Vec, GME, and MM-Embed.
  • III. U-FIRE BENCHMARK: U-FIRE consolidates 15 publicly available fashion datasets covering 9 core fashion image retrieval tasks into a unified instruction-augmented format.The benchmark also introduces 2 manually curated tasks for out-of-distribution generalization.
  • A. Instruction Annotation: Versatile fashion retrieval requires search instructions because raw query content alone does not specify the diverse retrieval objectives handled by one framework.Conventional single-task retrieval implicitly fixes objectives such as similarity or compatibility matching, whereas versatile retrieval unifies multiple objectives.
  • A. Instruction Annotation: Instruction templates are syntactically distinct yet semantically equivalent, and samples are randomly assigned templates to prevent overfitting to fixed sentence patterns.The strategy is intended to foster linguistic robustness.
  • A. Instruction Annotation: Each benchmark sample is represented as ⟨raw query content, search instruction, target image⟩, with dedicated gallery image sets constructed for each dataset.The consolidated benchmark contains over 400k samples: 325k training, 36k validation, and 46k testing.

B. Unseen Tasks for OOD Evaluation … B. MLLM-based Query/Target Encoding

U-FIRE adds two multimodal, reasoning-intensive unseen tasks to evaluate out-of-distribution generalization beyond multi-task learning on seen tasks. FashionLens uses an MLLM-based unified retrieval formulation with intention-aware scoring, special-token representations, and two components addressing feature interference and task-optimization imbalance.

  • B. Unseen Tasks for OOD Evaluation: U-FIRE introduces two unseen multimodal tasks requiring complex reasoning to measure generalization beyond training-task multi-task learning.Their test sets are not used during model optimization, unlike the unified dataset’s seen-task test sets.
  • B. Unseen Tasks for OOD Evaluation: Task 10 retrieves shop images from street photos plus natural-language modifications describing discrepancies in attributes such as color, material, or pattern.The task combines Street→Shop and Image+Modification Text→Image objectives using curated DeepFashion2 test-set image pairs and manual modification annotations.
  • B. Unseen Tasks for OOD Evaluation: Task 11 retrieves items compatible with visual context while satisfying explicit textual constraints, reflecting personalized preferences and multiple valid choices.Constraints may specify attributes such as color, style, or functionality.
  • IV. FASHIONLENS: FashionLens comprises an MLLM backbone, PGSQC, GGAS, and overall training objectives, with PGSQC targeting feature interference and GGAS targeting cross-task optimization imbalance.The framework section presents these components after problem formulation and MLLM-based encoding.
  • A. Problem Formulation: Versatile fashion retrieval is formulated as ranking candidates using intention-aware queries over a collection of heterogeneous retrieval tasks and datasets.The unified dataset contains K datasets covering O tasks, with K ≥ O because one task may include multiple datasets.
  • A. Problem Formulation: FashionLens learns a unified scoring function S: Q × G → R that assigns higher relevance scores to candidate images matching arbitrary intention-aware queries.G denotes the gallery image set.
  • B. MLLM-based Query/Target Encoding: An MLLM backbone processes heterogeneous modalities and performs semantic reasoning, enabling diverse query formats within one retrieval framework.The architecture is motivated by universal multimodal retrieval methods in general domains.
  • B. MLLM-based Query/Target Encoding: Learnable [RETq] and [RETt] tokens aggregate query and target semantics, producing normalized representations q0 and t on a unit hypersphere.The tokens are appended to the intention-aware query and corresponding target-image token sequences, respectively.

C. Proposal-Guided Spherical Query Calibrator · 1) Intention-Oriented Adaptation Proposal: · 2) Adaptive Spherical Linear Interpolation:

FashionLens addresses task interference by adapting the initial query representation toward search-intention-relevant features through a low-rank proposal and query-dependent spherical interpolation. The calibrator preserves original semantics while controlling adaptation strength according to each query’s representation.

  • C. Proposal-Guided Spherical Query Calibrator: Heterogeneous retrieval objectives cause a shared representation q0 to compromise across tasks, allowing dominant task features to overshadow intention-critical signals.Natural-language instructions also provide weaker conditioning than high-dimensional visual features, limiting independent query reorientation.
  • C. Proposal-Guided Spherical Query Calibrator: PGSQC cautiously adapts q0 because explicit supervision about intention-relevant features is absent and one-step calibration is unreliable.It combines the original representation with an adaptation proposal rather than fully replacing q0.
  • 1) Intention-Oriented Adaptation Proposal:: The intention-oriented proposal qp uses learnable low-rank matrices A ∈R^D×d and B ∈R^d×D to compress q0 into a latent space and reconstruct intention-relevant components.This information-bottleneck design focuses adaptation on relevant features while retaining the original representation’s dimensionality.
  • 1) Intention-Oriented Adaptation Proposal:: The residual connection q0 + q0AB preserves core MLLM semantics while the adapter learns what to change, and ℓ2 normalization places qp on q0’s unit hypersphere.The residual structure avoids uncontrolled deviations that could distort identity-relevant features.
  • 1) Intention-Oriented Adaptation Proposal:: An MLP generates query-conditioned projection matrices A and B, allowing the proposal to reflect distinct visual attributes or stylistic cues across individual queries.The query-adaptive parameters are jointly learned with the interpolation coefficient controlling adaptation strength.
  • 2) Adaptive Spherical Linear Interpolation:: Slerp combines normalized q0 and qp by rotating the query along the hypersphere, matching the cosine-similarity geometry used for retrieval instead of interpolating in Euclidean space.Because normalized embedding dot products equal cosine similarity, angular movement provides the relevant representation-space interpretation.
  • 2) Adaptive Spherical Linear Interpolation:: Adaptive Slerp uses a query-dependent coefficient λ(q0) to avoid under-correcting misaligned queries or overdistorting well-aligned ones.Queries with strong intention-relevant features need small λ, whereas severe feature interference requires larger λ.
  • 2) Adaptive Spherical Linear Interpolation:: The learnable λ(q0) ∈ (0, 1) determines how strongly the final query aligns with qp, using the angle between normalized q0 and qp on the hypersphere.Its query-dependent learning parallels the adaptive parameterization of A and B.

3) Adaptive Joint Parameterization:

The framework jointly predicts low-rank projections and interpolation from the initial query, coordinating adaptation direction and refinement strength. Structural regularization promotes complementary latent factors and stabilizes query adaptation.

  • Adaptive Joint Parameterization: An MLP jointly predicts low-rank projection matrices A and B and interpolation coefficient λ conditioned on the initial query representation q0.This replaces independent learning of the adaptation parameters.
  • Adaptive Joint Parameterization: Jointly learning (A, B, λ) shares information across parameters to coordinate the adaptation direction and degree of query refinement.A and B define the candidate direction, while λ controls how strongly the query follows it.
  • Regularization: An orthogonality constraint on down-projection matrix A encourages its R latent dimensions to capture complementary, non-redundant semantic factors.A defines the compact latent subspace for extracting intention-relevant information from the original query representation.
  • Regularization: The orthogonality constraint excludes up-projection matrix B because B reconstructs representations in the original space rather than defining the latent intention subspace.The exclusion is specifically tied to B’s reconstruction role.
  • Regularization: A Frobenius norm penalty on both projection matrices regularizes query-adaptation magnitude and stabilizes training.The penalty applies to both A and B.

D. Gradient-Guided Adaptive Sampling · 1) Difficulty Quantification via Retrieval Token Gradients:

FashionLens addresses optimization imbalance in multi-task fashion retrieval by measuring task difficulty through retrieval-token gradients and using smoothed difficulty estimates to guide sampling. This approach complements dataset-scale considerations by emphasizing each task’s real-time alignment difficulty.

  • D. Gradient-Guided Adaptive Sampling: Static dataset-size sampling can under-optimize low-resource tasks because large datasets dominate gradients and ignore task-specific learning difficulty.Fashion retrieval tasks differ substantially in query complexity and modality composition.
  • 1) Difficulty Quantification via Retrieval Token Gradients:: Full-parameter gradients are computationally prohibitive for large-scale MLLMs, motivating retrieval tokens as a compact proxy for task optimization demand.The learnable [RETq] and [RETt] tokens bottleneck multimodal aggregation and query–target alignment.
  • 1) Difficulty Quantification via Retrieval Token Gradients:: Task k’s instantaneous learning difficulty is quantified by summing the gradient magnitudes of its two learnable retrieval tokens at training step t.The signal is derived from the loss for task k at training step t.
  • 1) Difficulty Quantification via Retrieval Token Gradients:: Because mini-batch stochasticity makes instantaneous difficulty noisy, FashionLens maintains a smoothed difficulty estimate G(t)k.The estimate is introduced to stabilize the real-time difficulty signal.
  • 1) Difficulty Quantification via Retrieval Token Gradients:: The smoothed task-difficulty estimate is updated with an Exponential Moving Average (EMA).EMA is used to maintain the difficulty estimate over training steps.
  • 1) Difficulty Quantification via Retrieval Token Gradients:: A greater smoothed difficulty value indicates greater difficulty aligning query and target representations for task k and warrants increased training sampling.Sampling is therefore guided by the task’s estimated alignment difficulty.

2) Size-Aware Probability Refinement:

The method refines task-sampling probabilities by combining gradient-based difficulty with a sublinear dataset-size prior, while guaranteeing every task a minimum sampling chance. It implements this policy through task-level mini-batch sampling to preserve informative within-task contrastive signals.

  • Size-Aware Probability Refinement: Dataset size is incorporated into the task sampling score to counter inflated gradient norms from statistically unstable low-resource tasks.The score balances current optimization difficulty with adequate representation for tasks having more training data.
  • Size-Aware Probability Refinement: The sampling score combines gradient-based difficulty with a sublinear dataset-scale prior, using γ = 0.5 for square-root scaling.The temperature parameter η controls sensitivity to difficulty variations.
  • Size-Aware Probability Refinement: A minimum guarantee mechanism adds a small constant before normalization so every task retains a nonzero sampling probability.This keeps low-difficulty and limited-data tasks represented, sustaining gradient signals and preventing under-training of task-specific representations.
  • Size-Aware Probability Refinement: At each training step, batch-level sampling draws a mini-batch exclusively from the task with the highest sampling probability.This simplifies sampling and avoids trivial cross-task negatives, keeping within-mini-batch InfoNCE contrastive signals informative.

E. Training Objectives … B. Performance Comparison

The paper formulates retrieval training with InfoNCE and regularization losses, then evaluates FashionLens using specified implementation settings against broad, fashion-specific, and fine-tuned baselines. The comparison emphasizes efficiency and dataset- or task-specific strengths.

  • E. Training Objectives: Retrieval optimization uses the widely adopted InfoNCE loss for mini-batches of size N.The retrieval objective is defined for a mini-batch of size N.
  • E. Training Objectives: The final objective combines the retrieval loss with regularization losses weighted by balancing coefficients β1 and β2.The coefficients β1 and β2 control the contribution of the regularization terms.
  • V. EXPERIMENT: The experiments first specify implementation settings and then report results with corresponding analyses.This section covers experimental settings, performance comparisons, and result analysis.
  • A. Experimental Settings: FashionLens uses Qwen3-VL-4B as its MLLM backbone, with low-rank dimensions D = 2560 and d = 32.The backbone is selected for its balance between efficiency and effectiveness.
  • B. Performance Comparison: The baselines comprise general-domain universal multimodal retrievers, fashion-domain vision–language models, and models fine-tuned on U-FIRE.These categories cover models evaluated directly and models adapted specifically to the U-FIRE training set.
  • B. Performance Comparison: General-domain baselines include UniIR, VLM2Vec-V2, MM-Embed, and GME, which retain fashion retrieval capabilities from broad training corpora.The listed models use CLIP-L/14, Qwen2-VL-7B, or LLaVA-NeXT backbones and are evaluated directly on U-FIRE.
  • B. Performance Comparison: Fashion-domain baselines include Fame-ViL, FashionSAP, and DA-Fashion, while fine-tuned baselines include UniIR∗ and LoRA-adapted Qwen3-VL-4B.Fashion-domain models are evaluated without additional fine-tuning; the fine-tuned models represent full-parameter and LoRA-based paradigms.
  • B. Performance Comparison: DA-Fashion achieves the best FashionGen results, likely because FashionGen appears in its training data and Text→Image retrieval dominates its optimization.The comparison passage also notes that UniIR∗ substantially improves over the initial UniIR and that FashionLens uses only a 4B backbone.

C. Ablation Study

The ablation study evaluates FashionLens variants by separately removing or replacing gradient-guided sampling, proposal adaptation, Slerp, size-aware refinement, and instance-level proposal generation. Averaged across training tasks, GGAS provides the most substantial improvement and addresses optimization challenges from data imbalance and differing task difficulties.

  • Variant Design: The ablation compares a Qwen3-VL-4B LoRA baseline using naive q0 and random sampling against variants isolating GGAS, proposal use, interpolation, size refinement, and shared parameters.The baseline uses learnable special tokens and contrastive learning.
  • Variant Design: The proposal-only variant directly uses the intention-oriented adaptation proposal qp, testing whether Slerp is necessary to derive the final query representation q.This variant isolates the contribution of spherical interpolation.
  • Variant Design: The remaining variants replace Slerp with linear interpolation, disable size-aware probability refinement by setting γ = 0, or share proposal-generation parameters across samples.The shared-parameter variant removes the MLP and learns A, B, and λ globally.
  • Key Findings: GGAS yields the most substantial performance improvement among the ablations averaged over all training tasks.It uses gradient norms to prioritize hard tasks.
  • Key Findings: The results indicate that data imbalance and varying task difficulties severely hinder optimization in versatile retrieval.GGAS is presented as mitigating these issues through hard-task prioritization.

D. Visualization and Analysis

The analysis visualizes how FashionLens adapts queries and how task difficulty evolves during training. PGSQC adaptation varies with training-data scale, while GGAS reveals convergent yet task-dependent optimization difficulty.

  • D. Visualization and Analysis: The analysis visualizes intermediate results from FashionLens’s two key components.These components are PGSQC and GGAS.
  • On PGSQC: PGSQC shows a negative correlation between training-data scale and interpolation coefficient λ, even within the same task categories.For Text→Image (Task 1), FashionGen uses 49k samples and exhibits low λ, whereas Shoes uses 2.4k samples and yields much higher λ.
  • On PGSQC: A query representation farther from q0 necessitates a larger λ for further adaptation.The interpolation coefficient λ controls the magnitude of query adaptation.
  • On GGAS: GGAS visualizations show gradient norms generally decrease across datasets during training, as the model gradually converges.Task difficulty is measured by gradient norms.
  • On GGAS: Different rates of gradient decay across tasks confirm that fashion retrieval tasks have inherently different optimization complexity.The visualization tracks task difficulty for each dataset throughout training.

VI. CONCLUSION

FashionLens addresses fragmented fashion retrieval with a unified framework for versatile image retrieval. The paper establishes U-FIRE and introduces PGSQC to support standardized evaluation and diverse retrieval intents.

  • FashionLens is proposed as a unified framework for versatile fashion image retrieval that addresses fragmented retrieval limitations.
  • U-FIRE unifies 15 datasets into a comprehensive benchmark for standardized evaluation.
  • PGSQC dynamically calibrates query representations on the hypersphere to handle diverse retrieval intents.
Loading 2605.22552v1…