Source-linked AI summary
MELT: Improve Composed Image Retrieval via the Modification Frequentation-Rarity Balance Network
Guozhi Qiu, Zhiwei Chen, Zixu Li, Qinlei Huang, Zhiheng Fu, Xuemeng Song, Yupeng Hu
TL;DR
Composed image retrieval must interpret modification text relative to a reference image, while existing methods face rare-semantic neglect and hard-negative interference. MELT balances attention to rare semantics with diffusion-based denoising and reports strong results on FashionIQ and CIRR.
Problem
Method
MELT emphasizes rare modification semantics in multimodal contexts and applies diffusion denoising to high-similarity hard negatives.
Source: §1 · p. 1Results
MELT reports outstanding performance on both datasets, with average FashionIQ R@10 improving by 3.07% over suboptimal results.
Source: §3.2 · p. 4Takeaways & Limitations
The authors conclude that MELT facilitates multimodal composition and matching in CIR.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Composed Image Retrieval (CIR) uses a reference image and a modification text as a query to retrieve a target image satisfying the requirement of ``modifying the reference image according to the text instructions''. However, existing CIR methods face two limitations: (1) frequency bias leading to ``Rare Sample Neglect'', and (2) susceptibility of similarity scores to interference from hard negative samples and noise. To address these limitations, we confront two key challenges: asymmetric rare semantic localization and robust similarity estimation under hard negative samples. To solve these challenges, we propose the Modification frEquentation-rarity baLance neTwork MELT. MELT assigns increased attention to rare modification semantics in multimodal contexts while applying diffusion-based denoising to hard negative samples with high similarity scores, enhancing multimodal fusion and matching. Extensive experiments on two CIR benchmarks validate the superior performance of MELT. Codes are available at https://github.com/luckylittlezhi/MELT.
1. INTRODUCTION
CIR combines a reference image with modification text, whose semantics need not correspond one-to-one with the image. MELT targets rare modification semantics and hard-negative interference, and achieves state-of-the-art results on most metrics on FashionIQ and CIRR.
1. INTRODUCTION
- CIR retrieves target images from a reference image and modification text, which acts as a directive rather than a one-to-one semantic match.The composed feature must account for this semantic asymmetry to align with target images.Source: §1 · p. 1
- Frequency imbalance biases retrieval toward common objects and instructions, while hard negatives and noise can distort similarity rankings.Rare modifications and subtle regions may be neglected; fashion datasets can contain hard negatives with similar appearance or context.Source: §1 · p. 1, Figure 1 · p. 1
- MELT addresses asymmetric rare-semantic localization by locating image regions with maximal text attention and computing modification rarity under text guidance.Its challenge is to amplify infrequent, critical image tokens that occupy minimal areas and can be overwhelmed by high-frequency backgrounds.Source: §1 · p. 1
- MELT addresses hard-negative similarity estimation with multi-step diffusion denoising that progressively refines the initial similarity matrix.The challenge is to reduce noise-driven ranking errors through sequential denoising and robust composed matching.Source: §1 · p. 1, Figure 1 · p. 1
- MELT achieves state-of-the-art results on most metrics on both FashionIQ and CIRR.Its Rarity-Aware Token Refinement module reconstructs image semantics using text-guided rarity, while Diffusion-Based Similarity Denoising iteratively refines the similarity matrix.Source: §1 · p. 1
2. METHODOLOGY
MELT combines rarity-aware token refinement with diffusion-based similarity denoising to emphasize rare modification semantics and improve matching against hard negatives. Its modules construct text-guided composed features and refine similarity scores.
2. METHODOLOGY
- MELT pairs Rarity-Aware Token Refinement with Diffusion-Based Similarity Denoising to focus on rare modification semantics and reduce hard-negative interference.RATR highlights text-relevant image regions and refines local semantics; DSD denoises high-similarity hard negatives to make matching more robust.Source: §2 · p. 1, Figure 2 · p. 2
2.1. Problem Formulation
- CIR seeks an embedding in which the query formed by a reference image and modification text is close to its corresponding target image.The formulation expresses this as F(xr, xm) → F(xt).Source: §2.1 · p. 2
2.2. Rarity-Aware Token Refinement (RATR)
- RATR uses text-to-image attention to select modification-relevant reference tokens, scores rarity using residual energy and cosine similarity, and recalibrates attended tokens for rare modifications.It marks a sample rare when its fitting score exceeds the mean by γ standard deviations, then applies a learnable modification intensity to the selected tokens.Source: §2.2 · pp. 2–3
2.3. Diffusion-Based Similarity Denoising(DSD)
- DSD treats high-similarity hard negatives as noise in the similarity matrix and trains a denoiser to recover the true label distribution using multi-step DDIM [17].For each query, the positive is placed first and the remaining samples are sorted by similarity before denoising.Source: §2.3 · p. 3
3. EXPERIMENTS
MELT is evaluated on FashionIQ and CIRR, where it improves reported retrieval performance; ablations support contributions from rarity-aware token refinement and diffusion-based similarity denoising.
3.1. Experimental settings
- Evaluation uses FashionIQ and CIRR, with CIRR Recall@k, Recallsubset@k, and (R@5 + Rsubset@1)/2; training uses BLIP2 features and AdamW.The setup uses 32 learnable queries, hidden dimension 256, batch size 128, a 1-layer, 12-head attention encoder, and learning rate 1e-5; κ=0.3, λ=0.1, η=0.8, and γ=2.2.Source: §3.1 · p. 3
3.2. Perfermance Comparison
- Compared with suboptimal results, MELT raises FashionIQ average R@10 by 3.07% and R@50 by 2.27%; on CIRR, R@1 improves, R@5/R@10 each gain over 0.5%, and average performance rises 0.45%.FashionIQ’s larger improvement is attributed to rare samples from less common categories and difficult negatives from popular categories; in open-domain datasets, rare samples primarily affect R@1.Source: §3.2 · p. 4
3.3. Ablation Study
- Ablations show that rarity-aware attention, thresholding and token modification, as well as diffusion noise, denoiser loss and distillation, each contribute to performance.Removing rare-sample token modification causes a larger decline than removing attention or the rarity threshold; removing noise causes a slight decline, while removing denoiser loss causes a greater one. FashionIQ drops more than CIRR, and the diffusion module reduces interference from high-frequency hard negatives.Source: Table 2 · p. 4, §3.3 · p. 4
4. CONCLUSION
MELT addresses rare-sample neglect and interference from hard negatives and noise in composed image retrieval. It allocates more attention to rare modification semantics and denoises high-similarity hard negatives, achieving optimal performance across all metrics on two CIR benchmarks.
4. CONCLUSION
- MELT achieved optimal performance across all metrics on two composed image retrieval benchmark datasets.Source: §4 · p. 4
- MELT allocates more attention to rare modification semantics and applies diffusion denoising to high-similarity hard negatives to facilitate multimodal composition and matching.Source: §4 · p. 4
- MELT targets frequency bias causing rare-sample neglect and similarity-score interference from hard negative samples and noise.These limitations pose challenges of asymmetric rare semantic localization and robust similarity estimation under hard negatives.Source: §4 · p. 4