Source-linked AI summary
MELT: Improve Composed Image Retrieval via the Modification Frequentation-Rarity Balance Network
Guozhi Qiu, Zhiwei Chen, Zixu Li, Qinlei Huang, Zhiheng Fu, Xuemeng Song, Yupeng Hu
TL;DR
Composed Image Retrieval must align a reference image and modification text despite frequency bias and hard-negative interference. MELT targets these problems with rarity-aware token refinement and diffusion-based similarity denoising, achieving strong results across FashionIQ and CIRR.
Problem
Existing CIR methods neglect rare modification semantics because of frequency bias and suffer similarity interference from hard negatives and noise.
Method
MELT uses Rarity-Aware Token Refinement and Diffusion-Based Similarity Denoising to emphasize rare semantics and refine similarity estimates.
Results
MELT achieves state-of-the-art results on most metrics across the FashionIQ and CIRR benchmarks.
Takeaways & Limitations
MELT provides a CIR approach that combines rare-semantic emphasis with denoised similarity estimation for multimodal matching.
Abstract
from arXiv · showhide
Composed Image Retrieval (CIR) uses a reference image and a modification text as a query to retrieve a target image satisfying the requirement of ``modifying the reference image according to the text instructions''. However, existing CIR methods face two limitations: (1) frequency bias leading to ``Rare Sample Neglect'', and (2) susceptibility of similarity scores to interference from hard negative samples and noise. To address these limitations, we confront two key challenges: asymmetric rare semantic localization and robust similarity estimation under hard negative samples. To solve these challenges, we propose the Modification frEquentation-rarity baLance neTwork MELT. MELT assigns increased attention to rare modification semantics in multimodal contexts while applying diffusion-based denoising to hard negative samples with high similarity scores, enhancing multimodal fusion and matching. Extensive experiments on two CIR benchmarks validate the superior performance of MELT. Codes are available at https://github.com/luckylittlezhi/MELT.
1 School of Software, Shandong University, 2 Southern University of Science and Technology
The paper is indexed under Composed Image Retrieval, Rarity Measurement, and Linguistic-Visual Retrieval.
- The paper concerns Composed Image Retrieval.
- The paper addresses Rarity Measurement.
- The paper concerns Linguistic-Visual Retrieval.
1. INTRODUCTION
CIR retrieves target images by combining a reference image with modification text, but existing methods neglect rare modification semantics and suffer from noisy hard negatives. MELT addresses these issues through rarity-aware multimodal processing and diffusion-based similarity denoising.
- CIR combines a reference image and modification text to retrieve a target satisfying the textual modification.
- Frequency imbalance causes existing methods to favor common objects and instructions while neglecting rare modification semantics and subtle regions.
- Hard negative samples and noise can distort similarity matrices and produce ranking errors during single-pass scoring.
- Rare semantic localization is difficult because critical modification cues may occupy small image regions overwhelmed by frequent backgrounds.
- MELT emphasizes rare modification semantics and applies diffusion denoising to high-similarity hard negatives for multimodal fusion and matching.Its modules include rarity-aware token refinement and diffusion-based similarity denoising.
2. METHODOLOGY
MELT combines rarity-aware token refinement with diffusion-based similarity denoising to improve composed feature construction and matching in CIR. Its modules localize modification-relevant image semantics, identify rare modifications, and reduce hard-negative noise in similarity matrices.
- MELT architecture: MELT uses Rarity-Aware Token Refinement and Diffusion-Based Similarity Denoising to improve multimodal composition and matching.The first module refines image semantics, while the second denoises similarity scores affected by hard negatives.
- Feature extraction and alignment: Q-Former and MLP layers extract and align reference-image and modification-text features before multimodal interaction.The target image is processed into a corresponding feature representation for retrieval matching.
- Rarity-Aware Token Refinement: Text-to-image cross-attention selects the top-k reference-image tokens most correlated with the modification requirement.Modification text serves as Query, while reference-image features serve as Key and Value; the selected tokens are weighted to form a modification-focused feature.
- Rarity-Aware Token Refinement: Rarity scoring combines residual energy and cosine similarity, using EMA-estimated residual statistics to identify samples whose fitting score exceeds a threshold.Samples satisfying R > µR + γσR trigger correction of the most modification-attended image tokens before composed-feature construction.
- Diffusion-Based Similarity Denoising: Diffusion-Based Similarity Denoising treats high-similarity hard negatives as noise and progressively purifies the similarity matrix toward its ground-truth label distribution.The method reorders similarities with the positive sample first, constructs a forward diffusion process, and applies DDIM-guided denoising using composed and target features.
- Training objectives: MELT jointly optimizes denoising and retrieval matching with knowledge distillation and batch-based classification losses.The classification objective pulls composed features closer to their corresponding target features, while the overall objective includes learnable parameters and hyperparameters κ and λ.
3. EXPERIMENTS
MELT is evaluated on the CIRR and FashionIQ benchmarks, where it improves retrieval performance over baselines and benefits from all tested components. The larger FashionIQ improvement is attributed to its greater prevalence of rare and hard-negative samples.
- Experimental Settings: MELT is evaluated on open-domain CIRR and fashion-domain FashionIQ using Recall-based retrieval metrics.CIRR reports Recall@k, Recallsubset@k, and an average metric; the model uses BLIP2 features and AdamW training.
- Performance Comparison: 3.07% average R@10 and 2.27% average R@50 improvements are reported on FashionIQ over suboptimal results.On CIRR, R@1, R@5, and R@10 each improve by over 0.5%, while average performance increases by 0.45%.
- Performance Comparison: FashionIQ shows larger gains than CIRR because it contains more rare modification samples and hard negative samples.The comparison passage links the dataset difference to stronger semantic denoising effects and greater improvement in FashionIQ.
- Ablation Study: Removing rarity-aware attention reduces performance, showing that attention helps identify text-guided modification regions and measure modification rarity.Removing rare-sample optimization causes a larger decline than removing attention or the rarity threshold.
- Ablation Study: Removing Gaussian noise, semantic-denoiser loss, or diffusion-based similarity denoising lowers performance, supporting the contribution of each denoising component.The ablation results associate these declines with noise estimation, semantic-noise removal, and reduced interference from high-frequency hard negatives.
4. CONCLUSION
MELT addresses rare-sample neglect and hard-negative interference in composed image retrieval by emphasizing rare modification semantics and denoising high-similarity hard negatives. It achieves optimal performance across all metrics on two CIR benchmarks.
- Conclusion: MELT targets frequency bias and hard-negative noise by balancing attention toward rare modification semantics and denoising high-similarity negatives.The approach is designed to improve multimodal composition and matching.
- Conclusion: MELT achieves optimal performance across all metrics on two CIR benchmark datasets.