PaperlayerView exact revision on arXiv

Source-linked AI summary

OFFSET: Segmentation-based Focus Shift Revision for Composed Image Retrieval

Zhiwei Chen, Yupeng Hu, Zixu Li, Zhiheng Fu, Xuemeng Song, Liqiang Nie

arXiv:2507.05631v2cs.CV

TL;DR

CIR must combine reference images with modification text despite visual noise and shifting entity relevance. OFFSET addresses this with segmentation-based focus mapping and text-guided focus revision, and experiments on four benchmark datasets report state-of-the-art performance. The method's analysis also identifies a limitation: excessive channels can focus on irrelevant information.

  • Problem

    CIR methods overlook visual inhomogeneity and modification-text priority, causing query-feature degradation and visual focus bias.

  • Method

    OFFSET uses dominant portion segmentation and dual focus mapping for feature extraction, followed by modification-text-guided adaptive focus revision.

  • Results

    OFFSET achieves state-of-the-art performance, supported by extensive experiments on four benchmark datasets.

  • Takeaways & Limitations

    Focus mapping reduces noise interference, while textually guided revision enhances perception of the modification focus in composed features.

  • Takeaways & Limitations

    Using too many channels can focus on irrelevant information and disadvantage retrieval performance.

Abstract

from arXiv · show

Composed Image Retrieval (CIR) represents a novel retrieval paradigm that is capable of expressing users' intricate retrieval requirements flexibly. It enables the user to give a multimodal query, comprising a reference image and a modification text, and subsequently retrieve the target image. Notwithstanding the considerable advances made by prevailing methodologies, CIR remains in its nascent stages due to two limitations: 1) inhomogeneity between dominant and noisy portions in visual data is ignored, leading to query feature degradation, and 2) the priority of textual data in the image modification process is overlooked, which leads to a visual focus bias. To address these two limitations, this work presents a focus mapping-based feature extractor, which consists of two modules: dominant portion segmentation and dual focus mapping. It is designed to identify significant dominant portions in images and guide the extraction of visual and textual data features, thereby reducing the impact of noise interference. Subsequently, we propose a textually guided focus revision module, which can utilize the modification requirements implied in the text to perform adaptive focus revision on the reference image, thereby enhancing the perception of the modification focus on the composed features. The aforementioned modules collectively constitute the segmentatiOn-based Focus shiFt reviSion nETwork (\mbox{OFFSET}), and comprehensive experiments on four benchmark datasets substantiate the superiority of our proposed method. The codes and data are available on https://zivchen-ty.github.io/OFFSET.github.io/

1 Introduction

CIR combines a reference image and modification text to retrieve a target image, but existing methods overlook visual inhomogeneity and text priority. OFFSET addresses these issues with focus-mapping feature extraction and text-guided focus revision, achieving state-of-the-art performance on four benchmark datasets.

  • Task and motivation: CIR uses a reference image and modification text as a multimodal query to retrieve a target image.The modification text conveys the user's retrieval intent.
  • Limitations: Visual inhomogeneity causes dominant and noisy image regions to be treated equally, which can degrade query features and composition accuracy.Dominant regions may correlate with modification requirements, whereas irrelevant regions introduce noise during feature extraction.
  • Limitations: Existing methods overlook modification-text priority, creating a visual focus bias when text changes which image entities are relevant.An entity such as a tree may become dominant when the modification text makes it important.
  • Proposed approach: OFFSET combines dominant portion segmentation and dual focus mapping into a feature extractor, then performs textually guided adaptive focus revision.The method uses a role-supervised signal for segmentation and modification semantics to revise reference-image focus.
  • Evaluation: OFFSET achieves state-of-the-art performance, with experiments on four benchmark datasets validating its effectiveness and superiority.The experiments also evaluate the proposed model's components.

2 Related Work

Related work situates OFFSET at the intersection of Composed Image Retrieval and semantic segmentation for mask generation. The approach uses flexible mask generation to guide reference images and modification texts toward dominant portions.

  • Composed Image Retrieval: CIR integrates reference-image and modification-text semantics to retrieve a target image through flexible multimodal interaction.Existing approaches use either conventional feature extractors or vision-language models.
  • Semantic segmentation: Semantic segmentation generates object masks used in downstream tasks such as image editing, scene understanding, and object removal.Mask-generation methods can support limited-data or unseen-class scenarios.
  • OFFSET: OFFSET's architecture contains dominant portion segmentation, dual focus mapping, and textually guided focus revision, with the first two forming its feature extractor.The modules are presented as three key components of the proposed model.
  • Semantic segmentation: OFFSET uses CLIPSeg-based mask generation to guide reference images and modification texts toward dominant portions.This design matches the flexible requirements of the CIR task.

3 OFFSET

OFFSET combines segmentation-based focus mapping with textually guided focus revision to construct multimodal features for composed image retrieval. It segments dominant regions, maps visual and textual focus, and revises reference-image focus using modification semantics.

  • Problem Formulation: OFFSET addresses CIR by learning embeddings that bring a multimodal query and its target image close in a shared metric space.The query consists of a reference image and modification text.
  • Dominant Portion Segmentation: Dominant portion segmentation uses BLIP-2 captions and CLIPSeg to highlight dominant regions while masking visual noise.Captions are generated for reference and target images before segmentation.
  • Dual Focus Mapping: Dual focus mapping comprises Visual Focus Mapping and Textual Focus Mapping, guiding visual and textual features toward query-relevant portions.Visual mapping uses modality-aligned images, while textual mapping uses modification-text features.
  • Multi-Grained Focus Projection: Multi-Grained Focus Projection aligns focus-mapping outputs across granularities and channels to produce reference, target, and modification focused features.The resulting focused features use uniform focus channels before concatenation.
  • Textually Guided Focus Revision: Textually guided focus revision uses modification semantics to revise reference-image focus and move the composed feature toward the target-focused feature.The module addresses inaccurate focus localization caused by modality-independent visual mapping.

4.1 Experimental Settings

The experiments evaluate OFFSET on FashionIQ, Shoes, and CIRR using standard Recall@k protocols. Training uses a CLIP ViT-H/14 backbone with AdamW and fixed focus-related hyperparameters.

  • Datasets: Evaluation covers FashionIQ, Shoes, and CIRR, including two fashion-domain datasets and one open-domain dataset.These datasets are selected as public benchmarks for CIR evaluation.
  • Implementation Details: OFFSET uses CLIP ViT-H/14, AdamW, a 1e-4 initial learning rate, batch size 16, embedding dimension D=1024, and four focus channels.The CLIP learning rate is set to 1e-6, and the temperature factor τ is 0.1 across datasets.
  • Evaluation Metrics: Recall@k is the primary metric, with dataset-specific values of k and category-wise or subset averages.FashionIQ uses R@10 and R@50; Shoes uses k=1, 10, 50; CIRR includes R@1, 5, 10, 50 and subset metrics.

4.2 Performance Comparison

OFFSET consistently outperforms the compared baselines on FashionIQ, Shoes, and CIRR, with improvements across fashion-specific and open-domain retrieval tasks.

  • OFFSET achieves relative improvements of 1.00% on FashionIQ-Avg R@10, 0.89% on CIRR R_subset@1, and 0.79% on Shoes average over the best baselines.These comparisons cover fashion-domain and open-domain datasets.
  • VLP-based models generally outperform traditional feature-extraction models, supporting their use for visual-textual semantic alignment in CIR.This observation is reported alongside the cross-dataset comparison.
  • The comparison covers FashionIQ and Shoes tables using R@k performance metrics, with overall best results distinguished from the best baseline results.Table 1 evaluates FashionIQ and Table 2 evaluates Shoes; both use R@k(%).
  • OFFSET outperforms previous models on fashion-domain datasets and achieves state-of-the-art results on the open-domain CIRR dataset.The reported results indicate stable performance beyond domain-specific data.

4.3 Ablation Studies

Ablation studies show that OFFSET’s dual focus mapping, textually guided focus revision, and optimization functions each contribute to its performance across FashionIQ, Shoes, and CIRR.

  • G1: Ablation on Dual Focus Mapping: Removing dual focus mapping produces the worst result in its ablation group, while removing either visual or textual focus mapping also degrades performance.The results support jointly focusing visual and textual features on query-relevant portions.
  • The ablation study reports results for OFFSET variants across FashionIQ, Shoes, and CIRR, with Δ marking performance degradation relative to the per-column baseline.Table 4 organizes the module and optimization-function comparisons.
  • G1: Ablation on Dual Focus Mapping: Replacing multi-grained focus projection with average pooling reduces performance, highlighting its role in aligning local and global focus.The comparison is reported for derivative D#(4).
  • G2: Ablation on Textually Guided Focus Revision: Removing target-image visual focus mapping or multi-grained focus projection lowers performance, indicating that both modules contribute during textually guided focus revision.These comparisons correspond to derivatives D#(5) and D#(6).
  • G2: Ablation on Textually Guided Focus Revision: Removing focus revision creates a significant performance gap relative to OFFSET, demonstrating the effectiveness of revising multimodal-query focus.This comparison corresponds to derivative D#(7).
  • G3: Ablation on Optimization Functions: Removing batch-based classification loss reduces performance, while removing focus regularization harms focus consistency between the multimodal query and target.The optimization-function ablations are reported in G3 across the three benchmark datasets.

4.4 Further Analysis

Further analysis examines OFFSET’s focus-channel sensitivity, inference efficiency, and qualitative retrieval behavior. Performance peaks at an intermediate focus-channel setting, while OFFSET reduces inference time versus DQU-CIR and better matches modification details than its ablated variant.

  • Sensitivity to Focus Channel Number P: OFFSET performance generally improves as focus channel number P increases, then drops for larger values across FashionIQ, Shoes, and CIRR.A moderate number of focus channels captures diverse image features, whereas too many can focus on irrelevant information.
  • Efficiency Analysis: OFFSET decreases inference time by 40.27% compared with DQU-CIR while outperforming it on retrieval performance across the evaluated datasets.The comparison uses a single A40 GPU and includes inference time per sample, retrieval performance, and auxiliary models.
  • Efficiency Analysis: OFFSET achieves a 9.98% improvement on CIRR-Avg. over DQU-CIR while avoiding excessive additional inference overhead.CIRR-Avg. is reported as the especially improved retrieval result in the efficiency comparison.
  • Qualitative Analysis: In qualitative examples, OFFSET ranks target images first, whereas its w/o FM variant can rank a target outside the top five on CIRR.OFFSET also preserves unmodified regions and captures nuanced textual requirements more accurately than w/o FM.
  • Qualitative Analysis: OFFSET recognizes detailed modification requirements, including both transparent heel and sole regions, while w/o FM focuses only on the sole.This example illustrates the difference in interpreting the relationship between the reference image and modification text.

5 Conclusion

The conclusion identifies visual inhomogeneity and text-priority as neglected CIR phenomena and presents OFFSET as a combined focus-mapping and text-guided revision solution. Experiments on four benchmark datasets substantiate its efficacy, while future work targets additional downstream tasks.

  • Conclusion: Visual inhomogeneity degrades query features, while text-priority in multimodal queries produces visual focus bias in CIR.The conclusion frames these two phenomena as seriously neglected by the CIR community.
  • Conclusion: OFFSET uses focus mapping-based feature extraction to identify dominant regions, guide visual and textual feature extraction, and reduce noise interference.Its feature extractor addresses inhomogeneity through dominant-region identification and focus guidance.
  • Conclusion: OFFSET applies textually guided focus revision to adaptively revise the reference-image focus according to modification semantics.The revision is intended to enhance perception of the modification focus in composed features.
  • Conclusion: Experiments on four benchmark datasets substantiate the efficacy of OFFSET.The conclusion reports this as the overall experimental support for the proposed method.
  • Conclusion: Future work intends to extend OFFSET to downstream tasks such as information detection and prediction.The stated future directions remain within the broader multimodal application scope described by the paper.
Loading 2507.05631v2…