PaperlayerView exact revision on arXiv

Source-linked AI summary

FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrieval

Zixu Li, Zhiheng Fu, Yupeng Hu, Zhiwei Chen, Haokun Wen, Liqiang Nie

arXiv:2503.21309v1cs.CVcs.AI

TL;DR

Existing CIR datasets often use coarse modification text that misses fine-grained retrieval intent, causing imprecise positives and ambiguity among similar images. The paper builds fine-grained datasets through automated and manual annotation and introduces FineCIR for explicit modification parsing. FineCIR outperforms existing models on both fine-grained and traditional CIR benchmarks.

  • Problem

    CoarseMT inadequately captures fine-grained retrieval needs, while overlooked differences and visually similar candidates challenge precise CIR retrieval.

  • Method

    The paper constructs Fine-FashionIQ and Fine-CIRR with a fine-grained annotation pipeline and proposes FineCIR to explicitly parse modification semantics.

  • Results

    FineCIR outperforms existing models on both fine-grained and traditional CIR benchmark datasets.

  • Takeaways & Limitations

    Fine-grained datasets and explicit semantic parsing support CIR systems in capturing detailed modification intent across fine and coarse granularities.

Abstract

from arXiv · show

Composed Image Retrieval (CIR) facilitates image retrieval through a multimodal query consisting of a reference image and modification text. The reference image defines the retrieval context, while the modification text specifies desired alterations. However, existing CIR datasets predominantly employ coarse-grained modification text (CoarseMT), which inadequately captures fine-grained retrieval intents. This limitation introduces two key challenges: (1) ignoring detailed differences leads to imprecise positive samples, and (2) greater ambiguity arises when retrieving visually similar images. These issues degrade retrieval accuracy, necessitating manual result filtering or repeated queries. To address these limitations, we develop a robust fine-grained CIR data annotation pipeline that minimizes imprecise positive samples and enhances CIR systems' ability to discern modification intents accurately. Using this pipeline, we refine the FashionIQ and CIRR datasets to create two fine-grained CIR datasets: Fine-FashionIQ and Fine-CIRR. Furthermore, we introduce FineCIR, the first CIR framework explicitly designed to parse the modification text. FineCIR effectively captures fine-grained modification semantics and aligns them with ambiguous visual entities, enhancing retrieval precision. Extensive experiments demonstrate that FineCIR consistently outperforms state-of-the-art CIR baselines on both fine-grained and traditional CIR benchmark datasets. Our FineCIR code and fine-grained CIR datasets are available at https://github.com/SDU-L/FineCIR.git.

1. Introduction

CIR retrieves target images from a reference image and modification text, but coarse modification text misses fine-grained intent. The paper addresses imprecise positives and visual ambiguity through fine-grained annotation and explicit semantic parsing.

  • CIR uses a reference image and modification text to retrieve an image matching the intended alterations.
  • Existing open-domain and fashion-domain datasets mainly use CoarseMT, which inadequately conveys fine-grained retrieval needs.
  • Uncaptured visual differences create imprecise positive samples that can accept unintended modifications and reduce retrieval accuracy.
  • Short, coarse descriptions increase ambiguity among visually similar fashion images and burden users with filtering or repeated queries.
  • The proposed pipeline replaces CoarseMT with FineMT that comprehensively describes reference-target differences and produces Fine-FashionIQ and Fine-CIRR.
  • FineCIR explicitly parses entities, attributes, and relations, then aggregates entity tokens to align visual entities with modification semantics.
  • Experiments on four benchmark datasets report superior performance on both traditional and fine-grained CIR.

2. Related Work

Composed query datasets support retrieval from reference media and modification text, but most rely on coarse descriptions. The paper positions fine-grained annotation as a response to costly triplet labeling and limited dataset development.

  • Composed query datasets represent a multimodal query and target as a triplet for CIR, ZS-CIR, and CoVR tasks.
  • Most existing studies use CoarseMT, which lacks precision for fine-grained retrieval.
  • Existing CIR methods include traditional models that separately process modalities and VLP-based models that embed multimodal queries jointly.
  • High triplet-labeling costs have led some research toward ZS-CIR, which commonly trains text inversion networks on image-caption pairs.
  • The paper introduces an automated-and-manual annotation pipeline to add fine-grained annotations to existing CIR datasets.

3. Fine-grained CIR data annotation pipeline

The annotation pipeline selects suitable image pairs, generates and refines FineMT, and performs quality checks. Applied to FashionIQ and CIRR, it yields Fine-FashionIQ and Fine-CIRR while controlling annotation quality and text length.

  • The pipeline has three stages: Data Selection, Dataset Construction and Refinement, and Quality Check.
  • It applies to FashionIQ and CIRR, producing Fine-FashionIQ and Fine-CIRR for fine-grained CIR model development.
  • Existing image pairs are retained while FineMT is generated to describe reference-target differences, reducing construction overhead.
  • GPT-4o and human verification assess image-pair relevance, retrieval usefulness, and whether modifications are sufficient and meaningful.
  • BLIP-3 generates initial FineMT from image pairs and evaluation results, after which Llama3.2 reviews outputs for hallucinations.
  • FineMT introduces two concerns: text may dominate the reference image, and semantic image-pair similarity may make the reference alone sufficient for retrieval.

4. Method

FineCIR explicitly parses fine-grained modification text into structured semantics and aligns those semantics with visual entities during multimodal query composition. Its Exparse and Encompose components produce entity-aware composed representations for matching reference-image modifications to target images.

  • Explicit Modification Parsing: FineCIR uses Explicit Modification Parsing to convert FineMT into scene-graph-based entity-attribute-relation representations.The scene graph parser structures fine-grained semantics before subject-centric aggregation.
  • Explicit Modification Parsing: Subject-centric aggregation consolidates each subject’s attributes and associated object relations into its semantic representation.Object tokens integrate subject-object relation semantics, after which Graph Attention Networks aggregate the resulting information.
  • Entity-guided Composition Learning: Entity-guided Composition Learning addresses indeterminate relationships in which entities and modification clauses may correspond one-to-many in either direction.The module uses explicitly aggregated entity tokens to guide alignment between reference-image visual entities and FineMT semantics.
  • Entity-guided Composition Learning: Learnable queries, entity tokens, and FineMT features enter a Q-Former with reference-image features as cross-modal input to produce composed tokens.The composed token is the Q-Former CLS output, while target-image features yield target entity tokens for alignment.
  • Entity-guided Composition Learning: FineCIR applies batch-based classification to align composed tokens with corresponding target entity tokens, with temperature τ and batch size B controlling the matching loss.The final loss optimizes FineCIR parameters Θ∗.

5. Experiments

Experiments evaluate FineCIR on fine-grained and traditional CIR datasets using recall-based metrics, baselines, ablations, and qualitative comparisons. FineCIR achieves strong performance, including a 6.6% Avg.R@10 improvement on Fine-FashionIQ, while scene graphs and entity-guided components support fine-grained semantic understanding.

  • Evaluation: FineCIR is evaluated on Fine-FashionIQ, Fine-CIRR, FashionIQ, and CIRR using R@K-based metrics and comparisons with VLP and open-source CIR baselines.Fine-FashionIQ and FashionIQ use R@10 and R@50; Fine-CIRR and CIRR additionally use Rsubset@k and averaged metrics.
  • Quantitative results: 6.6% improvement on Avg.R@10 is achieved by FineCIR over baselines on Fine-FashionIQ.The reported advantage is attributed to the Explicit Modification Parsing module’s handling of complex FineMT semantics.
  • Method comparison: FineCIR achieves the best performance on all reported metrics and retains the best performance on traditional CIR datasets.The results are reported across both fine-grained and coarse-grained modification settings.
  • Ablation Study: Removing the scene graph causes the largest ablation performance drop, while subject-centric aggregation also benefits fine-grained subject-relation learning.Alternative scene-graph representations produce only slight drops relative to the complete model.
  • Case Study: FineCIR retrieves the target at Top-1 in qualitative Fine-FashionIQ and Fine-CIRR examples, compared with SPRC ranks of Top-4 and Top-2.The examples involve fine-grained modifications such as subject repositioning and background changes.
  • Additional analyses: Appendix experiments compare FineCIR’s training and inference efficiency with state-of-the-art methods and provide additional case studies.

6. Conclusion

The paper concludes that its annotation pipeline produces fine-grained CIR datasets and that FineCIR explicitly parses modification semantics. Experiments show improved performance on both fine-grained and traditional CIR benchmarks.

  • The annotation pipeline and two resulting datasets reduce imprecise positive samples and improve capture of user modification intent.
  • FineCIR explicitly parses modification semantics and outperforms existing models on fine-grained and traditional CIR datasets.

Supplementary Material

The supplementary material documents dataset construction, dataset statistics, comparisons with original CIR datasets, and additional methodological details. It also includes prompts and analyses supporting FineMT generation and quality checking.

  • The supplement explains differences between fine-grained and traditional CIR and details construction of the two proposed datasets.
  • It reports dataset statistics and comparisons between Fine-FashionIQ, Fine-CIRR, and the original CIR datasets.
  • Additional sections provide LLM prompts, FineMT-generation details, and changes in query counts during dataset construction.

A.1. FineMT in Solving CoarseMT’s Problems

FineMT addresses coarse modification text by describing image differences more comprehensively, reducing imprecise positives and improving discrimination among visually similar images. Examples across fashion and open-domain datasets show improved target rankings after FineMT relabeling.

  • Reducing imprecise positive samples: FineMT specifies detailed differences between reference and target images, reducing the likelihood of incorrectly labeled positive samples.More comprehensive modification requirements help distinguish true positives from suitable but incorrectly labeled candidates.
  • Fashion-domain examples: FineMT relabeling improves Fine-FashionIQ retrieval by identifying a necklace detail and moving one target from second to first place.
  • Open-domain examples: FineMT adds details such as standing in water, a dirt path, and background trees to open-domain buffalo queries.These additions help the query distinguish visually similar samples.
  • Open-domain examples: FineMT relabeling moves a Fine-CIRR target from fifth to first place and converts previously ambiguous cases into true negatives.
  • Coarse- versus fine-grained CIR: Fine-grained CIR uses detailed, often multi-sentence descriptions to express multidimensional modifications more effectively than coarse-grained CIR.The comparison links detailed descriptions with fewer imprecise positives and stronger certainty in capturing user intent.
  • Annotation process: The annotation process uses multimodal checking and query-guided refinement to assess image pairs and keep FineMT relevant to the reference image.The refinement step inputs the reference image together with FineMT to remove irrelevant or hallucinated content.
  • Annotation process: FineMT descriptions may be compressed when they exceed 77 tokens, retaining key information while reducing redundancy.

B.2. Analysis of Prompts for FineMT Generation

The FineMT pipeline combines BLIP-3 prompting with LLM-assisted evaluation, refinement, compression, and human checks to produce detailed modification descriptions. Prompt (c) was selected because it generated the most precise and comprehensive texts.

  • B.2. Analysis of Prompts for FineMT Generation: BLIP-3 prompts were designed to add image-pair details omitted from coarse modification text, including changes involving multiple objects.FineMT aims to describe modifications comprehensively rather than focusing on a single object.
  • B.2. Analysis of Prompts for FineMT Generation: Prompt (a), which lacked LLM evaluation, produced limited-detail texts that failed to capture comprehensive modifications.
  • B.2. Analysis of Prompts for FineMT Generation: The MLLM-and-human check evaluates whether image contents are related and whether the reference provides useful information for modification.
  • B.2. Analysis of Prompts for FineMT Generation: Query-guided refinement removes hallucinated or irrelevant content from FineMT when it does not align with the reference image.
  • B.2. Analysis of Prompts for FineMT Generation: LLM-assisted prompts (b) and (c) captured and elaborated fine-grained modifications from diverse perspectives.The comparison attributes their stronger detail to incorporating LLM evaluation.
  • B.2. Analysis of Prompts for FineMT Generation: Prompt (c) was selected because its generated texts were more precise and comprehensive across perspectives and details.
  • B.2. Analysis of Prompts for FineMT Generation: Fine-FashionIQ used additional prompts directing attention to garment components, holistic clothing interpretation, and reference-target comparison.The prompts address details such as straps and sleeve lengths.
  • B.3. Query Number Changes throughout the Construction: The construction process statistically tracked query-count changes for Fine-CIRR and Fine-FashionIQ through multiple checking and filtering steps.The query-count changes are presented as line graphs in Figures 13 and 14.

C.1. Dataset Details

Fine-FashionIQ and Fine-CIRR refine FashionIQ and CIRR into fine-grained datasets through checking and filtration. The resulting datasets retain detailed modification text while reducing unsuitable queries and controlling text length.

  • Fine-FashionIQ: Fine-FashionIQ is derived from FashionIQ’s fashion-domain data, covering 77,684 images across Dresses, Shirts, and Tops&Tees.
  • Dataset Statistics: The dataset statistics are summarized in Table 4, which compares fine-grained datasets with FashionIQ and CIRR using average modification-text length.
  • Fine-CIRR: Fine-CIRR is derived from CIRR’s 21,552 real images and includes a visually similar negative-image subset for distinguishing false negatives.
  • Dataset Statistics: Iterative checking and filtration produced fewer but higher-quality queries because some original annotations were unsuitable for fine-grained dataset construction.
  • Dataset Statistics: FineMT is longer than the original modification text because it provides finer-grained descriptions.
  • Dataset Statistics: Modification-text length was controlled to prevent text from overpowering visual information and to remain compatible with CLIP’s 77-token input limit.

D. Detailed Comparison on Traditional CIR Datasets

FineCIR is evaluated against multiple baselines on traditional FashionIQ and CIRR, where it demonstrates strong performance and captures modification semantics even with coarse-grained text.

  • Traditional CIR Benchmark Evaluation: FineCIR achieves outstanding performance on both FashionIQ and CIRR, capturing essential modification semantics even in coarse-grained modification text.The comparison is reported in Table 5 using R@K metrics.

E. Computation Cost Analysis

FineCIR prioritizes retrieval accuracy over minimum computational cost, accepting modest overhead for stronger fine-grained retrieval. Qualitative examples likewise show more accurate rankings than SPRC.

  • Computation Cost: The computation comparison measures training time, single-sample test time, and GPU memory on one NVIDIA A40 GPU.
  • Computation Cost: FineCIR’s w/o SG setting provides the best training and testing efficiency among the compared settings while achieving higher recall than Candidate and SPRC.
  • Computation Cost: FineCIR accepts modest computational overhead for up to an 11.4% absolute recall improvement over SPRC on fine-grained CIR benchmarks.
  • Qualitative Case Study: In Fine-FashionIQ examples, FineCIR retrieves the target at Top-1 by capturing detailed changes to the dress, including neckline, sleeves, skirt, and heels.
  • Qualitative Case Study: For an open-domain example, FineCIR ranks the target at Top-1 for person-position and background changes, whereas SPRC ranks it at Top-2.
  • Qualitative Case Study: For dog-position, posture, and background modifications, FineCIR ranks the target at Top-1 while SPRC places it outside the Top 5.
  • Qualitative Case Study: Across fashion and open-domain tasks, the qualitative results indicate that FineCIR more effectively captures and prioritizes fine-grained modification semantics than SPRC.
Loading 2503.21309v1…