PaperlayerView exact revision on arXiv

Source-linked AI summary

TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval

Zixu Li, Yupeng Hu, Zhiheng Fu, Zhiwei Chen, Yongqi Li, Liqiang Nie

arXiv:2604.21806v2cs.CV

TL;DR

CIR research has largely relied on short modification texts, leaving insufficient entity coverage and clause-entity misalignment for multi-modification queries. The paper constructs M-FashionIQ and M-CIRR and proposes TEMA, which combines text parsing with entity mapping. TEMA outperforms previous methods across original and multi-modification scenarios, while the datasets are designed for practical realism rather than performance gains alone.

  • Problem

    Conventional CIR uses short modification texts that cover few salient changes, creating insufficient entity coverage and clause-entity misalignment in practical multi-modification retrieval.

  • Method

    The paper constructs M-FashionIQ and M-CIRR and proposes TEMA, which uses an MMT Parsing Assistant and entity mapping for multi-modification CIR while accommodating simple modifications.

  • Results

    TEMA outperformed previous methods in both original and multi-modification scenarios across the evaluated CIR datasets.

  • Takeaways & Limitations

    The datasets and TEMA advance CIR toward real-world use by supporting instruction-rich multi-modification retrieval alongside original scenarios.

  • Takeaways & Limitations

    The longer annotations increase the difficulty of understanding modification intentions and do not necessarily lead to higher retrieval performance.

Abstract

from arXiv · show

Composed Image Retrieval (CIR) is an important image retrieval paradigm that enables users to retrieve a target image using a multimodal query that consists of a reference image and modification text. Although research on CIR has made significant progress, prevailing setups still rely simple modification texts that typically cover only a limited range of salient changes, which induces two limitations highly relevant to practical applications, namely Insufficient Entity Coverage and Clause-Entity Misalignment. In order to address these issues and bring CIR closer to real-world use, we construct two instruction-rich multi-modification datasets, M-FashionIQ and M-CIRR. In addition, we propose TEMA, the Text-oriented Entity Mapping Architecture, which is the first CIR framework designed for multi-modification while also accommodating simple modifications. Extensive experiments on four benchmark datasets demonstrate that TEMA's superiority in both original and multi-modification scenarios, while maintaining an optimal balance between retrieval accuracy and computational efficiency. Our codes and constructed multi-modification dataset (M-FashionIQ and M-CIRR) are available at https://github.com/lee-zixu/ACL26-TEMA/.

1 Introduction

CIR combines a reference image with modification text, but conventional short texts inadequately support multiple entities and clauses. The paper addresses these limitations with multi-modification datasets and TEMA, a framework designed for both simple and complex modifications.

  • CIR retrieves target images from a reference image plus modification text, using visual priors and textual instructions to express retrieval intent.The reference supplies appearance, layout, and style, while the text specifies how to modify it.
  • Short modification texts create insufficient entity coverage because detailed descriptions dominate while explicit references to modified entities remain sparse.Models may therefore focus on salient regions and omit other entities in multi-entity queries.
  • Multiple modification clauses can constrain the same entity, creating clause-entity misalignment in fine-grained retrieval settings.This setting requires preserving the subject and style while satisfying several attribute and relation constraints.
  • Existing CIR baselines show a pronounced performance drop on multi-modification queries when trained only on original FashionIQ.The paper attributes this likely to missing multi-modification annotations and the two identified limitations.
  • M-FashionIQ and M-CIRR replace short texts with human-verified, instruction-intensive multi-modification texts that expose multiple entities and clauses.The datasets retain the original reference and target images and evaluation protocols while providing denser training signals and more realistic tests.
  • TEMA combines a training-time MMT Parsing Assistant with an entity-mapping module that aggregates clauses and aligns them with visual entities.The Parsing Assistant is disabled during inference, while the entity-mapping module addresses multi-clause alignment.

2 Related Work

Prior CIR work includes traditional, vision-language, and newer foundation-model approaches, but existing studies do not explicitly model multi-entity to multi-clause alignment. TEMA targets this gap directly.

  • Existing CIR methods include traditional models and vision-language-pretrained models for retrieving images from reference images and modification text.
  • Some prior studies examine multi-object interference, while others construct training data without explicitly considering multi-modification requirements.The related work therefore distinguishes object interference from the broader multi-clause setting addressed here.
  • TEMA explicitly models multi-entity to multi-clause alignment through multi-modification text, parsing assistance, and entity mapping.

3 Multi-Modification CIR Datasets Construction

The paper constructs M-FashionIQ and M-CIRR by expanding established CIR datasets with instruction-intensive multi-modification texts. Automatic generation is combined with tailored prompting, manual review, and content filtering.

  • The construction pipeline is intended to produce benchmarks that better reflect nuanced user requests and practical multi-modification retrieval.
  • The proposed benchmarks are M-FashionIQ for fashion retrieval and M-CIRR for open-domain retrieval, both built from classic CIR datasets.The construction goal is closer alignment with practical scenarios rather than simply higher benchmark performance.
  • The pipeline replaces original short modification texts with MLLM-generated multi-modification texts while retaining the original reference and target images.The resulting texts explicitly represent multiple entities and clauses.
  • Dataset-specific prompts emphasize clothing attributes for FashionIQ and distinct objects in open-domain CIRR scenes.
  • A hybrid quality-control process uses textual manual review, machine assistance, and content filtering to reduce hallucinations and verify image-faithful modifications.Review covers consistency, accuracy, diversity, and quality before checking correspondence with the associated images.

4 Method

TEMA addresses multi-modification CIR by parsing modification texts for entity coverage and mapping multiple clauses to their corresponding entities. Its architecture combines a training-only Parsing Assistant with textual and visual entity mapping, then composes entity features with global multimodal features.

  • Motivation: TEMA targets insufficient entity coverage and clause–entity misalignment in multi-modification CIR.The framework is designed to preserve modified-entity information and align multiple clauses with the entities they describe.
  • MMT Parsing Assistant: The MMT Parsing Assistant uses an LLM-based summarizer and Consistency Detector to extract modified entities and verify summary coverage during training.The summary is iteratively refined when entities are missing or extraneous, and the assistant is disabled during inference.
  • Entity Mapping: MMT-oriented Entity Mapping extracts one-to-many correspondences between modified entities and clauses, consolidating clauses for the same entity under summary guidance.The textual mapping also uses summary-guided distillation to preserve entity information in generated text tokens.
  • Entity Mapping: Visual Entity Mapping aggregates reference-image feature channels corresponding to the same visual entity using learnable queries and a transformer.The process uses local and global reference-image features as transformer inputs.
  • Multimodal Query Composition: TEMA concatenates textual and visual entity features with global reference-image and MMT features to form final entity features before multimodal composition.The resulting representations support multi-granularity perception of multimodal queries.

5 Experiments

Experiments evaluate TEMA on multi-modification benchmarks, traditional CIR benchmarks, ablations, sensitivity settings, and qualitative PA outputs. TEMA performs strongly across both multi-modification and original CIR settings, while its components and design choices contribute materially to performance.

  • 5.2 Method Comparison: TEMA achieves superior performance on both M-FashionIQ and M-CIRR, outperforming baselines across fashion and open-domain multi-modification retrieval.The authors attribute this advantage partly to the PA and EM modules’ ability to handle nuanced multi-modification text.
  • 5.2 Method Comparison: BLIP-based models markedly outperform models with traditional backbones, indicating stronger handling of complex multi-modification text.The comparison includes models using traditional architectures such as ResNet and LSTM.
  • 5.3 Ablation Study: Removing PA, its consistency detector, or EM components reduces performance, with EM ablations causing particularly substantial drops.The results support the roles of PA-guided training and entity mapping for aggregating clauses associated with modified entities.
  • 5.4 Performance on Traditional CIR: TEMA outperforms previous baselines on traditional FashionIQ and CIRR while retaining performance beyond the multi-modification benchmarks.These experiments are presented as evidence of TEMA’s generalization ability across original and multi-modification CIR.
  • 5.5 Sensitivity Analysis: TEMA’s performance first improves and then declines as κ increases; similarly, excessive learnable-query channels N cause fluctuation and deterioration.The authors relate these trends to loss imbalance at high κ and confusion among entities at high N.
  • 5.6 Qualitative results for PA module: PA-generated summaries capture all to-be-modified entities while omitting some detailed descriptions, reducing distractions from lengthy text.The qualitative examples show summaries identifying entities such as dog breed and posture, which then guide entity aggregation.

6 Conclusion

The paper addresses two practical CIR limitations by introducing multi-modification datasets and TEMA, a framework supporting both original and multi-modification scenarios. TEMA outperformed previous methods in both settings.

  • The study addresses Insufficient Entity Coverage and Clause-Entity Misalignment in practical composed image retrieval.
  • It constructs the M-FashionIQ and M-CIRR multi-modification datasets.
  • TEMA is designed for multi-modification retrieval while also accommodating simple modifications.
  • TEMA outperformed previous methods in both original and multi-modification scenarios.

7 Limitations

The paper identifies limitations concerning dataset difficulty, training overhead, and retrieval scope. Its multi-modification datasets are intended to approximate real applications rather than solely maximize retrieval metrics.

  • Longer multi-modification annotations increase the difficulty of understanding modification intentions and do not necessarily improve retrieval metrics.
  • The PA module uses large language models during training and introduces minor computational overhead, although it is disabled during testing.
  • TEMA currently supports only single-turn retrieval, leaving effectiveness in multi-turn interactive CIR unexplored.
  • The constructed datasets prioritize a training and evaluation environment closer to real applications rather than solely increasing model performance.

8 Ethical Considerations

The paper describes dataset-release safeguards and responsible deployment practices, including privacy protections, restrictions on identifiable-face retrieval, and controls against misuse. The supplied passages also list appendices covering dataset construction, training, evaluation, and qualitative analyses.

  • The public dataset release will remove personally identifiable information and prohibit retrieval based on identifiable faces.
  • The dataset license explicitly disallows surveillance uses.
  • Recommended deployment safeguards include abuse detection, rate limiting, block lists, and context-aware access control policies.
  • The appendix includes sections on multi-modification datasets, dataset construction, training strategy, quantitative results, computation cost, prompts, and qualitative results.

A Multi-Modification Datasets

The paper constructs two multi-modification datasets from established CIR benchmarks: fashion-domain M-FashionIQ and open-domain M-CIRR. Their image and triplet collections support evaluation across distinct retrieval settings.

  • The datasets are constructed to evaluate models in multi-modification scenarios.
  • M-FashionIQ is based on FashionIQ and contains 77,684 fashion images across Dresses, Shirts, and Tops&Tees.
  • M-FashionIQ uses approximately 46K training images, 15K testing images, 18K training triplets, and approximately 6K testing triplets.
  • M-CIRR is based on CIRR and contains 21,552 real images from NLVR2.
  • M-CIRR contains 28,225 training triplets and 4,181 testing triplets, including a fine-discrimination subset focused on difficult negative images.

A.1 Dataset Statistics

The datasets expand FashionIQ and CIRR into multi-modification collections while preserving the original images and evaluation setup. Their texts are generated, checked, filtered, and used with entity-focused training objectives.

  • Dataset Construction: 24,016 triplets are processed to create M-FashionIQ and M-CIRR from expanded modification texts paired with original reference and target images.The resulting collections use expanded texts for training and validation-based evaluation.
  • Dataset Construction: MMTs replace short modification texts with human-verified descriptions that explicitly represent multiple entities and clauses.The annotations are generated by an MLLM and verified by human annotators.
  • Quality Control: Quality checks revise MMTs for consistency, accuracy, diversity, and overall textual quality.Annotators remove implausible, speculative, repetitive, or otherwise defective content.
  • Quality Control: A content filter removes statements describing target-image content without relating it to the reference image.The filter follows manual refinement and uses the corrected MMTs with the target image.
  • Training Setup: TEMA trains with a parsing assistant and entity mapping, while disabling the parsing assistant during inference to avoid additional dependencies and delays.The assistant guides textual entity features and entity aggregation during training.
  • LLM Robustness: Switching among integrated LLMs has minimal impact on TEMA’s overall performance, including when open-source models generate summaries and consistency checks.The reported alternatives include LLaMA 3 and Qwen2-VL series models.

D.2 Computation Cost Analysis

The paper analyzes TEMA’s computational cost and supporting implementation choices alongside prompt design for generating multi-modification texts. It also reports TEMA’s performance on original CIR benchmarks.

  • Computation Cost: TEMA is compared with Candidate using FLOPs, training time, test time, and GPU memory on a single NVIDIA A40 GPU with batch size 64.The analysis defines FLOPs as floating-point operations and distinguishes optimization time from inference time.
  • Computation Cost: TEMA is reported as superior to Candidate across the listed computation-cost measures.The supplied passage states an across-measure comparison but does not provide the individual values.
  • Original CIR Evaluation: TEMA is superior to existing state of the art on most metrics of the original FashionIQ and CIRR benchmarks.The supplied passage characterizes this as evidence of extensibility to traditional CIR settings.
  • Training and Inference: TEMA’s training process uses the parsing assistant and entity mapping, then applies feature extraction, summary-guided distillation, orthogonal regularization, and batch-based classification loss.The inference process disables the parsing assistant after the entity mapping module has learned from it during training.
  • MMT Generation: BLIP-3 prompts are tailored to garment components for M-FashionIQ and to objects, backgrounds, and relations for open-domain M-CIRR.The prompt design reflects FashionIQ’s garment-focused content and CIRR’s more varied scenes.
  • MMT Generation: Prompt (c) is selected because it produces more precise and comprehensive MMTs focused on one-to-many mappings and multiple constraints.The comparison reports that prompts (a) and (c) both captured detailed modifications, while (c) was judged stronger.

G.1 Mitigation on False-negative Samples

Multi-modification texts expose details omitted by short CIR annotations, reducing false negatives and improving retrieval examples. TEMA’s attention and qualitative cases further indicate coverage of modified entities, while residual annotation problems remain.

  • False-Negative Diagnosis: False negatives arise when visually similar candidates satisfy a multimodal query but are labeled negative because short modification texts describe too little.The issue reflects limited descriptive capability and numerous visually similar images.
  • M-FashionIQ: In M-FashionIQ, MMT relabeling makes the target the only positive sample in one example and raises its ranking from second to first in another.The original top-five results could satisfy the query despite being labeled negative when the necklace detail was omitted.
  • M-CIRR: In M-CIRR, MMTs add requirements such as standing in water, a dirt path, and background trees, raising one target from fifth to first.These additional details convert some original false negatives into true negatives under the more detailed query.
  • Attention Visualization: PA summaries attend to all listed modified entities in M-FashionIQ examples, including neckline, sleeves, shoes, skirt, and belt.The summaries omit detailed changes while retaining entity coverage.
  • Attention Visualization: PA summaries also attend to entities across M-CIRR scenes, including sheep, people, hamsters, backgrounds, and expressions.The visualizations are reported to support summary coverage of the MMT’s modified entities.
  • Qualitative Comparison: TEMA retrieves the target at top-1 in three qualitative cases where Candidate does not, although both models fail at top-1 in another case.The remaining failure is attributed to insufficient original annotation, and a small number of false-negative problems persist.
Loading 2604.21806v2…