Source-linked AI summary
XR: Cross-Modal Agents for Composed Image Retrieval
Zhongyu Yang, Wei Pang, Yingfang Yuan
TL;DR
CIR requires reasoning over a reference image and modification text, but existing embedding-based methods capture limited cross-modal cues and semantic relationships. XR addresses this gap with a training-free multi-agent workflow that imagines target representations, performs hybrid coarse filtering, and verifies candidates through questions. Across FashionIQ, CIRR, and CIRCO, XR consistently improves retrieval over strong baselines, while the framework remains limited to image–text composition and faces bias and efficiency challenges.
Problem
Existing CIR approaches capture limited cross-modal cues and lack semantic reasoning for queries combining reference images with modification text.
Method
XR coordinates imagination agents, similarity agents, and question agents for target synthesis, hybrid coarse filtering, and factual fine filtering.
Results
XR consistently improves retrieval over training-free and training-based baselines on FashionIQ, CIRCO, and CIRR.
Takeaways & Limitations
Cross-modal reasoning and coordinated agents are presented as important for aligning composed-image retrieval with user intent.
Takeaways & Limitations
XR is currently limited to image–text composition, while model-generated captions and verification questions may introduce bias and large candidate pools require efficiency optimization.
Abstract
from arXiv · showhide
Retrieval is being redefined by agentic AI, demanding multimodal reasoning beyond conventional similarity-based paradigms. Composed Image Retrieval (CIR) exemplifies this shift as each query combines a reference image with textual modifications, requiring compositional understanding across modalities. While embedding-based CIR methods have achieved progress, they remain narrow in perspective, capturing limited cross-modal cues and lacking semantic reasoning. To address these limitations, we introduce XR, a training-free multi-agent framework that reframes retrieval as a progressively coordinated reasoning process. It orchestrates three specialized types of agents: imagination agents synthesize target representations through cross-modal generation, similarity agents perform coarse filtering via hybrid matching, and question agents verify factual consistency through targeted reasoning for fine filtering. Through progressive multi-agent coordination, XR iteratively refines retrieval to meet both semantic and visual query constraints, achieving up to a 38% gain over strong training-free and training-based baselines on FashionIQ, CIRR, and CIRCO, while ablations show each agent is essential. Code is available: https://01yzzyu.github.io/xr.github.io/.
1 Introduction
Composed Image Retrieval combines a reference image with modification text, requiring cross-modal reasoning beyond conventional similarity matching. XR addresses this challenge with coordinated imagination, similarity, and question agents, and improves retrieval across three benchmarks.
- Motivation: CIR queries combine a reference image and modification text to preserve reference semantics while applying user-specified edits.This makes CIR relevant to fine-grained multimodal search in settings such as e-commerce and search engines.
- Motivation: Existing CIR methods struggle with fine-grained edit correspondences, multimodal cues, and cross-modal verification.Joint embeddings can miss edit-specific alignment, while single similarity-based matching may overlook information from one modality.
- XR Framework: XR progressively coordinates imagination, coarse filtering, and fine filtering to exploit cross-modal interactions.Imagination constructs target proxies; similarity agents score candidates using visual and textual cues; question agents verify factual consistency.
- XR Framework: Similarity agents provide broad retrieval coverage, while question agents refine candidates through factual verification.Their complementary roles preserve diverse evidence that single-score pipelines may overlook.
- Results: XR consistently improves edit-sensitive retrieval accuracy over strong training-free and training-based baselines on CIRR, CIRCO, and FashionIQ.The benchmarks span controlled reference-based queries and open-domain compositional settings.
2 Related Works
Prior CIR systems provide coarse similarity alignment but often lack fine-grained reasoning and iterative refinement. XR connects multimodal agent reasoning with retrieval through target imagination, cross-modal scoring, and factual verification.
- Multimodal Agent Systems: Multimodal agent systems support complex reasoning and coordination, but their practical coordination remains fragile.The related work positions agentic reasoning as promising but not yet fully reliable.
- CIR Methods: Embedding-based CIR methods achieve coarse alignment but blur fine-grained changes, while training-based models require costly supervision and retraining.Training-free methods generalize across domains but commonly use static fusion and one-shot pipelines.
- XR: XR bridges agentic reasoning and retrieval by imagining target images, scoring candidates across modalities, and enforcing factual verification.This workflow progressively connects reasoning with retrieval rather than treating them as separate capabilities.
3 Method
XR formulates CIR as progressively narrowing candidates through cross-modal imagination, similarity-based coarse filtering, and question-based factual verification. Its agents construct complementary target representations, combine textual and visual evidence, and re-rank candidates to preserve fine-grained semantic and visual constraints.
- 3.1 Preliminaries: CIR retrieves images that preserve a reference image’s visual characteristics while incorporating a user-specified textual modification.The query consists of a reference image and modification text, and the target set contains candidates approximating the ideal target image.
- 3.3 Imagination: XR’s imagination module generates complementary target captions and manipulation information from textual and visual cross-modal pairings.The text imagination agent produces textual manipulations, while the visual imagination agent produces visual-attribute indicators for the ideal target.
- 3.4 Coarse Filtering: Similarity agents perform cross-modal multiperspective scoring, aggregate text and vision scores with reciprocal rank fusion, and retain the top-k′ candidates.Text and vision agents assess candidates using generated target captions and candidate captions or images before ranking and filtering.
- 3.5 Fine Filtering: Question agents formulate True/False checks from textual modifications and imagined attributes to verify essential target properties in the shortlisted candidates.The question agent uses M_t, M_v, and T_m to create targeted questions, adding factual verification beyond similarity scoring.
- 3.5 Fine Filtering: XR combines question-based scores with aggregated similarity scores to re-rank the shortlisted candidates and produce the final target set.This progressive narrowing uses similarity for coarse alignment and questions for fine filtering, while cross-modal scoring provides multiple perspectives.
4 Experiments
XR is evaluated across three CIR benchmarks, multiple baselines, module ablations, and parameter settings. The experiments show consistent gains from coordinated cross-modal agents, with verification, fusion, and candidate-pool choices affecting accuracy and efficiency.
- Experiment Setup: XR is evaluated on CIRR, CIRCO, and FashionIQ using Recall@k for CIRR and FashionIQ and mAP@k for CIRCO.The benchmarks cover subset retrieval, large-scale retrieval with multiple ground truths, and three fashion categories.
- Experiment Setup: XR is compared with nine representative baselines, including both training-free and training-based CIR methods.The training-free baselines are CIReVL, LDRE, and ImageScope; the training-based baselines include PALAVRA, Pic2Word, SEARLE, iSEARLE, LinCIR, and FTI4CIR.
- Main Results: 57.10% R@50 on FashionIQ and 95.21% R@3 on CIRRsubset are achieved by XR with CLIP-ViT-B/32, while CIRCO reaches 30.95% mAP@50.XR also reaches 36.66% R@10 on FashionIQ, surpasses CIReVL by over 8 points in R@10, and exceeds the best CIRCO baseline by over 7 points.
- Ablation Studies: Each XR module contributes: visual similarity raises FashionIQ R@10 from 14.78% to 32.48%, combined similarity raises CIRCO mAP@25 from 4.14% to 19.29%, and verification raises CIRRsubset R@3 to 95.21%.Similarity agents provide broad alignment, while question-based agents reduce false positives and enforce modification faithfulness.
- Parameter Analysis: RRF with z=60 outperforms direct score summation, reaching 30.28% CIRCO mAP@25 and 83.15% CIRR R@10.Rank-based aggregation is more robust to heterogeneous modalities and noisy candidates than direct score averaging.
- Parameter Analysis: Three verification questions provide consistent gains, while candidate pools around k′=100 balance retrieval coverage and computational overhead.More than three questions slightly reduce performance, and gains plateau beyond k′=100; text–image fusion performs best at λ=0.15.
5 Conclusion
XR is a training-free cross-modal multi-agent framework for composed image retrieval that progressively refines results through semantic alignment and factual verification. Experiments show consistent improvements over training-free and training-based baselines, with coordinated agents producing stable and accurate retrieval.
- XR integrates imagination, coarse filtering, and fine filtering through similarity- and question-based agents.
- The framework progressively refines retrieval through semantic alignment and factual verification.
- XR consistently improves retrieval over both training-free and training-based baselines, particularly in fine-grained and distractor-rich scenarios.
- Ablation analyses find that coordinated agents yield more stable and accurate retrieval than their independent contributions alone.
A Detailed Experiment Results
The detailed FashionIQ analysis reports complete ablation and parameter-study results across shirts, dresses, and tops. It expands beyond the main paper’s representative metrics to provide a broader evaluation of XR against single-modality baselines.
- The FashionIQ analysis reports full scores across shirts, dresses, and tops.
- The supplementary results include a wider range of evaluation metrics than the main paper’s R@10 and R@50.
- The complete parameter and ablation results further demonstrate XR’s advantages over single-modality baselines.
B Statistical Significance Study
The statistical significance study evaluates XR across 10 independent random-seed runs using paired one-sided t-tests and Wilcoxon signed-rank tests. XR achieves the highest reported mean score in the comparison.
- 10 independent runs with different random seeds compare XR and baseline models under identical settings.
- Paired one-sided t-tests and Wilcoxon signed-rank tests assess whether XR performs better than each baseline.
- 57.16 ± 0.07 is XR’s highest reported mean score in the statistical comparison.
C Experimental Code
The authors plan to release experimental code, datasets, and detailed tutorials to support reproducibility. Documentation and guidelines are intended to facilitate replication and extension to other models and architectures.
- The project will release all experimental code, datasets, and detailed tutorials for reproducing the experiments.
- The release is intended to make replication straightforward for researchers and practitioners regardless of technical background.
- Comprehensive documentation and clear guidelines are intended to facilitate extension to other models and architectures.
D Limitations and Future Work
XR remains limited to image–text composition, with potential bias from generated captions and verification questions and efficiency challenges at very large candidate scales. Future work targets richer modalities, lighter agents, adaptive reasoning, and improved scalability.
- Limitations: XR is currently limited to image–text composition and has not been explored with richer modalities or temporal data.
- Limitations: Generated captions and verification questions from large models may introduce subtle biases that affect consistency.
- Limitations: Scaling XR to very large candidate pools requires further efficiency optimization.
- Future Work: Future directions include extending retrieval to video, audio, and interactive queries while developing lightweight, adaptive agents and diverse verification signals.
- Reliability and Transparency: XR's modular stages support reliability, transparency, auditing, and analysis of system behavior.
F Case Studies
Case studies across CIRR, FashionIQ, and CIRCO illustrate how XR handles complex scene edits, subtle attributes, and distractor-heavy retrieval settings. These qualitative examples complement the paper’s quantitative findings.
- Cross-benchmark comparison: Together, the case studies provide qualitative evidence that complements the quantitative findings.
- CIRR: CIRR demonstrates XR grounding complex scene edits through factual verification.The examples include bus orientation and reflective jackets.
- FashionIQ: FashionIQ demonstrates XR capturing subtle attribute modifications and validating them through text-based questioning.Examples include tone and lettering changes.
- CIRCO: CIRCO demonstrates XR remaining robust in distractor-heavy settings by verifying entity-level edits.Examples include food type and clothing.