Source-linked AI summary
Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing
Tingyu Song, Yanzhao Zhang, Mingxin Li, Zhuoning Guo, Dingkun Long, Pengjun Xie, Siyue Zhang, Yilun Zhao, Shu Wu
TL;DR
Existing composed image retrieval benchmarks cover a limited range of query categories, leaving it unclear whether model performance reflects compositional reasoning or benchmark biases. This paper constructs EDIR using image editing and finds that current models struggle to perform consistently across its subcategories.
Problem
Existing CIR benchmarks provide limited category coverage and scale, leaving uncertain whether model performance reflects compositional reasoning or benchmark biases.
Source: §1 · p. 1, §2.2 · p. 3Method
Results
Takeaways & Limitations
Comparison with existing CIR benchmarks confirms that EDIR reveals model weaknesses that other evaluations overlooked.
Source: §6 · p. 8The image-editing data synthesis pipeline remains computationally expensive, making large-scale generation challenging.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Composed Image Retrieval (CIR) is a pivotal and complex task in multimodal understanding. Current CIR benchmarks typically feature limited query categories and fail to capture the diverse requirements of real-world scenarios. To bridge this evaluation gap, we leverage image editing to achieve precise control over modification types and content, enabling a pipeline for synthesizing queries across a broad spectrum of categories. Using this pipeline, we construct EDIR, a novel fine-grained CIR benchmark. EDIR encompasses 5,000 high-quality queries structured across five main categories and fifteen subcategories. Our comprehensive evaluation of 13 multimodal embedding models reveals a significant capability gap; even state-of-the-art models (e.g., RzenEmbed and GME) struggle to perform consistently across all subcategories, highlighting the rigorous nature of our benchmark. Through comparative analysis, we further uncover inherent limitations in existing benchmarks, such as modality biases and insufficient categorical coverage. Furthermore, an in-domain training experiment demonstrates the feasibility of our benchmark. This experiment clarifies the task challenges by distinguishing between categories that are solvable with targeted data and those that expose intrinsic limitations of current model architectures.
1 Introduction
The introduction identifies coarse categories, insufficient query scale, and ambiguous labels as limitations of existing CIR benchmarks. It presents EDIR, built with a fine-grained taxonomy and image-editing synthesis, and uses it to expose gaps in model performance and benchmark coverage.
1 Introduction
- Existing benchmarks offer coarse category coverage, limited query scale, and ambiguous labels, partly because queries are annotated post hoc from retriever-selected targets.This construction can omit some modification categories and leave too few queries for others.Source: §1 · p. 1
- EDIR combines a taxonomy of five main categories and fifteen subcategories with an image-editing pipeline, yielding 5,000 queries and a gallery of 178,645 images.Synthesizing target images from textual modifications gives the pipeline precise control over query types and content.Source: §1 · pp. 1–2
- EDIR evaluation finds that top-performing multimodal embedding models cannot perform consistently across subcategories, while analysis identifies fine-grained coverage gaps and modality bias in existing benchmarks.The evaluation includes models trained on MLLMs and CLIP; modality bias lets models achieve high scores by over-relying on text.Source: §1 · p. 2
- An in-domain training experiment with EDIR-MLLM distinguishes challenges addressable with sufficient in-domain data from those reflecting intrinsic model weaknesses.The model is trained on the synthesized EDIR data.Source: §1 · p. 2
2 Related Works
Existing CIR benchmarks have limited categorical coverage and can favor text-only signals, making it unclear whether strong model performance reflects genuine multimodal compositionality. The related work motivates broader, finer-grained evaluation and situates it within evolving CIR methods.
2.1 Composed Image Retrieval Benchmarks
- FashionIQ is domain-specific, CIRR lacks fine-grained queries, CIRCO has imbalanced categories and ambiguous definitions, and I-CIR’s seven categories limit evaluation granularity.GeneCIS evaluates adaptation of similarity to textual conditions, while I-CIR introduces instance-level retrieval; existing benchmarks also show modality bias that can reward text-only signals (Huynh et al., 2025; Psomas et al., 2025).Source: §2.1 · p. 3
2.2 Composed Image Retrieval Methods
- CIR methods evolved from attribute classifiers to VLM-based approaches, with current work spanning text-inversion, data synthesis, and training-free pipelines.Recent data-synthesis work generates CIR training triplets but does not address the benchmark evaluation gaps identified here.Source: §2.2 · p. 3
- MLLM-based universal multimodal embeddings (Lin et al.; Jiang et al., 2024b) perform strongly on existing CIR benchmarks, but restricted evaluation leaves the source of their gains unclear.The unresolved question is whether gains reflect genuine compositional reasoning or exploitation of benchmark biases.Source: §2.2 · p. 3
3 EDIR Benchmark Construction
EDIR is a fine-grained composed image retrieval benchmark built through a controlled image-editing pipeline. It pairs a five-category, fifteen-subcategory taxonomy with query generation and quality checks to produce 5,000 queries.
3.1 Preliminary Setup
- The taxonomy defines five categories—Attribute, Object, Relationship, Global Environment, and Complex—across fifteen subcategories, using LAION-400M seed images filtered for quality.Qwen2.5VL-32B filters unsuitable seed images, including corrupted, blank, or document-style images.Source: §3.1 · p. 4
3.2 Raw Triplet Construction
- For each source image, an MLLM selects suitable subcategories and generates edit instructions; an image editor creates related images with shared base changes and distinctive edits for hard negatives.The shared visual context makes retrieval less trivial than a single-change target.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
3.3 Query Rewrite
- Queries are rewritten from edit instructions into natural language: simple queries use one distinctive change, while complex queries combine it with two shared base operations.The pipeline also constructs negation queries for categories such as Color and Shape, rather than using all four edits in a query.Source: Figure 2 · p. 4, §3.3 · p. 5
3.4 Data Quality Control
- Two MLLM filtering stages check edited images against composite instructions and then reference-target pairs against rewritten queries, removing 312,009 images and 889,013 triplets, respectively.The second check addresses partially correct images that can pass the first stage.Source: §3.4 · p. 5
3.5 Dataset Analysis
- EDIR contains 5,000 queries over a 178,645-image corpus, with each query paired with its target and three hard negatives from the same source image.Queries were sampled as 300 for each of 14 simple categories and 800 for Complex; the corpus also includes 150,000 additional edited images.Source: §3.5 · p. 5
- Human review of a random 12% sample found an 8.0% False Positive Rate, 7.3% False Hard Negative Rate, and 11.7% Global False Negative Rate.Annotators assessed target-query mismatches, hard negatives that also satisfy queries, and additional positives among top-five model retrievals.Source: §3.5 · pp. 5–6
4 Experiments and Results
EDIR reveals that current multimodal models struggle with fine-grained and compositional edits, while exposing gaps in existing benchmarks’ category coverage and use of both image and text. Error analysis links these weaknesses to limitations in foundation models and training data.
4.1 Experiment Setup
- The experiments evaluate MLLM-based and non-MLLM-based models using Recall@1.Source: §4.1 · p. 6
4.2 Results
- Non-MLLM models average 18.4%, while MLLMs outperform them, doing best on addition, replace, and action but worst on texture, remove, and shape.The authors attribute non-MLLMs’ low scores to CLIP-based models’ difficulty distinguishing fine-grained targets among visually similar edits.Source: §4.2 · p. 6
4.3 Error Analysis
- Errors reflect failures with negation, compositional reasoning, multiple constraints, and fine-grained visual details.The paper links negation and compositional-reasoning weaknesses to foundational MLLMs, and constraint and detail errors to training data that lacks such complexity.Source: §4.3 · p. 7
4.4 Benchmark Analysis
- EDIR’s model-performance correlations with prior benchmarks are positive but vary, and category-specific correlations are lower where existing benchmarks underrepresent skills such as removal and spatial edits.CIRCO has only 10 removal queries, while CIRR’s validation set has no spatial queries; EDIR provides broader coverage of these overlooked skills.Source: Figure 3 · p. 7, §4.4 · pp. 7–8
- On CIRCO, MLLM-based models perform better with text alone, indicating that its reference image can be redundant and exposing a modality shortcut.EDIR instead requires synthesis of image and text, providing a more robust test of composed image retrieval.Source: Figure 4 · p. 8, §4.4 · p. 8
5 In-domain Training and Analysis
In-domain training yields a new EDIR Recall@1 high for EDIR-MLLM and shows that fine-grained detail challenges improve more readily than complex compositional reasoning challenges. The experiment distinguishes challenges addressable with targeted training data from persistent model weaknesses.
5 In-domain Training and Analysis
- The team trained EDIR-MLLM, based on Qwen2.5-VL, for 2,500 steps on 225,000 triplets sampled evenly across 15 categories.The triplets came from an additional pool of approximately 1.1 million high-quality edit triplets; training used batch size 128.Source: §5 · p. 8
- A category is considered solvable if Recall@1 exceeds 60% or improves by more than 20 percentage points after in-domain training.The experiment uses this criterion to distinguish categories that targeted training can address from those that remain difficult.Source: §5 · p. 8
- EDIR-MLLM reaches 59.9% Recall@1 on EDIR, versus 36.9% for the average of other MLLM-based methods.The paper reports 59.9% as a new state-of-the-art Recall@1 on EDIR.Source: §5 · p. 8
- Color, material, texture, and action improve dramatically with in-domain examples, while count, spatial, and viewpoint show modest gains tied to intrinsic reasoning weaknesses.The analysis attributes the former challenges largely to inadequate training data and the latter to architectural limitations not easily resolved with in-domain data.Source: §5 · p. 8
6 Conclusion
EDIR is a fine-grained CIR benchmark built with image editing to expose compositional generalization gaps in current models. Evaluation and in-domain training show that it reveals overlooked weaknesses and distinguishes data-solvable issues from intrinsic model limitations.
6 Conclusion
- Evaluation of 13 multimodal embedding models reveals significant shortcomings on EDIR, while comparison with existing benchmarks exposes weaknesses they overlook.The results highlight a gap in current models’ compositional generalization.Source: §6 · p. 8
- EDIR provides 5,000 queries across fifteen detailed subcategories, synthesized through an automated pipeline leveraging image editing for granular CIR evaluation.The benchmark is intended to support development of more genuinely compositional and less biased CIR models.Source: §6 · p. 8
- An in-domain training experiment demonstrates EDIR’s solvability and distinguishes data-solvable issues from intrinsic model limitations.The experiment validates the unique challenges posed by EDIR.Source: §6 · p. 8
Limitations · A EDIR · A.1 Details of Construction
The section identifies limits in synthesis scale, query complexity, and EDIR’s diagnostic-only scope, then specifies the construction pipeline and selective negation strategy.
Limitations
- Programmatic image editing provides precise modification control, but its computational expense makes large-scale data synthesis costly and difficult.The authors identify scalability as a limitation and an avenue for future research.Source: Limitations · p. 9
- EDIR queries typically combine three distinct conditions, leaving scenarios with four or more interdependent instructions outside the benchmark’s current complexity.The authors identify more challenging queries as an opportunity for future benchmark development.Source: Limitations · p. 9
- EDIR is designed to diagnose model weaknesses rather than serve as a universal training solution; scalable training methods tailored to those weaknesses remain open.Source: Limitations · p. 9
A.1 Details of Construction
- Qwen25-VL-32B-Instruct (Bai et al., 2025) selects seed images and generates edit instructions, while Qwen-Image-Edit (Wu et al., 2025), version Qwen-Image-Edit-2509, generates target images.Source: §A.1 · p. 12
- Qwen3-32B (Yang et al., 2025) rewrites each edit instruction into a CIR query using either direct rewriting or a negation-form strategy.Source: §A.1 · p. 12
- Negation-form rewriting is limited to the color, shape, material, texture, style, weather, and time categories because not all categories suit negation.Source: §A.1 · p. 12
A.2 Error Analysis
Representative low-Recall@1 examples for the state-of-the-art RzenEmbed-7B reveal failures with negation, compositional reasoning, multiple constraints, and fine-grained visual details.
Failure in Handling Negation
- Retrieved images often preserve explicitly negated attributes or fail to remove requested objects or regions, showing difficulty mapping negation to the intended target state.The examples include changing a T-shirt’s original color and removing a region such as the empty wall above a bed.Source: §A.2 · p. 12, Figure 5 · p. 12
Deficiencies in Compositional Reasoning
- Retrieved images often match an object’s appearance but miss the global scene context, indicating difficulty jointly reasoning about foreground content and background.The example query asks for a similar object with a classroom background; poor performance is also noted for count, spatial, style, and viewpoint categories.Source: §A.2 · p. 12, Figure 6 · p. 13
Struggles with Multiple Constraints
- In complex queries, retrieved images may satisfy only some requirements, such as matching a jug, sponge, and garage-like background but missing the jug’s black-handle constraint.This illustrates difficulty composing and verifying multiple distinct requirements in one query.Source: §A.2 · p. 12, Figure 7 · p. 13
Insensitivity to Fine-Grained Details
- For texture, material, and shape queries, models overlook subtle visual cues and may retrieve images containing the object without its intended characteristics.In the example, retrieved images contain a jar but ignore its fine-grained details in the reference image.Source: §A.2 · p. 12, Figure 8 · p. 13
B Experiment Settings · B.1 Evaluation Details
Evaluation uses specified model configurations and retrieval instructions, with EDIR and comparison benchmarks assessed using their designated metrics and protocols.
B.1 Evaluation Details
- Non-MLLM models use CLIP-L/14 variants, including PIC2WORD, SEARLE, and MAGICLENS, to ensure a fair comparison.Source: §B.1 · p. 13
- MLLM-based models use a maximum sequence length of 2048 and a maximum of 1280 pixels.Source: §B.1 · p. 13
- The retrieval instruction asks models to find an image similar to the input image that satisfies the query.Source: §B.1 · p. 13
- EDIR is evaluated using Recall@1.Source: §B.1 · p. 13
- CIRR uses Recall@1 and CIRCO uses mAP@5.Source: §B.1 · p. 13
- FASHIONIQ is evaluated using Recall@10, while GENECIS uses Recall@1.Source: §B.1 · p. 13
- CIRR excludes the reference image from the retrieval corpus, and CIRR and CIRCO results are reported on their validation sets.The CIRR evaluation follows the protocol in Jiang et al. (2024a).Source: §B.1 · p. 13
B.2 Training Settings
The training data was synthesized by editing LAION-400M images and selecting 225,000 instances evenly across 15 categories; Qwen2.5-VL-7B-Instruct was then trained with LoRA and an InfoNCE-style loss.
B.2 Training Settings
- Editing 500,000 LAION-400M images yielded 1,087,710 candidate instances, from which 15,000 triplets per category across 15 categories formed a 225,000-instance training set.Each instance contains a reference image, query, target image, and three hard negatives sampled from the same source.Source: §B.2 · p. 13
- Qwen2.5-VL-7B-Instruct (Bai et al., 2025) was trained with LoRA and an InfoNCE-style loss.Training used batch size 128, a maximum of 1,280 image tokens and 1,500 sequence tokens, learning rate 3e-5, weight decay 0.01, and temperature 0.03; LoRA was applied only to q_proj, k_proj, v_proj, up_proj, down_proj, and gate_proj.Source: §B.2 · p. 13
B.3 Results · C Prompts. · — RULES —
Recall@3 improves model performance but does not eliminate EDIR’s model and category weaknesses; the appendix also documents prompts and rules for generating, rewriting, and checking edit queries.
B.3 Results
- Recall@3 performance increases for all models, yet MLLM-based models average close to 60 and zero-shot MMRet-MLLM, E5-V, and MAGICLENS remain weak.The supplied passage does not specify the unit for the average of close to 60.Source: §B.3 · p. 13, Table 5 · p. 14
- EDIR’s Recall@3 results still expose category-specific weaknesses, particularly for remove and view.Source: §B.3 · p. 13, Table 5 · p. 14
C Prompts.
- Qwen25-VL-32B-Instruct filters seed images, generates instructions for 5–6 categories with three instructions each, and supports direct and negation rewrites plus two-stage filtering.Source: C Prompts · p. 15, Figure 9 · p. 15, Figure 10 · p. 16
— RULES —
- Generation rules require valid JSON with exactly two root keys, 5–6 allowed categories, and 2–3 independent, atomic, concrete, plausible instructions per category, avoiding neglected categories.Instructions across categories should not conflict; remove, replace, cardinality, viewpoint, shape, time, and texture receive particular attention.Source: C Prompts · p. 16, — RULES — · p. 16
- Direct rewriting converts edit commands into descriptions of the desired final scene and removes comparisons with the original image.Source: — RULES — · p. 17, Figure 11 · p. 17
- A quality-control prompt checks whether the text accurately describes the source-to-target transition, rejecting mismatches, unexplained changes, subject inconsistencies, or over-description.A subject change is acceptable when the source and target subjects belong to the same basic category.Source: — RULES — · p. 19, Figure 13 · p. 19