Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing

Tingyu Song; Yanzhao Zhang; Mingxin Li; Zhuoning Guo; Dingkun Long; Pengjun Xie; Siyue Zhang; Yilun Zhao; Shu Wu

Source-linked AI summary

Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing

Tingyu Song, Yanzhao Zhang, Mingxin Li, Zhuoning Guo, Dingkun Long, Pengjun Xie, Siyue Zhang, Yilun Zhao, Shu Wu

arXiv:2601.16125v1Jan 22, 2026cs.CVcs.CLcs.IR

TL;DR

CIR benchmarks often provide limited category coverage and may not reveal whether model gains reflect genuine compositional reasoning. This paper introduces EDIR, a fine-grained benchmark synthesized through controllable image editing, and finds substantial capability gaps while using in-domain training to separate data-solvable challenges from intrinsic model weaknesses.

Problem
Existing CIR benchmarks have coarse or insufficient category coverage, leaving evaluation gaps and uncertainty about whether reported gains reflect genuine compositional reasoning or benchmark biases.
Method
The paper builds a five-category, fifteen-subcategory CIR taxonomy and uses image editing to synthesize controllable queries for the EDIR benchmark.
Results
Evaluation of 13 multimodal embedding models reveals significant capability gaps, with top-performing models unable to perform consistently across all EDIR subcategories.
Takeaways & Limitations
EDIR exposes model weaknesses overlooked by existing evaluations and distinguishes challenges addressable with targeted data from intrinsic architectural limitations.
Takeaways & Limitations
The data synthesis pipeline is computationally expensive, and EDIR’s Complex queries typically contain only three distinct conditions rather than four or more interdependent instructions.

Abstract

from arXiv · show

Composed Image Retrieval (CIR) is a pivotal and complex task in multimodal understanding. Current CIR benchmarks typically feature limited query categories and fail to capture the diverse requirements of real-world scenarios. To bridge this evaluation gap, we leverage image editing to achieve precise control over modification types and content, enabling a pipeline for synthesizing queries across a broad spectrum of categories. Using this pipeline, we construct EDIR, a novel fine-grained CIR benchmark. EDIR encompasses 5,000 high-quality queries structured across five main categories and fifteen subcategories. Our comprehensive evaluation of 13 multimodal embedding models reveals a significant capability gap; even state-of-the-art models (e.g., RzenEmbed and GME) struggle to perform consistently across all subcategories, highlighting the rigorous nature of our benchmark. Through comparative analysis, we further uncover inherent limitations in existing benchmarks, such as modality biases and insufficient categorical coverage. Furthermore, an in-domain training experiment demonstrates the feasibility of our benchmark. This experiment clarifies the task challenges by distinguishing between categories that are solvable with targeted data and those that expose intrinsic limitations of current model architectures.

1 Introduction

Existing CIR benchmarks provide limited and coarse-grained coverage of real-world modification requirements. The paper addresses this gap with EDIR, a fine-grained benchmark built through controllable image-editing synthesis and used to expose model and benchmark limitations.

Benchmark limitations: Existing benchmarks focus on narrow modification categories, limiting fine-grained evaluation of diverse real-world CIR requirements.Some category-based benchmarks also have insufficient scale and ambiguous labels, such as CIRCO’s overlapping “direct addressing” and “color” tags.
EDIR construction: EDIR organizes CIR requirements into five main categories and fifteen subcategories through a comprehensive taxonomy.The taxonomy is designed to support fine-grained evaluation across a broader range of query types.
EDIR construction: EDIR uses image editing to provide precise control over query types and modification content during data synthesis.The pipeline begins by applying textual modifications to synthesize target images before constructing the benchmark.
EDIR construction: The benchmark contains 5,000 high-quality queries paired with an image gallery of 178,645 images.
Evaluation findings: Evaluation of multimodal embedding models shows that even top-performing systems cannot consistently perform well across all EDIR subcategories.The authors attribute these shortcomings to inherent model limitations and a scarcity of suitable training data.
Evaluation findings: Existing benchmarks exhibit evaluation gaps, including insufficient fine-grained category coverage and modality bias that permits text reliance.
In-domain analysis: In-domain training distinguishes challenges that targeted data can address from those reflecting intrinsic weaknesses in current model architectures.The experiment trains EDIR-MLLM on synthesized data to analyze the solvability of EDIR’s fine-grained categories.

2 Related Works

CIR research has progressed from domain-specific and coarse benchmarks toward VLM- and MLLM-based methods. However, restricted benchmark designs leave it unclear whether reported gains reflect genuine compositional reasoning or exploitation of benchmark biases.

Benchmarks: FashionIQ is confined to fashion, while CIRR covers broader domains but lacks fine-grained queries for diagnosing model failures.
Benchmarks: CIRCO adds detailed categories and careful annotations but suffers from imbalanced query distributions and ambiguous category definitions.
Methods: CIR methods have evolved from specialized attribute classifiers to approaches built on vision-language models.
Methods: Current CIR approaches include text-inversion and data-synthesis methods, which respectively fuse image-derived tokens with text or generate CIR triplets.
Open evaluation gap: MLLM-based embedding models outperform existing CIR benchmarks, but limited evaluation coverage obscures whether gains reflect compositional reasoning or benchmark-bias exploitation.

3 EDIR Benchmark Construction

EDIR is constructed as a fine-grained CIR benchmark by combining a five-category taxonomy with an automated image-editing pipeline that generates controlled queries and hard negatives. Filtering and human validation produce 5,000 queries from a large set of edited-image triplets.

Benchmark Design: The benchmark represents each instance as a triplet containing a reference image, target image, and natural-language modification query.
Data Synthesis Pipeline: The synthesis pipeline filters seed images, generates edited-image triplets, and automatically rewrites edit instructions into natural-language CIR queries.
Taxonomy Definition: EDIR defines five major categories for fine-grained CIR evaluation: Attribute, Object, Relationship, Global Environment, and Complex.
Triplet Generation: For each source image, multiple edit instructions produce related images; one becomes the target and the others become hard negatives.
Dataset Statistics: 5,000 queries are sampled from 889,013 filtered high-quality triplets, with 300 queries per simple category and 800 for Complex.
Quality Validation: Human validation reports an 8.0% False Positive Rate, 7.3% False Hard Negative Rate, and 11.7% Global False Negative Rate.

4 Experiments and Results

Experiments evaluate non-MLLM and MLLM embedding models on EDIR, analyze their failure modes, and compare EDIR with prior CIR benchmarks. Results show strong category-dependent performance, persistent reasoning weaknesses, and evaluation gaps in existing benchmarks.

Model Results: Non-MLLM-based models achieve an average total score of only 18.4% on EDIR.Their CLIP-based architectures often identify the correct image group but fail to distinguish fine-grained targets, especially for remove and texture.
Model Results: MLLM-based models outperform Non-MLLM baselines, performing relatively strongly on addition, replace, and action but worse on texture, remove, and shape.
Error Analysis: The error taxonomy identifies failures in handling negation, compositional reasoning, and multiple constraints.Examples include removal and negative terms, count and spatial relationships, global scene transformations, and queries with several conditions.
Error Analysis: Underperformance reflects both intrinsic weaknesses in foundational MLLMs and inadequate training data for multiple constraints and fine-grained details.
Benchmark Analysis: Existing benchmarks lack balanced fine-grained coverage, including very few remove queries and no spatial queries in CIRR’s validation set.
Benchmark Analysis: On CIRCO, MLLM-based models perform better with text alone than with text and images, indicating a strong modality bias.EDIR instead requires synthesis of both modalities.

5 In-domain Training and Analysis

The in-domain experiment tests which EDIR categories can be solved with specialized training and which expose intrinsic model weaknesses. EDIR-MLLM substantially improves overall retrieval, but gains vary by category.

Training Setup: 225,000 training instances were sampled evenly across EDIR’s 15 categories and used to train EDIR-MLLM for 2,500 steps.The training set contains 15,000 triplets per category and uses a batch size of 128.
Overall Results: 59.9% Recall@1 makes EDIR-MLLM state of the art on EDIR, versus 36.9% for the average of other MLLM-based methods.The paper defines a category as solvable when Recall@1 exceeds 60% or improves by more than 20 percentage points after in-domain training.
Category Analysis: Color, material, texture, and action improve dramatically with targeted training, indicating that inadequate training data contributes to these difficulties.
Category Analysis: Count, spatial, and viewpoint show modest gains, indicating persistent difficulty with compositional reasoning despite in-domain data.

6 Conclusion

The paper concludes that EDIR provides a fine-grained, image-editing-derived benchmark for diagnosing CIR model weaknesses. Its evaluation and training analyses expose both data-solvable issues and intrinsic limitations.

Conclusion: EDIR evaluates CIR granularly with 5,000 queries spanning fifteen detailed subcategories.
Conclusion: Evaluation of 13 multimodal embedding models reveals significant shortcomings in compositional generalization.
Conclusion: Comparisons with existing benchmarks show that EDIR uncovers weaknesses those evaluations overlook.
Conclusion: In-domain training demonstrates EDIR’s solvability while distinguishing data-solvable issues from intrinsic model limitations.

Limitations

The authors identify limitations in EDIR’s synthesis pipeline and query complexity. Programmatic editing offers precise control, but the pipeline remains costly to scale, while complex queries have bounded complexity.

Data synthesis scalability: Computational expense makes large-scale data generation through the programmatic image-editing pipeline challenging.The pipeline provides precise control over modifications, but this control comes with substantial computational cost.
Query complexity: EDIR’s Complex queries are bounded in complexity.The limitation is stated as a scope boundary for the benchmark’s query composition.
Pipeline implementation: The synthesis pipeline uses Qwen25-VL-32B-Instruct for seed selection and instruction generation, Qwen-Image-Edit for target images, and Qwen3-32B for query rewriting.Two query-rewriting strategies are adopted: direct instruction rewriting and a second strategy described after the passage excerpt.

A.2 Error Analysis

The error analysis identifies four recurring weaknesses: negation handling, compositional reasoning, multiple-constraint satisfaction, and fine-grained visual sensitivity. These failures show where retrieved images diverge from query requirements.

Failure in Handling Negation: Models preserve negated attributes or fail to remove requested objects in explicit-negation and remove queries.
Deficiencies in Compositional Reasoning: Count, spatial, style, and viewpoint queries expose weak compositional reasoning, especially when foreground appearance and global scene context must align.
Struggles with Multiple Constraints: Complex queries are often only partially satisfied, such as matching objects and background while missing a fine attribute constraint.One example matches a jug, sponge, and garage-like background but misses that the jug handle is black.
Insensitivity to Fine-Grained Details: Models overlook subtle texture, material, and shape cues, retrieving images that contain the same object without its intended characteristics.

B Experiment Settings

The experiments evaluate multimodal models under standardized model and benchmark settings. EDIR uses Recall@1, while established benchmarks retain their standard metrics and protocols.

Model Settings: Non-MLLM models use CLIP-L/14 variants, while MLLM models use a 2048-token maximum sequence length and 1280 maximum pixels.All models receive the instruction: “Given an image, find a similar image satisfying the query.”
Benchmark Settings: EDIR is evaluated with Recall@1, while CIRR, CIRCO, FASHIONIQ, and GENECIS use Recall@1, mAP@5, Recall@10, and Recall@1, respectively.CIRR excludes the reference image from the retrieval corpus, and CIRR and CIRCO results are reported on validation sets.

B.2 Training Settings

The training data are synthesized by editing LAION-400M images and sampling category-balanced triplets. Qwen2.5-VL-7B-Instruct is trained with specified optimization, token, and parameter-efficient adaptation settings.

Data Construction: 500,000 LAION-400M images produce 1,087,710 training instances containing a reference image, query, target image, and three hard negatives.The final training set samples 15,000 triplets per category across 15 categories, yielding 225,000 instances.
Optimization: Qwen2.5-VL-7B-Instruct is trained with batch size 128, learning rate 3e-5, weight decay 0.01, 1,280 maximum image tokens, and 1,500 maximum sequence length.LoRA is applied to the q_proj, k_proj, v_proj, up_proj, down_proj, and gate_proj layers.

B.3 Results

The results section reports additional EDIR performance and describes the pipeline used to generate, rewrite, and quality-control fine-grained CIR queries. The benchmark emphasizes category-specific evaluation and filtering for accurate source-to-target descriptions.

Model Performance: Recall@3 substantially improves performance for all evaluated models, but MLLM-based models remain close to an average performance of 60 on EDIR.Zero-shot models including MMRet-MLLM, E5-V, and MAGICLENS still perform poorly.
Edit Instruction Generation: Qwen25-VL-32B-Instruct filters unsuitable seed images, then generates instructions across 5–6 categories with three instructions per category.The pipeline uses separate prompts for image suitability filtering and edit-instruction generation.
Edit Instruction Generation: Each generated record contains an image description and category-specific instruction lists, with exactly two top-level JSON keys and 5–6 selected categories.Allowed categories include color, material, shape, texture, addition, removal, replacement, spatial relations, action, viewpoint, style, time, and weather.
Query Rewriting: Direct rewriting converts edit instructions into searches describing the final desired scene while removing comparative terms such as “larger,” “more,” and “brighter.”A separate negation strategy focuses on the changed attribute rather than describing the final appearance.
Quality Control: A quality-control checkpoint compares the source image, text description, and target image to verify that the description accurately represents the transition.Descriptions fail when they mismatch the actual changes or omit a fundamental subject or scene inconsistency.