PaperlayerView exact revision on arXiv

Source-linked AI summary

COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relations

Zixu Li, Yupeng Hu, Zhiwei Chen, Haokun Wen, Xuemeng Song, Liqiang Nie

arXiv:2606.04604v1cs.CV

TL;DR

CIR methods can confuse visually similar images whose attributes do not match the composed query, while existing attribute-based approaches often require fixed labels or prototypes. COMBINER addresses this with disentangled attribute features, cross-modal unified prototypes, and attribute-based pairwise and neighbor relations, achieving optimal performance across metrics on three benchmark datasets.

  • Problem

    Existing CIR methods overlook visually similar but attribute-unrelated samples, while many attribute-based methods depend on fixed prototypes or explicit attribute annotations.

    Source: §I · p. 1

  • Method

    COMBINER disentangles attribute features, constructs Cross-modal Unified Prototypes, and models pairwise and neighbor relations using attribute similarity.

    Source: §I · p. 2, §V · p. 14

  • Results

    COMBINER achieves optimal performance across all metrics on three CIR benchmark datasets.

    Source: §I · p. 3

  • Takeaways & Limitations

    Attribute-prototype-based similarity distinguishes attribute-related samples from visually similar but attribute-unrelated samples for multimodal fusion and metric learning.

    Source: §I · p. 2

Abstract

from arXiv · show

Composed Image Retrieval (CIR) represents a challenging retrieval task that targets locating specific images through multimodal inputs. Despite recent progress in CIR techniques, prior approaches often overlook cases where images appear visually alike yet differ in attributes, potentially undermining both multimodal feature fusion and similarity modeling. To mitigate this limitation, we design a unified representation of cross-modal features based on attribute prototypes. Nevertheless, the task is far from straightforward, owing to three core issues: (1) entanglement in attribute-level semantics, (2) inconsistency across modalities, and (3) supervised signal missing. To tackle the above obstacles, we introduce a COMposed image retrieval network guided By attrIbute-based NEighbor Relations (COMBINER). Specifically, we first design an Adaptive Semantic Disentanglement module, which is capable of disentangling attribute features based on multimodal primitive features. Secondly, we propose a Unified Prototype-based Composition module, which can construct cross-modal unified prototypes (CUP) and facilitate multimodal feature composition. Finally, we introduce a Dual Relations Modeling module, which can mine pairwise and neighbor relations based on attribute similarity. Compared to traditional neighbor relations modeling CIR methods, COMBINER represents the first study addressing the phenomenon of visually similar but attribute-unrelated samples. It achieves a more accurate understanding of the semantic relations among samples by employing an attribute prototype-based similarity metric. Comprehensive experiments conducted on three benchmark datasets confirm the effectiveness of our proposed COMBINER. The implementation of our method will be accessed at https://github.com/Lee-zixu/COMBINER

I. INTRODUCTION

Composed Image Retrieval uses a reference image and modification text, but existing methods can confuse visually similar distractors with attribute-relevant samples. COMBINER addresses this through attribute-prototype representations, multimodal composition, and attribute-based relation modeling.

Motivation

  • CIR retrieves target images from a composed query combining a reference image with modification text.The reference approximates the target, while the modification describes the intended variation.Source: §I · p. 1

Problem

  • Existing pairwise and neighbor-relation methods can mishandle visually similar but attribute-unrelated images, treating them as negatives or pulling them closer incorrectly.A carpet-like candidate should not be brought close to a query requiring bedding.Source: §I · pp. 1–2, Figure 1 · p. 1
  • Attribute-prototype methods address this challenge, but prior approaches depend on fixed prototypes or explicit attribute labels and are difficult to apply to open-domain CIR without annotations.They also commonly assume that visual similarity in non-target attributes should be preserved.Source: §I · p. 2

Approach

  • COMBINER builds a unified cross-modal representation from attribute prototypes to identify attribute-related samples and separate attribute-unrelated visual distractors.Its similarity measure uses prototype feature distributions to distinguish target, attribute-related, and attribute-unrelated images.Source: §I · p. 2
  • The model combines Adaptive Semantic Disentanglement, Unified Prototype-based Composition, and Dual Relations Modeling for multimodal fusion and metric learning.These modules disentangle attribute features, construct the CUP, and model pairwise and neighbor relations using attribute similarity.Source: §I · pp. 2–3

Results

  • Experiments on three CIR benchmarks report optimal performance across all metrics, with COMBINER surpassing previously strongest methods that varied by dataset.The prior best methods included DQU-CIR on Shoes and FashionIQ and SPRC on CIRR.Source: §I · p. 3

II. RELATED WORK

COMBINER builds on CIR and prototype-learning research by addressing hard negatives that are visually similar but attribute-inconsistent. Its architecture combines adaptive attribute disentanglement, unified prototype composition, and dual relation modeling.

C. Unified Prototype-based Composition

Unified Prototype-based Composition addresses inconsistent image and text attribute prototypes by constructing a shared prototype space. It then uses aligned modality weights to form attribute-level composed query features.

D. Dual Relations Modeling

Dual Relations Modeling combines supervised pairwise learning with unsupervised, attribute-based neighbor modeling. Semantic clustering and distribution consistency regularization gather attribute-related samples while separating attribute-unrelated ones.

Pairwise Relations Modeling

Unsupervised Neighbor Relations Modeling

Distribution Consistency

IV. EXPERIMENT

The experiments evaluate COMBINER against existing methods, component contributions, hyper-parameters, inference efficiency, and qualitative behavior. Evaluation follows standard protocols on Shoes, FashionIQ, and CIRR.

  • The study asks whether COMBINER surpasses state-of-the-art methods and how its components affect performance.Source: §IV · p. 7
  • Additional experiments examine hyper-parameter sensitivity, inference efficiency, and qualitative performance.Source: §IV · p. 7

1) Datasets:

The evaluation uses three benchmark datasets spanning fashion and open-domain composed image retrieval.

Datasets

  • The study evaluates Shoes, FashionIQ, and CIRR for composed image retrieval.Shoes and FashionIQ are fashion-domain datasets, while CIRR is open-domain.Source: 1) Datasets · p. 7
  • The dataset selection includes two fashion datasets and one open-domain dataset.Source: 1) Datasets · p. 7
  • The authors use these datasets to support fair comparison with previous work.Source: 1) Datasets · p. 7

2) Implementation Details:

COMBINER is evaluated with standardized retrieval metrics against established CIR baselines across three datasets, with reported gains over prior methods.

Implementation Details

  • ViT-H/14 CLIP provides COMBINER’s feature backbone, with embedding dimension D = 1,024 and eight total attribute prototypes.The global and local prototype counts are both set to 4, giving U = 8.Source: 2) Implementation Details · p. 7

Evaluation Protocol

  • Evaluation reports dataset-specific Recall@k metrics, including R@k, Rsubset@k, and CIRR’s mean of R@5 and Rsubset@1.Shoes uses k = 1, 10, 50; FashionIQ reports R@10 and R@50 by category; CIRR reports R@k and Rsubset@k.Source: 3) Evaluation · p. 7

Performance Comparison

  • COMBINER consistently exceeds every baseline across FashionIQ, Shoes, and CIRR.Reported improvements include 2.1% for FashionIQ-Avg-R@10, 2.9% relative improvement for Shoes-R@1, and 1.2% for CIRR-R@1.Source: B. Performance Comparison (RQ1) · p. 8
  • Neighbor-relation modeling yields greater performance improvement on fashion datasets because they contain more visually similar samples.FashionIQ examples include dresses sharing the same color, creating visually similar candidates.Source: B. Performance Comparison (RQ1) · p. 8

C. Ablation Study (RQ2)

Ablation results support the roles of semantic disentanglement, unified prototype composition, dual relation modeling, and distribution consistency, while cross-backbone tests indicate generalizable gains.

Module Ablation

  • Removing ASD decreases performance, while removing UPC produces the worst module-variant performance.These results support disentangling attribute prototypes and unifying them for multimodal composition and metric learning.Source: 1) Module Ablation · p. 9
  • Removing DRM lowers performance, indicating that dual relations modeling improves retrieval by optimizing neighbor relations.Source: 1) Module Ablation · p. 9

Loss Ablation

  • Simultaneously pulling multimodal queries and target images toward semantic cluster centroids better gathers visually similar candidates into the same cluster.The joint setting outperforms variants that optimize only one of these two directions.Source: 1) Module Ablation · p. 9
  • Removing either distribution-consistency regularizer causes poor performance, while removing both produces the lowest performance.The regularizers optimize the distributions of semantic clustering centroids and multimodal queries.Source: 1) Module Ablation · p. 9

Backbone Robustness

  • Under identical CLIP backbones, COMBINER surpasses SADN by +3.47% on FashionIQ Avg R@10 and +2.15% on CIRR Avg.Comparisons across five backbones also show consistent superiority over corresponding same-backbone baselines.Source: 1) Module Ablation · p. 10

3) Loss Ablation:

The ablations show that COMBINER’s attribute-aware Dual Relations Modeling depends on specialized losses, clustering, and distribution alignment choices rather than generic deep metric learning alone.

Loss Ablation

  • Replacing COMBINER’s losses with HIST or AFNE DML losses substantially degrades performance, showing that generic DML does not address attribute-irrelevant negatives.COMBINER w/ HIST drops by more than 7% on FashionIQ-Avg (R@10).Source: 3) Loss Ablation · p. 10
  • The DRM module is designed specifically to separate visually deceptive but attribute-incorrect samples from genuine false negatives during metric learning.Attribute-irrelevant samples are treated as genuine negatives, unlike potential positives handled by false-negative-oriented weighting methods.Source: 3) Loss Ablation · p. 10

Clustering Algorithm Ablation

  • K-Means remains competitive with hierarchical clustering, whereas DBSCAN and spectral clustering noticeably reduce performance.The paper attributes these degradations to high-dimensional noise sensitivity for DBSCAN and difficulty constructing semantically appropriate graphs for spectral clustering.Source: 4) Clustering Algorithm Ablation · pp. 10–11

Alignment Strategy Ablation

  • The default KL divergence significantly outperforms JS divergence and Wasserstein-1 distance across all reported metrics.The authors interpret KL as supporting unidirectional supervision from target distributions to composed-feature distributions.Source: 5) Alignment Strategy Ablation · p. 11

D. Sensitivity Analysis (RQ3)

Sensitivity analyses examine trade-off weights, prototype counts, cluster counts, and inference efficiency across FashionIQ and CIRR, finding generally stable behavior with identifiable parameter boundaries.

Hyper-parameter Sensitivity

Prototype Sensitivity

  • Excessive attribute prototypes can reduce performance by disrupting subsequent neighbor-relation modeling, after which performance levels off.The reported pattern is a slight decline followed by stabilization as the prototype count increases beyond a suitable range.Source: 2) Sensitivity to Attribute Prototype… · p. 11, Figure 5 · p. 12

Cluster Sensitivity

Inference Efficiency

  • COMBINER improves retrieval without external models by learning attribute-based neighbor relations that separate visually similar samples with different attributes.The efficiency comparison reports inference time and retrieval performance against DQU-CIR on a single NVIDIA A40 GPU.Source: E. Inference Efficiency Analysis (RQ4) · p. 12, Table XII · p. 12

F. Qualitative Analysis (RQ5)

Qualitative analyses show that COMBINER attends to requested attributes, forms semantically consistent neighbors, and distinguishes attribute-related from visually similar but unrelated images, while exposing annotation and disentanglement limitations.

Case Study

  • COMBINER’s retrieved top-K images can satisfy modification semantics even when dataset annotations label another image as the sole target.This creates false-negative evaluation penalties despite semantically correct retrievals.Source: 1) Case Study · pp. 12–13
  • COMBINER can struggle when visually dominant attributes overshadow fine-grained structural details in highly coupled attribute combinations.The authors identify this as a capacity boundary of the Adaptive Semantic Disentanglement module.Source: 1) Case Study · p. 13

Similarity Visualization

  • Compared with the variant without ASD, COMBINER produces stronger target-query similarity contrasts and brighter main-diagonal patterns in similarity matrices.The visualization is used to compare multimodal queries with their corresponding target images.Source: 2) Similarity Matrix Visualization · p. 13, Figure 7 · p. 13

Attention Visualization

  • GradCAM visualizations show attribute-level attention on regions and properties specified by modification text across FashionIQ and CIRR examples.Examples include attending to short sleeves in a target and the absence of a waist strap in a reference.Source: 3) Attention Visualization · p. 13, Figure 8 · p. 13

Semantic Cluster Neighbors

Related papers

Loading 2606.04604v1…