PaperlayerView exact revision on arXiv

Source-linked AI summary

COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relations

Zixu Li, Yupeng Hu, Zhiwei Chen, Haokun Wen, Xuemeng Song, Liqiang Nie

arXiv:2606.04604v1cs.CV

TL;DR

CIR methods can confuse visually similar images whose attributes do not match the composed query, while existing attribute-based approaches often require fixed labels or prototypes. COMBINER addresses this with disentangled attribute features, cross-modal unified prototypes, and attribute-based pairwise and neighbor relations, achieving optimal performance across metrics on three benchmark datasets.

  • Problem

    Existing CIR methods overlook visually similar but attribute-unrelated samples, while many attribute-based methods depend on fixed prototypes or explicit attribute annotations.

  • Method

    COMBINER disentangles attribute features, constructs Cross-modal Unified Prototypes, and models pairwise and neighbor relations using attribute similarity.

  • Results

    COMBINER achieves optimal performance across all metrics on three CIR benchmark datasets.

  • Takeaways & Limitations

    Attribute-prototype-based similarity distinguishes attribute-related samples from visually similar but attribute-unrelated samples for multimodal fusion and metric learning.

Abstract

from arXiv · show

Composed Image Retrieval (CIR) represents a challenging retrieval task that targets locating specific images through multimodal inputs. Despite recent progress in CIR techniques, prior approaches often overlook cases where images appear visually alike yet differ in attributes, potentially undermining both multimodal feature fusion and similarity modeling. To mitigate this limitation, we design a unified representation of cross-modal features based on attribute prototypes. Nevertheless, the task is far from straightforward, owing to three core issues: (1) entanglement in attribute-level semantics, (2) inconsistency across modalities, and (3) supervised signal missing. To tackle the above obstacles, we introduce a COMposed image retrieval network guided By attrIbute-based NEighbor Relations (COMBINER). Specifically, we first design an Adaptive Semantic Disentanglement module, which is capable of disentangling attribute features based on multimodal primitive features. Secondly, we propose a Unified Prototype-based Composition module, which can construct cross-modal unified prototypes (CUP) and facilitate multimodal feature composition. Finally, we introduce a Dual Relations Modeling module, which can mine pairwise and neighbor relations based on attribute similarity. Compared to traditional neighbor relations modeling CIR methods, COMBINER represents the first study addressing the phenomenon of visually similar but attribute-unrelated samples. It achieves a more accurate understanding of the semantic relations among samples by employing an attribute prototype-based similarity metric. Comprehensive experiments conducted on three benchmark datasets confirm the effectiveness of our proposed COMBINER. The implementation of our method will be accessed at https://github.com/Lee-zixu/COMBINER

I. INTRODUCTION

Composed Image Retrieval uses a reference image and modification text, but existing methods can confuse visually similar distractors with attribute-relevant samples. COMBINER addresses this through attribute-prototype representations, multimodal composition, and attribute-based relation modeling.

  • Motivation: CIR retrieves target images from a composed query combining a reference image with modification text.The reference approximates the target, while the modification describes the intended variation.
  • Problem: Existing pairwise and neighbor-relation methods can mishandle visually similar but attribute-unrelated images, treating them as negatives or pulling them closer incorrectly.A carpet-like candidate should not be brought close to a query requiring bedding.
  • Problem: Attribute-prototype methods address this challenge, but prior approaches depend on fixed prototypes or explicit attribute labels and are difficult to apply to open-domain CIR without annotations.They also commonly assume that visual similarity in non-target attributes should be preserved.
  • Approach: COMBINER builds a unified cross-modal representation from attribute prototypes to identify attribute-related samples and separate attribute-unrelated visual distractors.Its similarity measure uses prototype feature distributions to distinguish target, attribute-related, and attribute-unrelated images.
  • Approach: The model combines Adaptive Semantic Disentanglement, Unified Prototype-based Composition, and Dual Relations Modeling for multimodal fusion and metric learning.These modules disentangle attribute features, construct the CUP, and model pairwise and neighbor relations using attribute similarity.
  • Results: Experiments on three CIR benchmarks report optimal performance across all metrics, with COMBINER surpassing previously strongest methods that varied by dataset.The prior best methods included DQU-CIR on Shoes and FashionIQ and SPRC on CIRR.

II. RELATED WORK

COMBINER builds on CIR and prototype-learning research by addressing hard negatives that are visually similar but attribute-inconsistent. Its architecture combines adaptive attribute disentanglement, unified prototype composition, and dual relation modeling.

  • CIR retrieves target images from queries combining a reference image with modification text.
  • Existing CIR methods often rely on global alignment or visual similarity, which can confuse attribute-unrelated distractors with relevant neighbors.
  • COMBINER uses Semantic Attribute Attention to adaptively disentangle multi-grained attribute prototypes from CLIP features.
  • The method uses unified prototypes to reduce modality heterogeneity and construct attribute-level composed features.
  • COMBINER models supervised pairwise and unsupervised neighbor relations using attribute-level features.

C. Unified Prototype-based Composition

Unified Prototype-based Composition addresses inconsistent image and text attribute prototypes by constructing a shared prototype space. It then uses aligned modality weights to form attribute-level composed query features.

  • Image and text attribute prototypes may occupy inconsistent feature spaces, preventing direct cross-modal composition.
  • The module concatenates reference-image and modification-text prototypes, then applies SAA to derive Cross-modal Unified Prototypes.
  • Unlike standard attention over entangled tokens, CUP operates on disentangled prototypes for fine-grained compositional operations.
  • Sigmoid-weighted modality contributions are computed in CUP space and used to aggregate the original prototypes into composed features.

D. Dual Relations Modeling

Dual Relations Modeling combines supervised pairwise learning with unsupervised, attribute-based neighbor modeling. Semantic clustering and distribution consistency regularization gather attribute-related samples while separating attribute-unrelated ones.

  • The module models supervised pairwise relations and unsupervised neighbor relations using attribute-level features.
  • Pairwise Relations Modeling: Batch-based classification pulls each composed feature toward its matching target feature and also aligns CUP-derived composed representations with targets.
  • Unsupervised Neighbor Relations Modeling: Attribute-based neighbor modeling uses K-means on pooled target prototype features to form semantic clusters and centroids.
  • Unsupervised Neighbor Relations Modeling: Cluster-oriented classification draws queries and targets toward shared centroids representing visually similar, attribute-related neighbors.
  • Distribution Consistency: KL-divergence regularization matches query-target similarity distributions over cluster centroids and pooled features.

IV. EXPERIMENT

The experiments evaluate COMBINER against existing methods, component contributions, hyper-parameters, inference efficiency, and qualitative behavior. Evaluation follows standard protocols on Shoes, FashionIQ, and CIRR.

  • The study asks whether COMBINER surpasses state-of-the-art methods and how its components affect performance.
  • Additional experiments examine hyper-parameter sensitivity, inference efficiency, and qualitative performance.

1) Datasets:

The evaluation uses three benchmark datasets spanning fashion and open-domain composed image retrieval.

  • Datasets: The study evaluates Shoes, FashionIQ, and CIRR for composed image retrieval.Shoes and FashionIQ are fashion-domain datasets, while CIRR is open-domain.
  • Datasets: The dataset selection includes two fashion datasets and one open-domain dataset.
  • Datasets: The authors use these datasets to support fair comparison with previous work.

2) Implementation Details:

COMBINER is evaluated with standardized retrieval metrics against established CIR baselines across three datasets, with reported gains over prior methods.

  • Implementation Details: ViT-H/14 CLIP provides COMBINER’s feature backbone, with embedding dimension D = 1,024 and eight total attribute prototypes.The global and local prototype counts are both set to 4, giving U = 8.
  • Evaluation Protocol: Evaluation reports dataset-specific Recall@k metrics, including R@k, Rsubset@k, and CIRR’s mean of R@5 and Rsubset@1.Shoes uses k = 1, 10, 50; FashionIQ reports R@10 and R@50 by category; CIRR reports R@k and Rsubset@k.
  • Performance Comparison: COMBINER consistently exceeds every baseline across FashionIQ, Shoes, and CIRR.Reported improvements include 2.1% for FashionIQ-Avg-R@10, 2.9% relative improvement for Shoes-R@1, and 1.2% for CIRR-R@1.
  • Performance Comparison: Neighbor-relation modeling yields greater performance improvement on fashion datasets because they contain more visually similar samples.FashionIQ examples include dresses sharing the same color, creating visually similar candidates.

C. Ablation Study (RQ2)

Ablation results support the roles of semantic disentanglement, unified prototype composition, dual relation modeling, and distribution consistency, while cross-backbone tests indicate generalizable gains.

  • Module Ablation: Removing ASD decreases performance, while removing UPC produces the worst module-variant performance.These results support disentangling attribute prototypes and unifying them for multimodal composition and metric learning.
  • Module Ablation: Removing DRM lowers performance, indicating that dual relations modeling improves retrieval by optimizing neighbor relations.
  • Loss Ablation: Simultaneously pulling multimodal queries and target images toward semantic cluster centroids better gathers visually similar candidates into the same cluster.The joint setting outperforms variants that optimize only one of these two directions.
  • Loss Ablation: Removing either distribution-consistency regularizer causes poor performance, while removing both produces the lowest performance.The regularizers optimize the distributions of semantic clustering centroids and multimodal queries.
  • Backbone Robustness: Under identical CLIP backbones, COMBINER surpasses SADN by +3.47% on FashionIQ Avg R@10 and +2.15% on CIRR Avg.Comparisons across five backbones also show consistent superiority over corresponding same-backbone baselines.

3) Loss Ablation:

The ablations show that COMBINER’s attribute-aware Dual Relations Modeling depends on specialized losses, clustering, and distribution alignment choices rather than generic deep metric learning alone.

  • Loss Ablation: Replacing COMBINER’s losses with HIST or AFNE DML losses substantially degrades performance, showing that generic DML does not address attribute-irrelevant negatives.COMBINER w/ HIST drops by more than 7% on FashionIQ-Avg (R@10).
  • Loss Ablation: The DRM module is designed specifically to separate visually deceptive but attribute-incorrect samples from genuine false negatives during metric learning.Attribute-irrelevant samples are treated as genuine negatives, unlike potential positives handled by false-negative-oriented weighting methods.
  • Clustering Algorithm Ablation: K-Means remains competitive with hierarchical clustering, whereas DBSCAN and spectral clustering noticeably reduce performance.The paper attributes these degradations to high-dimensional noise sensitivity for DBSCAN and difficulty constructing semantically appropriate graphs for spectral clustering.
  • Alignment Strategy Ablation: The default KL divergence significantly outperforms JS divergence and Wasserstein-1 distance across all reported metrics.The authors interpret KL as supporting unidirectional supervision from target distributions to composed-feature distributions.

D. Sensitivity Analysis (RQ3)

Sensitivity analyses examine trade-off weights, prototype counts, cluster counts, and inference efficiency across FashionIQ and CIRR, finding generally stable behavior with identifiable parameter boundaries.

  • Hyper-parameter Sensitivity: COMBINER’s performance changes with the loss weights ρ, κ, and µ, with dataset-specific sensitivity patterns observed in FashionIQ and CIRR.The analysis varies cluster-oriented classification, cluster-oriented consistency, and pool-oriented consistency hyper-parameters.
  • Prototype Sensitivity: Excessive attribute prototypes can reduce performance by disrupting subsequent neighbor-relation modeling, after which performance levels off.The reported pattern is a slight decline followed by stabilization as the prototype count increases beyond a suitable range.
  • Cluster Sensitivity: Performance first increases and then decreases as the semantic cluster count H grows, while the maximum-to-minimum difference remains below 1.0.The authors associate insufficient cluster centers with inadequate prototype learning and report robustness to H changes overall.
  • Inference Efficiency: COMBINER improves retrieval without external models by learning attribute-based neighbor relations that separate visually similar samples with different attributes.The efficiency comparison reports inference time and retrieval performance against DQU-CIR on a single NVIDIA A40 GPU.

F. Qualitative Analysis (RQ5)

Qualitative analyses show that COMBINER attends to requested attributes, forms semantically consistent neighbors, and distinguishes attribute-related from visually similar but unrelated images, while exposing annotation and disentanglement limitations.

  • Case Study: COMBINER’s retrieved top-K images can satisfy modification semantics even when dataset annotations label another image as the sole target.This creates false-negative evaluation penalties despite semantically correct retrievals.
  • Case Study: COMBINER can struggle when visually dominant attributes overshadow fine-grained structural details in highly coupled attribute combinations.The authors identify this as a capacity boundary of the Adaptive Semantic Disentanglement module.
  • Similarity Visualization: Compared with the variant without ASD, COMBINER produces stronger target-query similarity contrasts and brighter main-diagonal patterns in similarity matrices.The visualization is used to compare multimodal queries with their corresponding target images.
  • Attention Visualization: GradCAM visualizations show attribute-level attention on regions and properties specified by modification text across FashionIQ and CIRR examples.Examples include attending to short sleeves in a target and the absence of a waist strap in a reference.
  • Semantic Cluster Neighbors: K-means semantic clusters gather neighbors sharing requested attributes, such as blue panels or fruit and vegetables, despite differing reference-image content.These clusters provide qualitative evidence for attribute-consistent neighbor relations in FashionIQ and CIRR.
Loading 2606.04604v1…