PaperlayerView exact revision on arXiv

Source-linked AI summary

MulVec: Fine-Grained Role-Aware Matching for Training-Free Zero-Shot Composed Image Retrieval

Zihao Zhang, Dayan Wu, Xinze Liu, Hengjie Zhu, Yiliang Zhu, Ding Wang, Peng Fu, Zheng Lin, Weiping Wang

arXiv:2608.25305v1cs.CV

TL;DR

Training-free zero-shot composed image retrieval must combine requested changes with preserved source content, yet global matching can mix cues and miss fine-grained evidence. MULVEC compiles queries into four roles and applies role-specific matching to shared global-local candidate evidence. It improves CIRCO mAP@5 by up to 23.0% over the strongest compared method and gives the best CIRR and FashionIQ results in the comparison.

  • Problem

    Global matching can mix semantic cues and lose fine-grained details when retrieving an edited target from a reference image and text modification.

  • Method

    MULVEC compiles each query into Global, Desired, Preserve, and Forbidden roles and matches their probes against shared global and local candidate visual evidence.

  • Results

    23.0%: MULVEC improves CIRCO mAP@5 over the strongest compared method and achieves the best results on CIRR and FashionIQ.

  • Takeaways & Limitations

    Matched controls and operator ablations show that retrieval gains depend on both correct role assignment and role-specific matching, while local vectors strengthen the mechanism.

Abstract

from arXiv · show

Training-free zero-shot composed image retrieval finds a target image in a gallery from a reference image and a text edit without learning from task-specific image triplets. Existing methods typically describe the target as a whole and match this description with a global image representation. This global matching can mix different semantic cues and lose fine- grained details. We propose MULVEC, a role-aware method whose compiler produces a structured query record that is mapped to four retrieval roles: Global describes the full target, Desired states what should appear, Preserve states what should remain, and Forbidden states what should disappear. Frozen encoders map the query to one target description vector and role-specific probe vectors, while each candidate is represented by one global visual vector and a bank of local visual vectors. The retrieval roles then use this shared evidence for their respective purposes, and a fixed weighted sum of their scores ranks the entire gallery in a single retrieval pass. Across CIRCO, CIRR, and FashionIQ and three backbone scales, MULVEC improves CIRCO mAP@5 by up to 23.0% over the strongest compared method and gives the best CIRR and FashionIQ results in our comparison.

I. INTRODUCTION

Zero-shot composed image retrieval must satisfy requested changes while preserving relevant source content, but global matching can entangle semantic cues and miss fine-grained details. MULVEC addresses this with four role-specific retrieval functions and shared global-local candidate evidence.

  • Zero-shot composed image retrieval searches a gallery from a reference image and text edit without task-specific triplet training.
  • Existing methods can entangle requested constraints because a fused target description and global representation dilute decisive local evidence.
  • MULVEC maps each query to Global, Desired, Preserve, and Forbidden roles with distinct matching purposes.
  • Frozen encoders produce target and role-specific probe vectors, while candidates use global and local visual vectors ranked in one fixed weighted retrieval pass.
  • 23.0%: MULVEC improves CIRCO mAP@5 over the strongest compared method and achieves the best results on CIRR and FashionIQ.

II. RELATED WORK

Related work progresses from single-space and learned composition methods toward training-free query construction, multiple retrieval signals, and structured evidence. MULVEC is positioned within this progression as a training-free approach using explicit role-aware matching.

  • Supervised CIR methods learn composition from annotated reference–text–target triplets using shared embeddings and increasingly fine-grained interactions.
  • Zero-shot CIR methods may still learn task-specific mappings, projections, fusion modules, or composition components despite removing annotated target-domain triplets.
  • Training-free methods construct queries and retrieval scores at inference using rewritten captions, local-concept reranking, diverse edited captions, or direct reasoning.
  • Other training-free approaches combine textual and visual transformations, dual descriptions, proxy images, prompt directions, or separate reference and text similarities.
  • Recent methods structure retrieval with global and object-scale descriptions, semantic transitions, prescriptive and proscriptive constraints, or reference-aware bias correction.

C. Role-Aware Matching with Fine-Grained Evidence

Fine-grained candidate features expose small visual details, but MULVEC’s distinction is assigning different operators to Global, Desired, Preserve, and Forbidden roles. The same role-aware formulation can use global or global-plus-local evidence.

  • Local or set-valued features expose small visual details through region–word interaction, reranking, object-scale scoring, and set transportation.
  • SoFT separates must-have and must-avoid constraints with global filters, while SDR-CIR uses reference-aware bias around one composed target description.
  • MULVEC separately compiles Global, Desired, Preserve, and Forbidden so each role reads the same candidate evidence bank differently.
  • Desired rewards local support, Preserve checks local continuity against the reference, and Forbidden penalizes local support.
  • The system assigns each candidate a score from the reference and modification, then returns the gallery in descending score order without inference-time relevance labels.

B. Method Overview

MULVEC treats composed requests as role-specific evidence-matching problems rather than applying one shared matching rule. It combines structured query roles with global and local candidate evidence, then dispatches each role to its appropriate operator.

  • MULVEC maps Desired, Preserve, and Forbidden content to different scoring behaviors: reward appearance, check reference continuity, and penalize states that should disappear.
  • A frozen compiler maps the reference and edit into Global, Desired, Preserve, and Forbidden fields without accessing targets, candidates, or relevance labels.
  • A frozen vision–language encoder represents Global with one target vector and the other roles with role-specific probe-vector sets.
  • Each candidate is represented by one global visual vector and a bank of local visual vectors, while the local operators share a smooth late-interaction primitive.
  • Role-aware dispatch reads shared candidate evidence differently: Global uses global compatibility, Desired rewards required local content, Preserve uses reference-dependent continuity, and Forbidden measures removable source states.
  • A fixed weighted sum ranks the gallery once, with inactive Desired and Forbidden roles omitted and active weights renormalized.

A. Experimental Setup

The evaluation tests MULVEC as a training-free, direct single-pass method across open-domain and fashion retrieval benchmarks. Public comparisons are followed by matched controls and interventions designed to separate role handling from representation granularity.

  • MULVEC is evaluated on CIRCO, CIRR, and FashionIQ using their respective mAP and recall metrics.
  • The experiments first compare the complete framework, then use matched controls and targeted interventions to distinguish explicit role handling from simply adding candidate vectors.
  • All results use frozen OpenCLIP towers at ViT-B/32, ViT-L/14, or ViT-G/14 scale, with cached global and local image vectors.
  • The comparison is restricted to training-free methods that directly score the gallery in one pass; candidate-conditioned calls, iterative retrieval, verification, and rescue loops are excluded.
  • Compared methods span caption rewriting, reflective reasoning, composed-caption generation, prompt-direction transfer, dual signals, proxy images, and structured retrieval evidence.

B. Overall Retrieval Effectiveness

MULVEC achieves the strongest reported retrieval results across CIRCO, CIRR, and FashionIQ in the selected comparisons. Its gains remain consistent across backbone scales and across open-domain and category-specific settings.

  • 23.0%: MULVEC improves CIRCO mAP@5 over the best available comparator with ViT-B/32, alongside 22.4% for ViT-L/14 and 19.7% for ViT-G/14.
  • MULVEC leads all three CIRR full-gallery recalls and all three subset recalls at every evaluated backbone scale.
  • MULVEC achieves the highest category-specific and averaged FashionIQ R@10 and R@50 at all three backbone scales, leading all 24 reported metrics.
  • The results are consistent across open-domain CIRCO and CIRR and category-specific FashionIQ, suggesting the benefit is not tied to one query style or visual domain.
  • The same role-aware scorer remains useful from ViT-B to ViT-G with frozen towers, while case analyses show global-description optimization can miss local constraints.

C. Evidence for Role-Aware Matching

Matched controls test whether retrieval gains come from explicit role handling rather than simply adding vectors. Role-aware dispatch improves results across evidence granularities and datasets, while local vectors increase evidence resolution.

  • Matched control: Matched controls keep compiler outputs and frozen weights fixed while comparing a shared read against role-specific dispatch.The shared read concatenates nonempty Desired, Preserve, and Forbidden probes; role-aware dispatch applies positive, reference-aware, and negative operators.
  • Matched control: 7.08 and 5.58 points are the role-aware gains on CIRCO for Global only and Global + local evidence, respectively.The corresponding gains are 5.26 and 5.50 on CIRR, and 16.46 and 12.36 on FashionIQ.
  • Cross-dataset evidence: Role-aware dispatch improves the reported metric across both candidate-evidence settings on CIRCO, CIRR, and FashionIQ.This result is reported for both single-global and global-plus-local candidate representations.
  • Qualitative evidence: The qualitative examples involve multiple constraints, including age, added objects, style, and role binding.MULVEC retrieves labeled targets at rank 1 in the reported examples.
  • Evidence granularity: Local visual vectors improve both shared and role-aware reads, but their gains overlap rather than add fully.The strongest configuration combines local vectors with role-aware dispatch, showing that roles determine how evidence affects ranking while vectors increase its resolution.

2) Role Components and Role-Specific Operators:

Ablations show that the four roles and their asymmetric operators contribute differently to retrieval. Global is the strongest component overall, while correct role assignment, reference continuity, and Forbidden’s negative sign each matter independently.

  • Role components: Global gives the largest contribution on all three datasets when removed from the G/D/P/F combination.Removing it lowers CIRCO mAP@5, CIRR Recall@1, and FashionIQ macro Recall@10 by 15.15, 13.64, and 14.34 points.
  • Role components: Desired is consistently useful, Preserve matters most on FashionIQ, and Forbidden has metric-dependent effects.Forbidden removal lowers CIRR full-gallery Recall@1 by 1.20 points and CIRR subset Recall@1 from 71.06 to 69.29.
  • Role-specific operators: Each isolated intervention—role reassignment, removing reference continuity, or rewarding Forbidden—lowers all three reported metrics.These interventions preserve other components sufficiently to distinguish role assignment and operator effects.
  • Role-specific operators: Role reassignment shows that extra probes are insufficient unless their meanings control the matching operators.Preserved content requires a source-relative test, while removed content acts as counterevidence.
  • Design boundary: CIRCO 60 improves from best-positive rank 5 to 2 with Global only evidence and from 4 to 1 with Global + local evidence.For CIRCO 22, no configuration retrieves a labeled positive within the Top-50.

3) Softmax Temperature Sensitivity:

Temperature sensitivity tests whether local late interaction depends on a narrow setting. Across all three reported metrics, τ = 0.02 performs best, while the datasets differ in sensitivity.

  • Temperature sweep: τ = 0.02 gives the highest score on CIRCO, CIRR, and FashionIQ under the tested temperature grid.Only τ changes; compiler outputs, retrieval weights, and other settings remain frozen.
  • Temperature sweep: Adjacent temperatures 0.01 and 0.04 are lower by 0.70 and 0.52 points on CIRCO, 0.96 and 0.24 on CIRR, and 0.62 and 0.28 on FashionIQ.Small τ approaches hard local selection, whereas large τ spreads weight across weaker matches.
  • Dataset sensitivity: FashionIQ varies by only 0.65 points across the full grid, compared with 1.53 on CIRCO and 1.68 on CIRR.The two open-domain settings are more sensitive to local-evidence concentration in this experiment.

4) Role-Weight Sensitivity:

Role-weight sensitivity evaluates local robustness around the universal weights using exact full-gallery ranking and active-role normalization. Performance remains locally stable, with dataset-specific sensitivity to Global and Preserve.

  • Robustness: The robustness conclusion does not rely on one isolated weight setting.The sweep scales one frozen coefficient at a time across five values with query-specific active-role normalization.
  • Weight sweep: Across 16 non-anchor settings per dataset, the metric drops by at most 1.56 points on CIRCO, 0.45 on CIRR, and 0.55 on FashionIQ.The best sampled neighbor differs from the frozen point by −0.07, +0.19, and +0.02 points, respectively.
  • Dataset sensitivity: Reducing Global by 25% causes the largest CIRCO drop, while increasing Preserve by 25% causes the largest FashionIQ drop.This identifies different local sensitivities despite the shared frozen weighting scheme.
  • Dataset sensitivity: The response is locally stable but not uniform across datasets and role coefficients.CIRCO is most sensitive to weakening the target description vector, whereas FashionIQ is most sensitive to overemphasizing preservation.
Loading 2608.25305v1…