Optimizing VLP-aligned Multimodal Intent Representation with Correct Visual Instantiation for Zero-Shot Composed Image Retrieval

1 Introduction

• Fully supervised CIR relies on costly annotated reference–edit–target triplets and can generalize poorly to unseen compositions or domains, motivating zero-shot retrieval without paired supervision.Zero-shot CIR is intended to improve practicality and scalability in open, dynamic settings.Source: §1 · p. 2
• Under frozen VLP backbones, zero-shot CIR must form a retrievable composed-query representation that matches native text-space style while integrating the correct reference visuals and modification intent.The representation should capture user intent accurately and comprehensively enough to match target images.Source: §1 · p. 2
• Pseudo-word methods preserve reference content but coarse concatenation can inject irrelevant visual noise and hinder native VLPs from understanding composed intent.Noise can include background clutter, unrelated objects, or unchanged content; the resulting query resembles a concatenated edit instruction rather than a concise target description.Source: §1 · pp. 3–4
• MLLM-based methods may omit explicit visual instantiation cues from the reference image, making the intended object category and specific reference instance ambiguous, and often generate overly elaborate queries difficult for frozen VLP retrieval backbones.Source: §1 · p. 4
• VMIR-CVI combines target-oriented VLP-aligned reasoning, training-free visual-instance disentanglement, and hybrid fusion to form concise, visually grounded, VLP-compatible queries.VMIR guides MLLMs to reason over intent-consistent reference instances; TVID separates fine-grained cues from global features to suppress irrelevant noise; HIAF integrates those cues with reasoned text using pretrained VLP priors.Source: §1 · pp. 4–5

• VMIR-CVI is a unified zero-shot CIR paradigm that integrates fine-grained visual pseudo-word embeddings with VLP-aligned MLLM intent reasoning to bridge multimodal intents and targets.The framework is described as addressing reference inconsistency.Source: §1 · p. 5
• VMIR uses VLP-aligned chain-of-thought reasoning to generate retrieval-aligned textual queries with explicit, reasoned, target-consistent intent.Source: §1 · p. 5
• TVID is a training-free module that decouples instance-level cues from frozen VLP visual features without additional optimization.Source: §1 · p. 5
• HIAF is a lightweight module that integrates VLP-aligned textual intent and correctly instantiated visual cues into a coherent intent embedding via pretraining.The stated aim is accurate and robust zero-shot CIR.Source: §1 · p. 5
• VMIR-CVI consistently outperforms existing zero-shot CIR methods on CIRR, CIRCO, and FashionIQ, achieving new state-of-the-art results.Source: §1 · p. 5

Zero-shot CIR with Visual Pseudo-word Learning

• Existing pseudo-word methods map reference-image features into VLP embedding space as pseudo-word tokens, turning multimodal retrieval into text-to-image retrieval [33].The passages also note that these methods often map from global references and concatenate pseudo-mappings with modification text.Source: §2 · p. 5
• Pic2Word uses a lightweight network to map images into word tokens, enabling semantic fusion and flexible composition in language space.Source: §2 · p. 5
• SEARLE and iSEARLE combine optimized textual inversion with GPT-driven semantic regularization and knowledge distillation to improve inference efficiency and semantic consistency.Source: §2 · p. 5
• FTI4CIR addresses the limitation that global tokens struggle to capture fine-grained visual details by proposing a fine-grained approach.Source: §2 · p. 5
• Existing pseudo-word methods can ignore co-occurrence associations among visual instances, introduce visual noise, and use coarse concatenation with modification text that is suboptimal for VLP backbones.The limitation is stated for mappings learned from global reference features and for methods that concatenate pseudo-mappings with modification text.Source: §2 · p. 5

Zero-shot CIR with LLM-guided Intent Reasoning and Understanding

• LLM-guided methods analyze the semantic increment between reference images and modification text, while CIReVL combines VLM captions, LLM recombination, and CLIP retrieval.OSrCIR applies a reflective chain-of-thought framework to process image and text together.Source: §2 · pp. 5–6
• Existing MLLM reasoning methods rely on textual queries while neglecting effective visual reference cues, and rarely optimize reasoning content for VLP embedding spaces.The passages link these limitations to inconsistencies between reference and target intent instances and limited use of VLP performance.Source: §2 · p. 6

3 The Proposed Method

• VMIR-CVI combines MLLM-based intent-target caption reasoning with context-aware instance pseudo-word embeddings under a frozen VLP backbone.It generates VLP-aligned target descriptions, preserves fine-grained visual-instantiation cues, and reconstructs a hybrid-modal composed-intent representation.Source: §3 · p. 6

3.1 Preliminaries

• In zero-shot CIR, the composed query is expected to align with the implicit textual description of the target rather than receive direct supervision to approximate its visual embedding.The task retrieves a target from a reference image and modification text without annotated query–target pairs.Source: §3.1 · pp. 6–7

3.2 VLP-aligned Multimodal Intent Reasoning

• VMIR uses multi-step chain-of-thought prompting and few-shot VLP-native query examples to generate VLP-aligned target descriptions and infer which reference instances to retain.The retained-instance reasoning supports correct visual instantiation while reducing subsequent computation.Source: §3.2 · p. 7

3.3 Training-free Visual Instance Disentanglement

• TVID uses the pre-trained VLP visual encoder without additional training to select patches responding more strongly to their instance keyword than to competing keywords or the non-semantic reference.Attention-weighted pooling aggregates selected patches into instance-level visual representations.Source: §3.3 · pp. 8–9

3.4 Hybrid-modal Intent Alignment and Fusion

• HIAF pre-trains hybrid intent by combining VMIR descriptions and salient visual-instance keywords with context-aware instance embedding and adaptive fusion through a learnable Combiner.During pre-training, VMIR generates a VLP-native description and identifies salient visual-instance keywords from an image.Source: §3.4 · p. 9
• Pre-training optimizes the projector and Combiner with a CLIP-style bidirectional text-to-image and image-to-text contrastive objective.The paper states that pre-training yields a fusion strategy compatible with the VLP embedding space.Source: §3.4 · pp. 10–11
• At inference, VMIR supplies a VLP-aligned target description while TVID provides reference-instance cues for keyword replacement, and the Combiner fuses both representations for image retrieval.Retrieval ranks gallery images by similarity to the resulting hybrid intent representation.Source: §3.4 · pp. 10–11
• CIPE replaces instance keywords in the VLP-aligned description with projected visual-instance embeddings, preserving linguistic structure while grounding semantic tokens in visual cues.An MLP projection maps each visual-instance embedding into the VLP text space; the resulting pseudo-word replaces the corresponding keyword.Source: §3.4 · p. 10
• A learnable Combiner balances global textual semantics and instance-level visual grounding by softly weighting their shared-encoder representations.The lightweight MLP Combiner predicts softmax fusion weights from the projected concatenation of the pure-text and hybrid-sequence embeddings.Source: §3.4 · p. 11

Dataset

• CIRR and Fashion-IQ cover open-world scenes and specialized fashion domains, respectively; Fashion-IQ has Dress, Shirt, and Toptee categories.CIRR contains 36,554 triplets, and Fashion-IQ contains 30,134 triplets.Source: §4 · p. 11
• CIRCO is an open-domain benchmark with 1,020 query triplets, while CIRR and CIRCO emphasize diverse object interactions in real-world scenes.The authors describe evaluation on CIRR and CIRCO as better reflecting real-world generalization ability.Source: §4 · p. 11

Evaluation metrics

• CIRR uses Recall@K for K=1, 5, 10, and 50; CIRCO uses mAP@K for K=5, 10, 25, and 50.FashionIQ reports Recall@10 and Recall@50 for each category and their mean values.Source: §4 · p. 12

Implementation details

• HIAF pretraining uses 100,000 ImageNet2012 images and Qwen3-VL-30B-A3B-Instruct for captions and instance annotations.The Projector and Combiner are trained for 3 epochs with learning rate 2 × 10^-5 and batch size 32.Source: §4 · p. 12
• At inference, Qwen3-VL-30B-A3B-Instruct performs chain-of-thought reasoning to generate the target caption.The experiments fix the random seed to 43.Source: §4 · p. 12

Research questions

• The analyses ask how VMIR-CVI compares with state of the art and how VMIR, TVID, and HIAF affect retrieval, efficiency, and qualitative results.The comparison protocol examines benefits from accurate textual intent reasoning and explicit visual instantiation.Source: §5 · p. 12

Evaluation protocol

• Comparisons use zero-shot CIR; CIRR and CIRCO results are reported on test sets, while FashionIQ results use the validation set following prior work.The evaluation follows the dataset splits, metrics, and implementation settings described in Section 4 unless otherwise specified.Source: §5 · p. 12

Baselines

• The baseline set includes visual pseudo-word methods such as Pic2Word, SEARLE, LinCIR, Context-I2W, and LDRE, alongside MLLM/LLM reasoning methods.The listed reasoning approaches include CIReVL, OSr-CIR, and recent variants.Source: §5 · p. 12

5.1.1 Overall comparison protocol.

• The evaluation covers open-world CIRR, open-domain CIRCO with multiple valid targets, and FashionIQ attribute modifications, using ViT-B/32, ViT-L/14, and ViT-G/14 where baseline results are available.CIRR reports global and subset Recall@K; CIRCO uses mAP@K.Source: §5.1.1 · p. 12

5.1.2 Results on CIRR

• With ViT-G/14 on CIRR, VMIR-CVI improves average Recall@k by 2.21 points and average RecallSubset@k by 2.71 points, while matching the best Recall@50 result.The CIRR gains remain consistent as backbone capacity increases.Source: §5.1.2 · p. 13
• VMIR produces concise target-oriented descriptions, TVID separates intent-consistent visual instances from irrelevant content, and HIAF integrates visual cues with VLP-aligned text.The resulting representation is described as semantically explicit and visually grounded.Source: §5.1.2 · p. 14

5.1.3 Results on CIRCO

• VMIR-CVI achieves the best CIRCO mAP at every cutoff across all three VLP backbones, with statistically significant gains over the second-best methods.Reported gains span mAP@5 through mAP@50.Source: §5.1.3 · p. 14, Table 2 · p. 14

5.1.4 Results on FashionIQ

• On FashionIQ, VMIR-CVI surpasses OSrCIR by 2.83/3.89 and 3.14/3.41 points in average Recall@10/Recall@50 with ViT-B/32 and ViT-L/14, respectively.The strongest category-specific results are especially evident on Tops&Tees.Source: §5.1.4 · p. 16
• VMIR-CVI leads overall on CIRR and CIRCO across evaluated backbones and achieves the best average FashionIQ results across ViT-B/32, ViT-L/14, and ViT-G/14.FashionIQ results are slightly lower on Shirts under ViT-G/14.Source: Table 1 · p. 13, Table 2 · p. 14, §5.1.5 · p. 16

5.2 Ablation Studies (RQ2).

• The ablation protocol removes each major component, then tests module-specific alternatives, including reasoning strategies, visual-feature approaches, and HIAF components.The study uses ViT-L/14 as its backbone.Source: §5.2 · p. 16

5.2.1 Overall contribution of VMIR, TVID, and HIAF.

• Across FashionIQ, CIRCO, and CIRR, removing VMIR, TVID, or HIAF degrades performance; removing HIAF causes the largest drop, particularly on CIRCO.The results indicate that textual intent and visual cues must be aligned and fused into a unified query embedding.Source: §5.2.1 · pp. 16–17, Table 4 · p. 17, Table 5 · p. 17

5.2.2 Ablation on VMIR.

• Compared with Basic MCoT, VMIR raises average FashionIQ scores by around 4–6 points across categories and the average CIRR score by nearly 3 points.Basic MCoT also consistently improves over direct reasoning without MCoT across the three benchmarks.Source: Figure 4 · p. 17, §5.2.2 · p. 18

5.2.3 Sensitivity to different MLLMs.

• On FashionIQ, VMIR’s Qwen3-VL reasoning models show a clear scaling trend: 2B scores 40.26/37.93/45.87/26.69/58.16%, 8B 41.42/39.24/46.92/28.29/61.61%, and 30B-A3B 45.85/43.08/52.35/28.96/62.69%.The 8B-to-30B gain is relatively smaller on CIRCO.Source: Table 6 · p. 18, §5.2.3 · pp. 18–19

5.2.4 Ablation on TVID.

• TVID consistently outperforms Global Attention and Basic Attention across FashionIQ, CIRCO, and CIRR; on FashionIQ, category-wise average scores reach 45.85/43.08/52.35, improving over the stronger baseline by up to 2.63 points.On CIRCO, TVID also improves the average score to 28.96.Source: §5.2.4 · p. 19
• Global Attention can introduce clutter and irrelevant cues, while Basic Attention may localize salient but modification-irrelevant regions; TVID instead preserves intent-consistent instances.TVID provides cleaner instance-level visual evidence for the final hybrid-modal query.Source: §5.2.4 · pp. 19–20
• The HIAF analysis follows validation of VMIR and TVID and examines how it integrates reasoned textual intent with disentangled visual instance cues.The supplied passage introduces this ablation but does not report specific TVID ablation results.Source: §5.2.5 · p. 20

5.2.5 Ablation on HIAF.

• Removing CIPE drops average scores by 2–4 points on FashionIQ categories and 2.63 points on CIRR, showing that instance cues need contextual placement.Contextual placement within target-oriented descriptions preserves sentence structure while grounding visual concepts.Source: §5.2.5 · p. 20
• Removing target texts lowers average scores by 9.04 points on CIRCO and 9.47 points on CIRR, the largest reported degradation.The results indicate that MLLM-reasoned target descriptions provide the semantic backbone; visual instance cues alone cannot fully express compositional intent.Source: §5.2.5 · p. 20, §5.2.6 · p. 21
• Removing the Combiner consistently reduces performance, indicating that globally clear textual queries and visually specific instance-augmented queries provide complementary retrieval signals.The adaptive Combiner balances textual and visual evidence; the ablations identify it, CIPE, and reasoned target texts as necessary and complementary components.Source: §5.2.5 · p. 21

5.3 Efficiency Analysis (RQ3)

• VMIR-CVI uses 100K images and around 10GB GPU memory for alignment pre-training, compared with 3M images for Pic2Word and 2.47M texts for LinCIR.The VLP backbone remains frozen; only lightweight mapping and fusion modules are trained.Source: §5.3 · p. 21
• Inference latency is around 1.5s, comparable to mainstream MLLM-based zero-shot CIR methods, while VMIR-CVI achieves stronger retrieval performance.The method does not use iterative optimization or additional model fine-tuning at inference.Source: §5.3 · p. 21
• The efficiency analysis attributes VMIR-CVI’s performance gains mainly to more effective VLP-aligned multimodal intent representation, not heavy backbone tuning or expensive inference-time computation.The authors characterize the system as achieving minimal alignment pre-training cost with practical inference latency.Source: §5.3 · p. 21

5.4 Qualitative Analysis of VMIR and TVID (RQ4)

• VMIR generates concise target descriptions such as “black cap” and “a short-sleeve zebra print,” which the authors say help the VLP encoder perceive attributed objects more accurately.The qualitative comparison is against basic CoT reasoning and includes top-3 retrieval results.Source: §5.4 · p. 22
• TVID concentrates responses on precise visual regions, whereas CLIP-based soft attention produces diffuse activations over background regions or co-occurring objects.The examples use keywords including “Bottle,” “Black Cap,” and “Zebra Print.”Source: §5.4 · p. 22
• The authors conclude that VMIR improves query-level consistency with the VLP query paradigm while TVID enhances fine-grained visual decoupling at the reference level.Together, they align composed intent with visual evidence.Source: §5.4 · p. 22

5.5 Discussion and Limitations

• VMIR-CVI shows the strongest and most consistent gains on CIRR and CIRCO, while FashionIQ performance is best on average across backbones but slightly behind baselines on Shirts.The discussion connects CIRR and CIRCO results with precise target identification and retrieval over multiple valid targets.Source: §5.5 · pp. 22–23
• CIRCO results indicate robust intent-level retrieval over multiple valid targets, alongside precise target identification.The discussion notes that CIRCO evaluates ranking quality over multiple valid targets in diverse open-domain scenarios.Source: §5.5 · pp. 22–23
• On FashionIQ, VMIR-CVI performs best on average across backbones for fine-grained attribute edits, with category-level variation and a Shirts exception.It leads on Dresses, Tops&Tees, and overall, but trails the strongest baselines on Shirts.Source: §5.5 · p. 23
• Ablations describe VMIR as the retrieval-aligned semantic anchor, TVID as the source of instance-level visual grounding, and HIAF as their integrator.TVID extracts intent-consistent instance cues while suppressing irrelevant reference content.Source: §5.5 · p. 23
• Removing target text causes the largest HIAF degradation, while CIPE results favor placing visual pseudo-words in semantically appropriate positions.These findings support using visual instantiation to complement target-oriented semantics rather than replace it.Source: §5.5 · p. 23
• Performance depends on MLLM intent-reasoning quality, category-specific retrieval patterns, and fine-grained discrimination among near-duplicates.Subtle visual differences and ambiguous modification text make exact top-ranked image selection challenging; the authors identify more reliable reasoning, adaptive instance selection, and efficient fusion as future work.Source: §5.5 · p. 23

6 Conclusion and Future Work

• VMIR-CVI integrates VLP-aligned multimodal intent reasoning with correct visual instantiation to optimize multimodal intent representations.It targets the gap between composed queries and target images under frozen VLP backbones.Source: §6 · p. 23
• The framework combines retrieval-aligned target queries with disentangled fine-grained visual cues to capture semantic intent and discriminative visual evidence.Its fusion mechanism integrates these signals in the VLP embedding space while suppressing intent-irrelevant noise.Source: §6 · p. 23
• Across multiple VLP backbones and CIRR, CIRCO, and FashionIQ, VMIR-CVI consistently outperforms existing zero-shot CIR methods and establishes new state-of-the-art results.The conclusion reports extensive experiments on all three benchmarks.Source: §6 · pp. 23–24
• Performance depends on MLLM intent-reasoning quality, and frozen VLP features may be insufficient for complex scenes, subtle attributes, or ambiguous object interactions.Inaccurate target descriptions or retained-instance predictions may affect subsequent visual instantiation and fusion.Source: §6 · p. 24
• Future work will explore adaptive instance alignment, more reliable intent reasoning, and more efficient hybrid-modal fusion while preserving efficiency.The stated aim is to further improve robustness and generalization.Source: §6 · p. 24