PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models

Fan Xu, Luis A. Leiva

arXiv:2608.23102v1cs.CVcs.IR

TL;DR

Composed Image Retrieval requires combining reference images with textual modifications, while conventional multimodal fusion depends on task-specific training. PeFuse addresses this gap with training-free generative conversion using Diffusion Models and Multimodal Large Language Models. Across standard benchmarks, text-to-image reformulation is more effective than alternative conversions and achieves competitive or superior performance, while retaining modularity.

  • Problem

    CIR must align visual and textual content compositionally, but conventional fusion methods require dedicated training to combine modalities into target-aligned query embeddings.

  • Method

    PeFuse uses Diffusion Models and Multimodal Large Language Models to convert multimodal CIR queries into single-modality retrieval tasks through uni-directional and bi-directional pseudo-fusion.

  • Results

    Text-to-image reformulation is more effective than alternative conversion strategies and achieves competitive or superior performance against state-of-the-art models on standard CIR benchmarks.

  • Takeaways & Limitations

    Pseudo-fusion enables standard single-query retrieval systems to support zero-shot CIR in a flexible, plug-and-play manner without training new modules.

  • Takeaways & Limitations

    Chaining Diffusion models and MLLMs can propagate concept-drift errors, while hyperparameter tuning is labor-intensive and may not generalize across datasets.

Abstract

from arXiv · show

Composed Image Retrieval (CIR) is an emerging paradigm in content-based image retrieval that enables users to formulate compositional queries by combining a reference image with an auxiliary modality, usually text-based. This approach supports fine-grained search where the target image shares structural elements with the user-provided image while incorporating the modifications specified by the auxiliary text. Conventional CIR methods rely on multimodal fusion to combine visual and textual features into a joint query embedding, which requires training modules that align composed queries with the targets. In this work, we propose PeFuse (for pseudo-fusion), a training-free framework that leverages pretrained Diffusion Models and Multimodal Large Language Models to bridge modalities via generative conversion. We introduce two novel strategies: uni-directional and bi-directional conversion, which convert CIR into four single-modality retrieval problems. These methods reformulate CIR as either intra-modal or cross-modal single-query retrieval tasks, bypassing the need for dedicated task-specific training. Extensive experiments on standard benchmarks demonstrate that converting CIR into text-to-image retrieval tasks is more effective than alternative conversion strategies, achieving competitive or superior performance compared with state-of-the-art methods, while maintaining high flexibility thanks to replaceable components of the conversion pipeline. These results highlight the effectiveness of the pseudo-fusion paradigm for zero-shot CIR. Our code is publicly available at: https://github.com/StevenXuf/PeFuse4CIR.

1 Introduction

CIR enables fine-grained multimodal search, but conventional approaches typically require trained fusion modules to align composed queries with targets. PeFuse instead uses training-free generative conversion and benchmarks uni-directional and bi-directional strategies, finding text-to-image reformulation especially effective.

  • Motivation: CIR combines a reference image with an auxiliary modality to express desired modifications for fine-grained retrieval.The approach is useful for nuanced search intents in domains including e-commerce, digital asset management, and creative design.
  • Challenges: Effective CIR must model both individual modalities and how modifications alter the reference image’s meaning.
  • Challenges: Existing multimodal fusion methods integrate visual and textual representations into unified embeddings but generally depend on dedicated training with annotated CIR data.This reliance limits the training-free setting targeted by PeFuse.
  • Proposed approach: PeFuse pseudo-fuses queries through uni-directional and bi-directional conversion using Diffusion Models and Multimodal Large Language Models.The approaches convert multimodal queries into single-modality retrieval problems and can adapt off-the-shelf systems without architectural changes, fine-tuning, or training.
  • Results: Text-to-image reformulation achieves competitive or superior performance against state-of-the-art models on standard CIR benchmarks.
  • Evaluation: The study systematically benchmarks both conversion paradigms and analyzes model hyperparameters and component-based latency.

2 Related Work

Earlier CIR systems primarily learned joint embeddings, token mappings, or fusion operators, while later methods explored generative and language-model-based alternatives. PeFuse differs by systematically reformulating CIR into four single-query retrieval tasks with a modular, training-free design.

  • Learned fusion methods: Early CIR methods learned joint embedding spaces that directly compared fused image-text representations with candidate image embeddings.
  • Learned fusion methods: Transformer-based and token-based methods improved cross-modal alignment or represented reference images as pseudo-word tokens, typically through joint training.
  • Generative and language-model methods: Other approaches introduced learnable fusion operators, textual inversion, synthetic supervision, or MLLM-based multimodal fusion for CIR.
  • Training dependence: Task-specific or synthetic-data training in prior methods limits flexibility and generalizability to new domains or modalities.Training-free CIR methods were subsequently proposed to address this limitation.
  • PeFuse positioning: PeFuse reformulates CIR into four one-query intra-modal or cross-modal retrieval tasks using text-based or image-based representations.The framework systematically benchmarks these alternatives while using only one model where possible.
  • PeFuse positioning: By using pretrained Diffusion models and MLLMs, PeFuse provides a flexible, modular, plug-and-play solution for zero-shot CIR.

3 Methodology

PeFuse transforms composed image queries into unified unimodal representations using MLLMs and Diffusion models, enabling retrieval with pretrained models instead of task-specific fusion training. Uni-directional and bi-directional conversions reformulate CIR into four single-modality retrieval tasks.

  • Task formulation: PeFuse converts multimodal CIR queries into generated text or images, replacing explicit embedding fusion with unified unimodal representations.MLLMs generate textual descriptions, while Diffusion models generate images from composed queries.
  • Retrieval and ranking: The framework ranks gallery candidates by cosine similarity in a shared embedding space and returns the candidates in descending score order.The retrieval model produces embeddings for text and images, and the ranking procedure returns the top-k candidate image IDs for evaluation.
  • Uni-directional conversion: Uni-directional conversion uses MLLMs for image-and-modification-to-text conversion or Diffusion models for image synthesis, yielding text-to-image and image-to-image retrieval.These conversions project the query into a target modality and compare it with candidate images using a pretrained retrieval model.
  • Bi-directional conversion: Bi-directional conversion additionally converts target gallery images into text, reformulating retrieval as text-to-text or image-to-text matching.Generated target descriptions are matched with the query representations produced by uni-directional conversion.
  • Framework implementation: PeFuse is training-free and uses replaceable pretrained retrieval, MLLM, and Diffusion components to synthesize coherent target-modality data for implicit modality fusion.Its algorithm takes a reference image, text modification, retrieval mode, prompts, and pretrained components before sorting candidates.

4 Experiments

Experiments evaluate PeFuse’s conversion strategies, retrieval models, backbone scaling, and hyperparameter sensitivity across standard CIR benchmarks. Text-to-image conversion generally performs best, while MLLM-generated descriptions improve diffusion-based retrieval and model choices materially affect results.

  • Experimental setup: PeFuse uses MLLMs or Diffusion models for uni-directional conversion, while bi-directional conversion combines both components.The experiments minimize the number of models used while examining alternative pathways to single-modality retrieval.
  • Uni-directional conversion: OpenCLIP with a ViT-B/32 backbone surpasses most zero-shot and training-requiring methods on Fashion-IQ, with performance comparable to WeiMoCIR and slightly below LinCIR.Both CLIP and OpenCLIP outperform CIReVL when CIR is reformulated as text-to-image retrieval.
  • Uni-directional conversion: Text-to-image retrieval generally outperforms image-to-image retrieval across conversion strategies and retrieval models, except with SigLIP2.SigLIP2 is weakest for text-to-image retrieval but strongest for image-to-image retrieval, indicating substantial semantic differences across retrieval models.
  • Uni-directional conversion: Text-to-image retrieval remains strongest on CIRR, CIRCO, and GeneCIS, while image-to-image conversion is inferior on CIRCO and other conversion modes remain competitive on GeneCIS.On CIRCO, stronger retrieval models improve text-to-image performance, whereas diffusion-based image-to-image conversion requires further improvement.
  • Bi-directional conversion: Text-to-text retrieval generally exceeds image-to-text retrieval, but both underperform text-to-image retrieval; diffusion-generated query images also often contain artifacts that degrade retrieval.The paper attributes this pattern to artifacts captured by image encoders and to the greater semantic meaningfulness retained by MLLM-generated text.
  • Scaling laws: Higher backbone sizes generally improve text-to-image retrieval, although performance can fluctuate or decline for specific model sizes such as ViT-g/14.The reported exceptions include Recall@10 decreases and Recall@50 saturation on Fashion-IQ, plus slight Recall@1 and Recall@10 decreases on CIRR.
  • Hyperparameter analysis: MLLM text-generation retrieval is more sensitive to temperature than top-p or top-k, with lower temperature and moderate top-p plus higher top-k performing better.For diffusion conversion, MLLM-generated descriptions improve retrieval across hyperparameters, while image guidance scale has the strongest effect and higher values substantially worsen performance.
  • Hyperparameter analysis: Increasing diffusion inference steps slightly improves retrieval and photorealism but increases computation, making a moderate number of steps preferable for efficiency.Performance remains generally consistent across runs, while hyperparameter tuning is important for effective diffusion-based conversion.

5 Ablation and Latency Analysis

The analysis evaluates component choices, retrieval baselines, and computational costs for PeFuse on CIRCO. Results show that MLLM selection affects retrieval quality, while generative conversions add inference and pipeline overhead.

  • Ablation setup: The study replaces Qwen2.5-VL-7B-Instruct and SDXL-InstructPix2Pix with alternative pretrained models to measure component effects on retrieval performance.It also analyzes computational overhead and uses OpenCLIP (ViT-B-32) consistently across experiments.
  • Latency analysis: Table 6 reports memory consumption, average generative inference time, and end-to-end pipeline time for PeFuse conversion modes.The pipeline time includes dataset loading, model loading, data generation, feature extraction, and evaluation.
  • Baseline evaluations: Textual modifications retrieve target images slightly better than reference images alone, while combined queries fall between these baselines.Baseline retrieval representations are computationally efficient and deterministic on the CIRCO split.
  • MLLM ablation: Qwen2.5-VL-3B-Instruct performs best among the evaluated MLLMs for PeFuse (T→I), whereas LLaVA-1.5-7B-hf performs worst.The comparison highlights the role of MLLM choice in converting multimodal queries into text.
  • MLLM ablation: Qwen2.5-VL-3B-Instruct exceeds Qwen2.5-VL-7B-Instruct by 0.18% average mAP but takes 0.28 seconds longer per sample at inference.The 7B model is used for PeFuse (I→I) for consistency.

6 Model Deployment

PeFuse supports practical deployment by separating offline query generation from online vector retrieval. Candidate images and generated query representations are indexed for efficient matching.

  • Offline indexing: The candidate image pool is preprocessed by indexing visual representations in a FAISS vector database.This supports low-latency retrieval after the candidate representations have been prepared.
  • Offline indexing: MLLMs or Diffusion models can generate query data offline, whose representations are stored using the same indexing strategy.The deployment design shifts generative processing away from the retrieval-time matching step.

7 Limitations

The framework’s limitations arise from multi-model error propagation, prompt and hyperparameter sensitivity, and computational cost. The authors identify robustness, richer inputs, and lighter models as directions for future work.

  • Pipeline limitations: Integrating Diffusion models and MLLMs can cause concept drift and propagate errors through the pseudo-fusion pipeline.Chaining multiple models makes optimizing overall CIR performance challenging.
  • Input and prompting: The framework accepts non-preprocessed text modifiers, while paraphrasing, structured formatting, and prompt engineering may improve performance but risk LLM hallucinations.The authors state that these techniques require careful design.
  • Hyperparameter sensitivity: Diffusion models and MLLMs are highly sensitive to hyperparameters, whose tuning is labor-intensive and may not generalize across datasets.This limits the portability of tuning choices across diverse evaluation settings.
  • Scalability: Generating captions or new images can be computationally expensive at web scale.The authors identify this as a common limitation of LLM- and MLLM-based methods.
  • Future directions: Future work includes robust integration, multiple images, longer narratives, additional modalities, iterative fusion, and lightweight models for scalable deployment.The authors specifically mention video and progressively refined descriptions as examples of richer scenarios.

8 Conclusion

The paper systematically benchmarks training-free pseudo-fusion for zero-shot CIR and shows that heterogeneous queries can be reformulated for single-modality retrieval. Text-to-image conversion is the most effective conversion mode in the reported analysis.

  • Contribution: The work systematically investigates and benchmarks training-free uni-directional and bi-directional conversion for zero-shot CIR.It also quantifies generative-model hyperparameters and analyzes component ablations and latency.
  • Core finding: CIR can be reformulated by converting heterogeneous modalities into a single unified modality for intra-modal or cross-modal retrieval.This enables standard single-query retrieval systems and pretrained models to be used without training new modules.
  • Core finding: Text-to-image reformulation is more effective than other conversion modes in the reported CIR analysis.The conclusion emphasizes selecting appropriate MLLMs and Diffusion models for modality conversion.

A Datasets

The experiments use public benchmarks with dataset-specific splits. CIRCO, CIRR, and GeneCIS images are available, while Fashion-IQ has missing reference images caused by broken links.

  • Fashion-IQ is missing 98 shirt, 158 dress, and 97 toptee reference images because of broken dataset links.
  • CIRR, CIRCO, and GeneCIS have all images available.
  • The benchmarks comprise Fashion-IQ, CIRCO, CIRR, and GeneCIS, using Fashion-IQ validation data and test data for the other datasets.

B Training-based Methods

The appendix reports training-based baselines, scaling-law analyses, and implementation details for MLLM-generated descriptions. The study nevertheless focuses primarily on training-free methods.

  • The paper mainly focuses on training-free methods while reporting training-based baselines across the evaluated datasets.
  • Scaling laws are reported for Fashion-IQ categories, CIRR subsets, and GeneCIS using OpenCLIP for text-to-image retrieval.
  • MLLM prompts generate composed descriptions from reference images and text modifications, with multiple descriptions offering possible performance gains at extra computational cost.

E Qualitative Results Using MLLM for Diffusion Models

Qualitative results compare images generated from MLLM-produced descriptions with images generated from raw captions. MLLM-generated text produces visually superior outputs for both evaluated diffusion models.

  • MLLM-generated descriptions produce superior generated images compared with raw captions on the CIRCO validation split.The comparison is shown for SDXL-InstructPix2Pix and SDXL-Turbo.
  • Raw captions produce more noticeable artifacts for both diffusion models.
  • SDXL-Turbo-generated images contain more artifacts than SDXL-InstructPix2Pix-generated images.

F Failure Cases

The failure cases expose distinct weaknesses in text-to-image and image-to-image pseudo-fusion. Text queries can be overly detailed or misapply modifications, while generated images may preserve the reference too strongly.

  • Qwen-generated descriptions can capture target semantics but become overly detailed, causing mismatches with ground-truth images.
  • A generated textual query can incorrectly identify which objects a modification applies to, producing wrong retrieval results.
  • SDXL-InstructPix2Pix outputs may fail to implement the desired modifications and remain similar to the original reference images, degrading retrieval performance.
  • Raw-caption generation uses SDXL-InstructPix2Pix with image guidance scale 3.0, guidance scale 7.5, and 30 denoising steps, whereas SDXL-Turbo uses strength 0.5, guidance scale 0.0, and 30 steps.
  • None of the top-five PeFuse (T→I) results matches the corresponding query in the illustrated failure cases.The retrieval pool contains 123k CIRCO candidate images.
  • None of the top-five PeFuse (I→I) results matches the corresponding query in the illustrated failure cases.
Loading 2608.23102v1…