PaperlayerView exact revision on arXiv

Source-linked AI summary

CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion

Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Yoohoon Kang, Sangdoo Yun

arXiv:2303.11916v4cs.CVcs.IR

TL;DR

CIR methods face limited condition types and small training datasets, constraining the available training triplets. CompoDiff applies latent diffusion to composed retrieval and pairs it with an automatically generated 18.8M-triplet dataset. It achieves the best zero-shot performance on four benchmarks while supporting adjustable conditions and inference speed.

  • Problem

    Existing CIR approaches rely on small human-verified triplet datasets and are limited in handling conditions beyond text.

    Source: §1 · p. 1

  • Method

    CompoDiff uses latent diffusion for conditional retrieval and SynthTriplets18M, an automatically generated dataset of 18.8M image-condition-target triplets.

    Source: §1 · p. 2

  • Results

    CompoDiff achieves the best zero-shot performance on FashionIQ, CIRR, CIRCO, and GeneCIS.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Classifier-free guidance lets users adjust condition weights and inference speed without retraining the model.

    Source: §1 · p. 3, §3.3 · p. 7

    Existing benchmarks cannot evaluate negative-text or mask conditions, and quantitative evaluation on LAION-2B is infeasible because it requires expensive human verification.

    Source: §5.4 · p. 14

Abstract

from arXiv · show

This paper proposes a novel diffusion-based model, CompoDiff, for solving zero-shot Composed Image Retrieval (ZS-CIR) with latent diffusion. This paper also introduces a new synthetic dataset, named SynthTriplets18M, with 18.8 million reference images, conditions, and corresponding target image triplets to train CIR models. CompoDiff and SynthTriplets18M tackle the shortages of the previous CIR approaches, such as poor generalizability due to the small dataset scale and the limited types of conditions. CompoDiff not only achieves a new state-of-the-art on four ZS-CIR benchmarks, including FashionIQ, CIRR, CIRCO, and GeneCIS, but also enables a more versatile and controllable CIR by accepting various conditions, such as negative text, and image mask conditions. CompoDiff also shows the controllability of the condition strength between text and image queries and the trade-off between inference speed and performance, which are unavailable with existing CIR methods. The code and dataset are available at https://github.com/navervision/CompoDiff

1 Introduction

Existing CIR methods are limited by small human-verified datasets and text-only conditions. CompoDiff uses latent diffusion alongside SynthTriplets18M to support more versatile, controllable retrieval, and achieves the best zero-shot performance on four benchmarks.

1 Introduction

  • CompoDiff supports negative text, image masks, and mixed conditions, while prior fusion methods are limited to textual conditions.Users can adjust condition strength and inference speed through sampling settings, without retraining.Source: §1 · pp. 1, 3
  • CompoDiff treats retrieval as latent-space conditional image editing, using diffusion to map a reference-image embedding to a target-image embedding guided by conditions.Unlike image editing, CIR seeks a generalized concept that can retrieve matching items from a database.Source: §1 · p. 2
  • SynthTriplets18M provides 18.8M automatically generated triplets, over 500 times the size of existing CIR datasets.The dataset also contains 40 times more triplets and 12.5 times more keywords than InstructPix2Pix.Source: §1 · p. 2
  • ARTEMIS trained exclusively on SynthTriplets18M achieves a zero-shot score of 40.6, versus 38.2 for its FashionIQ-trained counterpart.This example shows that SynthTriplets18M can train an existing CIR model for zero-shot retrieval.Source: §1 · p. 3
  • CompoDiff achieves the best zero-shot performance on FashionIQ, CIRR, CIRCO, and GeneCIS, with significant gaps over prior methods.The evaluation uses models trained on SynthTriplets18M and a large-scale image-text paired dataset without access to target triplet datasets.Source: §1 · p. 3

2 Related Works

Prior CIR work combines image and text features or projects images into text space, while existing methods focus on text conditions. This paper positions guided latent diffusion and scalable synthetic triplets as alternatives for versatile, controllable retrieval.

Composed image retrieval

  • Existing CIR methods focus on text conditions and have difficulty with versatile scenarios, whereas this method uses latent diffusion with classifier-free guidance and a synthetic dataset.The paper describes its zero-shot setting as training on SynthTriplets18M and image-text paired data without target triplets; some prior work uses “zero-shot” to mean no triplet training.Source: §2 · pp. 3–4
  • Fusion-based CIR combines image and text features, while inversion-based zero-shot methods project an image into text space for text-to-image retrieval.Fusion methods often tune encoders; inversion-based methods use frozen encoders.Source: §2 · p. 3, Figure 2 · p. 4

Dataset creation with diffusion models

  • CoVR, VDG (Jang et al., 2024), and MagicLens (Zhang et al., 2024) synthesize modification texts for real-image pairs, while this dataset guides visual differences using instructions.The paper notes that plausible modification instructions between real images can be difficult to generate.Source: §2 · p. 4
  • Human collection of reference-target pairs and modification captions makes conventional triplet datasets expensive to scale.The paper instead generates synthetic triplets rather than relying on human annotators.Source: §2 · p. 4
  • The dataset synthesis process scales Brooks et al. (2022) from 1M to 18.8M triplets and adds object-level modifications for more diverse triplets.The authors report better CIR performance at the same 1M scale and say the process can scale to more samples.Source: §2 · p. 4

3 CompoDiff: Composed Image Retrieval with Latent Diffusion

CompoDiff treats composed image retrieval as conditional editing in frozen CLIP latent space, using a Transformer denoiser and classifier-free guidance. It is trained in two stages and supports multiple conditions and adjustable query weights at inference.

3.1 Preliminary: Diffusion model

  • Classifier-free guidance extends conditioning beyond positive text to negative text or masks and permits adjustment of condition intensity.The paper describes these as capabilities of providing multiple conditions through classifier-free guidance.Source: §3 · p. 4, §3.1 · p. 5

3 CompoDiff: Composed Image Retrieval with Latent Diffusion

  • CompoDiff diffuses in CLIP latent space with a Transformer denoiser, producing edited features directly usable for retrieval and supporting masked conditions and triplet relationships.Unlike the U-Net approaches described, its cross-attention mechanism supplies conditions to the denoiser.Source: §3 · p. 4, §3.1 · p. 5

3.2 Training

  • Training uses text-to-image pretraining on LAION-2B, then fine-tunes on LAION-2B conversion tasks and SynthTriplets18M triplet-based retrieval.Stage 2 also learns masked conversion by recovering full image embeddings from masked inputs; objectives are randomly selected in a 30%, 30%, and 40% mix.Source: §3.2 · pp. 5–6

3.3 Inference

  • At inference, classifier-free guidance combines reference-image, text, and mask conditions to form a target feature for image-to-image retrieval.Adjusting image and text weights shifts focus between reference-image similarity and text guidance; varying diffusion steps trades inference time against retrieval quality.Source: §3.3 · pp. 6–7, Figure 4 · p. 7

4 SynthTriplets18M: Massive High-Quality Synthesized Dataset

SynthTriplets18M is built by generating caption-level reference, instruction, and modified-caption triplets, converting them into image triplets, and filtering low-quality samples. The process yields 18.8M training triplets.

4.1 Keyword-based diverse caption generation

  • 586k object terms are substituted with CLIP-similar alternatives (0.5–0.7), then 48 templates yield ≈30M caption triplets.Source: §4.1 · p. 9, Table 2 · p. 9

4.2 Amplifying InstructPix2Pix (IP2P) triplets by LLM

  • LoRA fine-tunes OPT-6.7B on 452k IP2P caption triplets, after which the model generates ≈30M more caption triplets.Source: §4.2 · p. 9

4.3 Triplet generation from caption triplets

  • The process generates 60M caption triplets and uses Stable Diffusion variants with Prompt-to-Prompt to create corresponding image triplets.Using multiple diffusion models is intended to diversify images beyond IP2P's use of Stable Diffusion 1.5.Source: Figure 6 · p. 8, §4 · p. 8, §4.3 · p. 10

4.4 CLIP-based filtering

  • CLIP filtering uses 0.70 image-pair similarity, 0.2 caption alignment and directional similarity, plus 0.20 keyword-image or instruction-image checks.Source: §4.4 · p. 10
  • After CLIP filtering, SynthTriplets18M contains 18.8M image-condition-target triplets: 11.4M keyword-based and 7.4M LLM-based.Source: §4.4 · p. 10

4.5 Dataset Statistics

  • About half the captions have fewer than 13 tokens, 90% fewer than 20, and 0.8% more than 40; instruction-length distributions vary across CIR datasets.The authors presume this variation contributes to zero-shot CIR difficulty outperforming task-specific supervised methods.Source: Figure 7 · p. 9, Figure 8 · p. 9, §4.5 · p. 10

5 Experiments

The experiments describe CompoDiff’s implementation and zero-shot evaluation setup, comparison methods, and benchmark scope. The supplied passages also identify limits in current benchmark coverage and in quantitative evaluation on LAION-2B.

5.1 Implementation details

  • A 12-layer denoising Transformer applies text, mask, and image conditions only in cross-attention; T5-XL supplements CLIP text features for ViT-L.Source: §5.1 · p. 12

5.2 Experiment settings

  • Zero-shot evaluation covers FashionIQ, CIRR, CIRCO, and GeneCIS; CompoDiff is trained without access to the target triplet datasets.Source: §5.2 · p. 12, Table 3 · p. 14

5.3 Comparison methods.

  • Training-data requirements constrain comparisons: ARTEMIS and Combiner use triplets, whereas Pic2Word and SEARLE use image-only datasets; ARTEMIS also cannot use approximate nearest-neighbor search.Source: §5.3 · p. 13

5.4 CIR Datasets.

  • FashionIQ and CIRR instructions diverge from realistic edits and may require in-domain training; benchmarks omit negative and mask queries, while LAION-2B results lack feasible quantitative human verification.Source: §5.4 · p. 14

5.5 Qualitative comparisons on four Zero-shot CIR (ZS-CIR) benchmarks

CompoDiff outperforms prior methods on most reported zero-shot benchmarks, with inference speed adjustable through diffusion steps. The analysis also examines conditioning, dataset scale, text encoders, Stage 2 training, and condition weights.

5.5 Qualitative comparisons on four Zero-shot CIR (ZS-CIR) benchmarks

  • CompoDiff outperforms existing methods with significant gaps, especially on CIRR and CIRCO, and beats same-scale ZS-CIR methods on FashionIQ, CIRCO, and GeneCIS but not CIRR.The authors attribute the gains partly to SynthTriplets18M’s diversity; ARTEMIS trained on it also exceeds its FashionIQ-trained counterpart, 40.6 vs. 38.2 average recall.Source: §5.5 · pp. 14–15

5.6 Inference time

  • 42.17 at 0.12 sec with 5 steps trades off against 42.65 at 2.02 sec with 100 steps on FashionIQ; 4 or 5 steps remain competitive.The authors report CompoDiff throughput of 120–230 ms for a single image and describe inference as slower than encoder-only methods.Source: §5.6 · p. 15

5.7 Analysis

  • Cross-attention conditioning is three times faster than the naive concatenated-input design and gives better cross-modal retrieval performance than the prior-model comparison.The comparison covers COCO and its extensions; Stage 1-only CompoDiff performs better on image-to-text and text-to-image tasks than the two-stage model.Source: §5.6 · p. 15, §5.7 · p. 16
  • As SynthTriplets18M grows from 1M to 18.8M triplets, performance improves consistently across the evaluated models; keyword-based generation scales without manual labor.The authors note that large data volumes are not necessary to train CompoDiff, while scaling improves all methods.Source: §5.7 · p. 16, Table 6 · p. 16
  • Combining T5-XL with CLIP text features improves ViT-L/14 results, while T5 alone performs worse than CLIP alone.The authors use CLIP text features alone for ViT-G and suggest exploring learnable prompts as future work.Source: §5.7 · p. 17, Table 7 · p. 17
  • Alternating among three Stage 2 objectives improves overall performance, with LAION tasks supplementing SynthTriplets18M triplet training for generalizability.The authors say SynthTriplets18M is diverse but less diverse than LAION.Source: §5.7 · p. 17, Table 8 · p. 17
  • Increasing image weight emphasizes the reference image and increasing text weight emphasizes guidance, but either weight set too high can produce unexpected results.The authors select wI = 1.5 and wT = 7.5 as their content-condition trade-off.Source: §5.7 · p. 17

5.8 Qualitative examples

Qualitative examples show CompoDiff handling varied query conditions and retrieving images that reflect both image and text semantics. Its composed features can also be decoded into images.

5.8 Qualitative examples

  • CompoDiff handles positive and negative text, masked text, and combinations of these conditions with a partly masked reference image in LAION retrieval.The experiment uses FAISS to simulate billion-scale CIR scenarios.Source: §5.8 · p. 17
  • CompoDiff retrieves semantically stronger results than Pic2Word in the shown examples, reflecting both query-text meaning and image properties such as crowdedness.Pic2Word sometimes ignores text such as “grown up” or image properties such as crowdedness; its instruction handling was modified for the comparison.Source: §5.8 · p. 17, Figure 11 · p. 18
  • CompoDiff’s composed CLIP visual features can be decoded by unCLIP, producing images that reflect the input conditions alongside top-1 LAION retrievals.The authors suggest image generation could help when desired search results are unavailable.Source: §5.8 · pp. 17–18, Figure 12 · p. 18

6 Conclusion

The conclusion presents CompoDiff and SynthTriplets18M as contributions to complex CIR, highlighting zero-shot capability and versatility across conditions.

6 Conclusion

  • CompoDiff handles conditions such as negative text and image masks, while SynthTriplets18M provides 18.8M image, modification-text, and modified-image triplets.The authors report that existing CIR methods trained on the dataset became comparable zero-shot predictors to zero-shot CIR methods.Source: §6 · p. 18

Societal Impact

The societal-impact discussion distinguishes the paper’s intended CIR use from possible image-editing use and flags uncertainty about that secondary application.

Societal Impact

  • CompoDiff is designed for CIR, not image editing, and the authors do not guarantee effectiveness or safety for that unintended use.They report applying NSFW filters during SynthTriplets18M creation and call for continued research on AI’s ethical and societal implications.Source: Societal Impact · p. 19

Appendix

The appendix provides training hyperparameters, full experimental results, and additional qualitative examples.

Appendix A

  • Appendix A contains the detailed CompoDiff training hyperparameter settings.Source: Appendix · p. 23

Appendix B

Appendix C

A Hyperparameter details

The appendix lists CompoDiff's training hyperparameters and clarifies how its training stages correspond to zero-shot and supervised models.

A Hyperparameter details

B Full experiment results of Table 3

The full results report CompoDiff's zero-shot performance across CIRR, CIRCO, and GeneCIS, alongside fine-tuning comparisons for FashionIQ and CIRR.

FashionIQ

  • FashionIQ results use recall@K and compare zero-shot models with models trained on FashionIQ; the appendix also reports fine-tuning results for SynthTriplets18M-trained methods.Source: B Full experiment results of Table 3 · p. 23

CIRR

FashionIQ and CIRR

CIRCO

GeneCIS

  • CompoDiff achieves the best average recall on GeneCIS, leading in Change Attribute, Focus Object, and Change Object but trailing Pic2Word and SEARLE in Focus Attribute.The authors presume the Focus Attribute gap reflects a difference between its instruction distribution and SynthTriplets18M's.Source: B Full experiment results of Table 3 · p. 24

C More qualitative examples

The appendix extends the qualitative comparisons with Pic2Word and presents CompoDiff-generated images alongside retrieved LAION results.

Open world zero-shot CIR comparisons with Pic2Word

More versatile CIR examples on LAION

Related papers

Loading 2303.11916v4…