PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Composed Image Retrieval with Textual Inversion

Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, Alberto Del Bimbo

arXiv:2303.15247v2cs.CVcs.CLcs.IR

TL;DR

CIR methods depend on costly labeled datasets, motivating zero-shot retrieval without supervised training. SEARLE maps reference images to CLIP pseudo-word tokens and combines them with relative captions, while CIRCO supplies an open-domain benchmark with multiple ground truths; the method outperforms baselines across three datasets. [No external citations requested]

  • Problem

    Existing CIR methods rely on expensive manually annotated datasets, limiting scalability and use across domains.

  • Method

    SEARLE uses a textual inversion network to map reference images into pseudo-word tokens, concatenate them with relative captions, and perform CLIP retrieval without labeled training data.

  • Results

    SEARLE achieves substantial improvements over baselines, up to 7%, on FashionIQ, CIRR, and CIRCO.

  • Takeaways & Limitations

    CIRCO enables more comprehensive ZS-CIR evaluation by providing multiple annotated ground truths and reduced false negatives.

  • Takeaways & Limitations

    The reported pretraining uses 100K unlabeled ImageNet1K images, and the authors leave personalized image generation for future work.

Abstract

from arXiv · show

Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image and a relative caption that describes the difference between the two images. The high effort and cost required for labeling datasets for CIR hamper the widespread usage of existing methods, as they rely on supervised learning. In this work, we propose a new task, Zero-Shot CIR (ZS-CIR), that aims to address CIR without requiring a labeled training dataset. Our approach, named zero-Shot composEd imAge Retrieval with textuaL invErsion (SEARLE), maps the visual features of the reference image into a pseudo-word token in CLIP token embedding space and integrates it with the relative caption. To support research on ZS-CIR, we introduce an open-domain benchmarking dataset named Composed Image Retrieval on Common Objects in context (CIRCO), which is the first dataset for CIR containing multiple ground truths for each query. The experiments show that SEARLE exhibits better performance than the baselines on the two main datasets for CIR tasks, FashionIQ and CIRR, and on the proposed CIRCO. The dataset, the code and the model are publicly available at https://github.com/miccunifi/SEARLE.

1. Introduction

The paper introduces ZS-CIR to remove expensive labeled training data from composed image retrieval, proposes SEARLE to combine reference images with captions through pseudo-words, and presents CIRCO for broader evaluation. SEARLE reports improvements over baselines across FashionIQ, CIRR, and CIRCO.

  • Motivation: CIR retrieves target images using a reference image and relative caption, but existing methods rely on expensive manually annotated datasets.Creating reference–target pairs and descriptive captions is time-consuming and resource-intensive, limiting scalability and cross-domain use.
  • Zero-Shot CIR: ZS-CIR removes the need for supervised learning by combining the reference image and relative caption without labeled training data.The task is designed to address CIR without an expensive labeled training dataset.
  • SEARLE: SEARLE maps a reference image into a pseudo-word token in CLIP token embedding space and concatenates it with the relative caption for retrieval.Its textual inversion network is pretrained on unlabeled images through optimization-based textual inversion and distillation, then performs the mapping in one forward pass.
  • Results: Up to 7% improvement over baselines is reported on FashionIQ, CIRR, and CIRCO.The paper also states that SEARLE achieves state-of-the-art performance on all three datasets.
  • CIRCO: CIRCO is an open-domain CIR benchmark with multiple annotated ground truths and reduced false negatives, addressing limitations of existing datasets.It is introduced to support ZS-CIR and enable more comprehensive evaluation.

2. Related Work

Prior CIR research spans multiple domains and commonly uses multimodal composition methods, including CLIP-based approaches. SEARLE instead applies textual inversion and knowledge distillation to zero-shot retrieval, using fewer data than the concurrent Pic2Word approach.

  • CIR Methods: CIR research covers fashion, natural images, and synthetic images, with methods integrating image and text features through multimodal composition.Earlier approaches include residual gating, graph convolutional networks, and separate style-content modules.
  • CIR Methods: CLIP-based CIR methods combine pretrained image-text features or fine-tune the CLIP text encoder to improve retrieval performance.These approaches include combiner networks and task-oriented text-encoder fine-tuning.
  • Textual Inversion: Textual inversion maps images into pseudo-words for personalized synthesis and has also been applied to image retrieval.PALAVRA uses a mapping function followed by optimization but requires labeled image-caption data.
  • Related Zero-Shot CIR: Compared with Pic2Word, SEARLE trains with approximately 3% of the data and uses both distillation and regularization losses instead of only cycle contrastive loss.Pic2Word is trained on 3M CC3M images, whereas SEARLE uses substantially less data.
  • Knowledge Distillation: SEARLE transfers knowledge from computationally expensive optimization-based textual inversion to a lightweight network through distillation.The network acts as a surrogate for the optimization method.

3. Proposed Approach

SEARLE performs zero-shot composed image retrieval by mapping a reference image into a CLIP pseudo-word token and combining it with the relative caption. It uses optimization-based textual inversion with GPT-powered contextual regularization, then distills the resulting tokens into a faster feed-forward network trained on unlabeled images.

  • Core approach: SEARLE maps reference-image features into a pseudo-word token in CLIP token embedding space, reducing CIR to standard text-to-image retrieval.The pseudo-word is concatenated with the relative caption before encoding the composed query.
  • Textual Inversion Network ϕ Pre-training: The textual inversion network ϕ distills OTI-generated pseudo-word tokens from unlabeled images and predicts them in a single forward pass.Its training combines distillation with the same GPT-powered regularization used in OTI.
  • Textual Inversion Network ϕ Pre-training: ϕ replaces slower iterative OTI at inference while aiming to retain OTI’s expressive pseudo-word representations.The network is trained as a faster, less computationally heavy surrogate for OTI.
  • Optimization-based Textual Inversion: OTI iteratively optimizes each pseudo-word token to align image and text features using a cosine loss.The token is initialized randomly, inserted into a prompt, and optimized against the CLIP image representation.
  • Optimization-based Textual Inversion: GPT-powered contextualized regularization constrains pseudo-word tokens to the CLIP token embedding manifold and improves their interaction with actual words and relative captions.CLIP zero-shot classification supplies image concepts, and GPT expands concept prompts into contextualized phrases used for regularization.
  • Textual Inversion Network ϕ Pre-training: SEARLE’s textual inversion network is trained fully unsupervised on raw images rather than labeled images or captions.The reported pre-training setup uses 100K unlabeled ImageNet1K test-split images.

4. CIRCO dataset

CIRCO is an open-domain CIR benchmark designed to address false negatives and limited visual grounding in existing datasets. It provides multiple ground truths per query and a larger image index for more comprehensive, fine-grained evaluation.

  • Motivation: Existing CIR datasets often contain false negatives because each query has only one labeled target and all other images are treated as negatives.Many datasets also focus on specialized domains, while CIRR’s construction does not guarantee eliminating false negatives across the full dataset.
  • Dataset construction: CIRCO uses real-world COCO 2017 images and is the first CIR dataset with multiple annotated ground truths.Its annotations include shared characteristics of reference and target images to clarify ambiguities.
  • Dataset construction: CIRCO reduces missing ground truths to less than 10% through candidate retrieval, visual similarity search, and validity selection.The procedure combines the top 100 query-retrieved images with the top 50 images visually similar to the target.
  • Dataset statistics: CIRCO contains 1020 queries, split into 220 validation and 800 test queries, with an average of 4.53 ground truths per query.The validation ground truths are released, while test evaluation is hosted on an evaluation server.
  • Evaluation: CIRCO indexes all 120K COCO images, providing substantially more distractors than CIRR’s 2K-image test index.Its multiple ground truths also support fine-grained metrics such as mean Average Precision.

5. Experimental Results

The experiments evaluate SEARLE and related baselines on FashionIQ, CIRR, and CIRCO, then analyze the contributions of textual inversion and GPT-based regularization. SEARLE generally outperforms baselines, while dataset characteristics strongly affect retrieval behavior.

  • Evaluation setup: SEARLE and SEARLE-XL are evaluated on FashionIQ, CIRR, and CIRCO using standard evaluation protocols.The study also compares zero-shot baselines, competing methods, and SEARLE-OTI.
  • FashionIQ: SEARLE matches SEARLE-OTI with B/32 on FashionIQ while providing a significant efficiency gain without compromising performance.Both versions outperform the evaluated baselines; SEARLE-XL gains up to 7% in Recall@50 for the Dress category over Pic2Word.
  • CIRR: CIRR’s Text-only baseline performs best on RecallSubset, indicating that many relative captions do not usefully depend on the reference image.Image-only retrieval is equivalent to random guessing within the five-image subset.
  • CIRR: On CIRR, SEARLE variants with OTI and the textual inversion network obtain comparable results and improve over PALAVRA and Pic2Word with matched backbones.A RecallSubset comparison with Pic2Word is unavailable because its authors did not report that result.
  • CIRCO: On CIRCO, Image+Text outperforms Image-only and Text-only, while SEARLE significantly improves over all baselines, including Pic2Word.SEARLE-XL is reported but considered an unfair comparison because it helped select CIRCO’s multiple ground truths.
  • Ablation studies: Ablations show that GPT-based regularization benefits both optimization-based textual inversion and textual inversion-network training.Regularization helps pseudo-word tokens communicate with CLIP vocabulary tokens, while distillation from OTI-generated tokens outperforms learning from raw image and text features.

6. Conclusion

The paper introduces ZS-CIR, SEARLE, and CIRCO to reduce reliance on labeled CIR training data and improve evaluation with multiple ground truths. Future work will examine whether the method can support personalized image generation.

  • Conclusion: ZS-CIR addresses composed image retrieval without an expensive labeled training dataset, using SEARLE to combine image and caption information.SEARLE maps reference-image information into a pseudo-word token and integrates it with the relative caption.
  • Conclusion: CIRCO provides an open-domain CIR benchmark with multiple ground truths for each query, while both SEARLE variants outperform baselines and competing methods across evaluated datasets.The reported datasets include CIRR, FashionIQ, and CIRCO.
  • Future work: Future work will investigate the method’s potential for personalized image generation, including whether GPT-based regularization can help models use input text when synthesizing personalized objects.This possibility is presented as a future research direction rather than an established result.

A. Implementation Details

The implementation uses CLIP backbones, optimization-based textual inversion, a pretrained textual inversion network, GPT-generated regularization phrases, and order-averaged caption concatenation for FashionIQ.

  • Training configuration: Optimization-based textual inversion uses 350 iterations with learning rate 2e−2, while the textual inversion network uses learning rate 1e−4 and batch size 256.The network is trained for 100 epochs in SEARLE and 50 epochs in SEARLE-XL.
  • GPT regularization: GPT-Neo-2.7B generates 256 phrases for each of 20,932 Open Images V7 class names, with maximum length 35 tokens and temperature 0.5.Phrase generation takes approximately 12 hours on one NVIDIA A100 GPU and is performed once.
  • Inference: For FashionIQ, the two relative captions are concatenated with “and” in both possible orders, and the resulting features are averaged.This procedure is intended to reduce sensitivity to concatenation order.
  • Architecture: The CLIP feature and token-embedding dimensions are 512 for B/32 and 768 for L/14.The passage denotes these dimensions as d and d_w, respectively.

B. CIRCO Dataset

CIRCO is an open-domain CIR dataset built through a two-phase annotation process that creates image-caption-target triplets and supports multiple ground truths. Its images come from COCO’s unlabeled set, with balanced reference-image coverage across supercategories.

  • Dataset construction: CIRCO’s annotation process has two phases: constructing triplets with one target image, then adding additional ground truths.The process was carried out by the paper’s authors.
  • Dataset construction: CIRCO uses images from COCO 2017’s 123,403-image unlabeled set to provide open-domain real-life scenes without pretraining-image overlap.The unlabeled set was selected because COCO’s training set is often used for pretraining.
  • Dataset construction: Reference images are assigned to supercategories with CLIP ViT-L/14 zero-shot classification to support balanced query construction.The classification assumes each image is categorized by its main subject.
  • Triplet annotation: The annotation tool displays each reference image beside 50 candidate targets selected by CLIP similarity, excluding candidates with similarity above 0.92.Annotators can skip a reference image when no suitable target exists.
  • Triplet annotation: Annotators record a shared concept and write a relative caption that avoids naming the shared subject, requiring that subject to be inferred from the reference image.The shared concept clarifies ambiguities between the reference and target images.
  • Triplet annotation: The first phase produced 1,020 triplets containing a reference image, relative caption, and single target image.This forms the starting point for the second annotation phase.

B.2. Multiple Ground Truths Annotation

The second annotation phase expands each triplet with additional valid matches selected from a gallery. SEARLE retrieves candidate images for annotation, enabling multiple ground truths and an estimate of missing matches.

  • Annotation procedure: Annotators select all valid additional ground truths from a gallery and assign semantic aspects to each query.They begin from the triplet created in the first annotation phase.
  • Annotation procedure: SEARLE-XL retrieves annotation candidates using a pseudo-word for the reference image and a text-to-image query containing the shared concept and relative caption.Including the shared concept improves retrieval performance during annotation.
  • Annotation results: 4,624 ground truths were labeled, including 4,097 retrieved with SEARLE and 527 retrieved using similarity to the target image.Using Recall@100 of 82.15, the authors estimate approximately 4,987 ground truths in the full dataset.
  • Annotation results: 3,604 additional ground truths were labeled beyond the initial 1,020, reducing false negatives that would otherwise remain unannotated.The multi-ground-truth process also enables estimation of the missing-ground-truth fraction.

B.3. Dataset Analysis

CIRCO contains multiple ground truths per query and uses fine-grained mAP evaluation, while its captions are designed to describe genuine relative changes. The dataset also addresses false negatives and reference-insensitive captions found in CIRR.

  • Dataset statistics: CIRCO has 1,020 queries and 4,624 ground truths, averaging 4.53 ground truths per query; one query has 21, and two is the modal count.Figure 5 shows the distribution of queries by ground-truth count.
  • Dataset statistics: CIRCO captions average 10.4 words, and about 75% contain multiple statements compared with approximately 43% in CIRR.The caption length and semantic-concept distribution are reported as comparable with CIRR.
  • Comparison with CIRR: CIRR can contain false negatives and queries whose captions ignore the reference image because its automatically formed subsets may contain visually dissimilar subjects.The paper illustrates this issue with captions describing absolute target content, such as pillows versus penguins.
  • Comparison with CIRR: CIRCO lets annotators choose reference-target pairs without constraints, ensuring that captions describe relative changes rather than forced absolute descriptions.The authors present this strategy as improving dataset quality.
  • Comparison with CIRR: CIRCO contains 220 validation and 800 test queries, uses all 120K COCO images as distractors, and provides multiple ground truths with fewer queries than CIRR.The two-phase strategy is described as reducing false negatives and increasing quality.
  • Evaluation: CIRCO evaluation uses mAP@K for K from 5 to 50, averaging precision across queries and their annotated ground truths.The relevance function marks a ranked image as positive or negative according to its annotation.
  • Pseudo-word analysis: Pseudo-word tokens are evaluated for visual-content retrieval by querying an image database with the CLIP text prompt “a photo of S∗”.The corresponding input image is expected to rank first when its visual content is captured.
  • Pseudo-word analysis: The ablation results report effective visual information capture by v∗ across regularization techniques and improved CIR performance with the proposed method.Experiments are conducted on the CIRR validation set.

C.2. Training ϕ on Different Datasets

The textual inversion network is pretrained with raw images from different datasets to assess domain effects and generalization. Domain-specific pretraining helps FashionIQ, while variants trained on FashionIQ generalize promisingly to CIRR.

  • Experimental setup: The study compares textual inversion networks pretrained on ImageNet1K, CIRR, and FashionIQ raw images without using associated labels.The ImageNet1K test split contains about 100K images, versus 45,429 FashionIQ and 16,939 CIRR training images.
  • FashionIQ results: FashionIQ-pretrained SEARLE-FIQ and SEARLE-XL-FIQ improve performance over ImageNet-based variants on FashionIQ validation.Both variants also outperform OTI-based methods, which the authors associate with the distillation-based approach.
  • CIRR results: CIRR-pretrained variants obtain results comparable to ImageNet-based variants on CIRR validation, consistent with domain similarity between CIRR and ImageNet.FashionIQ-pretrained variants also generalize to the broader CIRR domain with promising performance.
  • CIRCO results: On CIRCO validation, the same considerations reported for the test set remain applicable across the evaluated variants.The paper reports these validation results for completeness.
  • Single-ground-truth comparison: With only the first annotated ground truth, Image + Text outperforms Image-only and Text-only on the CIRCO test set.The comparison uses Recall@K and does not use SEARLE-XL during evaluation.
  • Supervised comparison: Supervised Combiner performs best when training and testing datasets match, but supervised models struggle across domains while SEARLE achieves noteworthy zero-shot performance on both datasets.The authors connect the zero-shot setting with reduced dependence on manually annotated training data.

C.5. Qualitative Results

SEARLE integrates reference-image and caption features to retrieve relevant images across FashionIQ, CIRR, and CIRCO. The qualitative examples also expose false negatives in existing datasets and show the value of multiple ground truths.

  • SEARLE integrates visual features from the reference image with textual features from the relative caption to retrieve correct images.
  • Baselines often overemphasize either the reference image or the relative caption, whereas SEARLE combines both modalities.
  • SEARLE retrieves valid query matches that are unlabeled in FashionIQ and CIRR, revealing false negatives in these datasets.
  • On CIRCO, SEARLE retrieves more relevant top-5 images than PALAVRA for the compared queries.CIRCO highlights ground truths with green borders in the qualitative comparison.
Loading 2303.15247v2…