PaperlayerView exact revision on arXiv

Source-linked AI summary

Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval

Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, Tomas Pfister

arXiv:2302.03084v2cs.CV

TL;DR

Existing composed image retrieval methods depend on expensive labeled triplets and may be specialized to particular use cases. This paper introduces Zero-Shot Composed Image Retrieval and Pic2Word, which maps images to language tokens using image-caption pairs and unlabeled images. Across four CIR tasks, Pic2Word reports 10 - 100% relative improvement and performance on-par with or better than several supervised methods.

  • Problem

    Existing CIR methods require expensive labeled triplets and may not generalize beyond their specialized use cases.

  • Method

    Pic2Word trains a mapping network that converts image embeddings into language tokens using image-caption pairs and unlabeled images.

  • Results

    10 - 100% relative improvement is reported across four CIR tasks, with performance on-par with or better than several supervised CIR methods.

  • Takeaways & Limitations

    A single Pic2Word model shows strong generalization across diverse CIR tasks without labeled triplet training data.

  • Takeaways & Limitations

    Sketch-based retrieval remains difficult for natural images because natural images lack a suitable domain description in the pretraining captions.

Abstract

from arXiv · show

In Composed Image Retrieval (CIR), a user combines a query image with text to describe their intended target. Existing methods rely on supervised learning of CIR models using labeled triplets consisting of the query image, text specification, and the target image. Labeling such triplets is expensive and hinders broad applicability of CIR. In this work, we propose to study an important task, Zero-Shot Composed Image Retrieval (ZS-CIR), whose goal is to build a CIR model without requiring labeled triplets for training. To this end, we propose a novel method, called Pic2Word, that requires only weakly labeled image-caption pairs and unlabeled image datasets to train. Unlike existing supervised CIR models, our model trained on weakly labeled or unlabeled datasets shows strong generalization across diverse ZS-CIR tasks, e.g., attribute editing, object composition, and domain conversion. Our approach outperforms several supervised CIR methods on the common CIR benchmark, CIRR and Fashion-IQ. Code will be made publicly available at https://github.com/google-research/composed_image_retrieval.

1. Introduction

CIR combines a reference image with text to retrieve images matching the user’s intended modification, but existing approaches require costly triplet labels and task-specific models. Pic2Word introduces ZS-CIR, using image-caption pairs and unlabeled images to train one model across diverse CIR tasks.

  • Motivation: CIR combines a query image with text to retrieve images matching a desired modification.Applications include domain conversion, object or scene composition, and fashion-attribute manipulation.
  • Challenges: Existing CIR methods require triplets containing a reference image, text description, and target image, making dataset collection expensive.Both collecting related image pairs and writing modification descriptions incur substantial labeling cost.
  • Challenges: Models trained on labeled data are specialized to particular use cases and may not generalize across CIR tasks.
  • Zero-shot CIR: ZS-CIR aims to build a single model that performs diverse CIR tasks without expensive labeled-triplet collection.The proposed training sources are large-scale image-caption pairs and unlabeled images.
  • Pic2Word: Pic2Word maps an input image to a language token so image features and text descriptions can be flexibly composed using pretrained vision-language models.Its mapping network is trained with contrastive learning rather than triplet-labeled CIR data.
  • Results: 10 - 100% relative improvement is reported across four CIR tasks, reaching performance on-par with several methods trained on labeled data.

2. Related Work

Prior CIR work combines image and language features for retrieval, while vision-language foundation models provide broadly aligned image-text representations. Pic2Word differs from related image-token methods by targeting composed retrieval without class annotations or token fine-tuning.

  • Composed Image Retrieval: CIR retrieves images from a reference-image and text pair, with prior models commonly using late fusion of separately encoded visual and language features.Other work includes fashion retrieval, scene composition, and input-level feature fusion.
  • Vision-language Foundation Models: Vision-language foundation models such as CLIP and ALIGN pretrain paired image and language encoders on hundreds of millions of image-caption pairs.Their exposure to varied textual descriptions supports handling diverse task concepts.
  • Representing Images as Tokens: Earlier image-token approaches typically detect regions and optimize multimodal objectives during vision-language pretraining.These methods require a high-performing object detector during pretraining.
  • Representing Images as Tokens: Pic2Word focuses on composed image retrieval without class-wise annotations or fine-tuning a word token on few-shot labeled images.

3. Method

Pic2Word freezes CLIP’s vision and language encoders, learns a mapping from image embeddings to pseudo language tokens, and inserts those tokens into prompts for retrieval. Contrastive training makes the prompted language representation close to the corresponding visual representation, while inference supports several composition formats.

  • 3. Method: Pic2Word uses a frozen pretrained CLIP model and trains a mapping network to convert visual embeddings into pseudo language tokens.The predicted token is intended to faithfully represent the input image in CLIP’s language space.
  • 3.1. Preliminaries: CLIP aligns paired image-caption representations while separating unpaired examples through contrastive learning.Image and language features are normalized before image-caption retrieval.
  • 3.2. Learning the Pic2Word Mapping Network: The mapping network appends its predicted token to a prompt such as “a photo of” and feeds the resulting sequence into the frozen language encoder.The resulting language embedding is trained to represent the input visual embedding.
  • 3.2. Learning the Pic2Word Mapping Network: Contrastive optimization trains the mapping network so the prompted language embedding approaches the corresponding visual embedding using unlabeled images only.The other encoders remain fixed during this mapping-network optimization.
  • 3.2. Learning the Pic2Word Mapping Network: Pic2Word represents each image as one word token and can accept multiple images when forming a query.The paper notes that multiple tokens may better capture very fine image details.
  • 3.3. Inference: At inference, the pseudo token is inserted into predefined prompts, whose language embeddings are compared with visual features of candidate images.
  • 3.3. Inference: Inference prompts support domain conversion, object or scene composition, and sentence-based modification of the reference image.Examples include “[domain] of [*]”, “a photo of [*], [obj1] ...”, and “a photo of [*], [text]”.

4. Experiments

Pic2Word is evaluated across domain conversion, object composition, scene manipulation, and fashion attribute manipulation using zero-shot and supervised baselines. It generally outperforms zero-shot methods and matches or exceeds several supervised approaches, while performance depends on dataset-specific modality biases.

  • Experimental setup: Pic2Word is evaluated on four diverse CIR scenarios, including CIRR and Fashion-IQ benchmarks.The experiments cover domain conversion, object composition, scene manipulation, and fashion attribute manipulation.
  • Main results: Pic2Word outperforms all baselines by a large margin on ImageNet domain conversion.The result indicates strong composition of domain words with image features and is also illustrated qualitatively in Figure 3.
  • Main results: Pic2Word outperforms zero-shot baselines on COCO object composition, while supervised performance varies by training dataset.The Fashion-IQ-trained Combiner performs better than Pic2Word, whereas the CIRR-trained Combiner performs worse.
  • Main results: On CIRR and Fashion-IQ, Pic2Word outperforms zero-shot baselines and some supervised approaches.These datasets can contain informative captions or reference images that are weakly related to the target, allowing supervised models to learn dataset-specific modality preferences.
  • Analysis: The pseudo language token retrieves its input image with 99.8% top-1 recall and 100.0% top-5 recall on 13,164 CC3M validation images.This evaluation indicates that the token captures unique image features effectively.
  • Analysis: Pic2Word outperforms Combiner when fewer than 1,000 supervised triplets are available.The comparison uses reduced training samples for the supervised baseline in Fashion-IQ and CIRR.

5. Conclusion

The paper introduces zero-shot composed image retrieval and Pic2Word, which maps image features into pseudo language tokens for composing images with text. Across four datasets and diverse CIR tasks, Pic2Word generalizes strongly and matches or exceeds methods requiring labeled training data.

  • Pic2Word treats an image as a text token, allowing CLIP’s language encoder to compose image features with text descriptions.A pre-trained CLIP model provides the language and vision encoders, while Pic2Word learns the image-to-token mapping.
  • Pic2Word shows strong generalization across diverse CIR tasks and performs on-par with or better than recent methods requiring labeled training data.The analysis covers four datasets and tasks including domain conversion, object composition, scene manipulation, and fashion attribute manipulation.

A. Additional Discussion

DreamBooth and Textual Inversion generate images from user intent, whereas Pic2Word retrieves images from that intent. Pic2Word also differs in its data requirements and inference procedure.

  • Pic2Word retrieves images given user intent, while DreamBooth and Textual Inversion generate images with user intent.The distinction is framed as an application-level difference between retrieval and generation.
  • Pic2Word requires only unlabeled images, whereas DreamBooth and Textual Inversion rely on image sets containing the same object.
  • Pic2Word works in real time because inference requires a single forward pass of the visual encoder.

B. Experimental Details

The supplementary material documents evaluation datasets, the mapping-network architecture, qualitative-example construction, and baseline details. The mapping-network output is fed into the language encoder.

  • Evaluation Dataset: Evaluation datasets contain specified numbers of query and candidate images, with preprocessing following the main paper.
  • Mapping network design: The mapping network’s architecture is summarized in a Pytorch-style model description, and its output is fed into the language encoder.
  • Images used for qualitative examples: Qualitative examples exclude images that can identify a person, and one figure query uses an image outside the CC3M validation set.
  • Baseline and architecture reporting: Table C compares mapping-network designs and reports the strongest zero-shot baseline among three baselines for each dataset.

C. Additional Experiments

Additional experiments examine mapping-network design, training-data scale, fine-tuned CLIP baselines, CLIP versus BLIP, and qualitative retrieval behavior. They support the need for an expressive mapping network and show that composed representations capture image characteristics alongside textual modifications.

  • Architecture design: Removing nonlinear activation significantly reduces performance, while no architecture variant outperforms the default model across all datasets.The authors interpret this as evidence that faithfully predicting the pseudo language token requires a sufficiently expressive mapping network.
  • Training dataset: CC12M, approximately four times larger than CC3M, provides no clear advantage, indicating that CC3M is sufficient for training the mapping network.
  • Fine-tuned CLIP comparison: Fine-tuning CLIP on CC3M improves baseline performance, but the resulting baselines still fall short of Pic2Word.
  • CLIP and BLIP comparison: Text-only BLIP outperforms text-only CLIP, showing that pre-training differences can substantially affect performance.
  • Qualitative retrieval: In qualitative examples, target images appear within the top four for ImageNet queries, while web-image examples place all top-1 images in the candidate set.The examples indicate that composed representations express object characteristics specified by an image together with textual modifications.
  • Additional evaluation reporting: The supplementary tables report dataset-specific model, method, and retrieval metrics for additional comparisons and qualitative settings.
Loading 2302.03084v2…