PaperlayerView exact revision on arXiv

Source-linked AI summary

"This is my unicorn, Fluffy": Personalizing frozen vision-language representations

Niv Cohen, Rinon Gal, Eli A. Meirom, Gal Chechik, Yuval Atzmon

arXiv:2204.01694v3cs.CVcs.LG

TL;DR

Pretrained vision-language models offer open-world language capabilities, but how to use them to reason about user-specific visual concepts remains unclear. The paper introduces PerVL and PALAVRA, which extend the model’s input vocabulary with personalized concepts; it reports use in rich-language downstream tasks and notes inherited limitations of the underlying model.

  • Problem

    It is unclear how pretrained vision-language models can reason about user-specific visual concepts in unstructured language.

    Source: §1 · p. 1

  • Method

    PerVL learns personalized concepts from a few positive images and uses new input word embeddings so the pretrained model can reason about them in sentences.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    PALAVRA is reported to learn personalized concepts from positive examples and support rich-language vision-language tasks.

    Source: §1 · p. 3, §8 · p. 16

  • Takeaways & Limitations

    The extended representation can be used with rich language for tasks involving both novel and known concepts.

    Source: §8 · p. 16

    The approach inherits underlying vision-language model limitations; for example, CLIP struggles with spatial relations within a photo.

    Source: §8 · p. 17

Abstract

from arXiv · show

Large Vision & Language models pretrained on web-scale data provide representations that are invaluable for numerous V&L problems. However, it is unclear how they can be used for reasoning about user-specific visual concepts in unstructured language. This problem arises in multiple domains, from personalized image retrieval to personalized interaction with smart devices. We introduce a new learning setup called Personalized Vision & Language (PerVL) with two new benchmark datasets for retrieving and segmenting user-specific "personalized" concepts "in the wild". In PerVL, one should learn personalized concepts (1) independently of the downstream task (2) allowing a pretrained model to reason about them with free language, and (3) does not require personalized negative examples. We propose an architecture for solving PerVL that operates by extending the input vocabulary of a pretrained model with new word embeddings for the new personalized concepts. The model can then reason about them by simply using them in a sentence. We demonstrate that our approach learns personalized visual concepts from a few examples and can effectively apply them in image retrieval and semantic segmentation using rich textual queries.

1 Introduction

The introduction asks how pretrained vision-language models can reason about personal visual concepts using open-world language. It proposes PerVL and PALAVRA to learn such concepts from a few examples while retaining the model’s broader language-based reasoning.

1 Introduction

  • The broader motivation is to combine personalized concepts with the pretrained models’ open-world ability to use rich, free-form language.Such models already support tasks including captioning, retrieval, generation, segmentation, and robotic manipulation [54] [23] [26].Source: §1 · p. 1
  • The central challenge is enabling pretrained models to reason about user-specific visual instances in open-world vision problems.Users can often collect a few positive examples, but consistent negative distractors are harder to provide.Source: §1 · pp. 1–2
  • PerVL learns from a few images and a concept-type string, then applies the representation to downstream tasks without further task-specific supervision.The setup includes personalized retrieval, human-robot interaction, and video search scenarios.Source: §1 · p. 3
  • PALAVRA adds new word embeddings to the model’s input vocabulary, allowing personalized concepts to be used in sentences without modifying the encoder.This approach is motivated by prior methods’ risks of forgetting prior knowledge or difficulty accessing it alongside newly learned concepts.Source: §1 · pp. 2–3
  • The paper introduces PerVL, two benchmarks, and PALAVRA, which expands the model’s input vocabulary using positive image examples and a cycle-consistent loss.It also proposes using a textual encoder to improve generalization to new visual concepts.Source: §1 · p. 3

2 Related work

Related work personalizes or adapts vision-language models in several ways, including output-space adapters, soft prefixes, and implicit concept storage. The paper distinguishes PerVL through vocabulary-grounded representations intended for flexible downstream tasks.

2 Related work

  • Prior work [67] [86] [27] [49] commonly learns residual adapters, while [89] learns a soft prefix for classification.Fine-tuning can harm out-of-distribution generalization [78] [38], motivating approaches that do not fine-tune the pretrained model.Source: §2 · pp. 3–4
  • Adapter-based approaches modify CLIP’s output representation, while PALAVRA enriches the input vocabulary and retains the pretrained mapping.Other prior methods focus on narrow-vocabulary image classification, whereas this setup targets representations usable in varied downstream tasks.Source: §2 · p. 4
  • Some approaches combine fast-learned concepts with slow-learned concepts, but store the new concepts implicitly in network activations rather than grounding them in vocabulary.Related personalization work also maps concepts to tokens for captioning or generator inversion [19].Source: §2 · p. 4
  • The paper distinguishes its setup from meta-learning and incremental learning by targeting diverse downstream tasks, concept domains beyond training, and compatibility with CLIP-based tasks.Existing few-shot benchmarks lack instance-level annotations and include only a single task, so they do not directly fit this setting.Source: §2 · pp. 4–5

3 A new setup, Personalized Vision & Language

PerVL formalizes personalization of a pretrained vision-language model from a few positive images, without downstream-task supervision. It aims to add new concepts while preserving the model’s behavior on its original vocabulary.

3 A new setup, Personalized Vision & Language

  • PerVL personalizes a pretrained model with a few positive image examples and no supervision for the downstream task.At inference, the representation is used for a task involving the new concept.Source: §3 · p. 5
  • The setup expands the accepted vocabulary from V to V′ = V ∪ C, where C contains new concepts, while aiming to preserve behavior on V.The model accepts sentences and images as inputs.Source: §3 · p. 5
  • For each concept, personalization uses a small set of images without assuming access to negative training images.A string naming the concept type, such as “mug,” is also provided and can guide distinction from the broader type.Source: §3 · p. 5
  • At inference, the new concept is used in an instance for a downstream vision-language task that can be inferred using the pretrained model.For the CLIP case, the model compares sentence and image representations by cosine similarity.Source: §3 · p. 5

4 Methods

PALAVRA expands CLIP’s input vocabulary with personalized word embeddings rather than changing its output space. It learns an inversion mapping from image and augmented-text data, then initializes and tunes each concept embedding for use in downstream tasks.

4 Methods

  • PALAVRA adds personalized tokens to CLIP’s input vocabulary, unlike adapters that append layers after the encoder and change its output space.The approach is trained with fewer than 1M samples, compared with ∼400M images required to retrain CLIP; once trained, it can serve users with their own vocabularies.Source: §4 · pp. 5–6, Figure 2 · p. 6

4.1 Learning the inversion mapping fθ

  • The inversion mapping fθ maps sets of CLIP-space image embeddings into word-embedding codes, using a cycle loss to align template-text and image representations.Augmented text examples from a 20K-type vocabulary improve generalization to out-of-vocabulary concepts; an alignment matrix maps text representations toward the image area during training, not inference.Source: §4 · p. 6, Figure 3 · p. 7, §4.1 · pp. 7–8

4.2 Personalization: Learn an embedding of personalized concepts

  • For personalization, CLIP image examples produce an initial concept code through frozen fθ, which is then optimized with a contrastive loss against the concept-type embedding.The concept type acts as a “super-concept” because personalization uses no negative image examples.Source: §4 · p. 6, Figure 4 · p. 7, §4.2 · p. 9

4.3 Inference

  • At inference, the pretrained model remains unchanged; sentences use the learned concept embedding when they contain its [CONCEPT] token.Using [CONCEPT] followed by the concept type was found beneficial.Source: §4.3 · p. 9

5 Evaluation datasets for PerVL

The evaluation introduces personalized retrieval and segmentation benchmarks built from DeepFashion2 and Youtube-VOS. PALAVRA achieves the highest retrieval rates across both benchmarks and all reported retrieval metrics, while the Youtube-VOS benchmark tests concept recognition amid visual and query ambiguity.

5 Evaluation datasets for PerVL

  • The benchmarks use captions collected for DeepFashion2 images as search queries in image retrieval, and captions collected for Youtube-VOS frames with corresponding segmentation maps for referring-expression segmentation.The DeepFashion2 dataset contains 1,700 images from 100 unique fashion-item concepts; 450 images were assigned to an evaluation set.Source: §5 · p. 9, §5.1 · p. 10

5.1 Personalized fashion-item retrieval with DeepFashion2

  • DeepFashion2 captions range from detailed context to shorter, more ambiguous descriptions, testing retrieval with different amounts of query information.The benchmark uses 13 clothing categories as concept types.Source: §5.1 · pp. 9–10

5.2 Youtube-VOS for personalized retrieval and segmentation.

  • Youtube-VOS evaluation uses each video’s last frame and earlier frames for training, selecting scenes where the target concept appears at least twice in the evaluation frame.Workers captioned what distinguishes each instance from similar entities; the benchmark has about 500 concepts and 6,300 training samples, with scene-based validation and test splits.Source: §5.2 · pp. 10–11
  • PALAVRA achieves the highest retrieval rates on DeepFashion2 and Youtube-VOS across all retrieval metrics, significantly improving over the strongest baseline, AvgIM&Text.Performance for PALAVRA and Text Only substantially degrades with concept-only queries, demonstrating the importance of query information for distinguishing targets from distractors.Source: Figure 6 · p. 11
  • The Youtube-VOS retrieval variant matches textual queries to wide-box crops of instances, with distractors drawn across more scenes than in segmentation.Not all same-scene instances were labeled for retrieval.Source: §5.2 · p. 11

6 Experiments

The experiments evaluate personalized image retrieval and semantic segmentation against baselines on two PerVL benchmarks. PALAVRA leads on rich-query retrieval and improves segmentation, with robustness examined across object size and visual ambiguity.

6.1 Personalized image retrieval with a textual query

  • PALAVRA achieves the highest retrieval rate across all metrics and both benchmarks, improving 30%–50% over the strongest baseline, AvgIM+Text.The comparison uses challenging short rich queries; detailed rich-query results are reported in the appendix.Source: §6.1 · p. 13
  • Rich-query information is essential for retrieval, while Adapter and COLLIE outperform vanilla CLIP on concept-only queries but degrade with rich queries.The authors attribute the degradation to adaptation layers’ difficulty with free-form text unseen during training; these baselines are also sensitive to query prefixes.Source: §6.1 · p. 13

6.2 Semantic segmentation with a textual query

  • At IoU threshold 0.5, PALAVRA improves segmentation by 44.69% over the best competitor.Fig. 7a plots the percentage of test images exceeding each IoU threshold; PALAVRA leads across the full threshold range.Source: Figure 7 · p. 13, §6.2 · p. 14
  • PALAVRA performs better with concept-only than rich queries for segmentation; the authors hypothesize that complex context diverts CLIP’s attention and causes false negatives.In retrieval, context objects can instead help because they appear in target images and rarely in negative images.Source: §6.2 · p. 14
  • For small objects, segmentation worsens generally and image-based methods fare worse, while PALAVRA degrades less by relying on text.The model substantially outperforms baselines for visually distinct concepts and also improves for mostly ambiguous concepts.Source: Figure 7 · p. 13, §6.2 · pp. 14–15

7 Ablation Study

The ablations separate set-encoder training from fine-tuning and test text augmentation, loss components, initialization, and alignment. Results show that these components contribute differently across DeepFashion2 and YTVOS.

7 Ablation Study

  • Without tuning, PALAVRA improves over Text Only by 25% on DeepFashion2 (22.1 vs. 17.6) and YTVOS (47.1 vs. 37.6).The result indicates that the set encoder learns to predict visual-concept word embeddings that outperform vanilla CLIP embeddings.Source: §7 · p. 15
  • Extended-vocabulary text augmentation improves concept learning by ∼16% for DeepFashion2 (22.1 vs. 19.1) and ∼6% for YTVOS (47.1 vs. 44.4).This component is tested during training of the set encoder.Source: §7 · p. 15
  • Combining cycle loss with ground-truth regularization improves DeepFashion2 by ∼16% (22.1 vs 19.2) and YTVOS by ∼14% (47.1 vs 41.4); cycle loss alone fails to generalize.The authors hypothesize that ground-truth regularization keeps inversions in better-behaved regions of CLIP’s input space.Source: §7 · pp. 15–16
  • Replacing the set encoder’s prediction with random initialization degrades performance by ∼6% for YTVOS (57.1 vs 61.2) and ∼3% for DeepFashion2 (27.5 vs 28.4).The authors describe this as showing synergy between the two personalization steps.Source: §7 · p. 16
  • Integrating alignment matrix A improved results by ∼8% on DeepFashion2 (28.4 vs. 26.3) and ∼5% on YTVOS (61.2 vs. 58.1).Source: §7 · p. 16

8 Discussion

The approach expands a pretrained model’s input vocabulary with representations of personal concepts, supporting rich-language V&L tasks while retaining known concepts. Its limits include inherited weaknesses of the underlying model, such as CLIP’s difficulty with spatial relations.

8 Discussion

  • The approach extends a pretrained model’s input space with representations of personal concepts, letting it handle rich-language V&L tasks involving novel and known concepts.The authors hope this can support applications such as home robotics and organizing personal data.Source: §8 · pp. 16–17
  • Extended CLIP representations inherit CLIP’s difficulty understanding spatial relations within an image.The authors expect the approach could be extended to other V&L models.Source: §8 · p. 17

A Implementation details

The implementation appendix specifies training data and prompts, embedding alignment and regularization, and the optimization settings and controls used for the experiments.

A Implementation details

  • Retrieval used 5 training samples per concept and segmentation used 10; set-encoder training alternated batches of COCO images and augmented captions.The set encoder was trained for 300 epochs with batch size 256; word embeddings were optimized for 30 epochs.Source: A Implementation details · pp. 23–24
  • Cycle-loss queries sampled each epoch from templates spanning photo, illustration, sketch, diagram, and short concept-only prompts.Contrastive losses divided cosine similarities by temperature 0.25, selected on a validation set.Source: A Implementation details · p. 23
  • The set encoder’s regularizer encourages the predicted concept embedding to remain close to the embedding of its concept type.For multiword types, the type embedding uses the first word.Source: A Implementation details · p. 23
  • The alignment matrix was trained to map caption embeddings to image embeddings by minimizing L2 loss, and fine-tuning substituted A · ηc for ηc.Encoder outputs were normalized to unit norm; CLIP experiments used ViT-B/32.Source: A Implementation details · pp. 23–24
  • Hyperparameters were tuned one at a time on validation MRR, and the same fixed settings were used across methods.Five repetitions trained new set encoders; PALAVRA and baselines used consistent few-shot subsets, while COLLIE and adapter baselines received unique concept phrases for concepts sharing a type.Source: A Implementation details · p. 24

B Additional Results

Additional results examine shot count, detailed captions, prompt sensitivity, and qualitative segmentation; the appendix also describes an extension to other V&L models and segmentation baselines.

B Additional Results

  • DeepFashion2 retrieval improves as shots increase, while YTVOS saturates early, plausibly because its training images come from the same video and vary less.Figure A.1 plots MRR against the number of shots.Source: B Additional Results · pp. 24–25, Figure A.1 · p. 25
  • All compared retrieval methods improve with detailed captions, while PALAVRA remains ahead on DeepFashion2.Table A.1 reports retrieval results for detailed captions.Source: B Additional Results · p. 25, Table A.1 · p. 25
  • On DeepFashion2 retrieval, COLLIE achieves 13.4% average MRR with only the [CONCEPT] tag; sentence-prefix queries sharply reduce MRR to 5.3%, 4.6%, and 3.3%, respectively.Source: B Additional Results · pp. 25–26
  • PALAVRA sometimes segments distractors with the target or lets textual descriptions draw CLIP’s attention away from the main object.Qualitative examples also show successful identification despite visually similar distractors.Source: B Additional Results · p. 26, Figure A.2 · p. 26

C Personalization of Other Vision & Language

  • The authors describe how the approach could extend to other V&L models, using M-DETR as an example for mapping between CNN and RoBERTa spaces.They propose that fθ map CNN outputs to RoBERTa’s input space and that alignment matrix A map back to CNN outputs.Source: C Personalization of Other Vision &… · p. 26

D.1 Baselines and hyper parameters

  • The segmentation baseline uses text embeddings to match image regions, allowing them to be replaced by normalized average image embeddings or averaged image-and-text embeddings.For the PALAVRA comparison, the original CLIP text encoder uses learned input word embeddings for the concept token.Source: §D.1 · pp. 26–27
  • Segmentation settings were tuned on validation data; the single-image method used a resizing factor of 0.5 and three clicks from relevancy maps.Other parameters followed the baseline implementation.Source: §D.1 · p. 27

D.2 Rich queries versus concept-only queries

The appendix investigates why rich captions can underperform in segmentation and provides additional qualitative examples of PALAVRA’s segmentation behavior.

D.2 Rich queries versus concept-only queries

  • Rich queries initially underperform concept-only queries in segmentation, but outperform them when background relevance scores are masked except for valid retrieval candidates.The analysis attributes the initial pattern to CLIP attention drifting toward other objects mentioned in rich queries.Source: §D.2 · p. 27
  • For CLIP-guided personalized segmentation, captions should describe the target or its immediate vicinity rather than relations to other objects; COLLIE appears insensitive to query detail.The authors report that COLLIE’s segmentation performance does not appear sensitive to query detail, unlike PALAVRA and baseline CLIP.Source: §D.2 · p. 27

D.3 Qualitative Analysis

  • The qualitative analysis compares PALAVRA with COLLIE using examples of successful segmentations and failure modes.The appendix identifies Figures A.5 and A.6 as showing these examples, respectively.Source: §D.3 · p. 27

E Evaluation datasets

The appendix details the construction and annotation of personalized benchmarks from DeepFashion2 and YouTube-VOS, with concept-specific training examples, contextual captions, and held-out evaluation data.

E Evaluation datasets

  • Rich queries outperform concept-only queries when context objects are invalid segmentation candidates, but COLLIE largely ignores rich-query context in segmentation, where CLIP attention may drift toward mentioned objects.Source: Figure A.3 · p. 28, Figure A.4 · p. 28, Figure A.7 · p. 31

E.1 DeepFashion2

  • DeepFashion2 selection required at least five properly scaled images and 15 images per item, yielding about 1,700 images and 100 concepts; 652 contextual images were curated for annotation.Source: §E.1 · p. 31
  • The DeepFashion2 evaluation set contains 450 images and training set 1,250, with concepts randomly split into 50 validation and 50 test concepts.Source: §E.1 · p. 31

Annotations for DeepFashion2 with Amazon Mechanical Turk

E.2 Summarizing textual annotations with AMT

  • Short queries summarize detailed captions around one contextual object, intentionally reducing information; most DeepFashion2 experiments use these more challenging short queries.Source: §E.2 · p. 33

Overview

  • YTVOS object instances are personalized concepts: each video’s last frame is the evaluation target, while earlier crops train the model after filtering for zero mask IoU, 150-pixel displacement, and at least four examples.Source: Overview · pp. 33–34

Annotation with AMT

  • YTVOS captions distinguish each entity from similar instances, while retrieval candidates are wide crops around evaluation-frame instances, sized to retain context.Source: Annotation with AMT · pp. 34–35
Loading 2204.01694v3…