PaperlayerView exact revision on arXiv

Source-linked AI summary

CoLLIE: Continual Learning of Language Grounding from Language-Image Embeddings

Gabriel Skantze, Bram Willemsen

arXiv:2111.07993v3cs.CL

TL;DR

Language-image models can struggle with new or shifting language use, and retraining the foundation model is not feasible often enough. CoLLIE adjusts language embeddings with a learned transformation; the authors report learning from few examples, generalizing, and largely retaining zero-shot performance.

  • Problem

    Fixed language-image models lack a practical way to adapt to new or shifting language use, including meanings negotiated in context.

    Source: §1 · pp. 2–3

  • Method

    CoLLIE learns a transformation that adjusts language embeddings when needed while retaining a fixed foundation model.

    Source: §1 · p. 4, §7 · p. 20

  • Results

    The authors report fairly high performance with one example per new category, generalization to related language, and only a slight drop in original zero-shot performance.

    Source: §6 · p. 19

  • Takeaways & Limitations

    The authors present continual adjustment of language-image embeddings for new grounding as a problem not addressed by prior work and a benchmark for future models.

    Source: §6 · p. 19

    CoLLIE assumes the foundation model already represents the visual domain well, so its performance is constrained by that model.

    Source: §2 · p. 6

Abstract

from arXiv · show

This paper presents CoLLIE: a simple, yet effective model for continual learning of how language is grounded in vision. Given a pre-trained multimodal embedding model, where language and images are projected in the same semantic space (in this case CLIP by OpenAI), CoLLIE learns a transformation function that adjusts the language embeddings when needed to accommodate new language use. This is done by predicting the difference vector that needs to be applied, as well as a scaling factor for this vector, so that the adjustment is only applied when needed. Unlike traditional few-shot learning, the model does not just learn new classes and labels, but can also generalize to similar language use and leverage semantic compositionality. We verify the model's performance on two different tasks of identifying the targets of referring expressions, where it has to learn new language use. The results show that the model can efficiently learn and generalize from only a few examples, with little interference with the model's original zero-shot performance.

1. Introduction

Language use can shift during interaction, while fixed vision-language models lack a practical way to adapt to new meanings and preserve compositional understanding. CoLLIE addresses this by adjusting language embeddings through a separate transformation model while keeping the foundation model fixed.

1. Introduction

  • Fixed vision-language models cannot readily accommodate evolving or negotiated language use, although small misalignments may cause miscommunication.Language meaning can depend on context and task, and retraining the entire model frequently is not feasible.Source: §1 · pp. 2–3
  • Few-shot classifiers treat new labels as atomic categories, leaving unclear how to relate them to known labels or combine them compositionally.Learning “the red monitor” does not by itself imply understanding “the red display” or “the blue monitor.”Source: §1 · p. 4
  • CoLLIE keeps the zero-shot foundation model fixed and learns a separate transformation that adjusts language embeddings when needed.It predicts a difference vector to correct misaligned dimensions while leaving others unaffected.Source: §1 · p. 4
  • CoLLIE aims to learn new language-image mappings from only a few examples using a lightweight transformation model.The transformation model is described as relatively cheap to retrain.Source: §1 · p. 4
  • CoLLIE aims to generalize learned concepts to related concepts and use semantic compositionality.The paper gives combining newly learned language with known language as an intended capability.Source: §1 · p. 4
  • CoLLIE aims to preserve performance on prior tasks and prevent newly learned concepts from interfering with one another.These are stated design goals, rather than results established in this introduction.Source: §1 · p. 4

2. Related Work

Related work spans visual-language grounding, interactive reference resolution, and continual learning, but commonly assumes fixed language use or bounded classes. CoLLIE instead adapts language grounding while relying on a capable, fixed foundation model.

2. Related Work

  • Referring expression comprehension identifies a target in an image using compositional language, unlike classification over a fixed set of classes.The text describes ranking candidate referents by proximity in a joint image-text embedding space.Source: §2 · p. 5
  • Image captioning describes a scene, while referring-expression generation identifies a specific target and must balance distinctiveness with brevity.The latter should account for distractors when choosing how to describe the target.Source: §2 · p. 5
  • Interactive grounding work often assumes language remains fixed; dialog history can capture short-term adaptation, while longer-term change requires parameter updates.Retraining after each round improved performance in a small-domain study, but the passage says this is infeasible for models such as CLIP.Source: §2 · pp. 5–6
  • Continual-learning research has mainly addressed adding classes to image classifiers, with approaches including regularization, architectural change, and complementary learning systems.CoLLIE is closest to complementary learning in combining a fixed foundation model with a dynamic model.Source: §2 · p. 6
  • CoLLIE assumes the foundation model already represents the visual domain well, so its performance is constrained by that model.The authors accept storing newly learned examples and a small fixed set of negatives, retraining only the transformation model.Source: §2 · pp. 6–7

3. Data, Task and Metric

The evaluations use referring-expression retrieval to test whether CLIP can rank the intended image among candidate referents. They establish CLIP’s zero-shot performance on LAD and colored tangrams, where performance is notably lower on the tangrams.

3. Data, Task and Metric

  • Referring-expression comprehension ranks candidate referents by match to an expression, and the study uses separate images to control distractors.Mean Reciprocal Rank (MRR) is 1 divided by the correct candidate’s rank; 1 means first and 0.5 means second.Source: §3 · p. 7
  • CLIP’s zero-shot MRR is 0.310 on colored tangrams, compared with 0.773 on LAD.The tangram set contains 85 referents formed from 17 shapes and five colors; LAD uses 200 categories across four super-categories.Source: §3 · p. 7
  • The experiments use the publicly released CLIP model with a ViT-B/32 Vision Transformer architecture unless otherwise stated.LAD results average 20 iterations, each ranking one randomly selected image per category.Source: §3 · p. 7
  • A t-SNE view of colored-tangram CLIP embeddings shows apparent shape and color clusters, suggesting they may be associated with names.The analysis reduces the embeddings from 512 dimensions to two.Source: §3 · p. 8, Figure 3 · p. 8

4. The CoLLIE Transformation

CoLLIE transforms CLIP text embeddings to better match images under new language use while preserving zero-shot behavior when no adjustment is needed. It predicts an embedding difference and scales that correction according to whether the expression should be transformed.

4. The CoLLIE Transformation

  • CoLLIE maps a CLIP text embedding T to a transformed embedding T′ intended to better represent new language use and move closer to image embedding I.The transformation should usually leave the input similar to its original form to retain zero-shot performance.Source: §4 · p. 8
  • The transformation adds a predicted adjustment vector a(T), scaled by s(T), which ranges from 0 to 1.The scaling function helps retain zero-shot performance by limiting adjustments when they are not needed.Source: §4 · pp. 8–9
  • The adjustment function learns the difference I − T from accumulated text-image embedding pairs, using ridge-regularized linear regression.Predicting the difference is intended to correct dimensions that need adjustment without affecting dimensions that are already correct; the ridge parameter is λ = 0.001.Source: §4 · p. 9
  • The scaling function is trained to output near 1 for expressions to transform and near 0 otherwise, using accumulated examples as positives and 1,000 common English nouns as negatives.The authors use nouns because referring expressions are noun phrases and common nouns are expected to cover the embedding space fairly well.Source: §4 · p. 9
  • For the initial scaling-function tests, CoLLIE uses support vector regression with a linear kernel and constrains outputs to [0, 1].The authors state that alternative models will be evaluated later.Source: §4 · p. 10
  • In the monitor example, a correction learned from “the red monitor” is applied to “the blue monitor,” targeting misaligned shape dimensions while leaving aligned color dimensions alone.The CLIP weights remain frozen; only the added adjustment and scaling functions are retrained.Source: §4 · p. 10

5. Experiments

The experiments test whether CoLLIE can learn pseudo-words for known photographic objects and ground known words to novel colored tangrams. They assess learning, generalization, and retention of zero-shot performance.

5.1 Experiment I: Learning Pseudo-words for Realistic Objects

  • With one example per category, CoLLIE reaches 0.618 MRR, rising to 0.750 after five; on trained categories, final original-name MRR is 0.699 versus CLIP’s 0.760.On other categories, final MRR is 0.719 versus CLIP’s 0.769; without scaling, retention falls to 0.260 on trained categories and 0.242 on other categories.Source: §5.1 · pp. 11–12, Figure 7 · p. 12

5.1.1 Effects of Scaling and Adjustment Functions

  • SVR with RBF or linear kernels compares favorably with linear and logistic regression; a separate scaling function supports retention, while ridge regression with λ=0.001 balances learning and retention.Using common nouns as zero-vector targets instead of a separate scaling function yields much lower performance, especially for retention.Source: §5.1.1 · pp. 13–14

5.1.2 Effects of Training Size

  • As N increases, learning stays relatively stable but retention falls, especially with 100 negative examples; KNN improves over SVR, and more negatives may mitigate the drop.The authors identify scaling functions for large N as future work.Source: §5.1.2 · p. 14

5.2 Experiment II: Learning Language for Tangram Figures

  • 0.857 MRR versus CLIP’s 0.317 on unseen shape-color combinations after training on all 17 shapes in one color.The experiment was iterated 100 times.Source: §5.2 · p. 16
  • After 30 rounds, CoLLIE reaches 0.860 MRR on tangrams; unique shape-color combinations mean this result requires generalization, unlike the few-shot learner.The few-shot learner performs much worse because it cannot generalize across newly introduced referring expressions.Source: §5.2 · p. 15, Figure 9 · p. 17

5.2.1 Generalizing with Synonyms

  • 0.602 MRR on tangram synonyms, compared with CLIP’s 0.290, provides further evidence of generalization, though individual synonym results vary widely.Individual MRRs range from 0.056 to 0.920.Source: §5.2.1 · p. 17, Figure 10 · p. 18

5.2.2 Effects of Different Vision Backbones

  • CoLLIE’s tangram performance seems relatively unaffected by vision-backbone choice, although CLIP’s zero-shot performance is lower with ResNet-50x4.Table 4 reports MRR results across CLIP vision backbones and their embedding dimensions.Source: Table 4 · p. 18, §5.2.2 · p. 19

6. Discussion

The discussion presents CoLLIE as a continual language-use adaptation approach distinct from incremental class learning, with evidence of sample-efficient learning, generalization, and retention. It also identifies trade-offs and scope limits tied to scaling, the foundation model, and the transformation design.

6. Discussion

  • Unlike incremental class learning, CoLLIE addresses continual adaptation of language use in zero-shot joint embeddings and establishes a benchmark where prior comparisons were unavailable.Source: §6 · p. 19
  • CoLLIE reaches fairly high performance with one example per new category and generalizes to compositional and, to an extent, synonymous expressions.The experiments used shorter expressions and basic forms of compositionality; handling more complex constructions remains future work.Source: §6 · p. 19
  • Experiment I found a slight zero-shot drop but no catastrophic forgetting; simple, low-parameter transformations and small stored examples permit quick CPU updates.One Experiment II iteration, comprising 30 model updates, takes about 1 second on an Intel Core i7-1065G7 CPU.Source: §6 · p. 19
  • Adding examples requires retraining the transformation, which limits scalability depending on the use case; learning without storing examples remains future work.Source: §6 · pp. 19–20
  • The scaling function’s common-noun negative training may not prevent zero-shot performance drops, especially as concepts accumulate; stricter matching trades generalization for retention.Source: §6 · p. 20
  • CoLLIE depends on CLIP’s representations, so its performance is bounded by how well the foundation model represents visual properties, including detailed attributes and object counts.Source: §6 · p. 20
  • Transforming language embeddings supports resolving both newly learned and existing expressions, whereas transforming image embeddings would not resolve the new language use in Experiment I.The paper also notes recent work favoring a frozen image model and fine-tuned text model.Source: §6 · p. 20
  • Future directions include consolidating transformations, using context-specific transformations, and incorporating CoLLIE into interactive systems for partner-specific language use.Source: §6 · p. 20

7. Conclusion

CoLLIE adapts language embeddings from a pretrained language-image model to accommodate new language use. The paper reports learning from very few examples while generalizing beyond new labels to similar language use and semantic compositionality.

7. Conclusion

  • CoLLIE learns a transformation over language embeddings, predicting a difference vector and scaling it to adjust embeddings only when needed.Source: §7 · pp. 20–21
  • The paper establishes benchmarks for the trade-off among continual learning, retention, and generalization.Source: §7 · p. 21
  • The evaluation indicates that CoLLIE learns new language use from very few examples and generalizes to similar language use through semantic compositionality, unlike label-only few-shot learning.Source: §7 · p. 21

Reproducability Statement

The reproducibility statement identifies the implementation framework and the availability of experiment code and data pointers.

Reproducability Statement

Loading 2111.07993v3…