PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Stop Learning: Towards Continual Learning for the CLIP Model

Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, Haoxuan Ding

arXiv:2207.09248v2cs.CV

TL;DR

Fine-tuning CLIP for underperforming categories raises the question of whether it harms zero-shot learning and image-text matching. The paper studies this continual-learning problem, evaluates adapted methods, and proposes VR-LwF; it reports that VR-LwF largely maintains zero-shot capability while benefiting from fine-tuning.

  • Problem

    Fine-tuning CLIP on extra data for underperforming categories may damage its zero-shot learning and image-text matching capabilities.

    Source: A. Challenges · p. 3

  • Method

    The paper evaluates CLIP fine-tuning on updated classes, zero-shot learning, and image-text matching, adapts existing continual-learning methods, and proposes VR-LwF.

    Source: §I · p. 2, B. Evaluation of the CL-CLIP · p. 3

  • Results

    VR-LwF largely maintains zero-shot capability while benefiting from fine-tuning, whereas extensions of existing methods alleviate forgetting but sacrifice fine-tuning performance.

    Source: §I · p. 1, §VII · p. 11

  • Takeaways & Limitations

    The study establishes an evaluation benchmark and identifies CLIP continual learning as distinct from traditional continual learning.

    Source: §VII · p. 11

Abstract

from arXiv · show

The Contrastive Language-Image Pre-training (CLIP) Model is a recently proposed large-scale pre-train model which attracts increasing attention in the computer vision community. Benefiting from its gigantic image-text training set, the CLIP model has learned outstanding capabilities in zero-shot learning and image-text matching. To boost the recognition performance of CLIP on some target visual concepts, it is often desirable to further update the CLIP model by fine-tuning some classes-of-interest on extra training data. This operation, however, raises an important concern: will the update hurt the zero-shot learning or image-text matching capability of the CLIP, i.e., the catastrophic forgetting issue? If yes, could existing continual learning algorithms be adapted to alleviate the risk of catastrophic forgetting? To answer these questions, this work conducts a systemic study on the continual learning issue of the CLIP model. We construct evaluation protocols to measure the impact of fine-tuning updates and explore different ways to upgrade existing continual learning methods to mitigate the forgetting issue of the CLIP model. Our study reveals the particular challenges of CLIP continual learning problem and lays a foundation for further researches. Moreover, we propose a new algorithm, dubbed Learning without Forgetting via Replayed Vocabulary (VR-LwF), which shows exact effectiveness for alleviating the forgetting issue of the CLIP model.

I. INTRODUCTION

CLIP’s zero-shot and image-text matching capabilities make it useful, but fine-tuning categories where it underperforms may damage those capabilities. This study evaluates that forgetting problem, adapts existing continual-learning methods, and proposes VR-LwF.

I. INTRODUCTION

  • CLIP uses 400 million noisy image-text pairs and achieves zero-shot classification without labeled data on 16 datasets.At inference, class names are turned into prompts and compared with image embeddings.Source: §I · p. 1
  • Fine-tuning categories where CLIP falls short may harm its zero-shot learning and image-text matching capabilities.The paper frames this risk as catastrophic forgetting and asks whether continual-learning methods can alleviate it.Source: §I · p. 1
  • The study reports that fine-tuning significantly damages zero-shot and image-text matching, then explores extensions of LwF, GeoDL, IMM, and RKR.It also presents an evaluation protocol for assessing learning and forgetting after fine-tuning.Source: §I · pp. 1–2
  • VR-LwF matches prediction logits for randomly synthesized classes before and after an update and largely alleviates forgetting while retaining fine-tuning benefits.The introduction describes this as a simple-but-effective solution.Source: §I · pp. 1–2

A. The CLIP Model

CLIP encodes images and text separately and uses their similarity for matching and zero-shot classification. Continual-learning methods address forgetting through parameter plasticity, distillation, replay, or architecture changes, but some approaches do not fit CLIP’s pretraining constraints.

A. The CLIP Model

  • CLIP uses separate image and text encoders, then a contrastive-learning-like loss encourages correct matches between their embeddings.For zero-shot classification, class names are inserted into text prompts and compared with image features.Source: A. The CLIP Model · p. 2

B. Continual Learning

  • Continual learning aims to limit performance drops on earlier tasks and includes adaptive-plasticity, distillation, replay, and architecture-based methods.Adaptive methods preserve important parameters; distillation uses an old model as teacher, while replay stores or generates prior-task samples.Source: B. Continual Learning · pp. 2–3
  • EWC penalizes changes to parameters according to Fisher information, while Online-EWC retains only the last old-task penalty.MAS and SI estimate neuron importance from parameter perturbations; NPC lowers learning rates for stable neurons.Source: B. Continual Learning · p. 2
  • IMM [13] combines old and new models by averaging parameters or using a Laplacian approximation.IMM-Mean averages parameters; IMM-Mode combines the networks with a Laplacian approximation.Source: B. Continual Learning · p. 3
  • CLIP’s costly pretraining limits many continual-learning methods to fine-tuning, and exemplar or regularization methods needing prior training information are not optimal.Architecture-based methods instead modify the network, for example by expanding or masking parameters,,.Source: B. Continual Learning · p. 3

III. CONTINUAL LEARNING FOR THE CLIP MODEL

CLIP continual learning seeks to improve target classes while retaining zero-shot ability, a different aim from conventional continual learning. The study evaluates fine-tuning on updated classes, zero-shot classification, and image-text retrieval under one-session and multi-session settings.

A. Challenges

  • CLIP continual learning must improve updated classes while maintaining or improving zero-shot capability, unlike traditional continual learning’s focus on previously learned classes.Because CLIP has no explicit classes and covers many visual concepts, replay and methods needing pretraining information face particular constraints.Source: A. Challenges · p. 3
  • CLIP may forget zero-shot classes more easily because their logits can be close, so a small model change may alter a prediction.Replay is difficult without explicit classes, while EWC, SI, and MAS need information from costly prior pretraining.Source: A. Challenges · p. 3

B. Evaluation of the CL-CLIP

  • The protocol evaluates a fine-tuned model on updated classes, zero-shot learning, and image-text matching using one training set and three test datasets.The benchmark was introduced because no evaluation benchmark for CL-CLIP existed.Source: B. Evaluation of the CL-CLIP · p. 3, Figure 2 · p. 4
  • Updated-task training uses 80 MSCOCO object classes cropped by bounding boxes, with 596,974 training images and 35,360 testing images.Tiered-ImageNet supplies zero-shot evaluation, while MSCOCO provides image-text retrieval data with 5,000 test images and 25,000 captions.Source: B. Evaluation of the CL-CLIP · pp. 3–4
  • One-session training updates the model with all data available at once; multi-session training sequentially divides 80 classes into eight sessions.The multi-session setting is described as a more complex CL-CLIP situation.Source: B. Evaluation of the CL-CLIP · p. 4

A. Building Options

CLIP’s dual encoders allow continual-learning methods to update the image encoder, text encoder, or both. The multi-session MSCOCO setup organizes training and testing samples by session.

A. Building Options

  • The dual-encoder structure permits updating the image encoder, text encoder, or whole model.These options are named Image Only (IO), Text Only (TO), and Whole Model (WM).Source: A. Building Options · p. 4
  • The MSCOCO multi-session table lists prompted names in its second column and training and testing sample amounts in its third.The sessions split the 80 classes into eight sequential training sessions.Source: Table I · p. 4
  • The table’s session splits identify the training and testing samples for multi-session training.The caption specifies that the prompted names and sample amounts appear in the second and third columns, respectively.Source: Table I · p. 4

B. Extensions for CL-CLIP

The section adapts four continual-learning approaches to CL-CLIP: distillation-based LwF and GeoDL, adaptive-plasticity-based IMM, and architecture-based RKR.

B. Extensions for CL-CLIP

  • LwF adds distillation to align updated-model posterior probabilities with the previous model, applied to updated classes in CL-CLIP.For multi-session training, classification covers current classes while distillation covers both current and previous classes.Source: B. Extensions for CL-CLIP · pp. 4–5
  • GeoDL distills image and text features by minimizing their inner products across old and new models along estimated feature manifolds.Its original replaying samples are not applicable to CL-CLIP, so the extension uses distillation on image and text embeddings.Source: B. Extensions for CL-CLIP · p. 5
  • IMM-Mean averages parameters from the current and preceding models, while IMM-Mode requires previous-phase training information.The CL-CLIP extension forms a weighted average of the fine-tuned and original models; in MST, the old model is the previous session model.Source: B. Extensions for CL-CLIP · pp. 5–6
  • RKR freezes previously trained parameters and adds learnable modules, but the CL-CLIP extension uses only scaling factors after rectification caused severe zero-shot forgetting.The scaling factors are applied to each block’s output, and only added parameters are fine-tuned.Source: B. Extensions for CL-CLIP · p. 6

V. METHOD

Because exemplar replay is ill-suited to CLIP, the method replays pseudo-classes made from vocabulary tokens through the text encoder. VR-LwF distills their logits alongside classification training.

V. METHOD

  • Exemplar replay is unsuitable for CLIP because it lacks explicit classes and storing examples across its broad visual concepts is memory-consuming.The paper instead applies replay to the text encoder, whose mapping from class names to classifier weights can use vocabulary tokens without images.Source: §V · p. 6
  • VR-LwF treats randomly sampled vocabulary sequences as pseudo-classes and distills their logits from the previous and current text encoders.The sequences are formed by stacking sampled tokens without imposing grammar; random word-order shuffling can preserve reasonable retrieval performance and can be competitive with VSE++.Source: §V · p. 6, Table II · p. 6
  • The VR-LwF objective combines cross-entropy classification loss with replaying distillation loss.For multi-session training, former classes can be appended to replayed vocabularies, while VR-LwF distills only previous classes.Source: §V · p. 7

VI. EXPERIMENTS

The experiments assess whether CLIP exhibits catastrophic forgetting, whether selected continual-learning methods mitigate it, and how VR-LwF performs under two training settings.

VI. EXPERIMENTS

  • The experiments investigate whether CLIP suffers catastrophic forgetting and whether four selected continual-learning methods can alleviate it.The section also asks what experiences emerge under the two training settings.Source: §VI · p. 7
  • The study tests whether VR-LwF improves performance and examines the impact of its hyperparameters.Implementation details and evaluation metrics are introduced before the experiments.Source: §VI · p. 7

A. Implementation Details and Evaluation Metrics

The evaluation compares one-session and multi-session CL-CLIP training using updated-task accuracy, zero-shot accuracy, retrieval recall, backward transfer, and an accuracy average.

A. Implementation Details and Evaluation Metrics

  • OST fine-tunes for 15 epochs, while MST trains 10 epochs per session across 80 total epochs, equivalent to 10-epoch OST training steps.Both use Adam with batch size 100; VR-LwF uses pseudo-sentences of length 10, with 100 per batch.Source: A. Implementation Details and… · p. 7
  • UT-Acc measures updated-task accuracy, ZS-Acc measures zero-shot accuracy, and R@k measures text and image retrieval recall.MST UT-Acc averages across sessions and is not directly comparable with OST UT-Acc.Source: A. Implementation Details and… · p. 7
  • Bwt measures average accuracy drop on earlier sessions after later-task fine-tuning, with smaller values indicating worse forgetting.It is evaluated only for MST.Source: A. Implementation Details and… · p. 7
  • A-Acc averages UT-Acc and ZS-Acc, and validation selects hyperparameters that maximize this measure.Compared methods include whole-model, image-only, and text-only updating variants of fine-tuning and continual-learning methods.Source: A. Implementation Details and… · p. 7

B. Experimental Results

The experiments compare CL-CLIP methods in one-session and multi-session training, examining updated-task performance alongside zero-shot and image-text matching retention. VR-LwF-WM has the best reported average accuracy in both settings, while fine-tuning and other methods show trade-offs between these outcomes.

CLIP suffers from catastrophic forgetting

  • Direct fine-tuning improves updated-task performance but sharply reduces zero-shot accuracy and image-text matching metrics.The drops occur for WM, IO, and TO variants in both OST and MST; small Bwt also indicates forgetting of previously learned classes after new sessions.Source: B. Experimental Results · pp. 7–8

Different updating options perform dissimilarly

  • TO is safer for zero-shot performance but limits updated-task accuracy, IO generally forgets most, and WM works best with continual-learning methods.FT-TO ZS-Acc falls from 62.62% to 56.44%/55.45% in OST/MST, while FT-WM performs best when paired with CL methods.Source: B. Experimental Results · p. 8

There are CL methods to alleviate the forgetting issue, but only for OST

  • Existing continual-learning methods may alleviate forgetting in OST but are weak in MST, where GeoDL and LwF lose ground to VR-LwF.GeoDL-WM has OST A-Acc of 67.40%, close to VR-LwF-WM's 67.43%, but its MST gap from VR-LwF widens.Source: B. Experimental Results · p. 8

CL-CLIP is different from traditional continual learning

  • CLIP's shared text-encoder classifier mapping lets later training improve earlier classes, but continual-learning methods can constrain updated-task gains.VR-LwF matches Original CLIP on ZS-Acc and leads UT-Acc among CL extensions, yet remains around 15% below Joint-FT on UT-Acc.Source: B. Experimental Results · pp. 8–9

RKR keeps the original performance best, but learns least

  • RKR preserves zero-shot performance best, but its updated-task performance is usually 10% lower than other methods.RKR keeps CLIP frozen and trains additional parameters; adding rectification causes it to forget zero-shot abilities.Source: B. Experimental Results · p. 9

VR-LwF achieves the overall best performance

C. Ablations

The ablations find limited benefit from replaying natural captions over random-word pseudo-sentences, little influence from replay settings, and similar retrieval-result distributions on Flickr and COCO.

C. Ablations

  • Caption replaying slightly improved OST ZS-Acc and retrieval recalls, but not significantly, and performed worse than random-word replay on MST; replay length had little influence.Longer replay length seemed better for MST, while OST zero-shot performance dropped slightly; the authors suggest added noise may explain the OST drop.Source: C. Ablations · p. 10
  • Flickr and COCO retrieval results have the same distribution, suggesting the evaluation transfers across datasets.Fig. 5 plots TR@1(%) horizontally and IR@1(%) vertically, with OST above and MST below.Source: Figure 5 · p. 11, C. Ablations · p. 11
  • Increasing the number of replayed sentences did not distinctly change results, but larger replay counts increased training time cost.Source: C. Ablations · p. 11

VII. CONCLUSION

The conclusion presents a benchmark and protocols for continual learning with large pre-trained CLIP, and reports that VR-LwF largely maintains zero-shot capability while benefiting from fine-tuning.

VII. CONCLUSION

  • The paper presents a continual-learning benchmark and protocols for large pre-trained CLIP, finding that CL-CLIP differs from traditional continual learning and parameter-update schemes strongly affect performance.Source: §VII · p. 11
  • Extensions of existing continual-learning methods alleviate forgetting but sacrifice fine-tuning performance.Source: §VII · p. 11
  • VR-LwF largely maintains CLIP's zero-shot capability while allowing the model to benefit from fine-tuning.Source: §VII · p. 11

Related papers

Loading 2207.09248v2…