PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning without Forgetting for Vision-Language Models

Da-Wei Zhou, Yuanhan Zhang, Yan Wang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, Ziwei Liu

arXiv:2305.19270v2cs.CVcs.LG

TL;DR

Continual VLM learning must absorb new classes without forgetting prior knowledge, while making use of cross-modal information. PROOF freezes pretrained encoders, expands task-specific projections, and fuses visual and textual features; it reports state-of-the-art performance across nine benchmark datasets and a non-overlapping dataset.

  • Problem

    VLMs can forget former knowledge when continually trained on new classes, and relying only on text for classification neglects cross-modal features.

    Source: §1 · p. 1

  • Method

    PROOF freezes pretrained image and text encoders, expands task-specific projections while fixing prior ones, and uses fusion to adjust visual and textual features jointly.

    Source: §1 · p. 2

  • Results

    PROOF achieves state-of-the-art performance on nine benchmark datasets and a non-overlapping dataset, and shows strong performance in continual cross-modal retrieval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The proposed expandable projections and cross-modality fusion support incremental learning while preserving previous concepts across the evaluated settings.

    Source: §6 · p. 13

    The CIL setup assumes a pretrained vision-language model is available to initialize the classifier.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Class-Incremental Learning (CIL) or continual learning is a desired capability in the real world, which requires a learning system to adapt to new tasks without forgetting former ones. While traditional CIL methods focus on visual information to grasp core features, recent advances in Vision-Language Models (VLM) have shown promising capabilities in learning generalizable representations with the aid of textual information. However, when continually trained with new classes, VLMs often suffer from catastrophic forgetting of former knowledge. Applying VLMs to CIL poses two major challenges: 1) how to adapt the model without forgetting; and 2) how to make full use of the multi-modal information. To this end, we propose PROjectiOn Fusion (PROOF) that enables VLMs to learn without forgetting. To handle the first challenge, we propose training task-specific projections based on the frozen image/text encoders. When facing new tasks, new projections are expanded and former projections are fixed, alleviating the forgetting of old concepts. For the second challenge, we propose the fusion module to better utilize the cross-modality information. By jointly adjusting visual and textual features, the model can capture semantic information with stronger representation ability. Extensive experiments on nine benchmark datasets validate PROOF achieves state-of-the-art performance. Code is available at https://github.com/zhoudw-zdw/PROOF

1 INTRODUCTION

VLMs offer textual information for generalizable representations, but continual adaptation must preserve prior knowledge while using cross-modal features. PROOF freezes pretrained encoders, expands task-specific projections, and fuses modalities; it reports state-of-the-art performance on nine benchmarks and a non-overlapping dataset.

1 INTRODUCTION

  • Sequentially tuning VLMs can overwrite generalizability and former concepts, while text-only classification neglects cross-modal features.The introduction identifies avoiding forgetting and making full use of multimodal information as the two challenges.Source: §1 · p. 1
  • The paper frames its goal as making VLMs both retentive, preserving pretrained capabilities, and comprehensive, integrating information across modalities.These capabilities are intended to support incremental updates as data evolves.Source: §1 · p. 1
  • PROOF freezes pretrained image and text encoders and adds task-specific projections, expanding new projections while fixing old ones to preserve former knowledge.The projections encode task-specific information; the approach is intended to alleviate forgetting of old concepts.Source: §1 · pp. 1–2
  • PROOF uses cross-modal fusion to encode task-specific information and adjust query embeddings with context from different modalities.Its aggregated inference format considers cross-modal matching targets holistically.Source: §1 · p. 2
  • PROOF achieves state-of-the-art performance on nine benchmark datasets and a non-overlapping dataset, with strong performance in continual cross-modal retrieval.The introduction also reports comparisons against other cutting-edge methods for retrieval.Source: §1 · p. 2

2 RELATED WORK

Prior VLM tuning methods adapt models to downstream tasks, while CIL methods address forgetting through several strategy families. Existing pretrained-model CIL methods largely target vision-only models, leaving VLM class-incremental learning relatively unexplored.

2.1 Vision-Language Model (VLM) Tuning

  • VLM tuning methods use prompts, adapters, residuals, prototypes, and cross-modal matching, but generally overlook catastrophic forgetting across sequential downstream tasks.The related-work discussion says these approaches adapt VLMs to downstream tasks without addressing prior tasks.Source: §2.1 · p. 2

2.2 Class-Incremental Learning (CIL)

  • CIL methods address forgetting through replay, distillation, parameter regularization, bias correction, and dynamic network expansion.Replay may use exemplars, features, low-resolution images, logits, or generative models; distillation may regularize logits, features, or relations.Source: §2.2 · p. 2

2.3 CIL with VLM

  • Pretrained ViT CIL methods use prompts and related strategies, but are not designed for multimodal VLMs; VLM class-incremental learning remains relatively unexplored.S-Prompts addresses CLIP domain-incremental learning, while WiSE-FT cannot be extended to multiple tasks.Source: §2.3 · pp. 2–3

3 PRELIMINARIES

The preliminaries establish the incremental-learning setting, describe CLIP’s shared image-text embedding space, and contrast vision-based forgetting remedies with VLM tuning. They motivate combining cross-modal information with resistance to forgetting.

3.1 Class-Incremental Learning

  • CIL learns from a stream of disjoint class sets and builds one classifier for all seen classes, using current-task data and a fixed exemplar set from former classes.The paper assumes a pretrained vision-language model initializes the classifier.Source: §3.1 · p. 3

3.2 Vision-Language Model

  • CLIP jointly trains image and text encoders to map paired inputs into a shared embedding space, enabling a zero-shot classifier from templated class texts.Classification compares an image embedding with text embeddings for the candidate classes.Source: §3.2 · p. 3

3.3 Overcome Forgetting in Class-Incremental Learning

  • Vision-based prompt tuning freezes the image encoder and limits forgetting, whereas sequentially tuning CoOp’s shared prompt suffers catastrophic forgetting.The discussion contrasts these methods: vision-based approaches neglect text semantics, while transfer methods leverage cross-modal information but forget former concepts when tuned sequentially.Source: §3.3 · p. 3

4 PROOF: PROJECTION FUSION FOR VLM

PROOF adapts frozen VLM encoders with expandable task-specific projections and a cross-modal fusion module. It combines projected features with contextual information to support continual learning while preserving earlier projections.

4.2 Contextualizing Projections with Projection Fusion

  • The adapted query embedding is separately matched against visual prototypes and textual features to produce visual and textual predictions.The context prompt contributes task-specific information to self-attention but is not itself a matching target.Source: Figure 1 · p. 4, §4.2 · p. 5

4.1 Expandable Feature Projection

  • New task-specific projections learn downstream concepts while frozen encoders and earlier projections preserve prior knowledge.Projected features from different stages are aggregated, and projection parameters are described as negligible relative to the pretrained model.Source: §4.1 · p. 4

4.2 Contextualizing Projections with Projection Fusion

  • Self-attention jointly adapts the query embedding and context, which includes visual prototypes, textual classifiers, and expandable learned prompts.Visual prototypes are computed from frozen visual embeddings and remain fixed in subsequent tasks; earlier prompts are frozen as new prompts are initialized.Source: §4.2 · p. 5

4.3 Summary of PROOF

  • PROOF uses projected mapping, freezes former projections and prompts as new ones are initialized, and aggregates projected, visual, and textual matching logits.Training uses current-task data together with an exemplar set, while pretrained weights remain frozen.Source: §4.3 · pp. 5–6

5 EXPERIMENT

The experiments compare PROOF with established methods and examine its components and scope across settings. They also investigate performance on non-overlapping data and zero-shot degradation.

5 EXPERIMENT

  • The section compares PROOF with state-of-the-art methods on benchmark datasets to assess its ability to overcome forgetting.It also conducts ablations to analyze the effect of each model component.Source: §5 · p. 6
  • The experiments extend PROOF to other vision-language models and continual-learning scenarios.Source: §5 · p. 6
  • The section also evaluates a non-overlapping dataset and addresses zero-shot performance degradation.Source: §5 · p. 6

5.1 Experimental Setup

The setup evaluates PROOF across nine class-incremental benchmarks with shared class-order and training conventions. Comparisons use common CLIP backbones and include continual-learning, tuning, zero-shot, and joint-training baselines.

5.1 Experimental Setup

  • Nine benchmarks use 100 classes for CIFAR100, Aircraft, Cars, Food, and UCF101; 200 for CUB200, ObjectNet, and ImageNet-R; and 300 for SUN397.Training class order is shuffled with random seed 1993, and splits are denoted Base-x, Inc-y.Source: §5.1 · p. 6
  • Comparisons include CIL methods, sequential fine-tuning with LiT, PLOT, and CoOp, zero-shot CLIP, and joint-training Oracle.All methods use the same pretrained CLIP for fair comparison.Source: §5.1 · p. 6
  • Experiments use CLIP ViT-B/16 with OpenAI or OpenCLIP LAION-400M weights, and report last-stage accuracy AB and average performance across stages.Training uses batch size 64 for five epochs, SGD with momentum, and 20 herding-selected exemplars per class.Source: §5.1 · p. 6

5.2 Benchmark Comparison

PROOF leads the compared methods across the reported benchmarks and maintains top performance through incremental stages. Its results also show advantages over CoOp at the first stage and over vision-only methods.

5.2 Benchmark Comparison

  • PROOF consistently achieves the best performance among compared methods across nine benchmarks and both large- and small-base-class splits.The reported comparison uses CLIP with ViT-B/16 (OpenCLIP LAION-400M); the authors note that several prompt-based results were reproduced using CLIP’s visual branch.Source: §5.2 · p. 6
  • PROOF surpasses CoOp in first-stage performance and substantially improves on vision-only methods including L2P, DualPrompt, CODA-Prompt, and DAP.The authors attribute the latter comparison to co-adapting textual and visual information for incremental learning.Source: §5.2 · p. 6
  • PROOF’s performance curve consistently ranks at the top across methods, demonstrating its ability to resist forgetting.Source: §5.2 · p. 6

5.3 Ablation Study

The ablations report strong performance with OpenAI weights and examine PROOF’s components, context choices, prompt length, projection layers, parameter budget, and varying task sizes. Results favor combining projections and fusion, using multimodal context with prompts, and keeping context prompts small.

5.3.1 Different backbone weights

  • With OpenAI weights, PROOF performs best on all datasets among the compared methods.The table compares average and last performance, with and without exemplars, using the same pretrained CLIP initialization.Source: §5.3.1 · p. 6, Table 1 · p. 7

5.3.2 Compositional components

  • Combining expandable projections and cross-modal fusion outperforms either component alone, while adding context prompts yields the best performance among the tested variations.The study evaluates PROOF and its submodules on CIFAR100 B0 Inc10.Source: §5.3.2 · pp. 7–8, Figure 4 · p. 10

5.3.3 Number of context prompts

  • PROOF’s performance is robust to context-prompt length, supporting a default length of c = 3.The experiments vary c from 1 to 10, 30, 50, and 100 while retaining one context prompt per incremental task; the authors conclude that a small scale suffices for task-specific information.Source: §5.3.3 · p. 8, Figure 4 · p. 10

5.3.4 Variation of context information

  • Using visual and textual prototypes together improves performance over either alone, and adding context prompts achieves the best result among the tested context choices.The context variations are evaluated on CIFAR100 B0 Inc10.Source: §5.3.4 · pp. 8–9, Figure 5 · p. 10

5.3.5 Variation of projection types

  • A single linear layer achieves the best performance among the tested projection types, including SSF and Adapter.The authors suggest that a simple linear mapping can effectively bridge visual and textual domains.Source: §5.3.5 · p. 9, Figure 5 · p. 10

5.3.6 Parameter analysis

  • After merging projections, PROOF’s extra parameters fall from (2b+3)×d^2+B×d to 5×d^2+B×d, with a parameter scale similar to zero-shot CLIP.The authors state that the unmerged extra parameters are negligible relative to CLIP’s approximately 150 million parameters; projection merging combines the projections by summing weights.Source: §5.3.6 · pp. 9–11
  • PROOF achieves the best performance across the compared continual cross-modal retrieval tasks and metrics.The evaluation reports average and last performance for text retrieval and image retrieval.Source: Table 3 · p. 11, Figure 6 · p. 11

5.3.7 Different number of classes per task

  • PROOF shows substantial improvements when the number of classes per task varies in randomly sampled CIFAR100 and ImageNet-R task sequences.The compared methods use the same CLIP weight and number of exemplars.Source: Table 2 · p. 10, §5.3.7 · p. 11

5.4 Extending PROOF to other VLMs and other applications

PROOF extends beyond CLIP classification: with BEiT-3, it uses projection expansion for incremental cross-modal retrieval and performs best among the compared methods across retrieval tasks and metrics.

5.4 Extending PROOF to other VLMs and other applications

  • For BEiT-3 retrieval, PROOF uses projection expansion without cross-modal fusion because retrieval differs from classification.BEiT-3 functions as a dual encoder, like CLIP.Source: §5.4 · p. 11
  • Flickr30K is divided into five incremental subsets using action keywords, with a 5:1 training-to-testing ratio for pairs.The dataset provides five manually annotated descriptions for each image.Source: §5.4 · p. 11
  • PROOF consistently achieves the best performance across the compared cross-modal retrieval tasks and metrics, while incremental learning alleviates the forgetting caused by finetuning.The evaluation uses BEiT-3 and compares against finetuning, modified MEMO, and modified DER.Source: §5.4 · p. 11

5.5 CIL with non-overlapping dataset

The authors construct TV100 from post-2021 TV series to evaluate continual learning on classes absent from CLIP pretraining. PROOF outperforms competitors by a substantial margin on this non-overlapping dataset.

5.5 CIL with non-overlapping dataset

  • The dataset is designed to avoid overlap with CLIP pretraining by selecting series released after 2021 and excluding series with earlier seasons.The authors note some post-2021 shows may still feature concepts already known to CLIP, motivating hard-class selection.Source: §5.5 · p. 12
  • TV100 selects 100 hard classes from roughly 800 collected post-2021 TV-series classes, with pretrained CLIP achieving around 10% zero-shot accuracy on the selected dataset.Class difficulty is ranked using image zero-shot accuracy against the prompt “a photo of the TV series [CLASS].”Source: §5.5 · p. 12
  • PROOF still outperforms other competitors by a substantial margin, verifying its strong performance in real-world continual learning tasks.Experiments select Base0 Inc10 and Base50 Inc10 dataset splits.Source: §5.5 · p. 12

5.6 Exploring Zero-Shot Performance

The section evaluates zero-shot performance on unseen classes and upstream image-text matching as CLIP is adapted through CIL. PROOF† retains competitive zero-shot performance, trading some downstream adaptivity for a balance between seen and unseen classes.

5.6 Exploring Zero-Shot Performance

  • The evaluation measures seen-class accuracy, unseen-class accuracy, their harmonic mean, and average image-text matching score on 10,000 LAION-400M pairs across incremental stages.Unseen-class accuracy assesses zero-shot performance, while the LAION score indicates matching on upstream-task pairs.Source: §5.6 · p. 12
  • PROOF† uses a residual projection output that incorporates pretrained information into projected features, which the authors say helps maintain generalizability.The variation differs from PROOF in projection design.Source: §5.6 · pp. 12–13
  • PROOF† maintains competitive accuracy on unseen classes and a LAION score similar to zero-shot CLIP, while sacrificing adaptivity relative to PROOF.At the last incremental stage, PROOF† strikes a balance between seen- and unseen-class performance.Source: §5.6 · pp. 12–13

6 CONCLUSION

PROOF incrementally aligns visual and textual information through expandable projections and self-attention fusion, and experiments validate it across VLMs and continual-learning scenarios.

6 CONCLUSION

  • PROOF learns expandable projections to align visual and textual information incrementally, integrating new concepts without compromising previous ones.It also applies self-attention fusion to jointly adapt visual and textual information into instance-specific embeddings.Source: §6 · p. 13
  • Experiments validate PROOF across various VLMs and continual-learning scenarios, and a simple variation preserves zero-shot capability.Extending the model to exemplar-free scenarios is left for future work.Source: §6 · p. 13
Loading 2305.19270v2…