PaperlayerView exact revision on arXiv

Source-linked AI summary

Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models

Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, Yang You

arXiv:2303.06628v2cs.CVcs.LG

TL;DR

Continual learning can update pretrained vision-language models without retraining, but catastrophic forgetting degrades their zero-shot transfer ability and replay cannot access private pre-training data. ZSCL combines reference-dataset distillation with weight averaging and evaluates the approach on conventional and multi-domain benchmarks. It outperforms prior methods, including by 9.7% on MTIL Avg. scores.

  • Problem

    Continual training of pretrained vision-language models forgets pre-training knowledge and zero-shot transfer ability, while replay methods cannot access private pre-training data.

  • Method

    ZSCL distills the initial model on a semantically diverse reference dataset and ensembles weights during training to limit feature and parameter shifts.

  • Results

    9.7% higher Avg. scores are reported for ZSCL than other methods on MTIL, alongside improvements across conventional and multi-domain benchmarks.

  • Takeaways & Limitations

    ZSCL protects zero-shot transfer ability while preserving learned knowledge during continual learning of vision-language models.

  • Takeaways & Limitations

    ZSCL requires a reference dataset, motivating future work on preserving zero-shot transfer without an outside dataset.

Abstract

from arXiv · show

Continual learning (CL) can help pre-trained vision-language models efficiently adapt to new or under-trained data distributions without re-training. Nevertheless, during the continual training of the Contrastive Language-Image Pre-training (CLIP) model, we observe that the model's zero-shot transfer ability significantly degrades due to catastrophic forgetting. Existing CL methods can mitigate forgetting by replaying previous data. However, since the CLIP dataset is private, replay methods cannot access the pre-training dataset. In addition, replaying data of previously learned downstream tasks can enhance their performance but comes at the cost of sacrificing zero-shot performance. To address this challenge, we propose a novel method ZSCL to prevent zero-shot transfer degradation in the continual learning of vision-language models in both feature and parameter space. In the feature space, a reference dataset is introduced for distillation between the current and initial models. The reference dataset should have semantic diversity but no need to be labeled, seen in pre-training, or matched image-text pairs. In parameter space, we prevent a large parameter shift by averaging weights during the training. We propose a more challenging Multi-domain Task Incremental Learning (MTIL) benchmark to evaluate different methods, where tasks are from various domains instead of class-separated in a single dataset. Our method outperforms other methods in the traditional class-incremental learning setting and the MTIL by 9.7% average score. Our code locates at https://github.com/Thunderbeee/ZSCL.

1. Introduction

Continual learning can efficiently update vision-language models, but catastrophic forgetting also degrades their pre-training knowledge and zero-shot transfer. ZSCL addresses this with feature-space distillation, parameter-space weight ensembling, and evaluation on the multi-domain MTIL benchmark.

  • Motivation: Continual learning reduces retraining costs by incrementally learning new data, but models tend to catastrophically forget previous information.This forgetting is a central challenge for continual learning.
  • Motivation: Vision-language continual learning additionally forgets pre-training knowledge, causing degradation in zero-shot transfer ability.Existing continual-learning methods hardly prevent this degradation, while private pre-training data limits replay-based approaches.
  • ZSCL: ZSCL mitigates forgetting through feature-space distillation and parameter-space weight ensembling.The method uses the original model as teacher, a semantically diverse reference dataset without labels or matched image-text pairs, and weight averaging across training.
  • Evaluation: MTIL evaluates continual-learning methods across tasks from distinct domains rather than class-separated tasks within one dataset.The benchmark comprises 11 tasks ranging from animal-species to aircraft-series recognition.
  • Results: 7.7% and 6.0% higher Last accuracy are reported on 10-step CIFAR100 and TinyImageNet continual learning, respectively.On MTIL, ZSCL exceeds other methods by 10.9% on Transfer and 9.7% on Avg. scores.

2. Related Work

Prior work studies vision-language models and continual learning through joint embedding, replay, distillation, and downstream adaptation. ZSCL targets a gap left by methods that do not address zero-shot transfer degradation in continual learning of pretrained vision-language models.

  • Vision-Language Models: Vision-language models learn joint image-text embedding spaces through contrastive pre-training on large-scale image-text datasets.CLIP contains image and text encoders and treats paired image-text examples as positive pairs.
  • Continual Learning Methods: Continual-learning methods include parameter expansion, memory replay, distillation loss, and parameter regularization.ZSCL does not change the CLIP architecture because it aims to produce a more powerful model after continual learning.
  • Vision-Language Models for Downstream Tasks: Downstream vision-language methods improve task performance through strategies such as prompt learning, adapters, and model interpolation.Prior work has made relatively few attempts at continual learning for pretrained vision-language models.
  • Vision-Language Models for Downstream Tasks: Existing studies do not address zero-shot transfer degradation, while LwF-VR considers only feature space and cannot protect the vision backbone with random-sentence distillation.ZSCL combines revised feature-space distillation with parameter-space weight ensembling.

3. Approach

ZSCL preserves a pre-trained vision-language model’s feature space through reference-data distillation and limits parameter drift through weight ensemble during continual training.

  • Continual learning setup: Continual learning trains tasks sequentially, aiming for good performance across all tasks while handling task-incremental and class-incremental inference settings.In task-incremental learning, task identity is provided; in class-incremental learning, predictions use the combined class set.
  • CLIP preliminaries: CLIP classifies images by comparing image embeddings with text embeddings and fine-tunes them using temperature-scaled cross-entropy loss.The text encoder converts class names into embeddings, while prediction selects the class with the largest cosine similarity.
  • Distillation in feature space: Direct fine-tuning distorts out-of-distribution feature distributions because downstream datasets occupy a small feature-space scope, degrading zero-shot prediction.ZSCL therefore seeks to preserve relative similarities across a broader feature space during adaptation.
  • Distillation in feature space: Feature-space distillation matches the current model’s image-text similarity distributions to those of the original model using a reference dataset with diverse semantics.The reference data need not be labeled, drawn from pre-training, or contain matched image-text pairs; the training objective combines cross-entropy with image- and text-side distillation losses.
  • Weight ensemble in parameter space: ZSCL’s weight ensemble averages model weights sampled throughout continuous training instead of interpolating only the initial and final models.This extends weighted averaging to continual learning to limit loss of knowledge from the original model.
  • Weight ensemble in parameter space: Weight ensemble achieves better downstream-task performance than WiSE-FT and is more robust to hyperparameter choices.WiSE-FT uses a weighted average between the original and fine-tuned models, but its tradeoff depends on α; weight ensemble instead samples weights every I iterations.

4. Multi-domain Task Incremental Learning

The paper introduces MTIL, a cross-domain task-incremental benchmark, and defines metrics for measuring continual learning, retention, and zero-shot transfer across tasks.

  • MTIL Benchmark: MTIL contains 11 tasks from different domains, totaling 1,201 classes and requiring domain-specific knowledge.The benchmark evaluates two task orders: alphabetic Order-I and random Order-II.
  • MTIL Benchmark: Unlike conventional benchmarks that separate classes within one dataset, MTIL uses tasks collected from different data sources and domains.Traditional benchmarks include datasets such as MNIST, CIFAR100, TinyImageNet, and ImageNet.
  • Evaluation Metrics: MTIL organizes evaluation as a matrix whose rows represent training steps and columns represent performance on individual datasets.The matrix supports comparisons across task learning stages and datasets.
  • Evaluation Metrics: Transfer measures average performance on tasks not yet fine-tuned, quantifying how much zero-shot transfer ability is preserved.Avg averages accuracy across all datasets and timestamps, while Last averages all task performances after continual learning.

5. Experiments

Experiments evaluate ZSCL on conventional and multi-domain continual-learning settings, ablate its feature- and parameter-space components, and show improved transfer, average, and last-task performance.

  • Feature-Space Ablations: Reference datasets with diverse semantics improve distillation, while unlabeled, unmatched, and previously unseen image-text pairs are sufficient.ImageNet and Conceptual Caption are identified as effective sources, and reducing image or class diversity degrades performance.
  • Parameter-Space Ablations: Using the initial CLIP as teacher improves zero-shot, Mean, and Last performance, while weight ensembling provides a better Transfer–Last tradeoff.ZSCL with WC exceeds ZSCL* by 2.8% Avg. and 5.9% Transfer scores.
  • Multi-domain Task Incremental Learning: 9.1% Transfer improvement and only a 1.3% drop from initial CLIP are reported for ZSCL on MTIL, alongside a 10.1% Avg. gain.Without WC, ZSCL* achieves the best Last scores, while ZSCL improves transfer and average accuracy.
  • Main Properties: t-SNE visualization shows ZSCL maintaining the pretrained feature distribution with a small averaged feature distance after MTCL training.Other methods significantly distort the feature space relative to original CLIP.
  • Multi-domain Task Incremental Learning: 9.2% Avg. and 18.1% Last improvements are reported for ZSCL in MTIL Order-II, with a 1.2% Transfer loss.The method's Last score remains similar across the two task orders, unlike previous methods.

6. Limitation and Future Work

The paper identifies reference-dataset dependence as a limitation and leaves several extensions for future work, including synthetic references, updating pre-training information, and broader multimodal models.

  • Limitation: A reference dataset is required, motivating future work on synthetic image datasets.The paper suggests generating synthetic reference images with methods that synthesize datasets from a network.
  • Motivation: Continual learning is positioned as an efficient way to update large models as re-training costs increase.
  • Future Work: Handling corrections to pre-training information or updates to outdated information with a reference dataset remains future work.
  • Future Work: Extending the approach to next-token prediction in large-language-model-based multimodal systems remains an open research direction.

7. Conclusion

The paper presents ZSCL for protecting zero-shot transfer during continual learning of vision-language models. It combines feature-space distillation and parameter-space weight ensembling, and reports state-of-the-art performance on conventional and MTIL benchmarks.

  • Contribution: ZSCL protects zero-shot transfer ability during continual learning of vision-language models.
  • Method: Feature-space distillation of the initial model on a reference dataset significantly boosts performance.
  • Method: Weight ensembling across training stages alleviates forgetting in parameter space.
  • Results: ZSCL achieves state-of-the-art performance on conventional and new MTIL benchmarks.

A. Additional Benchmark Description

The benchmark documentation provides detailed information about the datasets used in the benchmark.

  • Dataset Information: Table 8 gives detailed information about the different datasets in the benchmark.

B. Additional Implementation Details

The implementation uses distinct batch sizes for MTIL and class-incremental learning, searches learning rates over three values, and tunes label smoothing and weight decay choices.

  • Training Setup: The implementation uses batch size 64 for MTIL and 128 for class-incremental learning.
  • Hyperparameters: Learning rates are searched among 10^-5, 10^-6, and 10^-7.
  • Hyperparameters: For MTIL, CIFAR100, and TinyImageNet, weight decay 0 and label smoothing 0.2 are generally good choices.

C. Additional MTIL Results

The MTIL benchmark reports complete results as a t × t matrix, with three indicators used to make comparisons across methods easier.

  • MTIL results form a t × t performance matrix, summarized in the main text by Transfer, Avg., and Last indicators.The complete ZSCL and ZSCL∗ matrices are provided in Tables 11 and 10, respectively; Order-II results appear in Table 12.

D. Additional Conventional Class Incremental Learning Results

Conventional class-incremental experiments re-implemented prior methods with CLIP and evaluated ZSCL on ImageNet benchmarks. ZSCL achieved higher average and final accuracy than competing methods across the reported settings.

  • Experimental setup: Prior CL methods were re-implemented with the CLIP backbone, comparing previous-versus-initial teachers and alternative model-averaging choices.The initial teacher performed better for LwF-based methods, while WiSE-FT variants averaged previous/current or initial/current models.
  • ImageNet results: 1.97% higher Avg. and 1.30% higher Last scores were achieved by ZSCL than other methods on IN100-B10.
  • ImageNet results: 3.54% higher Avg. and 6.62 higher Last scores were achieved by ZSCL than other methods on IN100-B50.
  • Evaluation scope: The ImageNet comparison covers class-incremental settings with 10 splits and reports average and last accuracy values.
Loading 2303.06628v2…