PaperlayerView exact revision on arXiv

Source-linked AI summary

RanPAC: Random Projections and Pre-trained Models for Continual Learning

Mark D. McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, Anton van den Hengel

arXiv:2307.02251v3cs.LGcs.CV

TL;DR

Continual learning must handle changing tasks while retaining prior performance, but pre-trained-model approaches face distribution gaps and adaptor forgetting. RanPAC uses frozen nonlinear random projections, prototype accumulation, and decorrelation to avoid parameter-update forgetting, reporting strong rehearsal-free results across continual-learning settings.

  • Problem

    Pre-trained-model continual learning still lacks a clear strategy that jointly offers strong performance, simplicity, and efficiency across diverse datasets and scenarios.

  • Method

    RanPAC combines class-prototype accumulation with a frozen nonlinear Random Projection layer and prototype decorrelation, avoiding trainable parameter updates during continual learning.

  • Results

    RanPAC reports the highest-to-date rehearsal-free continual-learning accuracies on the tested class- and domain-incremental datasets, with large margins over prior class-prototype strategies.

  • Takeaways & Limitations

    Pre-trained ViT-B/16 representations can support simple, rehearsal-free continual learning without risking forgetting in the pre-trained model.

  • Takeaways & Limitations

    RanPAC relies on a good generic feature extractor and is unlikely to be as powerful for continual-learning methods that train networks from scratch.

Abstract

from arXiv · show

Continual learning (CL) aims to incrementally learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. Most CL works focus on tackling catastrophic forgetting under a learning-from-scratch paradigm. However, with the increasing prominence of foundation models, pre-trained models equipped with informative representations have become available for various downstream requirements. Several CL methods based on pre-trained models have been explored, either utilizing pre-extracted features directly (which makes bridging distribution gaps challenging) or incorporating adaptors (which may be subject to forgetting). In this paper, we propose a concise and effective approach for CL with pre-trained models. Given that forgetting occurs during parameter updating, we contemplate an alternative approach that exploits training-free random projectors and class-prototype accumulation, which thus bypasses the issue. Specifically, we inject a frozen Random Projection layer with nonlinear activation between the pre-trained model's feature representations and output head, which captures interactions between features with expanded dimensionality, providing enhanced linear separability for class-prototype-based CL. We also demonstrate the importance of decorrelating the class-prototypes to reduce the distribution disparity when using pre-trained representations. These techniques prove to be effective and circumvent the problem of forgetting for both class- and domain-incremental continual learning. Compared to previous methods applied to pre-trained ViT-B/16 models, we reduce final error rates by between 20% and 62% on seven class-incremental benchmarks, despite not using any rehearsal memory. We conclude that the full potential of pre-trained models for simple, effective, and fast CL has not hitherto been fully tapped. Code is at github.com/RanPAC/RanPAC.

1 Introduction

Continual learning must adapt to changing tasks while preserving prior performance, but conventional updating risks catastrophic forgetting. RanPAC addresses this with frozen random projections, prototype accumulation, and decorrelation for rehearsal-free CL with pre-trained models.

  • Continual learning models must handle changing data or supervision while remaining performant on previously learned tasks.
  • Most existing CL methods train from scratch, while pre-trained models offer strong generic feature extractors for downstream adaptation.
  • Pre-trained-model CL strategies include prompting, selective fine-tuning, and class-prototype accumulation, generally without rehearsal buffers.
  • The best strategy for leveraging pre-trained models remains open across performance, simplicity, and efficiency, with fine-tuning carrying greater resource demands and forgetting risk.
  • RanPAC enhances class-prototype CL using a frozen nonlinear random-projection layer and decorrelation, bypassing forgetting during parameter updates.
  • RanPAC reports the highest-to-date rehearsal-free accuracies on tested class- and domain-incremental datasets, with large margins over prior class-prototype strategies.

2 Related Work

Prior pre-trained-model CL methods use prompting, selective fine-tuning, or class prototypes. RanPAC builds on prototype methods and frozen nonlinear random projections, whose feature interactions can improve linear separability.

  • Prompt-based methods train learnable prompt pools for transformer networks and have progressively improved performance.
  • Class prototypes suit CL because frozen features make accumulated prototypes independent of task order while requiring less memory than rehearsal buffers.
  • Frozen random projections avoid training their weights, support exact numerical learning for a linear output layer, and may create more linearly separable interactions.
  • RanPAC applies nonlinear activation to random projections because nonlinear random interactions may be more linearly separable than original features.

3 Background

The paper considers supervised class- and domain-incremental learning with inaccessible past data, using frozen pre-trained features, class prototypes, and incrementally updated second-order statistics.

  • 3.1 Continual learning problem setup: The supervised CL setup consists of T sequential stages whose disjoint training data cannot be revisited; class-incremental stages have disjoint labels, whereas domain-incremental stages share classes but shift distributions.
  • Features are extracted from a frozen pre-trained model, and labels are represented as length-K one-hot vectors across N training samples.
  • Nearest Class Mean classifiers average extracted features into class prototypes, but cross-entropy probing or fine-tuning can create recency bias and catastrophic forgetting.
  • Beyond cosine similarity, the method uses second-order feature statistics through incrementally updated Gram matrices and class-prototype quantities.
  • Gram-matrix calibration uses training-set statistics to compare features and prototypes, and these quantities can be updated progressively task by task.

4 The proposed approach and theoretical insights: RanPAC

RanPAC combines frozen nonlinear random projections with class-prototype accumulation and decorrelation to improve class separability for continual learning with pre-trained features. The approach uses streaming Gram-matrix and prototype updates, avoiding parameter updating while enhancing the representation used for classification.

  • Second-order statistics: Raw class-prototypes can be highly correlated across classes, producing poorly calibrated cosine similarities; decorrelation improves class separability.Compared with NCM, LDA or Eqn. (2) mostly removes correlations between class-prototypes.
  • Empirical intuition: On Split ImageNet-R, nonlinear projections with M = 2000 reduce histogram overlap, shift in-class similarities rightward, and coincide with accuracy surpassing the joint linear probe.Without nonlinearity, increasing M does not improve accuracy over omitting random projection, whereas nonlinear activations improve performance as M increases.
  • Random projections: Random projections map frozen pre-trained features into an expanded M-dimensional space, where increasing M makes projected norms and pairwise angles more distinctive.The projected distribution approaches an isotropic Gaussian as M increases, while distinct angles become more likely.
  • Nonlinearity: Element-wise nonlinear activation adds feature interactions and can simulate much higher-dimensional projections without exhaustively constructing all interaction terms.Random projection provides a computationally cheap alternative to explicitly forming flattened feature cross-products.
  • Continual-learning algorithm: For each training feature, RanPAC computes a nonlinear projected representation, accumulates its Gram matrix and class-prototype matrix incrementally, and forms a regularized closed-form classifier.The random projections are sampled once and frozen throughout all continual-learning stages; ridge regression uses (G + λI)^−1.

5 Experiments

Experiments evaluate RanPAC on class- and domain-incremental benchmarks using pre-trained ViT-B/16 models, standard average-accuracy metrics, and no rehearsal buffer. RanPAC consistently outperforms prompting and competing class-prototype methods, with strong gains from random projections.

  • Experimental setup: Experiments use pre-trained ViT-B/16 models on class- and domain-incremental benchmarks without rehearsal buffers.The evaluation reports final accuracy using the standard Average Accuracy metric and also examines ResNet and CLIP backbones in the appendices.
  • Class-incremental learning: 11%–28% error-rate reductions result from adding the random-projection layer when PETL is used, with gains of at least 8% otherwise except VTAB.These are ablation results for the class-incremental benchmarks in Table 1.
  • Experimental setup: Table 1 compares prompting and class-prototype strategies using Relative Error Rate and accuracy, with no rehearsal buffer for any listed method.The table includes ablations for NCM, random projections, and Phase 1.
  • Class-incremental learning: RanPAC surpasses the jointly trained linear probe on all seven class-incremental datasets.It also reaches within 2% raw accuracy of the best joint fine-tuning accuracy on three datasets, with larger gaps on two ImageNet variants and Cars.
  • Domain-incremental learning: RanPAC outperforms prompting strategies and highlights the value of random projections across domain-incremental datasets.The random-projection benefit is particularly strong for DomainNet, while PETL adds little value there, consistent with stages originating in different domains.

6 Conclusion

The paper concludes that RanPAC is a simple, rehearsal-free class-prototype strategy that substantially improves continual learning with pre-trained representations. Its effectiveness depends on access to a good generic feature extractor and trades additional parameters for simple, low-cost training.

  • Conclusion: RanPAC significantly reduces error rates on diverse continual-learning benchmarks without risking forgetting in the pre-trained model.The conclusion covers both class- and domain-incremental continual learning.
  • Limitations: RanPAC’s equations rely completely on a good generic feature extractor and are unlikely to be as powerful for networks trained from scratch.The authors suggest similar approaches might be useful with self-supervised or otherwise strong feature-extractor backbones.
  • Limitations: RanPAC uses more parameters than methods such as L2P, trading this cost for implementation simplicity and low-cost training.This is presented as an author-supported limitation and trade-off.
  • Future work: Future work includes task-agnostic continual learning, non-one-hot targets such as language embeddings, and regression through generic feature prototypes.The paper identifies substantial room for exploration in each direction.

Appendices

The appendices provide an overview of RanPAC, comparisons with alternative pre-trained-model strategies, and theoretical support for random projection and prototype decorrelation. They explain how higher-dimensional projections affect norms and inner products and how the output weights arise from regularized least squares.

  • Overview: RanPAC has two phases: optional first-session PETL training followed by frozen feature extraction, random projection, prototype accumulation, and output-weight computation.The projected features are h = ϕ(f ⊤W), and the output weights are computed after each task using the Gram matrix and matrix inversion.
  • Strategy comparison: RanPAC is categorized as a class-prototype strategy that adds a frozen nonlinear random-projection layer and uses second-order statistics to decorrelate prototypes.The appendix compares it with prompting and fine-tuning and notes that rehearsal buffers could potentially boost methods including RanPAC.
  • Random-projection theory: As projection dimension M increases, projected-vector norms are more likely to lie near their expected values and the projected distribution becomes a better Gaussian fit.This follows the appendix’s interpretation of the Chernoff bound.
  • Random-projection theory: Increasing M makes inner products between projected instances more likely to be distinct, supporting richer separation in the projected space.The appendix derives this behavior from the Gram-matrix expansion and Chernoff bounds.
  • Output weights: The score computation uses ypred = htestWo, where Wo = (G + λI)−1C is the closed-form l2-regularized least-squares solution.The formulation avoids bias calculations and updates the Gram matrix rather than feature-mean outer products.

B.4 Connection to Linear Discriminant Analysis, Mahalanobis distance and ZCA whitening

The section relates RanPAC’s decorrelated class-prototype classifier to LDA, Mahalanobis distance, and ZCA whitening, while distinguishing their underlying statistics and transforms.

  • RanPAC and LDA: RanPAC uses an inverted Gram matrix because its formulation is mean-square-error optimal and supports simpler, more efficient inference and continual accumulation.Inference avoids prototype-derived biases, while Gram-matrix and prototype updates are more efficient than covariance-matrix updates.
  • RanPAC and LDA: The classifier is equivalent to a linear classifier trained with mean square error and l2 regularization, with class-prototypes and second-order statistics accumulated across tasks.The equivalence applies in both continual-learning and non-continual settings.
  • Comparisons with related transforms: Cosine-similarity NCM uses first-order class means, whereas LDA and RanPAC use second-order statistics describing feature correlations.This distinguishes prototype averaging from correlation-aware classification.
  • Comparisons with related transforms: LDA matches RanPAC only when classes are equiprobable and G = S, which requires zero means for all features and is not generally true.LDA instead uses the covariance matrix S and class frequencies πy.
  • Comparisons with related transforms: With equiprobable classes, LDA minimizes Mahalanobis distance and is equivalent to minimizing Euclidean distance after ZCA-whitening test samples and class-prototypes.ZCA whitening uses the Mahalanobis transform DZ = S−0.5.
  • RanPAC and LDA: RanPAC decorrelates randomly projected features by choosing D so that the Gram matrix G = HH⊤ becomes the identity, yielding D⊤D = G−1.The resulting similarities match those from the minimum mean square error formulation.

C.2 Training details

Training uses frozen pre-trained features for the RanPAC classifier and optional SGD-based PETL or fine-tuning configurations, with reported efficiency close to inference-only processing.

  • RanPAC training: Phase 2 extracts features from a frozen pre-trained model, updates G and C, and computes Wo by matrix inversion and multiplication without SGD weight updates.The classifier is therefore trained through closed-form matrix operations rather than iterative backbone optimization.
  • PETL and baseline training: Phase 1 trains AdaptFormer, SSF, and VPT parameters with SGD, generally for 20 epochs, using specified batch size, learning rate, momentum, weight decay, and cosine annealing.These PETL settings apply when the optional first phase is used.
  • PETL and baseline training: Linear probes use 30 epochs, while full fine-tuning additionally applies a 0.0001 learning rate to the ViT backbone.Both configurations use batch size 128, learning rate 0.01 in the classification head, weight decay 0.0005, and momentum 0.9.
  • Data processing: Training augmentation uses random resized crops to 224×224 and random horizontal flips; inference uses resizing to short side 256 followed by center cropping.CIFAR100 is instead resized directly from 32×32 to 224×224 for inference.
  • Evaluation settings: The main experiments use seed 1993 to reproduce the comparison method’s results, while seeds 1994 and 1995 assess variability.RanPAC reports Average Accuracy as its primary metric, whereas the comparison repository calculates overall accuracy after each task.
  • Efficiency: Inference speed is negligibly different from the original pre-trained network, and Phase 2 is only slightly slower than passing all training data through the frozen backbone.For M = 10000, Gram-matrix inversions during λ selection take about 1 minute per task on a CPU.

Appendix D Parameter-Efficient Transfer Learning (PETL) methods

The PETL comparison evaluates AdaptFormer, SSF, and VPT under the same experimental framework, using specified architectural settings for VPT and AdaptFormer.

  • PETL methods: The experiments use AdaptFormer, SSF, and VPT; VPT uses the deep version with prompt length 5, and AdaptFormer uses projected dimension 64.These are the same three methods used in the referenced comparison.

E.1 Class Incremental Learning (CIL) Datasets

The evaluation covers seven class-incremental datasets with task schedules tailored to their class and domain structure, alongside defined accuracy and forgetting metrics.

  • Datasets: The CIL evaluation includes ImageNet-A, CUB, Omnibenchmark, VTAB, ImageNet-R, CIFAR100, and Stanford Cars.Dataset sources and split definitions follow the stated references and repositories.
  • Task configurations: Stanford Cars uses 16 classes in its first task and 20 classes in each of the following nine tasks when T = 10.Other CIL datasets are evaluated with T = 5, T = 10, and T = 20, while VTAB uses T = 5.
  • Task configurations: VTAB combines class- and domain-incremental characteristics by introducing disjoint class sets from different domains across five tasks.This structure distinguishes it from the three domain-incremental datasets described in the evaluation.
  • Dataset metadata: Table A2 summarizes CIL dataset references, sample counts, class counts, and validation-set sizes.Its headers define N as total training samples, K as total classes after all tasks, and # val samples as standard validation samples.
  • Metrics: Average Accuracy and Average Forgetting are the reported evaluation measures, with Rt,i denoting accuracy on task i after training on task t.For CIL, Rt,i is computed on the subset of classes belonging to Di.
  • Metrics: For DIL, metric computation depends on dataset conventions; CORe50 evaluates the entire validation set, making Rt,i constant across i.CDDB-Hard and DomainNet use the same validation treatment for the reported Table 3 results.

F.2 Variability and performance on each task

RanPAC’s final accuracy is largely stable across random task assignments without Phase 1, while Phase 1 introduces more variability but retains clear random-projection benefits. Performance also depends on task structure, domain, and projection dimension.

  • CIL variability: Without Phase 1, final Average Accuracy varies negligibly across random seeds and class-to-task assignments.This follows from the order-invariance of accumulated prototypes after all task data have been used.
  • CIL variability: With Phase 1, stochastic PETL training increases final Average Accuracy variability, although random projections remain beneficial.
  • DIL performance: CDDB-Hard shows domain-specific forgetting, and mean-over-domain accuracy can be substantially below overall accuracy.For this binary task, averaging domains may be misleading because validation-domain sizes differ and some domain accuracies remain poor.
  • DIL performance: DomainNet generally improves on each domain as new tasks arrive, suggesting its domains share feature representations that benefit from additional training data.
  • Task-count effects: For AdaptMLP, T = 5 tends to outperform T = 20, while T = 20 is often comparable to using no PETL.The difference is consistent with greater class diversity during first-session training when T = 5.
  • Task-agnostic CL: Task-agnostic CIFAR100 uses 200 micro-tasks with gradually shifting class subsets, requiring choices for first-session scope and regularization.The protocol has no clear task boundaries, unlike standard class-incremental learning.
  • Projection scaling: For split CIFAR100, M must exceed 1250 to surpass accuracy without random projections.

F.6 Comparison of PETL Methods and ViT-B/16 backbones

PETL and ViT-B/16 backbone choices materially affect RanPAC performance across datasets, so neither is uniformly optimal. The approach also extends beyond ViTs to ResNet and CLIP backbones, with CLIP-language targets improving CIFAR100 over zero-shot prediction.

  • PETL methods: PETL performance is dataset-dependent: AdaptMLP is best for CIFAR100, VPT for Cars, and SSF for ImageNet-A.
  • PETL methods: VTAB is an outlier where VPT with the ImageNet-21K backbone fails badly.
  • ViT-B/16 backbones: Backbone choice is dataset- and method-dependent: ImageNet-1K is best for ImageNet-A, ImageNet-21K for CIFAR100 and OmniBenchmark, and Cars varies by PETL method.
  • Alternative backbones: RanPAC applies to pre-trained ResNet50 and ResNet152 backbones, with NCM accuracies comparable to prior ResNet results under different task counts.
  • CLIP backbone: For CLIP vision backbones, random projections generally improve CIL accuracy over NCM alone, while DIL gains over ViT-B/16 Phase 2 occur only on CDDB-Hard and DomainNet.
  • CLIP regression targets: Using CLIP language representations as regression targets yields 77.5% final Average Accuracy at M = 5000 and T = 10, versus 68.6% zero-shot and 71.4% without RP.
  • Reproducibility: Accuracy can vary by approximately ±1% across random seeds, especially for PETL methods.The reported tables were updated to match results published in the code repository.
Loading 2307.02251v3…