PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Models are Continual Learners

Enrico Fini, Victor G. Turrisi da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, Julien Mairal

arXiv:2112.04215v2cs.CVcs.LG

TL;DR

Self-supervised models suffer severe degradation when unlabeled data arrives sequentially, because continual learning must preserve prior knowledge without revisiting old data. The paper introduces CaSSLe, which uses a predictor and the original self-supervised loss for distillation, and reports substantially better representations across continual-learning settings. The framework is compatible with several self-supervised methods and requires little additional tuning, though it assumes known task boundaries and adds computational cost.

  • Problem

    Self-supervised models lose efficacy in continual learning, where new unlabeled data arrives sequentially and old data may be unavailable for retraining.

  • Method

    CaSSLe adds a predictor that maps current representations to past representations and reuses the self-supervised loss as a distillation objective.

  • Results

    CaSSLe significantly outperforms continual self-supervised and adapted supervised continual-learning baselines, producing better representations and stronger forward transfer.

  • Takeaways & Limitations

    The framework enables continual training of six popular self-supervised models across class-, data-, and domain-incremental settings with little to no additional hyperparameter tuning.

  • Takeaways & Limitations

    CaSSLe assumes task boundaries are provided and increases training memory and time by roughly 30%.

Abstract

from arXiv · show

Self-supervised models have been shown to produce comparable or better visual representations than their supervised counterparts when trained offline on unlabeled data at scale. However, their efficacy is catastrophically reduced in a Continual Learning (CL) scenario where data is presented to the model sequentially. In this paper, we show that self-supervised loss functions can be seamlessly converted into distillation mechanisms for CL by adding a predictor network that maps the current state of the representations to their past state. This enables us to devise a framework for Continual self-supervised visual representation Learning that (i) significantly improves the quality of the learned representations, (ii) is compatible with several state-of-the-art self-supervised objectives, and (iii) needs little to no hyperparameter tuning. We demonstrate the effectiveness of our approach empirically by training six popular self-supervised models in various CL settings.

1. Introduction

The paper addresses catastrophic forgetting when self-supervised models learn sequentially from unlabeled data. CaSSLe converts self-supervised objectives into continual distillation, improving representations across diverse continual-learning settings.

  • Motivation: Offline self-supervised learning can match supervised representations, but sequentially arriving unlabeled data makes retraining on all previous data impractical or impossible.The problem arises when old data is unavailable and new knowledge must be integrated progressively.
  • CaSSLe: CaSSLe trains the current model to predict past representations with a prediction head, encouraging retention of previous knowledge.The framework reuses self-supervised losses as distillation mechanisms.
  • CaSSLe: CaSSLe is compatible with popular self-supervised losses and architectures and requires no additional hyperparameter tuning over the original method.These properties are presented as design features of the framework.
  • Evaluation: Six popular self-supervised methods are analyzed across class-, data-, and domain-incremental settings on CIFAR100, ImageNet100, and DomainNet.The study spans small, medium, and large-scale datasets.
  • Findings: In class-incremental learning, self-supervised methods typically approach or outperform supervised learning, whereas supervised learning retains a sizeable advantage in data- and domain-incremental settings.The comparison is reported as an empirical pattern across the evaluated settings.

2. Related Work

Related work spans self-supervised objectives and continual-learning methods, but existing continual-learning approaches often rely on supervision or are limited in scale and representation evaluation.

  • Self-Supervised Learning: Self-supervised methods learn augmentation-invariant representations using contrastive, clustering-based, or redundancy-reduction objectives.Examples include instance discrimination, prototype-based cross-entropy, and variance-invariance-covariance regularization.
  • Continual Learning: Continual-learning methods are commonly grouped into replay-based, regularization-based, and parameter-isolation approaches.These categories organize prior methods for mitigating catastrophic forgetting.
  • Continual Learning: Standard continual-learning evaluations train a linear classifier sequentially but do not capture the internal dynamics of hidden representations.This leaves representation behavior under continual learning insufficiently reflected by the protocol.
  • Unsupervised Continual Learning: Prior unsupervised continual-learning studies were largely restricted to digitlike datasets and unsuitable for large-scale scenarios.The cited examples focus on datasets such as MNIST and Omniglot.
  • Unsupervised Continual Learning: Recent work explores self-supervised or unsupervised continual learning, but some approaches are specifically designed for contrastive objectives.The related-work discussion identifies concurrent CSSL efforts with narrower objective compatibility.

3. Preliminaries

The preliminaries define self-supervised representation learning from augmented views and continual learning over sequentially changing data distributions, highlighting the unavailability of previous-task data.

  • Self-Supervised Learning: Self-supervised training extracts representations from two stochastically augmented views of each image and minimizes a loss between their representations.The encoder may use shared networks or exponentially moving-average copies.
  • Self-Supervised Learning: Strong augmentations are designed to preserve semantics while changing appearance, encouraging semantically similar images to occupy nearby feature-space regions.The augmentations may alter geometry, colors, and resolution.
  • Continual Learning: Continual learning trains a network on an ordered sequence of tasks, each associated with a different data distribution and class set.The standard objective sums task-specific losses over the sequence.
  • Continual Learning: Because the previous distribution is unavailable during a new task, the standard continual-learning objective cannot be minimized directly.Knowledge distillation is one indirect approach for transferring information between networks.

4. Continual Self-Supervised Learning

Continual self-supervised learning trains an augmentation-invariant feature extractor across sequential unlabeled tasks whose previous data are discarded. Evaluation uses a linear classifier after each task, with task boundaries assumed to be known.

  • Problem Setting: CSSL presents unlabeled tasks sequentially, discards previous-task data, and trains the model without labels after the first task.The objective is a feature extractor invariant to augmentations across all tasks.
  • Assumptions: Task boundaries are provided to the model for this setting.This assumption simplifies the new continual self-supervised learning problem.
  • Objective: The CSSL objective sums self-supervised losses across tasks sampled without labels.Its formulation inherits the task summation from continual learning while retaining the SSL loss.
  • Evaluation: After each task, a linear classifier is trained on the learned backbone and evaluated on the test set.This protocol is compatible with standard continual-learning metrics.

5. The CaSSLe Framework

CaSSLe converts self-supervised losses into continual-learning distillation by using a predictor to map current representations into the previous feature space. The framework preserves stability while allowing plasticity, and adapts across multiple SSL loss families without extra loss-weight tuning.

  • Framework design: CaSSLe adds a predictor network that maps current representations to their past state, avoiding direct feature matching that could discourage learning new concepts.A frozen copy of the previous model supplies past representations, while the current encoder produces representations for new data.
  • Framework design: The generic distillation loss reuses the selected SSL objective on predicted current features and detached past features.The final objective combines the original SSL loss with this distillation loss, and the loss can be made symmetric across views.
  • Framework design: CaSSLe requires no hyperparameter weighting between the distillation and SSL losses and is compatible with popular SSL architectures and objectives.The framework is described as simple to implement, with full loss-specific derivations provided separately.
  • Compatibility with SSL losses: InfoNCE distillation performs instance discrimination in the past feature space, while MSE-based distillation predicts past representations without additional regularization in the VICReg case.For InfoNCE, the predictor aligns present samples with their past representations while separating negatives; VICReg uses MSE as its invariance term.
  • Compatibility with SSL losses: Cross-entropy-based methods use cluster prototypes as a proxy for invariant representations, whereas cross-correlation objectives decorrelate feature components during distillation.The cross-entropy family includes SwAV, DeepClusterV2, and DINO; the analysis focuses mainly on Barlow Twins for cross-correlation objectives.

6. Experiments

Experiments evaluate CaSSLe across multiple continual self-supervised learning settings, datasets, metrics, and baselines. CaSSLe generally improves representation quality, transfer, and forgetting behavior, though gains depend on the SSL method and continual setting.

  • Experimental protocol: Experiments evaluate linear accuracy, forgetting, and forward transfer across CIFAR100, ImageNet100, and DomainNet continual-learning settings.The study covers class-, data-, and domain-incremental scenarios with five or six tasks, depending on the dataset.
  • Class-incremental results: CaSSLe produces better representations than competing strategies and improves forward transfer while reducing forgetting, especially on ImageNet100.On CIFAR100, SSL methods already show low forgetting relative to supervised learning, leaving less room for improvement.
  • Class-incremental results: CaSSLe outperforms other methods throughout five-task class-incremental CIFAR100 training and mitigates BYOL’s fine-tuning instability.EWC and Less-Forget also mitigate the instability, whereas LUMP initially drops before recovering.
  • Class-incremental results: ≈7% gains over MoCoV2 are reported for CaSSLe, compared with an ≈1% difference between MoCoV2+ and MoCoV2 on class-incremental CIFAR100.The comparison uses two- and five-task settings against Lin et al.’s method.
  • Cross-setting results: 6.8% average improvement on CIFAR100, 4% on ImageNet100, 2% on data-incremental ImageNet100, and 4.4% on domain-incremental learning are reported for CaSSLe.The data-incremental average excludes BYOL; the domain-incremental result uses domain-aware evaluation.
  • Downstream and semi-supervised results: CaSSLe improves downstream SSL performance by 3.4% on average over fine-tuning and improves all evaluated SSL methods in the downstream comparison.With limited labels, MoCoV2+ outperforms supervised learning using 10% labels.

7. Conclusion

The paper studies Continual Self-Supervised Learning and contributes CaSSLe, alongside a comprehensive analysis of CSSL properties. It also identifies task-boundary, resource, clustering, and broader-impact limitations.

  • The paper studies Continual Self-Supervised Learning, namely learning a set of tasks without labels continually.
  • CaSSLe reuses SSL methods and losses to learn continually, while the paper analyzes CSSL comprehensively.
  • Limitations: CaSSLe applies where task boundaries are provided, limiting its scope to task-aware settings.
  • Limitations: CaSSLe increases training memory and time requirements by roughly 30%.
  • Limitations: CaSSLe does not perform clustering and therefore requires either a supervised linear classifier or a clustering algorithm to map data to latent classes.
  • Broader impact: CSSL may facilitate powerful AIs for malicious uses, and biases from non-curated data may transfer to downstream tasks.

A. PyTorch-like pseudo-code

The pseudo-code combines the base SSL objective with a symmetric distillation objective between predicted current representations and frozen past representations. CaSSLe uses the same loss symmetrization as the underlying SSL method and adds no loss-weighting hyperparameter.

  • CaSSLe symmetrizes losses by summing both contributions, or averaging them in some cases, following the base SSL method.
  • CaSSLe uses a backbone-projector, a frozen backbone-projector, a predictor, stochastic augmentations, and an SSL loss function.
  • The training step computes the symmetric base SSL loss on representations from augmented views.
  • A frozen network produces past representations, which are compared with predictor outputs through a symmetric distillation loss.
  • The total objective is the sum of SSL and distillation losses, with no hyperparameter for loss weighting.

B. Derivation of distillation losses

CaSSLe derives distillation losses by applying the original SSL loss to predicted current representations and frozen representations. The framework supports cross-entropy, cross-correlation, contrastive, and MSE-based formulations.

  • The general distillation loss is the SSL loss evaluated between predictor outputs of current representations and frozen encoder representations.
  • The predictor g maps current representations z to the past-state space represented by frozen features ¯z.
  • Contrastive based: For contrastive distillation, negatives are drawn from both predicted and frozen features within the batch.
  • MSE based: MSE distillation compares predicted features with frozen features using mean squared error, with cosine similarity as an implementation option.
  • Cross-entropy based: Cross-entropy distillation encourages the current encoder to assign samples consistently to frozen centroids or prototypes.
  • Cross-correlation based: Cross-correlation distillation computes the correlation matrix from predicted and frozen features; VICReg uses only its invariance term.

C. Further discussion and implementation details of the baselines

The baseline study compares CaSSLe with selected continual-learning methods across several settings, adapting objectives and representations where necessary. Because of computational cost, not every method is evaluated in every setting, and baselines receive hyperparameter tuning.

  • Selection: Eight baselines spanning data-focused regularization, replay, and prior-focused regularization are tested on CIFAR100 with three SSL methods.
  • Selection: CaSSLe is compared with feature-distillation, replay, and regularization methods, while some related approaches are less aligned with the setting or focus on memory efficiency.
  • The study includes DomainNet domain-incremental evaluation across six named domains, with task-aware and task-agnostic strategies.
  • Implementation: Baseline implementations replace supervised objectives or classifier logits with SSL losses or projected features where required.
  • Implementation: Baseline tuning searches five distillation-loss weights, three EWC regularization weights, and three replay batch sizes, using a 500-sample replay buffer.

D. Additional results

Additional experiments show that CaSSLe’s advantages extend across supervised contrastive learning, evaluation protocols, task lengths, and network depths, while revealing limits in the breadth of loss-combination testing.

  • Supervised contrastive learning: CaSSLe is easily extended to supervised contrastive learning and improves over fine-tuning.The implementation is essentially the same as for the vanilla contrastive-based distillation loss.
  • DomainNet evaluation: CaSSLe always outperforms fine-tuning under both task-aware and task-agnostic DomainNet evaluation protocols.Task-agnostic accuracy is lower because class-conditional distributions are not perfectly aligned across domains.
  • k-NN evaluation: 58.7% vs 56.9%: SwAV + CaSSLe outperforms SwAV + fine-tuning on backbone features using Euclidean distance.The authors suggest the apparent projected-feature behavior may be an artifact of l2-normalization under cosine similarity.
  • Different numbers of tasks: With 10 CIFAR100 tasks, CaSSLe outperforms fine-tuning for Barlow Twins and by a very large margin for SimCLR.The longer sequence creates a low-data regime in which SSL models become more unstable; Barlow Twins finishes at roughly 50% accuracy, while SimCLR suffers.
  • Deeper architectures: CaSSLe outperforms fine-tuning by large margins with deeper architectures in both class-incremental and data-incremental settings.These experiments use deeper architectures than the ResNet18 networks featured in the main manuscript.
  • Loss combinations: Testing every SSL method with every possible distillation loss was computationally infeasible, so the supplementary study reports only a subset of combinations.The subset validates the strategy of using the same SSL loss for distillation.
Loading 2112.04215v2…