PaperlayerView exact revision on arXiv

Source-linked AI summary

Representational Continuity for Unsupervised Continual Learning

Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, Sung Ju Hwang

arXiv:2110.06976v3cs.LGcs.CV

TL;DR

Continual learning has mainly been developed for supervised settings, limiting applicability when task data are unlabeled, biased, or continuously changing. The paper studies UCL through representation analysis and introduces LUMP, finding stronger retention, performance, and out-of-distribution generalization than SCL.

  • Problem

    Existing continual learning advances focus on supervised learning, while real-world applications often provide biased, unlabeled, and continuously changing data.

  • Method

    The paper learns representations on unlabeled task sequences, analyzes their similarity and loss landscapes, and introduces LUMP to interpolate current and replayed instances.

  • Results

    UCL representations are more robust to catastrophic forgetting, perform better, generalize better to out-of-distribution tasks, and yield flatter, smoother loss landscapes than SCL.

  • Takeaways & Limitations

    The findings indicate that continual learning can learn useful, transferable representations without annotated data, while LUMP further mitigates forgetting.

  • Takeaways & Limitations

    The study does not consider high-resolution continual learning tasks and leaves evaluation on ImageNet for future work.

Abstract

from arXiv · show

Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent CL advances are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on unsupervised continual learning (UCL), where we learn the feature representations on an unlabelled sequence of tasks and show that reliance on annotated data is not necessary for continual learning. We conduct a systematic study analyzing the learned feature representations and show that unsupervised visual representations are surprisingly more robust to catastrophic forgetting, consistently achieve better performance, and generalize better to out-of-distribution tasks than SCL. Furthermore, we find that UCL achieves a smoother loss landscape through qualitative analysis of the learned representations and learns meaningful feature representations. Additionally, we propose Lifelong Unsupervised Mixup (LUMP), a simple yet effective technique that interpolates between the current task and previous tasks' instances to alleviate catastrophic forgetting for unsupervised representations.

1 INTRODUCTION

Continual learning methods have largely focused on supervised settings, while real-world data are often unlabeled, biased, and continuously changing. This work studies unsupervised continual learning and finds that its representations are more robust, generalize better, and can be improved with LUMP.

  • Existing continual learning methods address catastrophic forgetting primarily in supervised settings, despite the expense of obtaining high-quality labels.
  • Unsupervised representation learning commonly assumes large, unbiased, unlabeled datasets, an assumption challenged by limited and continuously changing real-world data.
  • UCL learns representations from unlabeled task streams and extends supervised continual learning strategies to evaluate SimSiam and BarlowTwins.
  • UCL representations show higher similarity between independent models, while their loss landscapes are flatter and smoother than those of SCL models.
  • LUMP interpolates current-task and previous-task instances, alleviating catastrophic forgetting without additional hyperparameters and outperforming supervised continual learning methods across multiple settings.
  • UCL representations are more robust to catastrophic forgetting, achieve better performance than SCL, and generalize better to out-of-distribution tasks.

2 RELATED WORK

Related work spans supervised continual learning strategies and unsupervised representation learning methods. The paper positions UCL at their intersection, addressing continual representation learning without labels.

  • Continual learning methods are commonly organized into regularization, architectural, and rehearsal approaches for mitigating catastrophic forgetting.
  • Unsupervised representation learning includes contrastive frameworks that learn from similarities between positive and negative pairs.
  • Contrastive methods often require large batches and negative pairs, restricting the scalability of these networks.

3 PRELIMINARIES

The paper defines continual learning over a sequence of tasks and reframes the objective from supervised classification to preserving learned representations from unlabeled task streams. It evaluates representation transfer using accuracy and forgetting metrics.

  • 3.1 PROBLEM SETUP: Continual learning operates on a sequence of T tasks, with supervised tasks containing labeled input-target pairs drawn from task-specific distributions.
  • 3.1 PROBLEM SETUP: SCL trains a representation network and classifier using cross-entropy loss on labeled examples from the current task.
  • 3.1 PROBLEM SETUP: UCL removes label supervision and learns representations from unlabeled examples across tasks while preserving knowledge from previous tasks.
  • 3.2 LEARNING PROTOCOL AND EVALUATION METRICS: The learning protocol first pre-trains representations sequentially, then evaluates their quality using downstream predictions on frozen representations.
  • 3.2 LEARNING PROTOCOL AND EVALUATION METRICS: Evaluation uses average accuracy across completed tasks and average forgetting based on each task’s maximum and final accuracy.

4 UNSUPERVISED CONTINUAL LEARNING

This section develops unsupervised representation learning for continual task sequences and adapts existing continual-learning strategies to preserve representations without labels. It also introduces LUMP, which mixes current and replayed past-task instances to mitigate forgetting.

  • Contrastive learning learns representations from positive and negative pairs but can require large batches, negative samples, architectural modifications, or non-differentiable operators.
  • SimSiam uses a shared encoder with projection and prediction heads, minimizing cosine similarity across two augmented views while relying on stop-gradient to avoid trivial solutions.
  • BarlowTwins reduces redundancy between embedding components by bringing the cross-correlation matrix of two augmented views closer to the identity matrix.Its loss trades off invariance and redundancy reduction through a positive λ parameter.
  • Unsupervised continual-learning adaptations include SI and DER, with UCL-DER combining representation learning and Euclidean matching of projected outputs to preserve past knowledge.DER for supervised learning instead matches network logits, but its cross-entropy replay objective cannot be used without labels.
  • Rehearsal performance can depend on α and supervised task information, motivating LUMP as an interpolation-based alternative for mitigating catastrophic forgetting.Standard Mixup interpolates examples and targets, whereas LUMP applies interpolation to current and replay-buffer instances for unsupervised continual learning.
  • LUMP interpolates current-task examples with uniformly sampled replay-buffer examples, augmenting past instances and approximating regularized loss minimization.The paper states that this revisits attributes shared by past and current tasks and successively mitigates catastrophic forgetting while learning discriminative and human-perceptual features.

5 EXPERIMENTS

Experiments compare supervised and unsupervised continual-learning representations across standard, few-shot, out-of-distribution, and representation analyses. Across these settings, UCL generally achieves stronger performance, lower forgetting, more stable representations, and smoother loss landscapes than SCL.

  • Quantitative results: UCL representations outperform SCL with substantially lower forgetting across Split CIFAR-10, Split CIFAR-100, and Split Tiny-ImageNet.With SimSiam, SI using UCL reduces forgetting by 68%, 54%, and 44% relative to the best SCL strategy on the three datasets.
  • Quantitative results: 71.4%, 69.7%, and 73.2% decreases in forgetting are obtained by FINETUNE with BarlowTwins on Split CIFAR-10, Split CIFAR-100, and Split Tiny-ImageNet.BarlowTwins also gives FINETUNE, DER, and LUMP lower forgetting than SimSiam with comparable accuracy across datasets.
  • Few-shot training: 15.78%p versus 32.21%p accuracy deterioration occurs for UCL versus SCL when Split CIFAR-100 training instances decrease from 2500 to 100.The accuracy gap widens as each task provides fewer training instances, and LUMP improves over all baselines across the few-shot experiments.
  • Out-of-distribution evaluation: UCL representations outperform supervised representations across all evaluated out-of-distribution datasets.On Split-CIFAR-10, UCL with SimSiam and SI improves absolute performance over the best SCL strategy by 4.58%, 6.09%, 15.26%, and 17.07% on MNIST, FMNIST, SVHN, and CIFAR-100.
  • Representation analysis: Two independent UCL models have higher feature similarity and lower parameter ℓ2 distance than two independent SCL models, while lower-layer representations are similar across models.UCL and SCL representations are similar in lower layers but diverge in higher layers across continual-learning strategies.
  • Qualitative analysis: UCL learns perceptually relevant, robust feature maps and a flatter, smoother loss landscape than SCL after continual learning.For feature maps, SCL representations appear noisy and incoherent on earlier tasks, whereas UCL representations remain more structured; the loss landscape is evaluated after training on early and later tasks.

6 DISCUSSION AND CONCLUSION

The paper finds that unsupervised continual learning (UCL) representations are more robust and generalize better than supervised continual learning representations. It analyzes these representations and introduces LUMP to improve performance while reducing catastrophic forgetting.

  • Surpassing supervised continual learning: UCL representations are more robust to catastrophic forgetting and achieve stronger performance than SCL representations across evaluated tasks and datasets.
  • Surpassing supervised continual learning: UCL generalizes better to out-of-distribution tasks and performs more strongly in few-shot learning settings than SCL.
  • Surpassing supervised continual learning: LUMP interpolates unsupervised instances from current and past tasks, obtaining higher performance with lower catastrophic forgetting across a wide range of tasks.
  • Dissecting the learned representations: Representation analysis finds high similarity between UCL and SCL in lower layers but dissimilarity in higher layers.
  • Dissecting the learned representations: UCL representations learn coherent, discriminative patterns and produce smoother loss landscapes than SCL representations.
  • Limitations and future work: The study does not evaluate high-resolution continual-learning tasks and leaves ImageNet evaluation and broader continual-learning setups for future work.

A SUPPLEMENTARY MATERIAL

The supplementary material documents implementation details, hyperparameter configurations, additional experiments, and visualizations with results.

  • The supplementary material provides implementation details.
  • It includes hyperparameter configurations in Appendix A.1.
  • It presents additional experiments, visualizations, and results in Appendix A.2.

A.1 EXPERIMENTAL DETAILS

The experiments use the DER codebase and established supervised and unsupervised representation-learning implementations. Training configurations differ between UCL and SCL, with dataset-specific UCL settings provided in Table A.5.

  • Implementations: Experiments use the DER open-source codebase and reproduce its supervised continual-learning results as baselines.
  • Implementations: The UCL experiments follow the original SimSiam and BarlowTwins representations.
  • Hyperparameter configurations: UCL methods use batch size 256 for 200 epochs, whereas SCL methods use batch size 32 for 50 epochs following prior settings.
  • Hyperparameter configurations: Table A.5 reports hyperparameter configurations for all datasets on the ResNet-18 architecture.

A.2 ADDITIONAL EXPERIMENTS

The supplementary experiments add loss-landscape and feature-map visualizations for Split CIFAR-100 and Split Tiny-ImageNet using ResNet-18. These visualizations examine representations and optimization landscapes after continual learning.

  • Loss landscape visualization: Figure A.6 provides an additional Split CIFAR-100 loss-landscape visualization for ResNet-18 using SimSiam UCL methods.
  • Feature-map visualizations: Figures A.7 and A.8 visualize second- and third-block feature maps for SCL and SimSiam UCL on Split CIFAR-100.
  • Feature-map visualizations: Figure A.9 visualizes second-block feature maps for SCL and SimSiam UCL on Split Tiny-ImageNet using ResNet-18.
Loading 2110.06976v3…