Source-linked AI summary
Co$^2$L: Contrastive Continual Learning
Hyuntak Cha, Jaeho Lee, Jinwoo Shin
TL;DR
Continual learning needs representations that remain useful across future tasks and resist catastrophic forgetting. Co2L learns such representations with contrastive learning and preserves them through self-distillation, outperforming jointly trained baselines across continual-learning scenarios and datasets.
Problem
The paper asks what knowledge is useful for future tasks and how to learn and preserve it under continual learning.
Method
Co2L is a rehearsal-based method that learns representations with asymmetric supervised contrastive loss and preserves them using instance-wise relation self-distillation.
Results
Co2L consistently outperforms baselines across Task-IL, Domain-IL, and Class-IL scenarios, datasets, and memory setups.
Takeaways & Limitations
Contrastively learned representations suffer less forgetting than jointly trained representations, supporting continual learning centered on transferable representations.
Takeaways & Limitations
The setup assumes a task sequence with task-specific class sets, while Domain-IL keeps the class set unchanged across tasks.
Abstract
from arXiv · showhide
Recent breakthroughs in self-supervised learning show that such algorithms learn visual representations that can be transferred better to unseen tasks than joint-training methods relying on task-specific supervision. In this paper, we found that the similar holds in the continual learning con-text: contrastively learned representations are more robust against the catastrophic forgetting than jointly trained representations. Based on this novel observation, we propose a rehearsal-based continual learning algorithm that focuses on continually learning and maintaining transferable representations. More specifically, the proposed scheme (1) learns representations using the contrastive learning objective, and (2) preserves learned representations using a self-supervised distillation step. We conduct extensive experimental validations under popular benchmark image classification datasets, where our method sets the new state-of-the-art performance.
1. Introduction
Continual learning must retain knowledge that transfers to future tasks rather than only preserving task-specific solutions. Co2L addresses this by learning transferable representations contrastively and preserving them through self-distillation.
- Motivation: The paper asks what knowledge remains useful for future tasks and how continual learning can learn and preserve it.It contrasts task-specific features such as color with potentially reusable features such as shape, polish, and texture.
- Motivation: Contrastive learning encourages representations invariant to selected input transformations instead of relying on task-specific supervision.Such representations have been effective for transfer, including ImageNet classification without labels.
- Challenges: Standard continual settings restrict informative negative samples and leave representation preservation under decoupled training insufficiently addressed.Class-incremental learners may access samples from only a small number of classes at each time step.
- Co2L: Co2L is a rehearsal-based continual learning algorithm that continually learns and preserves representations separately from the classifier.The framework uses current-task data together with samples stored in a memory buffer.
- Co2L: Co2L combines an asymmetric supervised contrastive loss for representation learning with self-distillation of instance-wise relations for representation preservation.Both components are presented as central contributions of the method.
- Results: 22.40% and 10.59% relative improvements result from distillation with and without buffered samples, respectively, on Seq-CIFAR-10.The ablation attributes consistent gains to preserved representations and efficient use of buffered samples.
2. Related Work
Prior continual learning work primarily replays samples, regularizes updates, or expands architectures, while representation-focused methods remain limited. Co2L instead studies buffered samples and self-distillation for decoupled representation learning.
- Continual learning: Continual learning methods commonly use rehearsal, regularization, or dynamic architecture expansion to mitigate forgetting.Rehearsal stores a fixed-sized buffer of past samples and replays them with current data.
- Continual learning: Experience Replay manages a fixed-sized buffer containing a small number of samples from past tasks.Those samples are replayed to prevent forgetting past knowledge.
- Continual learning: Existing rehearsal extensions mainly regulate updates or select representative and forgetting-prone samples.These approaches focus on preserving past predictions or avoiding conflicts with past-task objectives.
- Representation learning: Few studies address sample utilization and preservation for decoupled representation learning because representation objectives may not align with task-specific joint-training objectives.Co2L focuses on using buffered samples to learn representations continually in this decoupled setting.
- Knowledge distillation: Knowledge distillation is widely used against forgetting, but its design for decoupled representation-classifier training had not been studied.Co2L develops self-distillation for contrastive continual learning.
3. Problem Setup and Preliminaries
The paper formulates continual learning over task sequences with task-specific data and distinguishes Task-IL, Domain-IL, and Class-IL. Its contrastive-learning preliminaries augment samples, map them to normalized features, and optimize supervised contrastive loss.
- Problem setup: The experiments consider Task-IL, Domain-IL, and Class-IL as three continual-learning scenarios.These scenarios are the paper’s standard setup categories.
- Problem setup: Continual learning trains a model on a sequence of tasks indexed by t ∈ {1, 2, . . . , T}.Each task has a task-specific class set Ct and draws training input-label pairs from a task-specific distribution.
- Problem setup: In Domain-IL, the class set remains constant across tasks: C1 = C2 = · · · = CT.The scenario changes task distributions while retaining the same classes.
- Problem setup: Task-IL provides the task label during testing, whereas Domain-IL and Class-IL do not.The predictor is correspondingly written as ϕθ(x, t) for Task-IL and ϕθ(x) for the other scenarios.
- Contrastive learning: SupCon decomposes the model into a representation fϑ and a linear classifier w, then trains the representation without training w.It forms augmented views of each sample and preserves their labels.
- Contrastive learning: Augmented samples are mapped to a unit d-dimensional Euclidean sphere before optimizing supervised contrastive loss.The projection map is parameterized by φ, and ψ concatenates the representation and projection parameters.
- Contrastive learning: For each anchor, positive samples include its other augmentation and augmented samples with the same label.The temperature hyperparameter τ controls the contrastive similarity scaling.
4. Co2L: Contrastive Continual Learning
Co2L is a rehearsal-based continual learning scheme that learns representations with asymmetric supervised contrastive loss and preserves them through instance-wise relation distillation. It trains on current and buffered samples, using contrastive learning for new representations and a fixed previous-task model to regulate representation changes.
- Co2L jointly uses asymmetric supervised contrastive learning and self-supervised distillation in a decoupled representation-classifier training scheme.The compound objective combines representation learning with representation preservation.
- Each batch contains two independently augmented views of samples drawn from the union of current-task data and the replay buffer.At each task’s end, the buffer retains a uniformly sampled, class-balanced subset of training samples.
- The asymmetric loss uses current-task samples as anchors, while buffered past-task samples serve only as negative samples.This design is motivated by preventing overfitting to the small number of past-task samples in memory.
- The asymmetric loss empirically outperforms the original supervised contrastive loss and produces better-spread features for buffered samples.
- IRD preserves representations by distilling instance-wise relations between the current model and a reference model fixed after the previous task.It regulates changes in the overall structure of learned representations rather than exact feature-space changes.
- During new-task training, the reference model remains fixed while the current model minimizes the contrastive objective together with the IRD preservation loss.The reference model is the model trained at the end of the preceding task.
5. Experiment
Experiments evaluate Co2L across continual-learning scenarios and datasets, showing that contrastive representations transfer better to future tasks and resist forgetting. Ablations attribute performance gains to asymmetric supervised contrastive learning and instance-wise relation distillation.
- Experimental Setup: Experiments cover Task-IL, Class-IL, and Domain-IL using Seq-CIFAR-10, Seq-Tiny-ImageNet, and R-MNIST, with rehearsal baselines and multiple memory sizes.Co2L uses frozen learned representations followed by a separately trained classifier using last-task and buffered samples.
- Key Hypothesis: 62.76% vs. 32.77% average unseen-task accuracy shows that contrastively trained representations transfer better to future tasks than jointly trained ones.Classifiers were trained with all samples, including samples from unseen tasks.
- Main Results: Co2L consistently outperforms all baselines across scenarios, datasets, and memory sizes.The main classification results are reported in Table 1, averaged over ten independent trials.
- Ablation Studies: 22.40% relative improvement with replay and 10.59% without replay demonstrate that IRD significantly improves performance in both settings.The larger gain with replay suggests that preserving similarity structure among buffered and current-task samples is important.
- Ablation Studies: Asymmetric SupCon consistently improves over original SupCon, while t-SNE features show buffered samples better representing the entire class population.Figure 4 further indicates that IRD closes the performance gap between original and asymmetric SupCon when all past samples are available.
6. Conclusion
The paper proposes contrastive continual learning to learn and preserve representations in continual settings. Its asymmetric contrastive loss and instance-wise relation distillation support better performance than jointly trained baselines across learning setups.
- The proposed scheme learns representations under continual learning scenarios.
- Its asymmetric contrastive learning loss and instance-wise relation distillation help the model learn and preserve new and past representations.
- Co2L shows better performance than jointly trained baselines across various learning setups.
A. Training Details
Training uses the data augmentation scheme introduced in prior work for representation learning and linear evaluation. The default augmentations include cropping, horizontal flipping, color jittering, and grayscale conversion.
- Training follows a prior data augmentation scheme for representation learning and linear evaluation.
- RandomResizedCrop resizes cropped images to dataset-specific resolutions for Seq-CIFAR-10, Tiny-ImageNet, and R-MNIST.The crop scales are [0.2, 1.0], [0.1, 1.0], and [0.7, 1.0], respectively.
- RandomHorizontalFlip is applied with probability 0.5.
- ColorJitter is applied with probability 0.8 using maximum strengths {0.4, 0.4, 0.4, 0.1} for brightness, contrast, saturation, and hue.
A.2. Architecture
The architectures use dataset-specific encoders followed by projection MLPs for representation learning. Seq-CIFAR-10 and Tiny-ImageNet use ResNet-18, while R-MNIST uses a convolutional encoder.
- Seq-CIFAR-10 and Tiny-ImageNet use a non-pretrained ResNet-18 base encoder.
- Their encoder is followed by a 2-layer projection MLP mapping representations to a 128-dimensional latent space.The projection MLP has 512 hidden units.
- R-MNIST uses two convolutional layers and one fully connected layer as its base encoder.The convolutional layers use 20 and 50 filters with 5 × 5 kernels, followed by max pooling with stride 2.
- The R-MNIST encoder is also followed by a 2-layer projection MLP for representation learning.
A.3. Hyperparameter
Hyperparameters are selected through grid search using validation data, with selection criteria differing across continual-learning settings. Co2L uses specified learning, contrastive, distillation, epoch, batch-size, and distillation-power settings.
- Hyperparameters are selected by grid search on a validation set containing randomly drawn 10% of the training samples.
- The Co2L search considers learning rate η, batch size bsz, contrastive temperature τ, distillation temperatures κ and κ∗, and task epochs Et.
- The search spaces include η ∈ {0.1, 0.5, 1.0}, τ ∈ {0.1, 0.5, 1.0}, κ ∈ {0.1, 0.2}, and κ∗ ∈ {0.01, 0.05, 0.1}.
- The initial task uses E0 = 500 epochs, later tasks use Et>0 ∈ {50, 100}, and batch size is searched over {256, 512, 1024}.
- For Class-IL and Task-IL, the best hyperparameters maximize final accuracy averaged across both settings.
- All Co2L experiments use distillation power λ = 1.0.
A.4. Training Details for Co2L
Co2L uses separate optimization schedules for representation learning and linear evaluation. Representation learning restarts cosine-decay scheduling at each task, while linear evaluation uses exponential decay at fixed epochs.
- Representation learning: Representation learning uses linear warmup for 10 epochs followed by cosine learning-rate decay, restarted at every task.SGD uses momentum 0.9 and weight decay 0.0001.
- Linear evaluation: Linear evaluation trains a classifier for 100 epochs with SGD momentum 0.9 and no weight decay.The learning rate decays exponentially at epochs 60, 75, and 90 with decay rate 0.2.
- Linear evaluation: Linear evaluation uses learning rates {1.0, 0.1, 1.0} for {Seq-CIFAR-10, Seq-Tiny-ImageNet, R-MNIST}, respectively.
B. Experiments on IRD Alternatives
The IRD alternatives compare distillation targets at the encoder, embedding, and projection levels. The experiments report that matching projector outputs is more effective than matching encoder outputs or embedding representations for preserving continually learned representations.
- Co2L: Co2L learns and preserves representations using contrastive learning and an asymmetric representation-distillation loss, Lasym.
- Distillation alternatives: IRD distills instance-wise similarity outputs from the joint encoder-projector, whereas SEED directly distills encoder outputs.The comparison uses teacher and student encoders with corresponding temperature parameters.
- Distillation alternatives: Logit matching replaces IRD with losses that directly match representation maps from the past and current models.Two matching choices are defined on embedding and projection spaces.
- Results: Distilling the projector output significantly outperforms distilling at the encoder output and embedding output.
- Results: Projection-space distillation can better preserve learned representations because contrastive learning defines similarity on a unit d-dimensional Euclidean sphere.
- Evaluation: Table 6 reports classification accuracies for Seq-CIFAR-10 and Seq-Tiny-ImageNet averaged over ten independent trials.