PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Contrastive Learning for Image Classification

Zhiwei Lin, Yongtao Wang, Hongxiang Lin

arXiv:2107.01776v4cs.CVcs.AI

TL;DR

Streamed unlabeled data makes continual self-supervised learning vulnerable to catastrophic forgetting, while existing methods largely target packaged data. The paper addresses this in contrastive learning with rehearsal, sampling, self-supervised distillation, and an extra sample queue, improving accuracy across three benchmarks under 10 incremental steps.

  • Problem

    Streamed unlabeled data and resource constraints make continual self-supervised learning necessary, but catastrophic forgetting in contrastive learning is insufficiently studied.

  • Method

    A rehearsal-based continual contrastive learning framework combines representative sample selection, self-supervised knowledge distillation, and an extra sample queue separating old and new representations.

  • Results

    The method outperforms other continual contrastive learning methods, reaching 50.10% on CIFAR-100, 55.48% on ImageNet-Sub, and 62.79% on ImageNet-Full under 10 incremental settings.

  • Takeaways & Limitations

    The framework works on both small and large datasets and alleviates catastrophic forgetting in continual contrastive learning.

  • Takeaways & Limitations

    The method is currently available only for contrastive learning, and applying it to GAN, BYOL, or MAE requires additional revisions.

Abstract

from arXiv · show

Recently, self-supervised representation learning gives further development in multimedia technology. Most existing self-supervised learning methods are applicable to packaged data. However, when it comes to streamed data, they are suffering from a catastrophic forgetting problem, which is not studied extensively. In this paper, we make the first attempt to tackle the catastrophic forgetting problem in the mainstream self-supervised methods, i.e., contrastive learning methods. Specifically, we first develop a rehearsal-based framework combined with a novel sampling strategy and a self-supervised knowledge distillation to transfer information over time efficiently. Then, we propose an extra sample queue to help the network separate the feature representations of old and new data in the embedding space. Experimental results show that compared with the naive self-supervised baseline, which learns tasks one by one without taking any technique, we improve the image classification accuracy by 1.60% on CIFAR-100, 2.86% on ImageNet-Sub, and 1.29% on ImageNet-Full under 10 incremental steps setting. Our code will be available at https://github.com/VDIGPKU/ContinualContrastiveLearning.

1. INTRODUCTION

Streamed unlabeled data exposes self-supervised learning to catastrophic forgetting, a problem the paper addresses in contrastive learning through rehearsal, distillation, and an extra sample queue.

  • Motivation: Self-supervised models trained sequentially on streamed data suffer drastic performance drops on old data, while this problem remains seldom studied in unsupervised learning.The paper identifies catastrophic forgetting in several self-supervised methods under class-incremental learning.
  • Problem setting: The paper makes a first attempt to address catastrophic forgetting in contrastive learning under class-incremental learning.The contribution targets incremental data containing samples from new classes.
  • Method: The rehearsal-based framework combines a novel sampling strategy with self-supervised knowledge distillation to transfer information from old data.The sampling strategy and distillation are designed to preserve previously learned representations without relying on the full historical dataset.
  • Method: An extra sample queue reduces interference between old and new feature distributions as the network learns new data.It helps discriminate new data from old data in the embedding space.
  • Reported outcome: The method improves baseline performance on CIFAR-100 and ImageNet under continual learning settings.The introduction reports improvements across the evaluated datasets, with the detailed benchmark values presented later.

2. RELATED WORK

Self-supervised learning depends on large-scale unlabeled data, but continuously arriving data makes retraining costly; continual class-incremental learning therefore motivates rehearsal and distillation without labels.

  • 2.1. Self-supervised Learning: Self-supervised learning learns transferable representations from large-scale unlabeled data through designed pretext tasks, including instance discrimination and contrastive learning.Contrastive learning is presented as a prominent instance-discrimination approach.
  • 2.1. Self-supervised Learning: Continuously generated data makes collecting old and new data together for retraining time- and resource-consuming, motivating continuous self-supervised learning.Large datasets are described as indispensable for training self-supervised networks.
  • 2.2. Class Incremental Learning: Class-Incremental Learning increases the number of classes step by step, creating a setting for continual learning.The supplied passage introduces the class-incremental setting but does not provide its full definition.
  • 2.2. Class Incremental Learning: The paper implements Continual Contrastive Learning on MoCoV2, with its overall method pipeline shown in Fig. 2.The supplied figure caption identifies the pipeline, while the implementation passage names MoCoV2 as the base framework.
  • 2.2. Class Incremental Learning: Existing class-incremental methods commonly use rehearsal and knowledge distillation, but supervised rehearsal strategies rely on labels unavailable in self-supervised learning.The paper motivates feature-variance-based sampling as a label-free alternative.

3. PROPOSED METHOD

The method combines rehearsal, feature-variance sampling, self-supervised knowledge distillation, and an extra sample queue to mitigate forgetting in continual contrastive learning.

  • 3.2. Rehearsal with Knowledge Distillation: Rehearsal restores old samples, while feature-variance sampling selects representative data without relying on labels.Images are encoded, grouped with K-Means, and selected using feature variance after each training dataset.
  • 3.2. Rehearsal with Knowledge Distillation: Self-supervised distillation transfers old contrastive information by matching teacher and student similarity distributions with KL divergence.The teacher and student compute normalized similarity matrices from sampled old images before applying the distillation loss.
  • 3.2. Rehearsal with Knowledge Distillation: A momentum teacher is updated during training and yields better performance than a fixed teacher in the authors’ experiments.The momentum coefficient is set to mt = 0.996.
  • 3.3. Extra Sample Queue: The extra sample queue stores negative features from sampled old data to separate new and old representations in the embedding space.The queue is updated with selected old-data features during each iteration, and its contrastive loss discriminates new from old data.
  • 3.3. Extra Sample Queue: The total objective combines MoCo contrastive loss, extra-sample-queue loss, and knowledge-distillation loss with balancing weights.The experiments use λ1 = 0.9, λ2 = 0.1, and λ3 = 0.1.

4. EXPERIMENT

The method is evaluated for class-incremental learning on CIFAR-100 and ImageNet-Sub&Full, using fixed incremental splits, rehearsal memory, and final frozen-feature linear evaluation.

  • 4.1.1. Datasets: The evaluation uses CIFAR-100 and ImageNet-Sub&Full, whose classes are randomly ordered and divided into sequential incremental splits.ImageNet-Sub contains 100 randomly selected classes from ImageNet-Full.
  • 4.1.2. Implementation Details: Each old training set retains a constant number of stored images after every incremental step, following the class-incremental memory protocol.Classes are divided equally across T steps.
  • 4.1.2. Implementation Details: Models are evaluated after the final incremental step using top-1 linear classification accuracy on frozen encoder features.This follows prior self-supervised evaluation practice.
  • 4.2. Main Results: 1.70, 2.86, and 1.29 percentage points: the method raises top-1 accuracy to 50.10% on CIFAR-100, 55.48% on ImageNet-Sub, and 62.79% on ImageNet-Full under 10 incremental settings.These results outperform the compared continual contrastive learning methods in every Table 1 column.
  • 4.2. Main Results: The method also improves linear classification accuracy when applied to SimCLR and InsDisc on CIFAR-100.This result supports generalization beyond the MoCoV2 implementation used as the basic method.

4.3. Ablation Study

Ablations show that the sampling strategy, knowledge distillation, and extra sample queue each contribute to performance, while K and queue size introduce sensitivity.

  • Main components: The sampling strategy, knowledge distillation, and extra sample queue consistently boost performance over the ablated configurations.The sampling strategy and distillation preserve prior contrastive information, while the queue separates old and new feature regions.
  • K sensitivity: Top-1 accuracy changes with K on ImageNet-Sub, remaining stable from 5 to 15 but decreasing rapidly when K is far from the true class count.K is treated as prior knowledge because unlabeled data do not reveal the category number.
  • Queue size: The extra sample queue performs best at size 128, and larger queues do not improve performance consistently.Large queues update slowly, leaving stale feature vectors that can harm model updates.

4.4. Limitations

The method is currently limited to contrastive learning and still leaves a substantial gap to the continual-learning upper bound.

  • Scope: The method is only available for contrastive learning; applying it to GAN, BYOL, or MAE requires additional revisions.This scope boundary limits direct transfer to other self-supervised learning methods.
  • Performance boundary: A large gap remains between continual contrastive learning and its upper bound despite the method narrowing that gap.The authors identify this as an unresolved limitation and encourage further research.

5. CONCLUSION

The paper proposes a rehearsal-based continual contrastive learning framework that combines old-data rehearsal, self-supervised distillation, and an extra sample queue.

  • Conclusion: The framework rehearses a small number of old images selected with a novel sampling strategy while learning new data.Self-supervised knowledge distillation and an extra sample queue further support representation learning across old and new data.
  • Conclusion: Experimental results and analyses demonstrate the effectiveness of the proposed method.

A. PSEUDO-CODE

The pseudo-code trains the continual contrastive model on each incremental dataset using contrastive, extra-queue, and distillation losses, then updates the encoders, queues, teacher, and stored samples.

  • Inputs: The algorithm receives sequential datasets, an encoder, augmentations, a queue, an extra sample queue, and a restored dataset.
  • Contrastive training: Each batch produces positive and negative pairs for the MoCo and extra-sample-queue losses.
  • Knowledge distillation: Knowledge distillation computes student and teacher embeddings, similarity matrices, and a KL-divergence loss.
  • Updates: The total loss combines MoCo, extra-sample-queue, and distillation terms before updating the model and momentum encoder.
  • Continual updates: After each incremental step, the method updates the main queues and extra sample queue, updates the teacher with momentum, and adds sampled old data.

B. ADDITIONAL RESULTS

The method is evaluated with Forgetting and Forward Transfer metrics on ImageNet-Sub under 5- and 10-step settings, outperforming Finetuning on both metrics. These results further support its ability to alleviate catastrophic forgetting in self-supervised learning.

  • The evaluation reports Forgetting and Forward Transfer metrics in addition to linear classification accuracy.Forward Transfer is defined using linear evaluation accuracy after observing successive datasets and a random-initialization reference.
  • The method outperforms Finetuning on both Forgetting and Forward Transfer metrics, demonstrating reduced catastrophic forgetting in self-supervised learning.The comparison uses ImageNet-Sub under 5- and 10-incremental-step settings.

C. IMPLEMENTATION DETAIL

The method extends to SimCLR and InsDisc by adding knowledge distillation and extra-sample-queue loss terms. These terms use old-data negatives while retaining each method’s original positive samples and hyperparameter settings.

  • Knowledge distillation and extra-sample-queue losses are added to SimCLR and InsDisc without modifying their original contrastive-learning structure.The extra queue supplies old-data negative samples, which are combined with original positive samples to form the ESQ loss.
  • The balancing weights, temperature, and extra-sample-queue size for SimCLR and InsDisc match the MoCoV2 settings.
Loading 2107.01776v4…