PaperlayerView exact revision on arXiv

Source-linked AI summary

GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learning

Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, Pradeep Shenoy

arXiv:2111.11210v3cs.LGcs.AI

TL;DR

Continual learning must adapt across sequential tasks despite limited access to prior data and the resulting risk of catastrophic forgetting. GCR selects weighted replay coresets that approximate gradients over observed data, achieving gains over state-of-the-art methods offline and online while benefiting from supervised contrastive loss.

  • Problem

    Continual learning must preserve knowledge from prior tasks despite having access mainly to data from the current task.

  • Method

    GCR selects weighted replay coresets whose replay-loss gradients approximate the gradient of the full observed dataset and incorporates supervised contrastive loss.

  • Results

    GCR significantly outperforms comparable approaches by 2-4% accuracy offline and up to 5% online, with supervised contrastive loss independently contributing additional gains.

  • Takeaways & Limitations

    The gradient-based selection objective and representation loss each add value, and GCR applies across offline and online continual-learning settings.

  • Takeaways & Limitations

    Improving candidate selection beyond reservoir sampling would require storing intermediate logits for the entire current-task dataset, creating prohibitive storage requirements.

Abstract

from arXiv · show

Continual learning (CL) aims to develop techniques by which a single model adapts to an increasing number of tasks encountered sequentially, thereby potentially leveraging learnings across tasks in a resource-efficient manner. A major challenge for CL systems is catastrophic forgetting, where earlier tasks are forgotten while learning a new task. To address this, replay-based CL approaches maintain and repeatedly retrain on a small buffer of data selected across encountered tasks. We propose Gradient Coreset Replay (GCR), a novel strategy for replay buffer selection and update using a carefully designed optimization criterion. Specifically, we select and maintain a "coreset" that closely approximates the gradient of all the data seen so far with respect to current model parameters, and discuss key strategies needed for its effective application to the continual learning setting. We show significant gains (2%-4% absolute) over the state-of-the-art in the well-studied offline continual learning setting. Our findings also effectively transfer to online / streaming CL settings, showing upto 5% gains over existing approaches. Finally, we demonstrate the value of supervised contrastive loss for continual learning, which yields a cumulative gain of up to 5% accuracy when combined with our subset selection strategy.

1 Introduction

Continual learning seeks resource-efficient adaptation across sequential tasks while avoiding catastrophic forgetting caused by limited access to prior-task data. GCR addresses this challenge with gradient-based coreset replay and reports gains in offline and online settings.

  • Continual learning trains models incrementally across sequential scenarios while avoiding the costs of maintaining separate one-off models.
  • Limited access to prior-task data can cause catastrophic forgetting, in which later training erases information about earlier tasks.
  • Replay-based methods retain a small data sketch from previous tasks and include it in training throughout the model’s lifetime.
  • GCR selects and updates coresets by approximating model-parameter gradients over all data seen so far.
  • GCR achieves 2-4% absolute gains over state-of-the-art methods in offline continual learning and up to 5% gains in online or streaming settings.

2 Related Work

Related continual-learning work includes regularization, architectural modification, replay, and coreset-selection approaches. GCR differs from prior coreset methods by tying selection directly to the replay loss and using weighted buffer samples.

  • Regularization methods preserve earlier-task learning through weight, structural, functional, momentum-matching, or distillation-based constraints.
  • Architecture-based methods modify networks through recurrent designs, adaptive compression, overlapping filters, or shared-information discovery.
  • Replay methods store representative samples from earlier tasks for continued training, sometimes adding distillation or episodic-memory objectives.
  • Coresets are small informative weighted subsets that approximate attributes such as losses, gradients, logits, or dataset-wide sums.
  • A prior bilevel replay-buffer method is described as computationally expensive, intractable for large task scenarios, and poorly scaling with buffer size.
  • GCR ties coreset selection to the replay loss function and uses learned weights to improve buffer-data utilization.

3 Preliminaries

Continual learning trains one model across sequential tasks while limiting catastrophic forgetting despite limited access to earlier-task data. The formulation defines task data, model outputs, and replay-based objectives that use a small buffer of prior-task samples.

  • Each task provides an i.i.d. dataset with distinct classes, and different tasks are assumed to have no classes in common.
  • The model is described by parameters θ, a feature extractor Ωθ(x), logits hθ(x), and class probabilities fθ(x)=SOFTMAX(hθ(x)).
  • At step t, only current-task data are available, making direct optimization over the continual-learning objective difficult while the model must retain earlier-task knowledge.
  • Replay-based methods retain prior knowledge by training on a small buffer of previous-task samples alongside data from the new task.
  • DER augments replay with stored logits and an additional distillation loss, and outperforms earlier proposals in task-incremental and class-incremental offline settings.

4 GCR: Methods

GCR formulates replay-buffer selection as gradient approximation over data seen so far, then integrates weighted replay, task sampling, and supervised contrastive learning into continual-learning updates.

  • GradApprox for Replay Buffer Selection: GCR frames replay-buffer selection as an optimization problem that approximates gradients over the previously observed data.The formulation operates on a candidate pool and prior replay buffer, covering offline and online continual-learning settings.
  • GradApprox for Replay Buffer Selection: GradApprox selects K data points and weights whose weighted replay-loss gradients most closely match the gradient of the full dataset.The target gradient includes weights learned in the previous GradApprox round, and those weights must be used during subsequent learning.
  • GradApprox for Replay Buffer Selection: The optimization applies to specified replay losses, including DER’s distillation-plus-label loss and vanilla Experience Replay’s cross-entropy loss.In DER, GradApprox selects weights for both distillation and label-loss components.
  • GradApprox Implementation: GCR solves the weakly submodular objective with orthogonal matching pursuit, regularizes weights with an l2 penalty, and selects equal samples per class.These choices provide approximation guarantees, discourage large sample weights, and address class imbalance.
  • GCR Loss Objective: The GCR objective combines current-task prediction loss, replay distillation loss, replay label loss, and weighted supervised contrastive loss.The contrastive term encourages same-class embeddings to be closer than embeddings from different classes, using candidate-pool and replay-buffer samples as anchors.
  • GCR Loss Objective: GCR’s weighted-loss formulation and supervised contrastive loss independently and cumulatively improve over DER, while the selection procedure remains applicable to other replay losses.Historical model logits are stored for replay samples and candidate-pool samples to support DER-style distillation.
  • Further Optimization: Reservoir sampling limits candidate-pool storage, whereas gradient-based selection over the entire current task would require storing intermediate logits for every sample.The latter storage requirement is described as prohibitive.
  • Continual Learning Workflow: GCR trains on current-task data, then updates the replay buffer with a weighted K-sample summary drawn from current candidates and the previous buffer.Candidate points use reservoir sampling, while adaptive sampling draws tasks from the combined candidate pool and replay buffer.

5 Experiment setup

The experiments compare GCR with continual-learning baselines across offline class- and task-incremental settings using several sequential image datasets, controlled training, and repeated random seeds.

  • Evaluation scope: The evaluation compares GCR with state-of-the-art continual-learning baselines across different continual-learning settings.The baselines include Experience Replay, GEM, and MIR.
  • Metrics: Table 2 reports forgetting, where lower values are better, and compares the two best-performing algorithms, DER and GCR.The text states that lower forgetting indicates less catastrophic forgetting and that forgetting and accuracy covary closely.
  • Continual-learning settings: Offline Class-IL trains on all current-task data while prior-task data is accessed only through the replay buffer.Offline Task-IL uses task-specific classifier heads and requires a task identifier at inference.
  • Datasets: The datasets comprise Sequential CIFAR-10, Sequential CIFAR-100, Sequential Tiny-ImageNet, Sequential ImageNet-1k, and a 20-task CIFAR-100 extension.The standard splits use 5, 5, 10, and 5 tasks respectively, while the extended CIFAR-100 setting uses 20 tasks with 5 classes per task.
  • Training protocol: All experiments use ResNet18, minibatches of 32, stochastic gradient descent without a learning-rate scheduler, and tuned hyperparameters.Hyperparameters are selected by grid search on validation data formed from 10% of the training data.
  • Reporting: 15 random-seed runs report mean test accuracy with standard error of the mean, and Table 1 bolds the best-performing models using paired t-tests with p < 0.05.Table 1 covers offline Class-IL and Task-IL accuracy across CIFAR-10, CIFAR-100, Tiny-ImageNet, and buffer sizes.

6 Results

Across offline and online continual-learning evaluations, GCR generally improves replay performance, with especially clear gains in harder settings, larger task sequences, and smaller online buffers. Ablations also support the value of gradient-based selection and supervised contrastive loss.

  • More tasks: GCR shows robust gains on 20-task S-Cifar-100, with larger margins over ER and DER than in the 5-task setting.The results indicate that coreset selection scales effectively as the number of tasks increases.
  • Offline Class-IL: GCR gains roughly 1–5% over other methods in Class-IL, with statistically significant improvements.For S-Cifar-100, gains exceed 4.5% at buffer sizes 500 and 2000.
  • Offline Task-IL: GCR often outperforms alternatives in Task-IL, although DER and ER remain reasonably close.On S-Cifar-100 with buffer size 2000, the difference is close to 1.5%.
  • Generality: GCR also improves performance on high-resolution S-ImageNet-1k and combines effectively with other replay-based approaches.These findings extend the gradient-based coreset strategy beyond the main offline benchmarks.
  • Online streaming CL: In online CL, GCR’s advantage is most apparent with small buffers, including a gap of more than 5% over DER on S-Cifar-10 at buffer size 500.As buffer size increases, broader coverage can compensate for less effective data selection.
  • Ablation studies: Ablations show that GCR outperforms reservoir sampling and bilevel coresets, while supervised contrastive loss adds further value.For buffer size 2000, all pairwise comparisons were statistically significant with p < 1e −3.

7 Conclusion, Limitations, and Future Work

The paper concludes that GCR combines gradient-based replay-buffer selection with supervised representation learning to improve continual learning across offline and online settings. It reports stronger gains as task count and image complexity increase, while identifying integration with other continual-learning techniques and candidate-selection simplification as remaining challenges.

  • Conclusion: GCR selects replay coresets by optimizing gradient approximation and integrates this objective into replay-buffer selection and updates.The method also includes a supervised representation-learning loss in the continual-learning objective.
  • Conclusion: GCR improves accuracy by 2–4% in offline settings and by up to 5% in online settings across datasets, buffer sizes, and continual-learning settings.Ablations separately support contributions from the coreset objective and representation loss.
  • Conclusion: GCR’s gains become increasingly significant as the number of tasks increases and extend to more complex images.The paper reports effective application in both offline and online settings.
  • Limitations and future work: Integrating GCR with exemplars, function regularization, and self-supervision remains an open challenge.The paper also notes that its candidate selection is simplified using a throw-away candidate buffer.

B.1 Forgetting metric

The paper evaluates forgetting in offline and online continual learning, while emphasizing that forgetting must be interpreted alongside final accuracy. It also notes that the OCS comparison is incomplete because reproducibility and broader-setting results are limited.

  • Table 6 reports forgetting metrics for offline Class-IL and Task-IL continual learning.
  • Table 7 reports forgetting metrics for online continual learning.
  • Forgetting should be compared alongside final accuracy because low forgetting can reflect failure to learn subsequent tasks.The paper gives iCaRL as an example of poor overall accuracy despite favorable forgetting metrics.
  • The OCS comparison reports better GCR performance, but the comparison is incomplete because OCS code and broader evaluation results were unavailable.The limitation covers other settings, datasets, and buffer sizes explored in the paper.

C Generality of gradient-based coresets

The paper tests whether gradient-based coreset selection generalizes across replay methods and to a substantially harder dataset. The reported results support gains with ER and on S-Imagenet-1k, although all methods remain relatively inaccurate on the latter.

  • GCR coreset selection retains its gains when combined with the ER replay method.The authors report that the gains from gradient approximation apply to other replay-based methods.
  • GCR outperforms ER and DER by a significant margin on S-Imagenet-1k.The dataset contains high-resolution Imagenet images divided into five tasks of 200 categories each.
  • All three methods have fairly low overall accuracy on S-Imagenet-1k because the task is substantially harder than S-Cifar100.

D Implementation details

The implementation details identify the searched hyperparameters and the benchmark configuration used for offline Class-IL experiments. The supplied passages provide table labels and captions rather than the underlying parameter values.

  • Table 11 lists the hyperparameter values selected from grid search for the experiments.
  • Table 12 lists the hyperparameter search spaces for offline and online settings.
  • The offline Class-IL configuration includes buffer size and results for S-Cifar10, S-Cifar100, and S-Tinyimg.
Loading 2111.11210v3…