PaperlayerView exact revision on arXiv

Source-linked AI summary

CARES: Coverage-Aware Replay Exemplar Selection for Continual Learning

Arundhati S. Shanbhag, Brian B. Moser, Federico Raue, Tobias C. Nauen, Andreas Dengel

arXiv:2610.32799v1cs.LG

TL;DR

Replay-based continual learning depends on choosing representative buffer samples, while model-dependent coreset selection can add recurring computational overhead. CARES selects class-aware, density-weighted coverage in a frozen pretrained feature space without continual-learner training signals. Across online task- and class-incremental settings, it is competitive or superior while reducing selection cost.

  • Problem

    Replay effectiveness depends on buffer composition, but repeated model-dependent selection can impose substantial computational overhead during online updates.

    Source: §1 · pp. 1–2

  • Method

    CARES selects replay examples using class-aware, density-weighted coverage in a frozen pretrained feature space, without optimizing or using training signals from the continual learner.

    Source: §1 · p. 2, §6 · pp. 9–10

  • Results

    Across online task- and class-incremental settings, CARES achieves competitive or superior final accuracy while reducing selection cost; on Split-Tiny-ImageNet, it is 2.3–8.8× faster than existing coreset-based replay methods.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Stable pretrained representation geometry provides a computationally efficient alternative for repeated replay-buffer construction, particularly in online streams.

    Source: §6 · p. 10

    CARES assumes access to a suitable frozen pretrained encoder, and its selection quality depends on the structure induced by that feature space.

    Source: §5 · p. 9

Abstract

from arXiv · show

Replay-based Continual Learning mitigates catastrophic forgetting by keeping a buffer of past samples, making buffer composition critical under a fixed memory budget. Recent coreset-based replay methods improve buffer quality using model-derived gradients, losses, or auxiliary optimization, but incur substantial computational overhead as memory is repeatedly updated. We propose CARES, a continual model-optimization-free approach for buffer construction that instead exploits the geometry of a frozen pretrained feature space using class-aware, density-weighted coverage. We evaluate CARES across balanced, imbalanced, and label-noisy streams under online class- and task-incremental settings. On Tiny-ImageNet, CARES is 2.3 to 8.8 times faster than existing coreset-based selectors. Although CARES achieves this speedup using no training signal from the continual learner, it ranks among the state of the art across balanced, imbalanced, and label-noisy streams under online class- and task-incremental settings, outperforming the strongest baseline by up to 7.58 percentage points under class imbalance while also attaining higher average accuracy on Tiny-ImageNet. Code will be released upon publication

1 INTRODUCTION

Replay buffers help mitigate forgetting, but their effectiveness depends on which samples are retained and repeated model-dependent selection can be costly. CARES instead uses class-aware, density-weighted coverage in a frozen feature space, without continual-learner training signals.

1 INTRODUCTION

  • Under a fixed memory budget, rehearsal effectiveness depends on which samples are retained, while pure coverage can favor isolated examples and density-only selection can favor redundant dominant modes.CARES motivates density-aware coverage as a compromise between these objectives.Source: §1 · pp. 1–2
  • Because buffer updates recur throughout an online stream, optimization-heavy gradient- and loss-based selectors can accumulate substantial computational overhead.These methods can produce effective buffers, particularly under extreme memory limits.Source: §1 · p. 2
  • CARES combines the existing replay memory with newly observed examples, embeds candidates using a frozen pretrained encoder, and applies density-aware coverage within classes.It allocates the fixed replay budget across observed classes before reconstructing memory.Source: §1 · p. 2
  • On Split-Tiny-ImageNet, CARES is 2.3–8.8× faster than existing coreset-based replay methods while achieving higher final accuracy across balanced, imbalanced, and label-noisy settings.The approach uses no model-derived losses, gradients, or auxiliary model training for selection.Source: §1 · p. 2

2 RELATED WORK

Related work spans replay strategies, coreset selection, and buffer construction in feature space. CARES is distinguished by its explicit coreset objective over frozen embeddings, combining class-aware allocation with density-aware coverage.

Replay-based CL

  • Replay methods select or retrieve stored examples using interference, uncertainty, gradients, or diversity; DER and DER++ add distillation-style objectives, while GSS promotes gradient diversity and MIR retrieves examples predicted to incur the largest loss increase.Collectively, these methods underscore that buffer composition, not just buffer size, drives rehearsal effectiveness.Source: §2 · pp. 2–3

Feature-Space Buffer Construction

  • Unlike iCaRL’s herding, REMIND’s compressed hidden representations, and Rainbow Memory’s feature-space uncertainty, CARES uses an explicit coreset objective combining class-aware allocation and density-aware coverage.The compared methods use the continually trained network’s features, frozen early layers, or uncertainty, respectively.Source: §2 · p. 3

Coreset-Selection for Replay

  • Coreset selection formalizes which limited set of samples to retain, and recent replay methods optimize model-dependent proxies such as gradients, diversity, learned distributions, or reducible loss.These methods commonly incur repeated continual model-dependent computations.Source: §2 · p. 3

3 CARES

CARES constructs a fixed-capacity replay memory by combining retained and newly staged examples, embedding candidates with a frozen encoder, and selecting class-wise under a density-aware coverage criterion. This adapts geometric coreset selection to repeated online updates.

3.1 SETTING AND PROBLEM DEFINITION

  • The ideal replay memory cannot be selected directly online because future tasks are unknown, the full history cannot be retained, and evaluating candidate memories would require repeatedly optimizing the continual model.Practical selection therefore relies on tractable proxies computed from data available at each update.Source: §3.1 · p. 4

3.2 CONTINUAL REPLAY-MEMORY CONSTRUCTION

  • At each update, CARES combines the previous replay memory with a bounded staging pool of newly observed examples, embeds the candidates with a fixed encoder, and replaces the memory with selected candidates.Previously retained examples compete directly with new candidates; selection uses no predictions, losses, gradients, or parameters from the continually trained model.Source: §3.2 · p. 4

3.3 DENSITY-AWARE COVERAGE SELECTION

  • Within each class, CARES approximately maximizes density-weighted facility location under a cardinality constraint, using coverage neighborhoods and candidate density scores.The score downweights candidates with unusually large neighbor distances and those in highly redundant regions.Source: §3.3 · p. 5
  • CARES allocates the replay budget approximately equally across represented classes, giving an additional slot to classes with the largest candidate pools and selecting within each class.This prevents classes with more candidates from automatically dominating the global geometric objective.Source: §3.3 · pp. 5–6
  • On Split-CIFAR-100, CARES matches the strongest baseline on the balanced stream and achieves the highest final average accuracy under imbalance and label noise.The supplied table caption reports these comparisons for Table 1.Source: Table 1 · p. 6

4 EXPERIMENTS

The experiments assess CARES in online task- and class-incremental protocols, including balanced, imbalanced, and label-noisy streams. Results cover robustness, replay capacity, runtime, sensitivity, and component ablations.

4.1 EXPERIMENTAL SETUP

  • The experiments use online TIL and CIL protocols with different datasets, stream conditions, and replay capacities, reporting final average accuracy and forgetting.Protocol-specific comparisons retain the reference setting's baselines and configurations, so absolute accuracies are not compared across tables. The setup specifies single-pass streams with blurred task boundaries for Protocol 2 and ℓ2-normalized features from a frozen ImageNet-pretrained Inception-V3 encoder for memory construction.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2 ROBUSTNESS TO IMBALANCE AND LABEL NOISE

  • On Split-CIFAR-100, CARES reaches 54.88% under class imbalance, 7.58 percentage points above BCSR; on Split-Tiny-ImageNet, it leads all three stream conditions.On Tiny-ImageNet, its gains over BCSR are 5.36, 4.35, and 8.65 points on balanced, imbalanced, and label-noisy streams, respectively. CARES does not uniformly minimize FGTT; its primary advantage in Protocol 1 is improved final retained performance.Source: Table 2 · p. 7, §4.2 · p. 7

4.4 RUNTIME EFFICIENCY

  • At 0.39 hours on Split-Tiny-ImageNet, CARES is comparable to K-means Embedding and iCaRL at 0.41 hours, and is 2.3×, 6.8×, and 8.8× faster than GCR, BCSR, and OCS.Figure 1 presents the balanced Protocol 1 accuracy–runtime trade-off at K = 200; CARES achieves higher final accuracy than these methods.Source: §4.4 · p. 7, Figure 1 · p. 8

4.3 PERFORMANCE ACROSS MEMORY CAPACITIES

  • At K = 200, CARES reaches 6.60% on Split-TinyImageNet versus 6.19% for GCR; at K = 2000, it leads Split-CIFAR-10 and Split-CIFAR-100 with 54.88% and 25.17%.It is competitive on Split-CIFAR-10 and Split-CIFAR-100 at K = 200 and benefits strongly from added replay capacity.Source: §4.3 · p. 7, Table 3 · p. 8

4.5 SENSITIVITY ANALYSIS

  • On balanced Split-CIFAR-100, accuracy remains relatively stable across a broad range of γ values, while increasing the staging pool beyond a moderate multiple of K provides no consistent gain.The results indicate that CARES does not require precise tuning or an excessively large temporary candidate pool.Source: Table 4 · p. 8, §4.5 · p. 8
  • On Split-CIFAR-100 under Protocol 2, CARES exhibits similar performance across different pretrained encoders, indicating limited sensitivity to the feature extractor used for selection.Table 5 reports final class-incremental accuracy for the encoder variants.Source: §4.5 · p. 8, Table 5 · p. 9

4.6 ABLATION STUDIES

  • Under class imbalance, class-wise selection substantially outperforms both global variants; density weighting adds gains on balanced and label-noisy streams, though not uniformly.The ablation compares class-balanced random selection, global coverage, density-weighted global coverage, and independent per-class geometric selection.Source: Table 6 · p. 9, §4.6 · p. 9

5 LIMITATIONS AND FUTURE WORK

CARES is less consistently advantageous when replay capacity is very small, and it assumes access to a suitable frozen pretrained encoder. The staging pool also adds temporary memory beyond the permanent replay budget.

5 LIMITATIONS AND FUTURE WORK

  • CARES is less consistently advantageous under very small replay budgets, where learner-dependent signals can provide more targeted choices.CARES becomes increasingly competitive as replay capacity allows the selected set to represent feature-space geometry more fully.Source: §5 · p. 9
  • CARES assumes a suitable frozen pretrained encoder, and staging requires temporary memory beyond the permanent replay budget.In the evaluated setting, performance was relatively stable across representations, and moderate staging capacities were sufficient; excessively large candidate pools showed no consistent benefit.Source: §5 · p. 9

6 CONCLUSION

CARES selects replay exemplars through class-aware, density-weighted coverage in a frozen pretrained feature space, without continual-learner optimization signals. Across online task- and class-incremental settings, it achieves competitive or superior final accuracy while reducing selection cost.

6 CONCLUSION

  • CARES selects exemplars using class-aware, density-weighted coverage in a frozen pretrained feature space, without learner-derived losses, gradients, backward passes, or auxiliary training.Selection is decoupled from optimization of the continual learner.Source: §6 · pp. 9–10
  • Across online task- and class-incremental settings, CARES achieves competitive or superior final accuracy while substantially reducing memory-selection cost versus learner-dependent coreset methods.Its advantage is particularly pronounced under class imbalance, and relative performance improves as replay capacity increases.Source: §6 · p. 10
  • Class-aware allocation is critical for geometric selection under imbalanced streams, while density weighting provides complementary gains in balanced and label-noisy settings.Source: §6 · p. 10
  • Effective replay-buffer construction need not depend on continual-learner signals; stable pretrained representation geometry offers a computationally efficient alternative when selection repeats throughout an online stream.Source: §6 · p. 10

SUPPLEMENTARY MATERIAL

At each task, CARES maintains a bounded staging pool alongside the previous replay memory. When their combined candidates exceed capacity, it embeds them with a frozen feature extractor and selects a reconstructed memory using class-aware, density-weighted coverage.

A CARES ALGORITHM

  • Incoming examples enter a bounded task-specific staging pool through reservoir sampling, while learner training uses current examples and sampled items from the previous replay memory.Source: A CARES ALGORITHM · p. 13
  • At the memory update, CARES combines the previous replay memory with the staging pool; if the candidate set exceeds capacity K, it computes normalized frozen embeddings and applies CARES-SELECT.If the combined set is no larger than K, it becomes the replay memory directly.Source: A CARES ALGORITHM · p. 13
  • The procedure repeats this staging and memory-reconstruction cycle across tasks, returning the final replay memory alongside the continually updated learner.Source: A CARES ALGORITHM · p. 13

B EXPERIMENTAL PROTOCOL DETAILS

The experiments use two online continual-learning protocols with different task structures and evaluation conditions. Protocol 1 provides task identity at inference, whereas Protocol 2 evaluates class-incrementally without task identity.

B.1 PROTOCOL 1:

  • Protocol 1 divides Split CIFAR-100 into 20 five-class tasks and Tiny-ImageNet into 20 ten-class tasks, with replay capacities 100 and 200, respectively.It uses incoming batches of 50 and 100 samples, respectively; task identity is available at inference, and results average three runs.Source: §B.1 · pp. 13–14

B.2 PROTOCOL 2

  • Protocol 2 uses five tasks for Split CIFAR-10 and Split CIFAR-100 and ten tasks for Split-TinyImageNet, with 2, 20, and 20 classes per task, respectively.Streams contain 6,250 samples with blurred task boundaries and a single pass; evaluation is class-incremental without task identity, with replay capacities K ∈{200, 500, 2000} and results averaged over 15 runs.Source: §B.2 · p. 14

C ADDITIONAL RESULTS

The additional-results tables cover balanced, imbalanced, and label-noise evaluations on Split-CIFAR-100 and Split-Tiny-ImageNet, plus online CIL accuracy.

C ADDITIONAL RESULTS

  • Table 7 reports Split-CIFAR-100 results under balanced, imbalanced, and label-noise settings, using AT and FGTT.Higher AT and lower FGTT are better; results are mean ± standard deviation.Source: Table 7 · p. 14
  • Table 8 reports Split-Tiny-ImageNet results under balanced, imbalanced, and label-noise settings, with average accuracy and forgetting.Results are mean ± standard deviation; higher average accuracy and lower forgetting are better.Source: Table 8 · p. 15
  • Table 9 reports online class-incremental learning average accuracy over 15 seeds.Best results are shown in bold.Source: Table 9 · p. 15

D IMPLEMENTATION DETAILS

The implementation details specify shared experimental hardware and dataset-specific hyperparameter settings; scaling results compare persistent replay capacity with staging-pool size.

D IMPLEMENTATION DETAILS

  • Experiments used one NVIDIA RTX A6000 GPU and four CPU cores, with the same hardware configuration across methods and settings unless stated otherwise.Table 11 lists Protocol 2 hyperparameter settings.Source: D IMPLEMENTATION DETAILS · p. 15, Table 11 · p. 16
  • Protocol 1 hyperparameters are shared across stream conditions where applicable, with learning rate, weight decay, coverage, and staging-pool size selected separately by dataset and condition.Table 10 lists Protocol 1 settings.Source: Table 10 · p. 15

E.1 REPLAY AND STAGING-POOL SCALING

  • Increasing replay capacity from K = 200 to 1000 raises final CIL accuracy from 9.83% to 19.88% and TIL accuracy from 63.60% to 73.63%.At K = 500, increasing P from 200 to 1000 raises CIL accuracy from 16.06% to 16.72% and TIL accuracy from 69.26% to 70.12%, while memory-update overhead rises from 11.2 s to 23.5 s.Source: §E.1 · p. 16, Table 12 · p. 16

E.2 JOINT SENSITIVITY TO COVERAGE AND STAGING-POOL CAPACITY

On balanced Split CIFAR-100, CARES performs strongly across a broad range of coverage and staging-pool settings, with no consistent gain from very large pools.

E.2 JOINT SENSITIVITY TO COVERAGE AND STAGING-POOL CAPACITY

  • The sensitivity experiment fixes K = 100, learning rate at 0.02, and weight decay at 5 × 10−5 while varying γ and P.Coverage values are 0.2, 0.3, 0.4, 0.6, and 0.8; staging pools range from K to 20K.Source: §E.2 · p. 16
  • The highest mean accuracy occurs at γ = 0.4 and P = 2K, while neighboring configurations perform similarly.Strong performance spans a broad region rather than a narrow operating point.Source: §E.2 · pp. 16–17
  • Increasing P to 10K or 20K provides no consistent accuracy improvement over moderate staging-pool sizes.The figure reports final task-incremental accuracy for P relative to replay capacity K = 100.Source: §E.2 · p. 17, Figure 2 · p. 17

Related papers

Loading 2610.32799v1…