Source-linked AI summary
Gradient based sample selection for online continual learning
Rahaf Aljundi, Min Lin, Baptiste Goujaud, Yoshua Bengio
TL;DR
Continual learning needs to control catastrophic forgetting while processing non-stationary streams without reliable task boundaries or i.i.d. assumptions. This paper selects a fixed replay-buffer subset by reducing continual-learning constraints, using parameter gradients to measure sample diversity and a greedy alternative for efficient selection. The resulting strategies perform comparably to, or better than, methods relying on task boundaries and offer an advantage on blurry or imbalanced streams.
Problem
Continual-learning replay buffers often depend on task boundaries and i.i.d. assumptions, while exact constraint selection requires computation and storage that grow with the number of previous examples.
Method
The paper formulates replay-buffer population as constraint selection, approximates solid-angle minimization with a surrogate based on gradient diversity, and develops a greedy selection alternative.
Results
The proposed selection strategies perform comparably to or better than task-boundary-based methods and improve over reservoir sampling by up to 15% in an imbalanced disjoint-MNIST setting.
Takeaways & Limitations
Gradient-space diversity can support fixed-size replay-buffer selection without task boundaries, including settings with blurry boundaries or imbalanced data.
Takeaways & Limitations
The approach assumes the feasible constraint intersection is generally nonempty and does not consider outliers; its solid-angle objective is also difficult to minimize directly.
Abstract
from arXiv · showhide
A continual learning agent learns online with a non-stationary and never-ending stream of data. The key to such learning process is to overcome the catastrophic forgetting of previously seen data, which is a well known problem of neural networks. To prevent forgetting, a replay buffer is usually employed to store the previous data for the purpose of rehearsal. Previous works often depend on task boundary and i.i.d. assumptions to properly select samples for the replay buffer. In this work, we formulate sample selection as a constraint reduction problem based on the constrained optimization view of continual learning. The goal is to select a fixed subset of constraints that best approximate the feasible region defined by the original constraints. We show that it is equivalent to maximizing the diversity of samples in the replay buffer with parameters gradient as the feature. We further develop a greedy alternative that is cheap and efficient. The advantage of the proposed method is demonstrated by comparing to other alternatives under the continual learning setting. Further comparisons are made against state of the art methods that rely on task boundaries which show comparable or even better results for our method.
1 Introduction
Continual learning must prevent catastrophic forgetting in non-stationary streams, but major approaches rely on task boundaries or assumptions that do not hold in fully online settings. The paper formulates replay-buffer population as constraint selection and proposes methods designed for streams without task boundaries or i.i.d. assumptions.
- Catastrophic forgetting is the central challenge in continual learning, motivating methods for storing and reusing information from previous data.
- Prior-focused methods regularize parameters, but parameters can drift from earlier feasible regions over long chains of similar tasks.
- Parameter-isolation methods assign different parameters to different tasks, including dynamic architectures that freeze or grow the network.
- Both prior-focused and parameter-isolation strategies are limited for general online streams because task boundaries and i.i.d. assumptions are often unavailable.
- The paper formulates replay-buffer population as constraint selection, proposes a surrogate objective and greedy algorithm, and evaluates these strategies against alternative selection methods.
2 Related Work
Replay-based continual-learning methods retain prior examples for rehearsal or constraint construction, but prominent approaches allocate memory by task and therefore require task boundaries. Reservoir sampling addresses settings without such information, motivating more general buffer-selection strategies.
- Replay-based methods store previous examples directly in a buffer or through a generative model, then replay them for rehearsal or use them as constraints.
- iCaRL and GEM allocate memory to store task-specific core sets, requiring task boundaries to divide storage evenly across tasks.
- Reservoir sampling has been used when task boundaries and i.i.d. assumptions are unavailable, providing a comparison point for more general replay-buffer selection.
3 Continual Learning as Constrained Optimization
The paper casts online continual learning as constrained optimization and reduces an ever-growing constraint set through fixed replay-buffer selection. It approximates the feasible region by minimizing solid angle, using gradient diversity as a surrogate and a cheaper greedy alternative.
- Online learning minimizes current-example loss without increasing losses on previously learned examples, under a non-stationary stream without an i.i.d. assumption.
- The constraint set grows linearly with prior examples, making exact optimization increasingly costly in computation and storage and motivating a fixed replay buffer.
- Replay-buffer population is formulated as selecting M examples whose reduced feasible region approximates the region defined by all previous-example constraints.
- The feasible region is an intersection of gradient-defined half-spaces forming a polyhedral convex cone, and solid angle measures its size for approximation.
- The surrogate increases pairwise gradient angles, is empirically monotonic with solid angle, and is equivalent to maximizing gradient-direction variance and sample diversity.
- A greedy heuristic preserves diverse examples using cosine-similarity scores, offering a cheaper alternative to integer quadratic programming for fixed-buffer selection.
4 Experiments
The experiments evaluate sample-selection strategies in online continual learning without task-boundary information across three benchmarks, including imbalanced streams. The proposed gradient-based methods are compared with random, clustering, IQP, reservoir, and task-aware replay baselines.
- Benchmarks: The evaluation uses disjoint MNIST, permuted MNIST, and disjoint CIFAR-10, with test accuracy reported after sequential data streams.Disjoint MNIST and CIFAR-10 split labels into five tasks, while permuted MNIST uses ten pixel permutations.
- Compared Methods: The study compares random selection, feature- and gradient-space online clustering, GSS-IQP, and GSS-Greedy under fixed buffer sizes.GSS-IQP solves an integer quadratic programming surrogate, while GSS-Greedy is designed as a cheaper alternative.
- Compared Methods: GSS-Greedy reduces online selection cost because it avoids re-processing all recent and buffered samples.Its main computational cost is estimating gradients for selected candidates.
- Performance of Sample Selection Methods: Both proposed selection methods stably outperform alternative buffer sizes across the evaluated benchmarks, while GSS-Greedy is especially strong on disjoint CIFAR-10.GSS-Clust performs comparably to or favorably against feature clustering on permuted MNIST.
- Comparison with Reservoir Sampling: On imbalanced disjoint MNIST streams, the proposed strategies outperform reservoir sampling, with reported improvement reaching 15%.The streams contain one task with 2,000 examples and other tasks with 200 each; reservoir sampling suffers on under-represented tasks.
- Comparison with State-of-the-art Task Aware Methods: Against task-aware replay methods, the proposed approach performs similarly despite using no task information.The comparison includes GEM and iCaRL, which use task boundaries or task-specific replay organization.
5 Conclusion
The paper presents gradient-space sample selection for online continual learning without task boundaries, using a greedy approach that efficiently represents previously seen data.
- The method selects a finite set of samples that represents previously seen data without requiring task-boundary information.
- The proposed greedy selection approach targets diversity in gradient space and constantly outperforms other selection strategies.
- The selection strategy performs as well as task-boundary-based algorithms when choosing representative examples.
- The method offers an advantage when task boundaries are blurry or data are imbalanced.
A Clarifications of points in the main paper
The paper estimates the solid-angle objective through sampling and replaces direct solid-angle minimization with an integer quadratic surrogate based on normalized gradient similarities.
- The solid angle is estimated by sampling Bernoulli random variables because no tractable formula is available.With N = 10^9 variables, the estimated confidence interval has length around 10^-4.
- The surrogate formulation selects a fixed set of samples that minimizes the solid angle formed by the constraints.
- The surrogate minimization is solved as an integer quadratic programming problem using normalized gradient similarities.The normalized gradient matrix is G = ⟨g_i,g_j⟩/(∥g_i∥∥g_j∥).
- The selection vector X encodes the chosen samples, with selected samples corresponding to entries equal to 1.
B.1 Performance under blurry task boundary
Under blurry task boundaries in disjoint CIFAR10, the experiments swap examples across tasks and report that both proposed methods outperform other selection strategies.
- The blurry-boundary setting retains 90% of each task’s data and introduces 10% from other tasks.The experiment uses a buffer of 500 samples and trains on 1k samples per task.
- Table 5 reports each task’s accuracy at the end of the sequence for disjoint CIFAR10 with blurry task boundaries.
- Both proposed methods perform better than the other selection strategies in this blurry-boundary experiment.
B.2 Constrained Optimization Compared to Rehearsal
The constrained GSS-IQP update improves accuracy over rehearsal but is computationally expensive, while rehearsal with a larger buffer is cheaper and achieves similar results.
- The constrained optimization becomes infeasible with large buffers because gradients for buffer samples must be estimated and projected at each learning step.
- Figure 5 presents a greedy sample-selection ablation study using test accuracy.
- GSS-IQP(Constrained) improves over GSS-IQP(Rehearsal) by 3–5% but requires longer training because its cost scales polynomially with buffer size.
- GSS-IQP(Rehearsal) with buffer size 200 reaches 78.9%, compared with 76.26% for GSS-IQP(Constrained) with buffer size 100.
B.3 Effect of n in Greedy Sample Selection
GSS-Greedy scores new samples by comparing their gradients with gradients from buffered samples, and n controls the accuracy–cost trade-off in this estimate.
- GSS-Greedy computes each new sample’s score from cosine similarities between its gradient and n gradients sampled from the buffer.The score measures similarity to samples already stored in the replay buffer.
- 67.3 average test accuracy occurs at n = 1 on the disjoint MNIST sequence because the similarity estimate is highly noisy.Noisy scores cause new samples to enter the buffer more often, increasing forgetting of previous tasks.
- Increasing n generally improves average test accuracy by producing a better similarity estimate.
- Very large n values slightly reduce average test accuracy because high rejection rates favor representative samples from earlier tasks.
- The experiments use n = 10 as a compromise between score approximation quality and computational cost.
B.4 Batch size effect.
Batch size affects online continual-learning performance: larger batches reduce parameter updates, while very small batches make update estimates noisier.
- The experiment evaluates GSS-Greedy on disjoint MNIST with buffer size M = 300 across different batch sizes.Batch size is tested as an online continual-learning setting in which each task receives one pass through the training data.
- Large batch sizes reduce parameter updates and impair performance on learned samples.
- Very small batch sizes produce noisier parameter-update estimates.