Source-linked AI summary
Orthogonal Gradient Descent for Continual Learning
Mehrdad Farajtabar, Navid Azizan, Alex Mott, Ang Li
TL;DR
Continual learning exposes catastrophic forgetting because sequential neural-network updates can overwrite knowledge from prior tasks when their data is unavailable. The paper introduces OGD, which projects new-task gradients into directions that minimally change previous predictions, and reports state-of-the-art performance across continual-learning benchmarks without storing historical data.
Problem
Sequential training can degrade performance on previously learned tasks when earlier data cannot be revisited, creating the problem of catastrophic forgetting.
Method
OGD maintains gradient directions of previous-task predictions and projects new-task loss gradients onto their orthogonal space before updating the network.
Results
OGD matches or exceeds other state-of-the-art methods across benchmark experiments, with particularly strong results on split MNIST.
Takeaways & Limitations
OGD retains information over many tasks without storing historical data, while using the neural network's capacity to learn new data with minimized interference.
Takeaways & Limitations
OGD's required storage grows with the number of tasks, and all evaluated methods fail considerably on dissimilar tasks such as MNIST rotations larger than 90 degrees.
Abstract
from arXiv · showhide
Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains. Whenever these problems require learning in a continual or sequential manner, however, neural networks suffer from the problem of catastrophic forgetting; they forget how to solve previous tasks after being trained on a new task, despite having the essential capacity to solve both tasks if they were trained on both simultaneously. In this paper, we propose to address this issue from a parameter space perspective and study an approach to restrict the direction of the gradient updates to avoid forgetting previously-learned data. We present the Orthogonal Gradient Descent (OGD) method, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the neural network output on previous task does not change and the projected gradient is still in a useful direction for learning the new task. Our approach utilizes the high capacity of a neural network more efficiently and does not require storing the previously learned data that might raise privacy concerns. Experiments on common benchmarks reveal the effectiveness of the proposed OGD method.
1 Introduction
Continual learning requires models to acquire new information while previously presented data is unavailable, but sequential neural-network training can cause catastrophic forgetting. OGD addresses this by modifying gradient updates to avoid interference with previous tasks.
- Continual learning requires acquiring new information while previously presented information is unavailable for retrieval.
- Sequential neural-network training typically degrades performance on previously trained tasks when earlier data cannot be revisited, a phenomenon called catastrophic forgetting.
- SGD-based optimizers depend on current minibatches and therefore produce gradients that are oblivious to knowledge from previous tasks.
- OGD preserves previous knowledge by projecting new-task loss gradients into directions orthogonal to gradient directions associated with prior predictions.
- Experiments on three continual-learning benchmarks report state-of-the-art performance without storing historical data.
2 Preliminaries
The preliminaries define sequential tasks, model outputs, empirical task loss, and the distinction between loss gradients and model gradients. Figure 1 illustrates gradient projection relative to a previous-task prediction gradient.
- Tasks arrive sequentially, with previous-task data inaccessible while the model trains on the current task.
- Each data point consists of an input x in R^d and a label y, while f(x; w) denotes the model prediction with parameters w in R^p.
- For classification, f(x; w) has c components, and f_j(x; w) is the logit associated with class j.
- The empirical risk for task t sums the differentiable per-example loss over the task's training set, with softmax cross entropy given as a common classification choice.
- The loss gradient and model gradient are distinct: the former differentiates loss, while the latter differentiates network predictions with respect to parameters.
- Figure 1 contrasts the original task-B gradient g with its projection ˜g onto the space orthogonal to a previous-task logit gradient.
3 Orthogonal Gradient Descent
Orthogonal Gradient Descent (OGD) mitigates catastrophic forgetting by constraining new-task updates to directions that minimally change previous-task predictions. It stores gradient directions from earlier tasks, projects new gradients onto their orthogonal complement, and retains a descent direction for the new task.
- Core idea: OGD constrains parameter updates to the space orthogonal to previous-task prediction gradients, minimizing changes to earlier outputs.Directions aligned with prediction gradients cause the largest local changes, while orthogonal directions cause the least change.
- Practical variants: OGD-GTL stores only the gradient of the ground-truth logit, while OGD-AVE stores the gradient of the average logits; both reduce storage by a factor of c.The experiments use OGD-GTL and store only a subset of gradients per task, such as 200 for MNIST.
- Gradient projection: New-task loss gradients are orthogonalized against an accumulated basis of previous model gradients before parameter updates are applied.The basis is constructed from gradients on earlier-task samples and logits, then new gradients are projected perpendicular to it.
- Gradient projection: The projected update remains a descent direction for the new task, so sufficiently small learning-rate steps reduce its loss while remaining perpendicular to the stored subspace.The paper derives ⟨−˜g, g⟩ = −∥˜g∥2 ≤ 0.
- Scope and limitation: The method extends to multiple sequential tasks and can be used when gradient steps should minimally interfere with previously learned data points.Its storage requirement grows with the number of tasks, and larger gradient collections eventually provide diminishing returns.
4 Experiments
Experiments on Permuted, Rotated, and Split Mnist evaluate OGD against continual-learning baselines under sequential task training. OGD generally preserves prior-task performance, matching or approaching multi-task learning and outperforming weaker baselines.
- Experimental setup: Experiments compare OGD with EWC, A-GEM, SGD, and multi-task learning across Permuted, Rotated, and Split Mnist benchmarks.All Mnist experiments use five epochs per task and a consistent architecture and training setup.
- Permuted Mnist: On three-task Permuted Mnist, OGD retains task 1 performance similarly to A-GEM, slightly below MTL and significantly above SGD and EWC.The evaluation uses three pixel permutations trained sequentially for five epochs each.
- Permuted Mnist: Across five Permuted Mnist tasks, OGD performs significantly better than EWC and SGD while remaining on par with A-GEM.Accuracy is measured after training on task 5 for examples from each task.
- Rotated Mnist: On two-task Rotated Mnist, OGD and A-GEM show similar performance as rotation angle increases and the task becomes harder.Accuracy is measured on original, unrotated Mnist after training on rotated digits.
- Rotated Mnist: Across five Rotated Mnist tasks, OGD outperforms other methods on 10, 20, and 30 degree rotations.The tasks use rotations from 0° through 40° in 10° increments.
- Split Mnist: On two-task Split Mnist, OGD performs best across five label partitions and beats A-GEM, while preserving the same performance ordering across configurations.Each task contains five disjoint labels, and accuracy is measured on task 1 after training on task 2.
- Split Mnist: On five two-class Split Mnist tasks, OGD performs very closely to MTL and consistently outperforms the other baselines.The evaluation covers multiple label partitions using a multi-headed architecture.
5 Related Work
Related continual-learning methods address catastrophic forgetting through architecture expansion, weight regularization, memory or generative replay, and gradient-based constraints. These approaches differ in whether they preserve old parameters, constrain updates, or retain information from previous tasks.
- Expansion methods allocate new neurons, layers, or modules for new tasks while preserving or reusing representations learned previously.
- Regularization methods constrain weight updates using task-importance estimates, including Fisher-information approximations and posterior-based uncertainty measures.
- Repetition-based methods replay stored previous data or samples from a generative model alongside current-task data to reduce forgetting.
- GEM treats previous-task episodic-memory losses as inequality constraints, preventing their increase while allowing them to decrease.
- Gradient-based approaches protect previous knowledge by shielding updates or restricting them to directions orthogonal to prior-task inputs, but they require stored data or a generative model.
- Continual learning is closely connected to meta-learning and few-shot learning, which use learned or data-driven biases to improve adaptation to new tasks.
6 Conclusion and Outlook
The paper concludes that OGD projects updates into directions that minimally interfere with prior predictions and performs strongly across continual-learning benchmarks. It also identifies storage, task dissimilarity, learning-rate sensitivity, and broader applicability as important considerations.
- OGD projects current gradient steps into the orthogonal space of predictions on previous data points to minimize interference while learning new tasks.
- OGD matches or exceeds other state-of-the-art methods across benchmark experiments and retains information over many tasks.
- Because gradients cannot be stored for full datasets, some forgetting remains; storing more or prioritizing important directions may improve performance but increases resource demands.
- All evaluated methods fail considerably on highly dissimilar tasks, while OGD can also be sensitive to learning rate and sometimes underperform A-GEM at large learning rates.
- OGD does not require explicit task identification and may apply to gradual task shifts or standard learning settings without many training epochs.
A.1 Variants of Orthogonal Gradient Descent
The appendix evaluates OGD variants by changing which output gradients are stored and how many are retained, with results reported for rotated and permuted Mnist.
- OGD-ALL stores gradients with respect to all model logits, OGD-AVG uses the average logit, and OGD-GTL uses the ground-truth logit.
- The experiment stores 20, 200, or 2000 gradients to assess how the collection size affects OGD performance.
- Table 4 reports task-1 accuracy after fully training on task 2, separately for rotated Mnist and permuted Mnist.
A.2 Increased Training Epochs
The appendix studies whether additional training epochs change continual-learning performance on permuted Mnist. The main Mnist experiments use five epochs per task, balancing training adequacy with avoidance of over-fitting.
- The study varies the number of training epochs to measure its effect on different methods in a two-task permuted-Mnist setting.
- Five epochs per task are used in the main Mnist experiments because they achieve 93% accuracy on vanilla Mnist while remaining short enough to avoid over-fitting.
- Figure 5 compares OGD and other methods as training epochs increase for each permuted-Mnist task.
A.3 Split Mnist
Across three additional split-MNIST instances, the relative ordering of methods remains unchanged despite different class partitions and task orders. OGD stays close to multitask learning, ahead of A-GEM, EWC, and SGD.
- Robustness across task arrangements: Across three additional split-MNIST instances, the method ranking is preserved despite different class splits and task presentation orders.The tests vary both how classes are assigned to tasks and the sequence in which tasks are presented.
- Method comparison: OGD and MTL are very close in performance across these split-MNIST tests.
- Method comparison: A-GEM follows OGD and MTL, while EWC and SGD rank lower in the reported comparisons.
- Evaluation setup: Tables 5, 6, and 7 report test accuracy after models are trained on all tasks sequentially.Each table uses a different ordering of Mnist classes assigned to tasks while retaining the same general settings as Table 3.