PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Learning in Low-rank Orthogonal Subspaces

Arslan Chaudhry, Naeemullah Khan, Puneet K. Dokania, Philip H. S. Torr

arXiv:2010.11635v2cs.LG

TL;DR

Continual learning seeks to retain knowledge across sequential tasks, but shared vector spaces can permit interference and forgetting. The paper assigns tasks to orthogonal low-rank subspaces and preserves gradient relationships through isometric mappings optimized on the Stiefel manifold, reporting improvements over experience-replay baselines on standard classification benchmarks.

  • Problem

    Continual learning must remember sequentially arriving tasks, while existing methods still learn tasks in a joint vector space that the paper associates with interference and forgetting.

  • Method

    ORTHOG-SUBSPACE projects tasks into mutually orthogonal low-rank subspaces and uses isometric transformations learned on the Stiefel manifold to maintain gradient orthogonality.

  • Results

    ORTHOG-SUBSPACE improved considerably over strong memory replay-based baselines across standard continual-learning image-classification benchmarks, with gains reported both with and without memory.

  • Takeaways & Limitations

    Learning tasks in orthogonal subspaces is reported as an effective strategy for reducing interference among tasks in continual learning.

  • Takeaways & Limitations

    Continual learning assumes zero forgetting, but partially forgetting older, more biased data may be acceptable or preferable in some scenarios.

Abstract

from arXiv · show

In continual learning (CL), a learner is faced with a sequence of tasks, arriving one after the other, and the goal is to remember all the tasks once the continual learning experience is finished. The prior art in CL uses episodic memory, parameter regularization or extensible network structures to reduce interference among tasks, but in the end, all the approaches learn different tasks in a joint vector space. We believe this invariably leads to interference among different tasks. We propose to learn tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Further, to keep the gradients of different tasks coming from these subspaces orthogonal to each other, we learn isometric mappings by posing network training as an optimization problem over the Stiefel manifold. To the best of our understanding, we report, for the first time, strong results over experience-replay baseline with and without memory on standard classification benchmarks in continual learning. The code is made publicly available.

1 Introduction

Continual learning must preserve knowledge across sequentially arriving tasks, but existing approaches still learn tasks in a shared vector space where interference can cause forgetting. ORTHOG-SUBSPACE instead assigns tasks orthogonal subspaces and reports gains over experience-replay baselines.

  • Motivation: Continual learning aims to remember observed tasks while adapting to a sequential stream of new tasks.Sequential task arrival creates challenges unlike standard single-task supervised learning.
  • Prior approaches: Existing approaches reduce forgetting through parameter regularization, modular network components, or related training modifications.Modular approaches add components as tasks arrive but rely on knowing the correct module at test time.
  • Proposed approach: ORTHOG-SUBSPACE learns different tasks in mutually orthogonal, low-rank vector subspaces to reduce interference.Task-specific projection matrices have rank r with r ≪ m and are generated offline.
  • Results: 13% higher average accuracy and 66% lower forgetting are reported versus the strongest experience-replay baseline in a three-layer network.These improvements apply to the shallower-network setting described in the introduction.
  • Results: 8% higher average accuracy and 50% lower forgetting are reported than the strong experience-replay baseline on deeper networks with the same episodic memory.The figure illustrates orthogonal subspaces and reduced overlap across network layers.

2 Background

The paper formalizes continual learning as sequential, mostly one-pass classification with limited replay and introduces orthogonal projections and isometric, Stiefel-manifold-constrained optimization. The optimization projects gradients onto a tangent space and updates weights along a curve that remains on the manifold.

  • Continual learning setup: The continual learner receives task-labeled examples sequentially and can retain only a few samples from each task in a replay buffer.The predictor combines a feature extractor with a classifier and is evaluated over the task sequence.
  • Model: The feature extractor is an L-layer feed-forward neural network parameterized by layer weight matrices, followed by a classifier mapping features to outputs.The feature space is H and the loss is defined over predictions and targets.
  • Continual learning setup: The setup focuses on classification streams experienced once, with changing input or output distributions and a task descriptor available at training and test time.The task descriptor identifies the correct classification head.
  • Metrics: Average accuracy and average maximum forgetting are the two reported continual-learning statistics.Average maximum forgetting measures each task’s performance decrease from its peak to the end of the experience.
  • Orthogonal subspaces: Orthogonal projections map features into low-dimensional subspaces, while orthonormal columns simplify the projection matrix to P = OO^⊤.A rank-r projection uses r ≪ m dimensions within the m-dimensional feature space.
  • Isometries: An isometry preserves distances and angles, and composing two isometries yields another isometry.The paper uses these properties to motivate transformations that preserve relationships between gradients.
  • Stiefel manifold: Network training constrains weight matrices by W_l^⊤W_l = I, defining a Stiefel manifold for the feasible weights.For convolutional layers, the weights are reshaped into matrix form before applying the constraint.
  • Optimization: Stiefel-manifold optimization first projects the gradient onto the tangent space, then follows a Cayley-transform descent curve that remains on the manifold.The Cayley transform uses a curve-length parameter and can be approximated by fixed-point iteration without expensive matrix inversion.

3 Continual Learning in Orthogonal Subspaces

ORTHOG-SUBSPACE assigns tasks to distinct low-rank orthogonal subspaces, then preserves orthonormal network transformations during training to reduce interference. Task projections separate gradients at the projected layer, while Stiefel-manifold updates keep earlier-layer transformations isometric and empirically reduce cross-task gradient overlap.

  • Task-specific orthogonal subspaces: ORTHOG-SUBSPACE projects each task's feature representation into a distinct low-dimensional subspace using task-specific rank-r projection matrices.The subspaces are formed from disjoint orthonormal basis vectors, with r ≪ m; projections can be generated offline and associated with task identifiers.
  • Gradient interference: At the projected layer, task gradients are orthogonal by construction, eliminating gradient interference there.Backpropagation can reduce orthogonality in earlier layers, especially for deeper networks.
  • Training procedure: Training combines current-task and replay-buffer gradients before applying layer-wise Stiefel-manifold updates and updating the classifier head.The algorithm samples a past task and example, sums current and past gradients, updates each feature-extractor layer, and stores current samples in a ring buffer.
  • Isometric gradient propagation: Isometric transformations preserve inner products across layers, keeping gradients from different tasks near-orthogonal beyond the projected layer.The approach therefore uses orthonormal weight matrices throughout the continual learning experience.
  • Stiefel-manifold training: The method preserves orthonormal weights by projecting effective gradients onto each layer's tangent space and retracting updates with an iterative Cayley transform.This optimization is performed on the Stiefel manifold while the classifier head receives a standard gradient update.

4 Experiments

The experiments evaluate ORTHOG-SUBSPACE on four supervised continual-learning classification benchmarks against regularization, memory, and other baselines. Across shallow and deep networks, orthogonal subspaces improve accuracy and reduce forgetting, with the strongest configuration combining projection, memory, and orthogonalization.

  • Benchmarks and metrics: The evaluation covers Permuted MNIST, Rotated MNIST, Split CIFAR, and Split miniImageNet using average accuracy and forgetting.The MNIST tasks apply pixel permutations or image rotations, while Split CIFAR and Split miniImageNet assign five random classes to each task.
  • Shallow networks: On Permuted and Rotated MNIST without memory, ORTHOG-SUBSPACE gains 7.1% and 9.2% average accuracy over ER-Ring and reduces forgetting by 66% and 42%.These comparisons concern relatively shallower networks and absolute average-accuracy gains.
  • Deeper networks: On CIFAR100 and miniImageNet with memory and preserved orthogonality, ORTHOG-SUBSPACE gains 4.7% and 1.6% average accuracy over ER-Ring and reduces forgetting by 50% and 16.6%.The reported comparisons concern deeper networks and tiny episodic memory.
  • Memory size: Increasing episodic-memory size narrows the performance gap between ER-Ring and ORTHOG-SUBSPACE, because relearning from large memory can sufficiently mitigate forgetting.The authors state that their conclusions for tiny memory remain valid.
  • Ablation study: The ablation achieves its best accuracy and forgetting performance when memory is combined with subspace projection and Stiefel-manifold orthogonalization.Projection alone is competitive but remains below ER-Ring, while adding memory improves performance before orthogonalization provides the best result.
  • Gradient analysis: With Stiefel-manifold weights, gradient inner products are more concentrated around zero than without the constraint.This empirically supports the claim that isometric transformations keep gradients from different tasks near-orthogonal across layers.

5 Related work

Continual learning methods address forgetting through regularization, modularity, or memory, but these strategies face brittleness, memory growth, or impractical gradient-storage costs. ORTHOG-SUBSPACE instead targets task interference through orthogonal subspaces and isometric transformations.

  • Gradient interference: Figure 3 compares gradient inner-product histograms across layers, where distributions farther left indicate more orthogonality and less interference.
  • Existing approaches: Continual learning literature includes regularization, modular, and memory-based approaches for reducing forgetting.Regularization limits parameter drift, modular methods add task-specific components, and memory methods replay stored examples.
  • Existing approaches: Regularization methods can be brittle for many tasks and may require multiple passes over each task dataset.Their parameter-importance locality assumption can be violated, and their effectiveness may depend on repeated dataset access.
  • Existing approaches: Modular approaches can achieve zero forgetting but require memory that increases with the number of tasks.
  • Comparison with OGD: OGD stores task-gradient information whose size scales with network size, whereas ORTHOG-SUBSPACE stores lower-dimensional subspace matrices and a small replay buffer.ORTHOG-SUBSPACE also projects the feature-extraction layer and maintains orthogonality through Stiefel-manifold learning rather than subtracting gradient projections at every step.

6 Conclusion

The paper concludes that ORTHOG-SUBSPACE learns tasks in mutually orthogonal subspaces while using isometric transformations to preserve gradient orthogonality. It reports considerable improvement over memory-replay baselines on standard image-classification continual-learning benchmarks.

  • Conclusion: ORTHOG-SUBSPACE learns different tasks in orthogonal subspaces.
  • Conclusion: Isometric transformations keep gradients in the projected layer orthogonal in earlier layers.
  • Conclusion: The transformations are learned by posing network training as optimization over the Stiefel manifold.
  • Conclusion: ORTHOG-SUBSPACE improved considerably over strong memory replay-based baselines on standard continual-learning image-classification benchmarks.

7 Broader Impact

The broader impact discussion notes that continual learning can support learning from new data without constant retraining, but its suitability depends on whether retaining all older information is required. In some settings, forgetting older and potentially more biased data may be acceptable or preferable.

  • Potential benefits: Continual learning can support efficient learning on new data without constant retraining on previous data.
  • Potential benefits: Continual learning is useful when models operate across multiple environments where simultaneous retraining on all environments is infeasible.
  • Scope and suitability: Some applications may accept or prefer partial forgetting when older data is more biased than recent data.
  • Scope and suitability: Practitioners should use continual-learning approaches only when zero forgetting of previous data is suitable for the application.

Appendix

The appendix contains a proof that isometry preserves angles, a derivation of the Stiefel-manifold tangent-space gradient projection, additional experiments, and the hyperparameter grid.

  • Appendix contents: Appendix A proves that isometry preserves angles.
  • Appendix contents: Appendix B derives the closed-form gradient projection on the tangent space of the Stiefel manifold.
  • Appendix contents: Appendix C provides further experimental results.
  • Appendix contents: Appendix D lists the hyperparameter grid considered in the experiments.

A Isometry Preserves Angles

The supplied passages establish that isometries preserve distances and inner products, and they characterize tangent-space structure and gradient projection on the Stiefel manifold.

  • A Isometry Preserves Angles: An isometry preserves inner products, and preserving inner products is equivalent to being an isometry.The theorem states the equivalence directly, while the proof derives distance preservation from inner-product preservation.
  • A Isometry Preserves Angles: The Stiefel manifold consists of n × p matrices X satisfying X⊤X = I, with n > p.Its columns are p orthonormal vectors in Rn.
  • A Isometry Preserves Angles: A matrix Z = XA + X⊥B lies in the tangent space at X if and only if A is skew-symmetric.The decomposition separates components within the span of X from components in its orthogonal complement.
  • A Isometry Preserves Angles: Under the canonical inner product, the tangent-space projection of gradient G at X is πTX(G) = AX, where A = GX⊤ − XG⊤.The proof derives this projection by decomposing G and tangent vectors in the X and X⊥ bases.

C More Results

Table 3 reports continual-learning accuracy and forgetting for larger episodic-memory settings on Split CIFAR and Split miniImageNet.

  • C More Results: Table 3 reports both Accuracy (2) and Forgetting (3) results.The table covers continual-learning experiments with larger episodic memory sizes.
  • C More Results: The experiments store 2, 3, or 5 samples per class per task.These memory settings are reported for the larger episodic-memory comparison.
  • C More Results: The top table is for Split CIFAR, while the bottom table is for Split miniImageNet.

D Hyper-parameter Selection

The experiments use benchmark-specific hyper-parameter grids, with selected values reported in parentheses; the supplied figure passage concerns gradient inner products rather than selection criteria.

  • D Hyper-parameter Selection: The hyper-parameter section reports grids and places the best values for each benchmark in parentheses.This describes the selection-reporting convention used for the experiments.
  • D Hyper-parameter Selection: Multitask, Finetune, EWC, AGEM, and MER are tuned over learning-rate grids spanning 0.003 to 1.0.EWC additionally uses a regularization grid from 0.1 to 1000.
Loading 2010.11635v2…