PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Networks in Null Space of Feature Covariance for Continual Learning

Shipeng Wang, Xiaorong Li, Jian Sun, Zongben Xu

arXiv:2103.07113v3cs.LG

TL;DR

Continual learning must balance plasticity for new tasks with stability on previous tasks, but deep networks suffer catastrophic forgetting. Adam-NSCL addresses this dilemma by projecting Adam updates into an approximate null space of previous-task features; experiments show it is effective and outperforms compared methods.

  • Problem

    Continual learning requires simultaneous plasticity for new knowledge and stability on previous tasks, yet deep networks suffer catastrophic forgetting.

  • Method

    Adam-NSCL projects Adam-generated parameter updates into the approximate null space of previous-task input features, using uncentered feature covariance matrices.

  • Results

    Adam-NSCL is effective and outperforms compared continual learning methods; on 10-split-CIFAR-100, it achieves the highest ACC 73.77% with competitive BWT -1.6%.

  • Takeaways & Limitations

    The approach provides a continual learning training algorithm for settings where previous-task datasets are unavailable.

  • Takeaways & Limitations

    The method uses an approximate null space because guaranteeing zero singular values is unrealistic, and further theoretical analysis is left for future work.

Abstract

from arXiv · show

In the setting of continual learning, a network is trained on a sequence of tasks, and suffers from catastrophic forgetting. To balance plasticity and stability of network in continual learning, in this paper, we propose a novel network training algorithm called Adam-NSCL, which sequentially optimizes network parameters in the null space of previous tasks. We first propose two mathematical conditions respectively for achieving network stability and plasticity in continual learning. Based on them, the network training for sequential tasks can be simply achieved by projecting the candidate parameter update into the approximate null space of all previous tasks in the network training process, where the candidate parameter update can be generated by Adam. The approximate null space can be derived by applying singular value decomposition to the uncentered covariance matrix of all input features of previous tasks for each linear layer. For efficiency, the uncentered covariance matrix can be incrementally computed after learning each task. We also empirically verify the rationality of the approximate null space at each linear layer. We apply our approach to training networks for continual learning on benchmark datasets of CIFAR-100 and TinyImageNet, and the results suggest that the proposed approach outperforms or matches the state-ot-the-art continual learning approaches.

1. Introduction

Continual learning requires networks to learn sequential tasks while retaining previous performance, creating a plasticity-stability dilemma. Adam-NSCL addresses this by projecting Adam updates into layer-wise approximate null spaces and performs competitively in benchmark evaluations.

  • Continual learning requires plasticity for new tasks and stability for previous tasks, but deep networks commonly suffer catastrophic forgetting.
  • Adam-NSCL trains without previous-task datasets by constraining each layer’s parameter update to the approximate null space of previous input-feature covariance.The approximate null space is obtained from singular vectors associated with the smallest singular values.
  • The method incrementally computes uncentered feature covariance after each task, avoiding the need to retain all previous input features.
  • Experiments on continual-learning benchmarks find Adam-NSCL effective and outperforming state-of-the-art continual-learning methods.
  • The paper develops theoretical conditions, presents the training algorithm, and evaluates its efficacy experimentally.

2. Related Work

Continual-learning methods address forgetting through regularization, distillation, replay, architecture changes, or update rules. Adam-NSCL instead uses approximate feature-covariance null spaces, avoiding replayed data and unstable matrix inversion associated with OWM.

  • Regularization-based methods penalize parameter changes and store importance weights for previous tasks.
  • Distillation-based methods use previous-task models as teachers, often relying on current, external, coreset, or synthetic data when old datasets are inaccessible.
  • Replay-based methods combine current data with selected or generated previous-task data, with selection potentially creating dataset imbalance.
  • Architecture-based methods expand or mask networks for new tasks, potentially producing computationally expensive and memory-intensive models.
  • Algorithm-based methods modify parameter updates; GEM and A-GEM require stored previous-task data, while OWM uses an unstable matrix inversion.
  • Adam-NSCL trains in the approximate null space of previous feature covariance, balancing plasticity and stability without regularizers, architecture revision, or replayed data.Compared with OWM, it uses theoretical and empirical null-space analysis and avoids unstable matrix inversion.

3. Analysis of Stability and Plasticity

The paper formulates stability and plasticity conditions for sequential training without previous-task data. Stability is enforced through layer-wise null spaces of uncentered feature covariance, while plasticity requires updates to remain aligned with current-task training directions.

  • 3.1. Preliminaries: The analysis models sequential tasks, network parameters, layer-wise updates, and input features, including convolution through matrix multiplication.
  • 3.2. Conditions for continual learning: Training on a current task without previous-task data makes simultaneously retaining old performance and learning new knowledge challenging.
  • 3.2. Conditions for continual learning: A parameter update in each layer’s null space preserves the network outputs on previous-task data across sequential training.
  • 3.2. Conditions for continual learning: The exact feature-null-space condition is memory-prohibitive, motivating covariance-based approximation and incremental maintenance of layer-wise covariance matrices.
  • 3.2. Conditions for continual learning: Condition 1 replaces storing all previous features with the null space of their uncentered feature covariance matrix.The covariance has fixed layer-dependent size, unlike the feature matrix whose row count grows with seen data.
  • 3.2. Conditions for continual learning: If Condition 1 holds, performance on previous-task data is not degraded after learning the current task.
  • 3.2. Conditions for continual learning: Condition 2 requires a positive inner product between the parameter update and the current-task update direction, ensuring that training loss decreases.

4. Network Training in Covariance Null Space

Adam-NSCL trains each new task by projecting Adam-generated parameter updates into layer-wise approximate null spaces derived from accumulated feature covariances. The method incrementally updates these covariances, uses SVD to identify small-singular-value directions, and balances stability with plasticity through the approximation threshold.

  • Adam-NSCL projects each candidate Adam update into the approximate null space of accumulated feature covariance at every network layer.This layer-wise projection is designed to balance stability on previous tasks with plasticity for the current task.
  • The algorithm incrementally updates uncentered feature covariance matrices after each task using features extracted at every linear layer.For a new task, features from the preceding task are accumulated and combined with the total number of seen data.
  • 4.1. Approximate null space: SVD identifies singular vectors associated with sufficiently small singular values, whose range space defines the approximate null space used for projection.The threshold is selected relative to the smallest singular value through hyperparameter a.
  • 4.1. Approximate null space: The exact null-space condition may be too strict because zero singular values are not guaranteed, motivating an approximate null space based on small singular values.The approximation is framed through principal-component directions and the residual contribution of the selected small-singular-value subspace.
  • 4.1. Approximate null space: For pretrained ResNet-18 on ImageNet ILSVRC 2012, all layer-wise proportions R are below 0.05, supporting the approximate-null-space assumption.The small-singular-value components contribute negligibly under the reported threshold a = 50.
  • The hyperparameter a controls the stability–plasticity trade-off by changing the size of the approximate null space.Larger a enlarges the update space and increases plasticity, but can weaken the null-space assumption and reduce stability.

5. Experiments

Experiments evaluate Adam-NSCL on three sequential-task benchmarks against diverse continual-learning methods, finding strong accuracy, competitive forgetting, and support for its approximate-null-space rationale. Additional analyses examine stability, plasticity, and training-loss retention.

  • Experimental settings: Experiments use 10-split-CIFAR-100, 20-split-CIFAR-100, and 25-split-TinyImageNet, with the latter forming a longer sequence of 25 tasks.The CIFAR-100 tasks have disjoint classes; TinyImageNet uses 64×64 RGB images.
  • Experimental settings: Evaluation reports average accuracy (ACC) and backward transfer (BWT), where negative BWT measures forgetting on previous tasks.With similar ACC, larger BWT is better.
  • Comparative results: 73.77% ACC and -1.6% BWT are achieved by Adam-NSCL on 10-split-CIFAR-100, with the highest ACC among compared methods.GEM and A-GEM have better BWT but substantially lower ACC values of 49.48% and 49.57%.
  • Comparative results: 58.28% ACC and -6.05% BWT are achieved by Adam-NSCL on 20-split-CIFAR-100, with the best ACC and competitive BWT.MUC-MAS, SI, and OWM have marginally better BWT but 16.68%, 13.01%, and 8.3% lower ACC, respectively.
  • Comparative results: Adam-NSCL achieves 4.88% and 7.48% larger ACC than OWM on 10-split- and 20-split-CIFAR-100, respectively, with similar BWT.On 25-split-TinyImageNet, it has significantly better ACC and comparable BWT than OWM.
  • Stability and plasticity analysis: As the approximation parameter a increases, BWT worsens while ACC first increases and then decreases in the middle and bottom Figure 5 panels.The analysis varies a over 10, 20, 30, 40, and 50.
  • Evolution of training loss: Training losses on previous tasks are retained after learning new tasks, supporting stability on sequential 10-split-CIFAR-100 tasks.Figure 6 tracks losses for T1, T2, and T3 after subsequent task learning.

6. Conclusion

The paper addresses continual learning when previous-task datasets are inaccessible by proposing Adam-NSCL, which combines theoretical stability and plasticity conditions with null-space-constrained updates. Extensive experiments report that the algorithm outperforms compared continual-learning methods, while future work targets better null-space approximation and theoretical analysis.

  • Adam-NSCL addresses the plasticity-stability dilemma when datasets from previous tasks are inaccessible.
  • The method proposes theoretical conditions for stability and plasticity during sequential network training.
  • Adam-generated candidate updates are projected into the approximate null space of previous tasks’ uncentered feature covariance matrix.
  • Extensive experiments show that Adam-NSCL outperforms the compared continual-learning methods.
  • Future work will improve null-space approximation and conduct theoretical analysis of the algorithm.

Appendix A

Appendix A states and proves a lemma showing that null-space parameter updates preserve previous-task outputs during training. The proof uses the network’s recursive structure and iterative training process across layers and steps.

  • Lemma 1 states that updates lying in each layer’s input-feature null space preserve previous-task behavior during training.
  • The proof initializes current-task parameters with the previous parameters and establishes the result for the first training step.
  • The recursive network structure propagates preservation from earlier linear layers through later layers.
  • Repeating the argument for subsequent steps shows that previous-task features and outputs remain unchanged throughout training.

Appendix B

Appendix B explains why the plasticity condition makes a parameter update a descent direction. Using Taylor’s theorem, it shows that the loss decreases for sufficiently small positive learning rates.

  • Condition 2 requires the inner product between the parameter update and gradient-descent update to be positive.
  • The proof analyzes the loss after updating parameters by defining it as L(w_t,s − α∆w_t,s).
  • Taylor’s theorem establishes the local loss change as the learning rate approaches zero.
  • For all sufficiently small α in (0, ᾱ), the updated loss is lower than the current loss.

Appendix C

Appendix C invokes Kronecker-product properties to establish the stated expression involving vectorized layer updates, identity matrices, and gradient terms.

  • The proof uses properties of the Kronecker product to derive the expression claimed in Section 4.1.
  • The notation defines vec(·) as vectorization, I as the identity matrix, and ⊗ as the Kronecker product.

Appendix D

The paper distinguishes Adam-NSCL from OWM through its feature-covariance null space, incremental covariance updates, and different stability–plasticity trade-offs. It reports higher ACC than OWM across three benchmarks with comparable BWT.

  • Algorithmic differences: Adam-NSCL defines its null space from the feature covariance matrix, whereas OWM uses a projection matrix formed from previous features.The covariance-based null space is accumulated after each task.
  • Algorithmic differences: Adam-NSCL incrementally updates its covariance matrix without approximation error, while OWM updates its projection using recursive least squares.OWM’s matrix inversion introduces an approximation error that can accumulate.
  • Hyperparameter behavior: Adam-NSCL uses hyperparameter a to approximate the covariance null space and balance stability and plasticity, whereas OWM requires layer-wise tuning of β_l.The paper reports that a is easy to set, while β_l is difficult to tune for OWM.
  • Experimental comparison: 4.88%, 7.48% and 8.3% higher ACC than OWM are reported across three benchmarks, with comparable BWT.The paper also refers to an experimental comparison on ImageNet with deeper networks.
Loading 2103.07113v3…