PaperlayerView exact revision on arXiv

Source-linked AI summary

Gradient Projection Memory for Continual Learning

Gobinda Saha, Isha Garg, Kaushik Roy

arXiv:2103.09762v1cs.LGcs.CV

TL;DR

Artificial neural networks forget earlier tasks because unconstrained updates alter previously learned information. The paper uses SVD on activations to store compact Gradient Projection Memory bases and constrains new gradients orthogonally to important past-task subspaces. Across diverse image-classification sequences, this approach achieves high accuracy while largely eliminating forgetting, but fixed network capacity limits very long sequences of dissimilar tasks.

  • Problem

    Continual learning requires neural networks to acquire new tasks without forgetting past tasks, a capability existing approaches address through growth, weight importance, or old-data replay.

  • Method

    GPM uses SVD on post-task network activations to obtain bases of important gradient subspaces and constrains new learning to orthogonal gradient directions.

  • Results

    Across diverse image-classification tasks and network architectures, GPM reports high classification performance while mitigating forgetting and using memory efficiently.

  • Takeaways & Limitations

    Orthogonal updates to activation-derived subspaces preserve past information while retaining learnability for new tasks.

  • Takeaways & Limitations

    With fixed network capacity, dissimilar tasks populate GPM rapidly, limiting learning when its gradient space reaches capacity.

Abstract

from arXiv · show

The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems. Existing approaches to enable such learning in artificial neural networks usually rely on network growth, importance based weight update or replay of old data from the memory. In contrast, we propose a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks. We find the bases of these subspaces by analyzing network representations (activations) after learning each task with Singular Value Decomposition (SVD) in a single shot manner and store them in the memory as Gradient Projection Memory (GPM). With qualitative and quantitative analyses, we show that such orthogonal gradient descent induces minimum to no interference with the past tasks, thereby mitigates forgetting. We evaluate our algorithm on diverse image classification datasets with short and long sequences of tasks and report better or on-par performance compared to the state-of-the-art approaches.

1 INTRODUCTION

Continual learning exposes catastrophic forgetting because unconstrained gradient updates alter representations learned for earlier tasks. The paper proposes Gradient Projection Memory, which constrains new updates using activation-derived gradient subspaces without retaining old data or gradients.

  • Motivation: Unconstrained gradient-based optimization changes learned representations for new task distributions, causing catastrophic forgetting.The problem arises when updates minimize the current task objective without constraints.
  • Existing approaches: Regularization methods preserve past knowledge through weight penalties, but their performance suffers on longer task sequences.These methods impose a stability-plasticity trade-off on parameters important to previous tasks.
  • Existing approaches: Memory-based approaches constrain new gradients using old examples or stored gradient directions, requiring replay data or gradient memories.The cited approaches differ in whether they retain episodic examples or gradient directions.
  • Proposed approach: GPM partitions gradient space into Core Gradient Space and Residual Gradient Space, then stores activation-derived bases obtained with SVD after each task.The stored bases provide explicit constraints without retaining old examples or old gradient directions.

2 RELATED WORKS

Continual-learning methods primarily address forgetting through network expansion, parameter regularization, or memory. GPM differs by storing compact activation-derived gradient-subspace bases rather than old data or complete gradient directions.

  • Expansion-based methods: Expansion-based methods allocate separate parameter subsets or add subnetworks for new tasks, preserving old parameters through architectural growth.Examples include freezing a base model, adding lateral subnetworks, or splitting and duplicating important units.
  • Regularization-based methods: Regularization-based methods penalize changes to parameters deemed important for previous tasks in fixed-capacity models.Importance can be estimated from Fisher information, loss sensitivity, or output sensitivity to inputs.
  • Memory-based methods: Memory-based methods rehearse stored examples or synthesize old data with generative models to mitigate forgetting.These methods depend on retaining or reconstructing information from prior tasks.
  • Memory-based methods: GEM projects new gradients using constraints derived from episodic-memory samples, while A-GEM simplifies this to one memory-estimated direction.Experience Replay and Meta-Experience Replay jointly train on new and stored samples.
  • Relation to prior work: GPM avoids raw-data storage, whereas OGD stores gradient directions; GPM instead stores activation-derived core-gradient bases with much lower memory requirements.The paper contrasts GPM with OGD and notes that OGD requires a locality assumption for small learning settings.

3 NOTATIONS AND BACKGROUND

The paper formalizes sequential supervised learning, layerwise representations, and SVD-based matrix approximation. These notations support extracting compact bases from network activations for continual-learning constraints.

  • Continual-learning setup: Continual learning is modeled as T supervised tasks learned sequentially, with each task having a descriptor and dataset of input-label pairs.The network is evaluated across tasks indexed by τ.
  • Network notation: An L-layer network computes layerwise transformations, while x_i,τ denotes raw input at the first layer and its representation at later layers.The notation distinguishes vector inputs for fully connected layers from matrix inputs for convolutional layers.
  • Network notation: The network parameters are represented as task-indexed layerwise weights, with W^0 denoting initialization.The notation tracks parameter sets across sequential task learning.
  • SVD background: SVD factorizes A into orthogonal singular-vector matrices and a diagonal matrix of sorted singular values.The matrix rank determines the number of nonzero singular components.
  • SVD background: A k-rank approximation retains the smallest number of singular components satisfying a Frobenius-norm threshold.The threshold hyperparameter ε_th controls the approximation criterion.
  • Convolution notation: Figure 1 represents convolution as matrix multiplication during forward and backward passes.This representation prepares the derivation of convolutional gradient spaces.

4 INPUT AND GRADIENT SPACES

The paper establishes that gradient updates lie in spaces spanned by layer inputs: input vectors for fully connected layers and input patches for convolutional filters. This relationship enables activation-based characterization of important gradient subspaces.

  • General relationship: SGD updates lie in the span of input data points, a relationship analyzed for both fully connected and convolutional layers.The analysis applies generally across network layers and tasks.
  • Fully connected layer: For a single-layer linear network, the mean-squared-error setup uses input x, label y, and weights W.The loss is defined over the prediction error W x − y.
  • Fully connected layer: The fully connected gradient is ∇W L = (W x − y)xT = δxT, so updates lie in the span of the input vector x.The relation also holds for mini-batches, nonlinear units, and cross-entropy losses, with a different error term δ.
  • Convolutional layer: Convolution can be reshaped into matrix multiplication, where each row represents an input patch vector p_j.The reshaped input and filter matrices produce the output feature map.
  • Convolutional layer: During backpropagation, convolutional filter gradients are computed from the reshaped input and error matrix, placing updates in the space spanned by input patches.The gradient has the same shape as the reshaped filter matrix.

5 CONTINUAL LEARNING WITH GRADIENT PROJECTION MEMORY (GPM)

GPM identifies task-relevant gradient subspaces from network representations and constrains later updates to their orthogonal complements. Across diverse continual-learning experiments, it preserves prior-task performance while using limited memory and often reducing training cost.

  • Core method: GPM uses network representations and SVD to identify bases of task-relevant gradient subspaces stored in memory.Representation matrices are constructed from activations, decomposed with SVD, and truncated using a threshold to retain significant directions.
  • Core method: For each new task, GPM retrieves stored CGS bases and subtracts projected gradient components so updates lie orthogonal to past-task CGS.This explicitly constrains gradient directions without storing old examples or old gradient directions.
  • Experimental setup: GPM evaluates continual learning on PMNIST, split CIFAR-100, split miniImageNet, and 5-Datasets using sequential task settings and SGD training.The experiments include short and long sequences, with deeper-network evaluation on split miniImageNet.
  • Results: 93.91 ± 0.16% average accuracy is achieved on PMNIST, with least forgetting except OWM, while using 69% of GPM Max memory and about 400 times less memory than OGD.The PMNIST comparison also reports approximately 10% better accuracy than OGD.
  • Limitations: The method’s capacity limits learning sequences of hundreds of dissimilar tasks because GPM size is fixed by network architecture and its gradient space can fill quickly.In the 5-Datasets experiment, 78% of the gradient space is constrained after five less-related tasks.
  • Efficiency: GPM is usually faster because reference gradient bases are precomputed once per task, although the update consumes 30% of total time for short miniImageNet training.GPM update accounts for 0.2%, 3%, and 6% of total time on PMNIST, CIFAR-100, and 5-Datasets, respectively.
  • Results: GPM achieves 57.72 ± 0.37% average accuracy with BWT of -0.01 on 20-task CIFAR-100 Superclass using the smallest network.It outperforms RCL and APD while using 84% and 30% fewer network parameters, respectively.

C.1 DATASET STATISTICS

Dataset statistics are presented in Table 4, while Table 5 summarizes the five-dataset setting and its image-channel normalization.

  • Table 4 and Table 5 provide summaries of the datasets used in the experiments.
  • Table 5 reports statistics for the 5-Datasets experiment.
  • Monochromatic images are replicated across all RGB channels, producing 3 × 32 × 32 inputs.

C.2 ARCHITECTURE DETAILS

The experiments use AlexNet-like and reduced ResNet18 architectures, with ReLU hidden units and softmax cross-entropy outputs.

  • AlexNet-like architecture: The AlexNet-like network has three convolutional layers followed by two fully connected layers.The convolutional layers use 64, 128, and 256 filters, while each fully connected layer has 2048 units.
  • AlexNet-like architecture: The AlexNet-like architecture applies batch normalization, max-pooling, ReLU activations, and dropout.Batch normalization is used except in the classifier layer; max-pooling follows the convolutional layers.
  • Reduced ResNet18 architecture: The reduced ResNet18 architecture modifies the first convolution and classifier-side average pooling for the miniImageNet and 5-Datasets experiments.The first convolution uses stride 2 for miniImageNet, and 4 × 4 average-pooling is replaced with 2 × 2 pooling in both settings.
  • All networks use ReLU in hidden units and softmax with cross-entropy loss in the final layer.

C.3 CIFAR-100 SUPERCLASS EXPERIMENT

The CIFAR-100 superclass experiment uses a modified LeNet-5 setup with fixed training, validation, optimization, and stopping procedures.

  • The experiment uses a modified LeNet-5 architecture with 20-50-800-500 neurons.
  • Baseline results are reported from APD, with no data augmentation or preprocessing used.
  • Five percent of each task’s training data is reserved for validation.
  • Training uses batch size 64, initial learning rate 0.01, a maximum of 50 epochs, decay, and early termination.

C.4 BASELINE IMPLEMENTATIONS

The study evaluates several continual-learning baselines using official implementations or adapted code, while tuning the interference-control threshold by architecture and dataset.

  • Baseline implementations: GEM, A-GEM, ER Res, and OWM are implemented from their respective official implementations.
  • Baseline implementations: EWC and HAT use the official implementation provided by Serrà et al., while OGD adapts code from Bennani et al.
  • Threshold selection: The threshold hyperparameter ϵth controls interference through approximating significant representations from past tasks.
  • Threshold selection: The experiments use ϵth values from 0.95 to 1 because representation characteristics vary across architectures and datasets.

C.6 TRAINING TIME MEASUREMENT

Training-time measurements are reported on specified NVIDIA GPUs, while GPM memory dimensions are determined by layer input or patch-vector sizes.

  • Per-epoch training times for ten sequential PMNIST tasks were averaged across tasks on an NVIDIA GeForce GTX 1060 GPU.
  • Training times used a GTX 1060 for PMNIST and a GTX 1080 Ti for all other datasets.
  • Baseline and approach hyperparameters are tabulated using dataset-specific abbreviations such as perm, cifar, minImg, and 5data.
  • GPM matrices can reach square dimensions determined by fully connected input vectors or convolutional input patches.
  • The implementation reports both per-layer GPM matrix sizes and maximum parameter counts for the experimental architectures.

D.1 RESULT TABLES

The result tables and figures cover PMNIST, class-incremental digit learning, and sequential image-classification experiments, including variability and GPM basis growth. Reported results show stable task-1 accuracy and threshold-dependent interference reduction, while replay-based DGDMN remains strongest in the digit setting.

  • PMNIST: Five independent runs provide standard deviations for PMNIST results, including single-epoch and multi-epoch settings.
  • GPM growth: Table 10 reports the number of new GPM bases added at different layers after each PMNIST and 10-split CIFAR-100 task.
  • Class-incremental learning: In class-incremental Digit learning, EWC forgets catastrophically, GPM outperforms DGR, and DGDMN outperforms all methods.
  • Sequential learning: Task-1 accuracy remains stable while learning 20 sequential miniImageNet tasks.
  • Interference control: Increasing ϵth decreases interference-activation spread and minimizes forgetting across Conv1, Conv3, and FC1.
Loading 2103.09762v1…