PaperlayerView exact revision on arXiv

Source-linked AI summary

Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting

Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Weijer, Antonio M. Lopez, Andrew D. Bagdanov

arXiv:1802.02950v4cs.CV

TL;DR

Sequential task learning can cause catastrophic forgetting, while EWC’s practical diagonal-Fisher assumption may discard important parameter correlations. The paper reparameterizes networks through factorized rotations that approximately diagonalize the Fisher Information Matrix, making EWC more effective. Across MNIST, CIFAR-100, CUB-200, and Stanford-40, the rotated method improves over standard EWC and is competitive with exemplar-free state-of-the-art methods.

  • Problem

    Sequential learning can cause catastrophic forgetting, and EWC’s diagonal Fisher assumption may discard important information needed to prevent it.

    Source: §I · p. 1, B. Limitations of EWC · p. 3, §VI · p. 7

  • Method

    The method reparameterizes network layers by rotating parameter space through intermediate features so the Fisher Information Matrix becomes more compact and approximately diagonal.

    Source: §I · p. 1, B. Limitations of EWC · p. 3, §VI · p. 7

  • Results

    R-EWC consistently improves over EWC, with absolute accuracy gains of 2.1% to 5% on larger two-task datasets while retaining similar second-task accuracy.

    Source: C. Comparison with EWC on two tasks · p. 6, §VI · p. 7

  • Takeaways & Limitations

    Rotating parameter space makes EWC more effective at reducing catastrophic forgetting and yields performance comparable to or better than other exemplar-free weight-consolidation methods.

    Source: §I · p. 1, §VI · p. 7

    The approach still relies on EWC’s diagonal approximation, which is motivated partly by computational constraints and can be unrealistic in the original parameter space.

    Source: B. Limitations of EWC · p. 3

Abstract

from arXiv · show

In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters. This reparameterization takes the form of a factorized rotation of parameter space which, when used in conjunction with Elastic Weight Consolidation (which assumes a diagonal Fisher Information Matrix), leads to significantly better performance on lifelong learning of sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and Stanford-40 datasets demonstrate that we significantly improve the results of standard elastic weight consolidation, and that we obtain competitive results when compared to other state-of-the-art in lifelong learning without forgetting.

I. INTRODUCTION

Lifelong learning presents tasks sequentially without retaining access to earlier task data, making catastrophic forgetting a central challenge. The paper addresses EWC’s diagonal-Fisher limitation through a rotation-based reparameterization that improves forgetting prevention.

I. INTRODUCTION

  • Lifelong learning trains networks on sequential task groups while exposing them to data from only the current group.This differs from conventional training, where examples from all expected tasks are generally available.Source: §I · p. 1
  • Catastrophic forgetting occurs when adapting network weights to new tasks causes previously learned tasks to be forgotten.The paper frames preventing this interference as the main challenge of sequential learning.Source: §I · p. 1
  • Existing lifelong-learning methods either retain exemplars from earlier tasks or avoid storing prior training data through regularization-based approaches such as EWC.Related methods include exemplar rehearsal, inequality constraints, output regularization, and feature preservation.Source: §I · p. 1, §II · p. 2
  • EWC selectively regularizes parameters using the Fisher Information Matrix, but assumes that this matrix is diagonal, a condition the paper identifies as almost never true.The diagonal assumption makes EWC practical but can misrepresent important parameter directions.Source: §I · p. 1
  • The proposed method rotates parameter space without changing the forward-pass output, making the backward-pass Fisher Information Matrix approximately diagonal for more effective EWC.The paper reports significantly improved prevention of catastrophic forgetting and performance over standard EWC.Source: §I · p. 1, Figure 1 · p. 1

III. ELASTIC WEIGHT CONSOLIDATION

EWC frames sequential task learning as Bayesian updating and approximates the posterior around previously learned parameters with a Gaussian shaped by the Fisher Information Matrix. Its practical diagonal approximation converts a full quadratic regularizer into parameter-wise penalties.

III. ELASTIC WEIGHT CONSOLIDATION

  • EWC learns task K by combining the new task likelihood with the posterior formed from the preceding K−1 tasks.This sequential Bayesian factorization treats the previous posterior as the prior for the new task.Source: A. Overview · p. 2
  • Because the true posterior is intractable, EWC uses a Laplace approximation that represents it as a Gaussian around previously learned parameters.The Fisher Information Matrix approximates the inverse covariance in this approximation.Source: A. Overview · p. 2
  • In practice, EWC assumes the Fisher Information Matrix is diagonal, replacing the quadratic regularizer with scaling by diagonal entries.This approximation reduces storage and computation from O(N^2) to O(N), where N is the number of parameter-space elements.Source: A. Overview · p. 2
  • The Fisher Information Matrix measures how parameter-space directions relate to forgetting after a network reaches a trained configuration.EWC penalizes movement in high-Fisher-information directions and favors movement along lower-information directions.Source: A. Overview · p. 3

B. Limitations of EWC

EWC’s diagonal Fisher approximation is computationally efficient but can be unrealistic in the original parameter space because it discards parameter correlations. Rotating the space improves diagonal alignment and preserves more Fisher-matrix energy.

B. Limitations of EWC

  • The diagonal Fisher approximation reduces computation and storage from O(N^2) to O(N), but may be unrealistic for sequential learning in the original parameter space.The paper identifies efficiency as the practical reason for using the diagonal assumption.Source: B. Limitations of EWC · p. 3
  • Rotating parameter space to align average log-probability gradients with coordinate axes makes the diagonal approximation more reasonable.In the rotated space, EWC can optimize the new task while better preserving the old task.Source: B. Limitations of EWC · p. 3
  • 40.8% of the Fisher-matrix energy is retained by the diagonal in a second-layer MNIST multilayer perceptron example.The full matrix is clearly non-diagonal, so the diagonal approximation misses significant correlations between weights.Source: B. Limitations of EWC · p. 3

IV. ROTATED ELASTIC WEIGHT CONSOLIDATION

The method reparameterizes the network so its forward response is unchanged while the Fisher Information Matrix becomes more nearly diagonal, making EWC more effective.

IV. ROTATED ELASTIC WEIGHT CONSOLIDATION

  • The desired reparameterization preserves the network’s feedforward response while better satisfying EWC’s diagonal-FIM assumption.The resulting diagonal FIM can be efficiently estimated in the new parameter space.Source: §IV · p. 3
  • The approach performs gradient-descent minimization for new tasks in the reparameterized space.EWC is therefore applied using the approximately diagonal FIM of that space.Source: §IV · p. 3
  • SVD-based direct rotation is impractical because it is expensive, ignores sequential network structure, and requires an unavailable FIM.These issues motivate an indirect rotation implemented within the network architecture.Source: §IV · p. 3

A. Indirect rotation

Indirect rotation uses local feature transformations to reparameterize fully connected layers without changing forward computation, while concentrating more FIM energy on diagonal terms.

A. Indirect rotation

  • Assuming independence between output gradients and inputs factorizes the FIM into one factor for backpropagated gradients and another for inputs.This factorization motivates separate rotations of input and output spaces.Source: A. Indirect rotation · p. 3
  • The rotations are implemented as two additional fixed local linear layers, x′ = U1x and y = U2y′.Because the rotations are local, they can be integrated into the network architecture.Source: A. Indirect rotation · p. 4, Figure 3 · p. 4
  • After reparameterization, training estimates the FIM and learns W′ using transformed inputs and outputs.The procedure is otherwise the same as the original EWC training procedure.Source: A. Indirect rotation · p. 4
  • The method preserves both networks’ forward passes by expressing W as U2W′U1, keeping the sequential structure intact.The new learning problem uses transformed inputs, outputs, and weights.Source: A. Indirect rotation · pp. 3–4
  • The layer-wise extension applies the same procedure using each layer’s inputs and backpropagated output gradients.This assumes a block-diagonal FIM and produces layer-specific reparameterizations.Source: A. Indirect rotation · p. 4

B. Extension to convolutional layers

The fully connected rotation procedure extends to convolutional layers by applying channel-space rotations through additional 1 × 1 convolutional layers and processing kernel slices.

B. Extension to convolutional layers

  • Convolutional rotations use two additional 1 × 1 convolutional layers to transform the input and output channel spaces.This adapts the fully connected construction with only slight modifications.Source: B. Extension to convolutional layers · p. 4
  • For each spatial coordinate, input and output fibers are d1-dimensional and d2-dimensional vectors, respectively.Their self-correlation matrices are averaged over spatial coordinates.Source: B. Extension to convolutional layers · p. 4
  • A mode-i fiber is the vector obtained by fixing every tensor index except i.A tensor slice fixes all indices except two.Source: B. Extension to convolutional layers · p. 4
  • Algorithm 1 incrementally fits each task with EWC using rotated parameters and a Fisher matrix, then combines parameters before computing new rotations.The task data are supplied as per-class training sets.Source: B. Extension to convolutional layers · p. 5
  • The convolutional procedure computes decompositions, rotates kernel slices, and tiles the resulting slices into the final rotated kernel tensor.The rotated slices are defined from the original kernel’s spatial slices.Source: B. Extension to convolutional layers · p. 5

V. EXPERIMENTAL RESULTS

The paper evaluates the proposed approach against EWC and other baselines across multiple experiments.

V. EXPERIMENTAL RESULTS

  • The experiments compare the proposed approach with EWC and other baselines.The passage introduces the experimental evaluation without reporting outcomes.Source: §V · p. 5

A. Experimental settings

The evaluation uses sequential class-incremental tasks across MNIST, CIFAR-100, CUB-200 Birds, and Stanford-40 Actions, with dataset-specific preprocessing and architectures. Experiments assume task labels are unknown at inference.

Datasets

  • Each dataset is divided equally into class groups that are learned as sequential tasks.The evaluated datasets are MNIST, CIFAR-100, CUB-200 Birds, and Stanford-40 Actions.Source: A. Experimental settings · p. 5
  • CUB-200 images use cropped bounding boxes resized to 224×224, while Stanford-40 images are resized to 256×256 and randomly cropped to 224×224.Source: A. Experimental settings · p. 5

Training details

  • For the fine-grained datasets, the method finetunes an ImageNet-pretrained VGG-16 with global pooling added after the final convolutional layer.The modification reduces memory use and computational complexity.Source: A. Experimental settings · p. 5

Inference setting

  • The experiments use a single network head because task labels are unknown at inference, increasing output neurons as new tasks are added.Source: A. Experimental settings · p. 5

B. Disjoint MNIST comparison and ablation study

On disjoint MNIST, R-EWC outperforms fine-tuning and EWC across regularization settings, while rotating fully connected layers is sufficient for substantial improvement. Larger regularization preserves the first task but makes the second harder to learn.

MNIST comparison

Trade-off parameter

Ablation study

  • All R-EWC configurations that rotate fully connected layers significantly outperform EWC.Source: Table I · p. 6

Conclusion

  • R-EWC maintains similar task 2 performance to EWC while producing much less catastrophic forgetting on task 1.The regularized FIM is usually between 10^-2 and 10^-4, with λ=100 suggested as a more balanced trade-off.Source: B. Disjoint MNIST comparison and… · p. 6

C. Comparison with EWC on two tasks

Across larger datasets split into two sequential tasks, R-EWC improves overall accuracy over EWC while forgetting less on the first task and retaining similar second-task accuracy. The multi-task comparison extends this advantage as additional tasks are learned.

Two-task comparison

More-task comparison

E. Comparison with the state-of-the-art

On CIFAR-100, R-EWC outperforms the compared exemplar-free baselines and approaches, but does not reach, an upper bound requiring all previous data. The paper attributes its broader advantage to reparameterizing the FIM so EWC can work more effectively.

State-of-the-art comparison

Conclusion

  • The method reparameterizes layers to produce a more compact and more diagonal FIM, addressing information discarded by EWC’s diagonal approximation.Source: §VI · p. 7
  • The fixed additional layers approximate parameter-space rotation without increasing the number of parameters.Source: §VI · p. 7
  • Across several tasks and settings, R-EWC consistently improves over EWC and obtains results comparable to or better than other exemplar-free weight-consolidation methods.Source: §VI · p. 7

Related papers

Loading 1802.02950v4…